contextjet-ai/awesome-llm-observability/collect-user-feedback

Collect user feedback

Your users are running the best eval you have, for free, every day. Capturing their reactions and attaching them to the trace turns production into a continuous source of labeled data. Most teams either skip this or collect it and never use it.

Capture both explicit and implicit signals

Explicit (ask directly):

• Thumbs up/down on a response (the simplest, highest-signal control).
• A short reason on thumbs-down (dropdown: wrong / unhelpful / unsafe / other).
• Accept / edit / reject on a suggested output (great for copilot-style apps).

Implicit (infer from behavior, no extra UI):

• Did the user retry, rephrase, or immediately give up? (dissatisfaction).
• Did they copy/use the output, or continue the task? (satisfaction).
• Did a downstream action succeed (the code ran, the ticket resolved)?

Implicit signals are noisier but you get them on 100% of traffic instead of the small % who click thumbs.

Wire it to the trace (this is the key step)

Attach every feedback signal to the trace/span it is about (by trace ID), with the user, timestamp, and the reason. Most observability platforms (Langfuse, Phoenix, Opik, LangSmith) have a feedback/scores API for exactly this. Without the trace link, feedback is a number with no context; with it, a thumbs-down is a fully debuggable example.

Use it (do not just collect it)

1. Monitor feedback rate + thumbs-down rate over time; alert on spikes (see set-up-drift-alerts).
2. Triage thumbs-down traces into candidate test cases (see trace-based-testing) and your eval set (build-eval-dataset).
3. Correlate feedback with your automated eval scores to check your judge actually agrees with humans (calibration).
4. Close the loop: the corrections/edits users make are gold-standard reference outputs, use them.

Handle it responsibly

• PII / privacy: feedback text can contain sensitive data, redact before it hits a third-party backend (redact-pii-for-tracing).
• Bias: thumbs are a biased sample (angry and delighted users click most). Do not treat thumbs-up rate as ground-truth quality; use it as signal, validate with evals.

Verify

• A thumbs-down in the UI shows up attached to the right trace in your backend.
• Thumbs-down traces are being triaged into the eval/regression set, not just counted.
• A dashboard shows feedback trends with an alert on a drop.

Anti-patterns

• Collecting feedback that never links to a trace (a number you cannot act on).
• Collecting it and never feeding it back into evals or fixes.
• Treating thumbs-up rate as objective quality (selection bias).
• Logging raw feedback with PII to a SaaS backend without redaction.

Grounding

Human feedback as the signal for LLM quality underpins RLHF and preference modeling (Ouyang et al., InstructGPT, arXiv:2203.02155) and human-aligned evaluation (Zheng et al. 2023, arXiv:2306.05685). Here it feeds the lighter-weight eval + observability loop (trace-based-testing, build-eval-dataset).