Your demo isn’t evidence
Make a wrong answer visible.
One answer is not a reliability measure
A demo shows that a system can produce a right answer. It does not tell you how often the system fails, which requests trigger those failures, or whether the next change makes them worse.
Define the behavior that matters
Use representative tasks and explicit acceptance criteria. Include malformed tool output, partial responses, timeouts, and requests the system should refuse or escalate.
Make comparisons repeatable
Keep a versioned evaluation set and connect outcomes to traces. Compare quality, latency, and cost before and after a change. A lower bill is useful only when the system still meets its requirements.
Connect findings to operations
Alert on meaningful failure patterns. Give the team a runbook, a named escalation path, and enough trace context to reproduce what happened.