Skip to main content

For AI product teams

Keep the failure useful after the fix.

A production failure is a starting point for an evaluation case. Preserve the trace and expert verdict, replay representative work and give the release owner evidence to review.

Scope your agent engagement Follow the workflow ↓

Failed trace

Separate what happened from what should have happened.

Bring the relevant inputs, tool calls and output. Identify the missing constraint before changing the prompt or swapping a component.

Expert verdict

Write down why the output failed.

Connect the reviewer’s correction to its source and expected behavior. Retain disagreements and cases where a valid answer requires escalation.

Evaluation case

Make the correction checkable in the next version.

Turn the reviewed failure into a representative case with explicit criteria. Agree which surrounding behaviors must still hold.

Replay

Compare the proposed change against the baseline.

Replay the failure and related tasks. Inspect task quality, unresolved errors, review effort, latency and cost against the criteria agreed for the engagement.

Release decision

Ship with a reason the owner can defend.

Bring the replay results and open cases to the release reviewer. Decide whether to proceed, revise or stop; a passing example alone does not resolve every production failure.

Fictional example · illustrative, not a customer result

Failure-to-release record

Compare the current handoff with a proposed, source-connected record.

Before · context lost

  • Trace: agent drafts a customer reply before inspecting the cited source.
  • Fix: add “check the source” to the prompt; the original case disappears.

After · proposed pilot artifact

  • Verdict: the reply made an unsupported assertion.
  • Eval case: source missing → escalate; source available → cite reviewed evidence.
  • Replay: check the original failure and adjacent valid replies.
  • Release review: compare results and retain unresolved cases.

Review question
Would the next version catch this failure, and can the release owner see what remains untested?

Prepare the conversation

Bring the work. Agree on what to evaluate.

Customer inputs

  • A failed trace and representative neighboring tasks
  • Reviewer judgments and sources for expected behavior
  • Current evaluation setup and release criteria

Evaluation questions

  • Is the failure in context, reasoning, tools or the workflow boundary?
  • Does the evaluation test the correction rather than mirror the patch?
  • Which open failures block the proposed release?

Scope the next step

AI-native services

Starting at $100k+ · Scoped engagement

Help your team build agents and workflows, and measurably improve the systems you already run.

Inputs, integrations, boundaries, deliverables and evaluation criteria are agreed during scoping.

Scope your agent engagement Compare the offers →