I am building a small evaluation loop for an AI workflow. The goal is to turn
production-like failures into regression cases instead of fixing each one only
in the prompt.
The cases I want to capture include:
the agent chose the wrong tool;
retrieval returned a plausible but irrelevant document;
the structured output was incomplete;
the model followed an unsafe instruction from retrieved content;
a tool timed out after the model had already started a response;
the workflow repeated a side effect after a queue retry.
For each case, I want to save the input, the expected behavior, the observed
behavior, the failure category, and the release or prompt version. Some cases
should check the final answer. Others should check retrieval, tool choice,
output structure, or whether the system stopped safely.
How do you decide that a failure is specific enough to add to an evaluation
set? Do you keep separate datasets for retrieval, tool calls, and final
behavior, or one end-to-end set with labels?
My rule of thumb: a failure becomes a case when I can write the expected behaviour as a check that a future change could break. “The answer was bad” isn’t specific enough. “For this input, doc X should be in the top 5” or “this request should call lookup_order, not search_faq” is.
I also add one case per root cause, not per incident. Five failures from the same bad chunk are one retrieval case.
On structure, I keep one dataset with a layer label (retrieval, tool_choice, output_schema, safety, side_effect) and grade each case at its own layer:
retrieval: expected doc IDs in top k
tool choice: expected tool and key arguments
output: schema validation, no model grader needed
safety: did it refuse or stop, checked as a boolean
End-to-end answer checks are a separate, smaller set, because they’re slow and noisy.
Two of your examples I’d treat as tests rather than evals. The tool timeout mid-response and the repeated side effect on retry are about your orchestration, not the model. Replay them with recorded tool responses and an idempotency key, and assert the side effect happened once.
Also save the retrieved documents with each case. If the index changes later, you can still tell whether the model or the retrieval regressed.
+1 to @rosealvinanex on the retry/side-effect one being a test, not a model eval. i built exactly that: an idempotency key minted in the harness before the action, replay the dropped response, and the assert is “exactly one side effect,” nothing about the model’s output. the tool-timeout-mid-response is the same shape.
on AMOHYZ’s question about working out which step caused it: what made this tractable for me was an append-only record of every step (what the agent read, what it proposed, what fired) kept separate from the agent, so i could replay the run and see where the decision went wrong, not just that the final answer was bad. for stale-data failures, recording the facts the agent read at decision time is what lets you point at the exact step.
one thing to add to the dataset question: keep the legitimate-but-suspicious cases, not just the failures. a change that looked like it should block but was fine is what tells you if your check got too strict, and it’s the case that catches a regression where a fix starts over-blocking. i score two numbers separately for that, how many real failures are caught vs how often a safe run is wrongly flagged. eval set’s here if useful: GitHub - rayyanmaan/agent-guardrail · GitHub
I stopped trying to find the broken step. Test the end result instead. After the run, check what happened: was the card charged once, was the ticket created, did the output have the right fields. Save that as the test. A future change that breaks the agent breaks this result too. The broken step can be found later, the saved case is the evidence.
i don’t think these are opposed, they’re two layers. asserting on the end result (charged once, ticket created, right fields) is exactly the per-case check i run, that’s the ground truth and i agree it’s where the test lives. the append-only record isn’t for finding the broken step live, it’s the evidence you reach for when that end-result assertion fails and “it charged twice” doesn’t tell you why. you test the outcome, and you keep the trace so a red case is debuggable instead of just red. the “find it later” you mention is a lot easier when the trace was recorded up front rather than reconstructed after.
Yes exactly. Result + traces (OTel) is enough, I’ll generate (or my agent frameworks generate) OTel from the runs that are saved with the end result (deliverables + the traces), to show if the checks are failing.