# What should become an evaluation case after an agent failure?

**URL:** <https://forum.langchain.com/t/what-should-become-an-evaluation-case-after-an-agent-failure/4963>\
**Category:** Observability & Evals\
**Created:** [October 28, 2026, 5:29pm UTC](https://forum.langchain.com/t/what-should-become-an-evaluation-case-after-an-agent-failure/4963 "2026-10-28T17:29:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dante\_io](https://avatars.discourse-cdn.com/v4/letter/d/8baadc/32.png) [@dante\_io](https://forum.langchain.com/u/dante_io)\
**Post date:** [October 28, 2026, 5:29pm UTC](https://forum.langchain.com/t/what-should-become-an-evaluation-case-after-an-agent-failure/4963/1 "2026-10-28T17:29:24Z")

</div>

I am building a small evaluation loop for an AI workflow. The goal is to turn  
production-like failures into regression cases instead of fixing each one only  
in the prompt.

The cases I want to capture include:

- the agent chose the wrong tool;
- retrieval returned a plausible but irrelevant document;
- the structured output was incomplete;
- the model followed an unsafe instruction from retrieved content;
- a tool timed out after the model had already started a response;
- the workflow repeated a side effect after a queue retry.

For each case, I want to save the input, the expected behavior, the observed  
behavior, the failure category, and the release or prompt version. Some cases  
should check the final answer. Others should check retrieval, tool choice,  
output structure, or whether the system stopped safely.

How do you decide that a failure is specific enough to add to an evaluation  
set? Do you keep separate datasets for retrieval, tool calls, and final  
behavior, or one end-to-end set with labels?

I wrote down the broader reliability pattern here:  
[https://oduonye.com/blog/posts/reliable-ai-starts-where-the-demo-ends/](https://oduonye.com/blog/posts/reliable-ai-starts-where-the-demo-ends/).

---

<div class="post-metadata">

**Author:** ![rosealvinanex](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/rosealvinanex/32/3717_2.png) [@rosealvinanex](https://forum.langchain.com/u/rosealvinanex)\
**Post date:** [October 30, 2026, 1:58pm UTC](https://forum.langchain.com/t/what-should-become-an-evaluation-case-after-an-agent-failure/4963/2 "2026-10-30T13:58:32Z")

</div>

My rule of thumb: a failure becomes a case when I can write the expected behaviour as a check that a future change could break. “The answer was bad” isn’t specific enough. “For this input, doc X should be in the top 5” or “this request should call `lookup_order`, not `search_faq`” is.

I also add one case per root cause, not per incident. Five failures from the same bad chunk are one retrieval case.

On structure, I keep one dataset with a `layer` label (retrieval, tool\_choice, output\_schema, safety, side\_effect) and grade each case at its own layer:

- retrieval: expected doc IDs in top k
- tool choice: expected tool and key arguments
- output: schema validation, no model grader needed
- safety: did it refuse or stop, checked as a boolean

End-to-end answer checks are a separate, smaller set, because they’re slow and noisy.

Two of your examples I’d treat as tests rather than evals. The tool timeout mid-response and the repeated side effect on retry are about your orchestration, not the model. Replay them with recorded tool responses and an idempotency key, and assert the side effect happened once.

Also save the retrieved documents with each case. If the index changes later, you can still tell whether the model or the retrieval regressed.