I am looking for practical feedback from people using LangSmith-style traces, evals, and incident review workflows.
I am testing a small open-source artifact called `change_proposal`: a plain JSON file that sits after observability/evals and records:
- what behavior was observed;
- what outcome/eval signal made it important;
- what change is proposed;
- what evidence supports it;
- how to validate it;
- how to roll it back.
Repo: GitHub - vittorfp/agent-change-proposals: Evidence-backed improvement proposals for AI agents. · GitHub
Example: agent-change-proposals/examples/declared-intent/change_proposal.example.json at main · vittorfp/agent-change-proposals · GitHub
Discussion: Should evidence-backed agent change proposals exist? · vittorfp/agent-change-proposals · Discussion #6 · GitHub
This is not meant to replace LangSmith, tracing, or eval runners. I am trying to understand the handoff after traces/evals reveal a problem: how does that become a reviewable prompt/tool/retrieval/guardrail change?
The feedback I would value most:
- Would this artifact fit PR review, eval review, CI, incident follow-up, or none of those?
- What trace/eval evidence would make a proposed behavior change trustworthy enough to review?
- For CI gating, should fields like immutable evidence refs, baseline/candidate artifact IDs, evaluator and dataset versions, metric threshold, sample count, and rollback trigger live under `validation`, typed `evidence`, or a separate gate/decision object?
- Is this useful glue around observability/evals, or just extra ceremony around something teams already solve another way?
Blunt feedback is welcome. I am trying to decide what should stay plain JSON, what should become stricter, and what would make the artifact actually fit real agent improvement workflows.