Been building with LangChain agents and kept running into the same problem: once an agent runs, I have no real record of what it decided, which tools it called, why, how long each step took, or what it cost. If something goes wrong, there’s no audit trail to look at.
So I built AgentLens — a small open-source CLI that wraps your agent with a callback handler, captures every step (LLM calls, tool calls, timing, token usage) to a local trace file, and generates a plain-English Markdown report on demand.
A few things about it:
-
100% local — no trace data ever leaves your machine, no signup, no cloud dependency
-
Works with any LangChain agent via a standard callback handler
-
Generates a report like this:
AgentLens Audit: sample_run
Summary
- Steps: 2
- Total duration: 652ms
- Errors: 0
Step-by-step timeline
- llm_call: ChatGroq — 644ms
- tool_call: calculator — 8ms
Anomalies
It’s very early (Phase 1, basically) — no dashboard, no multi-run comparison yet, just the CLI. Posting here because I’d genuinely like feedback from people actually running LangChain agents in production: is this something you’d reach for, and what’s missing that would make it actually useful for your workflow?
GitHub: GitHub - subhash-0000/AgentLens · GitHub
Quick update on this — I dogfooded AgentLens against a real 4-agent LangChain + Groq production pipeline (not just the toy sample script), and it surfaced some real gaps I’ve since fixed:
- Real cost tracking: costs were showing $0.00 across the board because the capture logic was grabbing the LangChain wrapper class name (
ChatGroq) instead of the actual model string. Fixed — now correctly resolves real model names and looks up accurate per-token pricing.
- Per-agent identification: in a multi-agent system where every agent shares the same model, the report couldn’t tell them apart. You can now pass
tags=["YourAgentName"] in the invocation config to get distinct, human-readable names per agent in the audit report (e.g. QuestionGenerator (openai/gpt-oss-20b) instead of 4 identical ChatGroq entries).
- Fixed a false-positive-heavy anomaly rule: the original “flag anything over 20% of total cost” rule fired on 3 out of 4 steps in a normal run, since that’s just what happens when cost is split across a small number of similarly-sized steps. Replaced with a rule relative to the average cost per step, which actually distinguishes real outliers from normal variance.
All of this came from actually running it against a real pipeline with real fallback/retry logic, not synthetic tests — happy to share specifics if anyone’s curious about the failure modes I hit along the way.
GitHub: GitHub - subhash-0000/AgentLens · GitHub
Still very open to feedback, especially from anyone running multi-agent LangChain systems in production — what would make this actually useful for your workflow?
Does the step-by-step trace also preserve parent/child run relationships, so a failed tool call can be traced back to the agent invocation that triggered it? That’s the connection I’d want when reconstructing a multi-agent failure.
Good catch until a few days ago, the answer was no. Just added it.
Every event now carries both its own LangChain run_id and a parent_run_id, so a failed tool call (or any nested call) traces directly back to the agent invocation that triggered it. Here’s a real excerpt from a live 4-agent pipeline run — not synthetic data:
QuestionGenerator
run_id: 01a106f4-8773-75c1-b2be-a42d88f42bd7
parent: 01a106f4-8770-74e0-a288-a8b137f74adb
tokens: 360 in / 990 out, $0.000324
AnswerGenerator
run_id: 01a106f4-8e7d-75a1-ad40-3b0bee2c8ba3
parent: 01a106f4-8e7a-7312-8e79-9f7ba225f4ee
tokens: 593 in / 1073 out, $0.000366
Each agent’s run_id is nested under its own parent chain invocation, so if AnswerGenerator’s tool call failed, the trace would show that failure’s parent_run_id pointing straight back to this exact agent invocation — not just “something failed somewhere.” I verified this specifically with a deliberate failure injection (a broken tool call nested under a tagged agent) before trusting it, and the parent linkage held correctly through the error path too, not just the success path.
Happy to dig into any edge case you’re specifically worried about — multi-level nesting, concurrent agents, whatever’s actually biting you in practice.
Yes — and the gap I’d want closed on top of the trace file is tamper-evidence. A local trace tells you what happened; it doesn’t prove it to anyone else. The failure mode that bit me in production: logs are mutable, so “the agent called X with Y” is a claim, not evidence. What worked for us: every tool result gets a sha256 receipt over the canonical output bytes plus a timestamp, and each step record hash-chains to the previous one — now the trace can’t be silently edited after the fact, and a third party can recompute the hash from the published bytes instead of trusting a server. Two concrete suggestions for phase 2: (1) keep raw tool inputs/outputs content-addressed, not just summarized — summaries are the layer that lies; (2) label the Claude narrative as interpretation, not record. The narrative is the part I’d trust least — it’s the one layer that can hallucinate the story of what the agent was trying to do. Machine-readable evidence canonical, prose clearly marked as opinion. I’d reach for this the day the trace file is something I can hand to someone who doesn’t trust me.
Dogfooding on a real 4-agent pipeline is the right test. The important distinction I’d add to AgentLens is between observability and evidential validity: a complete-looking trace can show what ran while still failing to establish that the claimed task outcome was valid—or that no adverse effect co-occurred.
I analyzed 11,624 complete AgentDojo executions; 200 runs had task success and attack success simultaneously. A report evaluating only completion, errors, latency, or cost would have counted those as wins.
For AgentLens, I’d add four checks:
- Claimed outcome ↔ observable state change
- Candidate/chosen tool call + policy/version at decision time
- Retry/fallback chain + duplicate or adverse side effects
- Trace completeness and tamper evidence
That also suggests anomaly rules should be conditioned on expected outcome and risk, not only relative cost.
Since you already have real fallback/retry traces, send one redacted trace plus one sentence describing what you believe it proves. I’ll return a one-page independent evidence check within 24 hours at no charge—no source code or platform access needed.
Reproducible artifact behind the 11,624-run result: GitHub - albdrani37-hash/agentdojo-trace-audit: A reproducibility and audit layer for reconstructing and verifying AgentDojo trace-derived claims · GitHub
The parent_run_id addition is the part I’d want most. I’ve built multi-agent orchestration platforms for clients, and the recurring pain was the end of the month: nobody could say which agent in the tree had burnt the budget, and we were stitching it together with Excel sheets and shell scripts.
Two ideas now that you have run_id and parent_run_id. First, roll cost up the tree, so an orchestrator’s line shows its own spend plus everything it delegated. That’s usually the question someone asks after a bad bill. Second, the handler can do more than observe: set raise_error = True on it and an exception raised in on_llm_start stops the call, so a per-run cost ceiling becomes possible while staying fully local.
Curious how you’re keeping the per-token price table current as Groq adds and renames models?
@subhash-0000 nice that you added parent_run_id so fast after it was asked, and keeping everything local is a good call. +1 to @rambozambo’s tamper-evidence suggestion, especially “summaries are the layer that lies”.
one thing that bit me building something similar: a hash chain only helps if the agent process can’t write the trace. if the callback handler writes the file from inside the agent, a bad run can rewrite it and recompute every hash and it still verifies. in mine the agent has no write handle, a separate component appends, and sqlite triggers reject UPDATE and DELETE. there’s a tamper test suite too, including an edit where the attacker recomputes the row’s own hash so only the link to the next row breaks: https://github.com/rayyanmaan/agent-guardrail. could be useful if you add the chain.
is the trace meant to hold up for someone else, or mainly for debugging your own runs? and @rambozambo, when you hit the mutable logs problem, was the worry the agent process itself or other jobs with access to the same storage?
@rayyanmaan Both, but if I have to pick the one that actually bit us: anything else with write access to the same storage. In a multi-agent setup the trace store is shared by design, so the agent process itself is only one of several writers. A retry job, a cleanup script, another agent with the same mount: any of them can rewrite a row, and if that writer can also recompute the chain, the chain still verifies afterward. Your setup is the right shape: the agent gets no write handle, appends happen in a separate component, and UPDATE and DELETE are rejected outright. The test in your suite where the attacker recomputes the row’s own hash and only the link to the next row breaks is exactly the case a naive chain fails quietly. One addition from our side: publish the head hash somewhere the writers cannot reach, a second system or a public page, anywhere that is append-only from their side. Then a full rewrite has to explain why the head moved, and verification stops depending on trusting the store at all.
@rambozambo this is the part i needed. you’re right that my chain assumes the ledger is the only record of the latest hash, and it’s actually in my limitations: truncating the newest entries or rewriting everything from a point on both leave a chain that still verifies, because nothing outside the store remembers where the head was. anchoring the head somewhere the writers can’t reach is the thing that closes that, and it’s cleaner than anything i’d worked out. verification stops depending on trusting the store at all.
the part i’d want to get right is how often you publish the head. every append is safest but that’s a lot of writes, and publishing periodically leaves a window where a rewrite isn’t caught yet. when you did this in production, where did you actually put the head (a second db, a public page, something else), how often, and how did you think about that gap between publishes?
@rayyanmaan Per job, not per append. Every append is the safest setting and mostly theatre: the writes cost more than the risk they retire for ordinary runs. In our build the anchor attaches to the record itself: every receipt carries its own anchor envelope from creation, so the head state a reader checks is anchored per record, not on a schedule. The head goes to a public page on a separate system, one the trace writers have no credentials for. They can read it, they cannot move it. On the gap: be honest that it exists and price it. Anything still waiting on its anchor confirmation is only as strong as the store itself, so the interval is a risk decision, not a correctness one. A run whose result someone will pay for gets its wrap record anchored like every other record, and nobody should act on it until that record reads published. A long background run can live with a longer unconfirmed tail, because a rewrite in that window is detectable the moment the anchor lands and the head does not match. The mistake to avoid is anchoring on the same cadence for both and calling it a policy.
@rambozambo anchoring per record instead of on a schedule is the bit that makes this click. tying the anchor to the thing it protects is the same shape as what i already do on the check side: i don’t recheck the whole world, i recheck the facts the specific action depends on, right before the effect. so the anchor just becomes another precondition on that effect. a payable action doesn’t fire until its own record reads published, and a low-stakes run can carry a longer unconfirmed tail because a rewrite there shows up the moment the anchor lands and the head doesn’t match. that also lines up with the two run types i already have.
the thing i’d want to work out is the latency. if a payable effect can’t fire until its anchor lands on the separate system, that’s a round trip in front of every irreversible action. did you anchor synchronously before the effect and eat that cost, or make the confirmation cheap enough that gating on it basically isn’t felt?
@rayyanmaan Neither extreme, and the split is worth naming. The anchor is synchronous but cheap: it is a hash over the exact bytes being saved, computed at the moment of saving, from data already in hand. No extra round trip, no re-execution, nothing serialized in front of the effect beyond hashing the thing you were already writing. The expensive-sounding part, an independent party recomputing the hash and comparing it against the record, happens later, whenever someone actually checks. It never sits on the effect’s path.
What a payable effect gates on, in our setup, is the record reading published, and that state attaches to the record at creation rather than arriving in a later batch. So the sequence is: save the result, the anchor rides with it, the effect waits only for that one record to read published. The wait exists, I will not pretend otherwise, but it is a wait on a small record you just wrote, not a consensus round and not a re-verification of the whole chain.
And the honest limit, same as before: the anchor proves the saved bytes were not changed after the save. It says nothing about whether the result was right. Gating buys you exactly what the run produced, never a verdict that the run was correct.
that clears up the latency for me. i was picturing the anchor as a round trip in front of the effect, but if it’s just a hash over the bytes i’m already writing, computed at save time, then nothing extra sits on the path, and the independent recompute happening later is off the hot path entirely. the only real wait is the one record reading published, which is a wait on something small i just wrote, not a check of the whole chain. that’s a clean split.
and the honest limit lands too: the anchor proves the saved bytes weren’t changed after the save, not that the result was right. that actually matches how i already frame the ledger, it proves what happened, not that what happened was a good decision. tamper-evidence is about the integrity of the record, not the correctness of the call. thanks for walking through this, the per-record anchor is going into my next sprint.