AgentLens — free, local-only CLI to audit what your LangChain agent actually did

OSS Product Help LangChain
,
1

Been building with LangChain agents and kept running into the same problem: once an agent runs, I have no real record of what it decided, which tools it called, why, how long each step took, or what it cost. If something goes wrong, there’s no audit trail to look at.

So I built AgentLens — a small open-source CLI that wraps your agent with a callback handler, captures every step (LLM calls, tool calls, timing, token usage) to a local trace file, and generates a plain-English Markdown report on demand.

A few things about it:

  • 100% local — no trace data ever leaves your machine, no signup, no cloud dependency

  • Works with any LangChain agent via a standard callback handler

  • Generates a report like this:
    AgentLens Audit: sample_run

    Summary

    • Steps: 2
    • Total duration: 652ms
    • Errors: 0

    Step-by-step timeline

    • llm_call: ChatGroq — 644ms
    • tool_call: calculator — 8ms

    Anomalies

    • None detected

      Optional: point it at Claude and it’ll add a 2-3 sentence plain-English narrative of what the agent was actually trying to do

    It’s very early (Phase 1, basically) — no dashboard, no multi-run comparison yet, just the CLI. Posting here because I’d genuinely like feedback from people actually running LangChain agents in production: is this something you’d reach for, and what’s missing that would make it actually useful for your workflow?

    GitHub: GitHub - subhash-0000/AgentLens · GitHub

2

Quick update on this — I dogfooded AgentLens against a real 4-agent LangChain + Groq production pipeline (not just the toy sample script), and it surfaced some real gaps I’ve since fixed:

  • Real cost tracking: costs were showing $0.00 across the board because the capture logic was grabbing the LangChain wrapper class name (ChatGroq) instead of the actual model string. Fixed — now correctly resolves real model names and looks up accurate per-token pricing.
  • Per-agent identification: in a multi-agent system where every agent shares the same model, the report couldn’t tell them apart. You can now pass tags=["YourAgentName"] in the invocation config to get distinct, human-readable names per agent in the audit report (e.g. QuestionGenerator (openai/gpt-oss-20b) instead of 4 identical ChatGroq entries).
  • Fixed a false-positive-heavy anomaly rule: the original “flag anything over 20% of total cost” rule fired on 3 out of 4 steps in a normal run, since that’s just what happens when cost is split across a small number of similarly-sized steps. Replaced with a rule relative to the average cost per step, which actually distinguishes real outliers from normal variance.

All of this came from actually running it against a real pipeline with real fallback/retry logic, not synthetic tests — happy to share specifics if anyone’s curious about the failure modes I hit along the way.

GitHub: GitHub - subhash-0000/AgentLens · GitHub

Still very open to feedback, especially from anyone running multi-agent LangChain systems in production — what would make this actually useful for your workflow?

3

Does the step-by-step trace also preserve parent/child run relationships, so a failed tool call can be traced back to the agent invocation that triggered it? That’s the connection I’d want when reconstructing a multi-agent failure.

4

Good catch until a few days ago, the answer was no. Just added it.

Every event now carries both its own LangChain run_id and a parent_run_id, so a failed tool call (or any nested call) traces directly back to the agent invocation that triggered it. Here’s a real excerpt from a live 4-agent pipeline run — not synthetic data:

QuestionGenerator
run_id: 01a106f4-8773-75c1-b2be-a42d88f42bd7
parent: 01a106f4-8770-74e0-a288-a8b137f74adb
tokens: 360 in / 990 out, $0.000324

AnswerGenerator
run_id: 01a106f4-8e7d-75a1-ad40-3b0bee2c8ba3
parent: 01a106f4-8e7a-7312-8e79-9f7ba225f4ee
tokens: 593 in / 1073 out, $0.000366

Each agent’s run_id is nested under its own parent chain invocation, so if AnswerGenerator’s tool call failed, the trace would show that failure’s parent_run_id pointing straight back to this exact agent invocation — not just “something failed somewhere.” I verified this specifically with a deliberate failure injection (a broken tool call nested under a tagged agent) before trusting it, and the parent linkage held correctly through the error path too, not just the success path.

Happy to dig into any edge case you’re specifically worried about — multi-level nesting, concurrent agents, whatever’s actually biting you in practice.