Curious how others are approaching governance for LangChain agents in production — specifically:
- PII leaking through RAG pipelines
When a retriever pulls documents containing customer data (SSNs, emails, credit cards), that PII flows into the model context and can leak in responses. Are people scanning retriever output before it hits the LLM? Building custom callbacks? Using an external DLP service?
- Runaway costs
An agent with web search + code execution can burn through budget fast if it loops. How are you enforcing hard budget caps (not just alerts-after-the-fact)? Per-session? Per-user? Is anyone using the new middleware system for this?
- Tool authorization beyond allowlists
tool_allowlist middleware handles which tools are permitted, but what about argument-level governance? Example: allowing web_search but blocking queries containing internal company names. Or allowing send_email but only to approved domains.
- Audit trails for compliance
For those in regulated industries (healthcare, finance) — how are you producing evidence that every agent action was governed? Are LangSmith traces sufficient for SOC2/HIPAA, or do you need something more structured (SARIF, signed decision records)?
What I’ve been exploring:
I’ve been building governance middleware that handles these at the agent lifecycle level — deterministic policy evaluation at each hook point (before_model, wrap_tool_call, after_tool, after_model). The key constraint I landed on: no LLM in the governance path. Everything is regex + policy rules, so it’s fast (under 5ms) and reproducible.
The pattern that works: multi-stage defense where PII is redacted before the model ever sees it, rather than trying to filter the response after the fact.
Would love to hear what patterns others have found. Are most people rolling their own, or using the middleware ecosystem?
Hello @nagasatish ,
Really good framing, especially the “no LLM in the governance path” constraint, that’s what keeps policies auditable and fast enough to run on every hook.
PII in RAG: Post-hoc filtering is the wrong layer, agreed. Best defense is two-tier: strip/tokenize PII at ingestion (never let it into the vector store), then run deterministic scanners on tool output before the next model call. LangChain’s PIIMiddleware does the second part well, regex detectors with redact/mask/block, plus apply_to_tool_results=True so retriever output gets scrubbed automatically.
Costs: ModelCallLimitMiddleware / ToolCallLimitMiddleware are good first stops, but they cap call counts, not dollars. For real budget caps you’ll want a small custom wrap_model_call middleware that tracks response.usage_metadata and raises before the next call. LangSmith alerts help you notice overspend, but they don’t prevent it.
Tool arg governance: Allowlisting a tool only answers “can this run at all”, argument-level control needs custom wrap_tool_call middleware inspecting request.tool_call["args"] before handler(request) fires. HumanInTheLoopMiddleware is the right call for high-impact actions like payments or emails.
Audit trails: LangSmith traces are great for debugging, but compliance teams usually want structured, immutable records — policy ID, matched rule, decision, timestamp, written to a separate audit sink. Traces supplement that, they don’t replace it.
Your instinct is right: deterministic policy at before_model / wrap_tool_call / after_tool / after_model, ordering PII redaction before the model call and arg checks before execution.
@keenborder786 This is exactly the breakdown I was hoping for — thanks for the detailed response.
The two-tier approach for PII (strip at ingestion + scan at tool output) makes sense. The gap I keep running into is that most teams inherit vector stores they didn’t build — migrations, shared indexes, third-party data sources — so you can’t always guarantee PII was stripped at ingestion. That’s why the after_tool / tool output scanning feels like the non-negotiable minimum.
On costs — agreed that ModelCallLimitMiddleware caps calls, not spend. The wrap_model_call approach tracking response.usage_metadata is what I ended up with too. The tricky bit is multi-model agents where one request fans out to 3-4 different models with different token prices. Need the cost tracker to be model-aware.
The audit trail point resonates the most. LangSmith traces are great for debugging but they’re:
Mutable (can be deleted)
Unstructured (no policy ID → decision mapping)
Tied to one vendor’s platform
For compliance you need something like: {policy_id, matched_rule, action, risk_score, correlation_id, timestamp} written to an immutable sink. That’s the part I’ve been packaging as structured decision receipts.
FWIW, I’ve been building this as an open-source middleware that bundles all four stages — it’s called TealTiger and it’s listed in the LangChain middleware integrations. One install, multi-stage defense, no LLM in the path. Happy to share more if anyone’s experimenting with similar patterns.
@nagasatish thank you, if it helped you will really appreciate it if you could close this thread by marking the answer as solution.
for a true hard cap, I’d enforce it as a pre-debit reservation at the agent’s step boundary. Before each model plus tool step, estimate the worst-case cost and atomically reserve it against both the session budget and the user’s daily budget. If either reservation fails, inject a forced-final-answer state rather than throwing and leaving the run half-broken. Then reconcile against actual usage afterward.
That estimate is the hard part. How are you thinking about bounding a tool step you haven’t run, especially when its output size or downstream calls vary? Those ledger rows can also serve as the signed allow/deny record for the audit trail.
The pre-debit reservation model is the right shape for a *true* hard cap — reserve worst-case before the step, reconcile against actuals after. It’s the only way to guarantee you never overspend rather than noticing you did. Two things I’ve had to work through building this:
**Bounding the worst-case cost of a step you haven’t run.** You can’t predict it exactly, so I stopped trying to and instead bound it with declared per-tool ceilings. Each tool carries a `max_tokens_out` (or `max_cost`) in its policy config; the reservation for a step is `input_tokens_priced + tool_ceiling + model_output_ceiling`, priced per the model that step will use. It’s deliberately pessimistic — you reserve more than you’ll usually spend and release the difference on reconcile. For tools that fan out to downstream calls, the ceiling has to cover the whole subtree, which in practice means either a declared max fan-out or treating “unbounded fan-out” tools as requiring approval rather than a reservation. Tools with no declared ceiling fail closed to a conservative default. It’s not free — someone has to set the ceilings — but it makes the cap enforceable instead of aspirational.
**Multi-model pricing.** Agreed this is the sharp edge. The reservation and the reconcile both have to be model-aware — a single request fanning out to a cheap router model + an expensive reasoning model can’t share one price. I key the ledger rows by `(model, step)` and price each leg independently against a pricing table, so `usage_metadata` from each model reconciles against its own reservation. A flat per-request estimate quietly breaks the moment the agent mixes models.
**Forced-final-answer over throwing** — strongly agree. Raising mid-run leaves the graph half-executed and the audit trail ambiguous (“was this denied, or did it crash?”). Injecting a terminal state gives you a clean, recorded outcome: budget-exceeded is a *decision*, not an exception. And yes — those ledger rows double as the signed allow/deny audit record, which is the same structured `{policy_id, matched_rule, action, risk_score, correlation_id, timestamp}` shape I mentioned upthread. One reservation row = one governance decision = one audit entry is a nice property.
One correction to my own earlier framing, since it matters for anyone implementing this against the current middleware: I listed `after_tool` as a hook point, but `AgentMiddleware` doesn’t actually have a post-tool hook — the lifecycle is `before_agent` / `before_model` / `after_model` / `wrap_model_call` / `wrap_tool_call` / `after_agent`. So **post-tool result scanning has to run inside `wrap_tool_call`**: call `handler(request)`, then scan the returned `ToolMessage` before returning it. Same place the reservation reconcile naturally lives, actually — you’ve already got the tool result in hand there. (For async agents that’s `awrap_tool_call`; the sync-only version raises `NotImplementedError` under `ainvoke`/`astream`.) Worth being precise about because “scan after the tool” is conceptually a stage but not a separate hook.