How are you handling auditability and policy enforcement in multi-step agents?

OSS Product Help LangChain
1

Hey LangChain community,

As more of us move agents into production — especially those with tool calling, memory, and real-world actions — the gap between “it works in the notebook” and “it’s safe, auditable, and compliant in production” becomes obvious.

I’ve been working on Anchor, an open-source deterministic governance engine designed for this exact layer.

Core Capabilities:

  • One signed constitution.anchor that governs both code (static AST + behavioral sandbox) and live runtime decisions
  • Cryptographic Decision Audit Chain (tamper-evident, hash-chained)
  • Regulatory polyglottism — one record satisfies EU AI Act Art 12, RBI FREE-AI, CFPB Reg B, SEC priorities, etc.
  • Sovereign Relay architecture so raw prompts/responses never leave your infrastructure

Links:

I’m particularly interested in hearing from the community:

  1. How are you currently handling runtime policy enforcement and auditability in complex agent graphs?
  2. What failure modes have you seen (or worry about) when agents get write access?
  3. Would a standardized governance layer be useful in the LangChain ecosystem?

Would especially love feedback from people running agents in regulated or high-stakes environments.

Thanks!

4

Anchor looks like a strong answer for the decision/action side of this tamper-evident audit chains and regulatory mapping for what the agent did.

One gap that sits one layer earlier: most policy enforcement and decision-audit frameworks assume the input context the agent reasoned over was valid at the time. In practice, retrieval is time-blind — a superseded regulation or an outdated internal policy scores the same semantic similarity as the current version, so the agent can be perfectly policy-compliant in its actions while reasoning from stale source material.

A complete audit trail probably needs both: Anchor-style decision/action logging, plus a point-in-time freshness record on what the agent was allowed to see before it acted. Without the second layer, “the agent followed policy” and “the agent acted on accurate information” are two different, unverified claims.

We built the freshness-scoring half of this for RAG pipelines specifically. happy to compare notes on where the two could compose.

5

Thanks for taking the time to read it @VLSiddarth ,
this is a really thoughtful observation.

I agree that decision auditability and retrieval provenance answer different questions. An agent can faithfully execute a policy while reasoning over information that’s already outdated.

Interestingly, we’re working on a companion project called Canon, which deterministically monitors governance sources, tracks rule changes through versioned evidence packages, and produces tamper-evident records before updates are approved into Anchor. While it’s currently focused on governance frameworks rather than RAG corpora, the underlying idea is very similar: proving exactly what information was considered valid at the time a decision was made.

I think there’s definitely room for these approaches to complement each other. I’d be interested to learn more about how you’re scoring freshness and recording provenance in your pipeline.

6

Great thread - most of what Anchor describes maps onto primitives that already ship in LangChain v1 / LangGraph, and being precise about which layer does what is where the real gaps (and Anchor’s value) show up.

  1. Runtime enforcement. The framework’s design principle is that policies which can’t live in a prompt are enforced by middleware, not the model: “Guardrails intercept data as it flows through the agent loop… before tool results reach the model’s context” (docs). The deterministic policy-decision-point is wrap_tool_call - it sits between the model’s decision and the side effect and can deny/rewrite the call regardless of what the model intends:
@wrap_tool_call
def policy_gate(request, handler):
    if request.tool_call["name"] in WRITE_TOOLS and request.runtime.context.user_role not in {"admin","operator"}:
        return {"role":"tool","content":"DENIED","tool_call_id":request.tool_call["id"]}
    return handler(request)

Built-ins already cover a lot: PII (block/redact/hash), tool-call limits (runaway loops), and role-based dynamic tool filtering (middleware).

Audit. LangSmith captures every tool call/decision/output with custom metadata (policy_version, user_id), exportable via OTel (Fleet audit trail). Checkpointers add replayable state - the docs literally call them “an audit trail and a recovery point to inspect the exact state that led to an action” (durability). The genuine gap Anchor fills: this is an audit log, not a tamper-evident chain. Native isn’t hash-chained, so your Decision Audit Chain is non-redundant if you need cryptographic non-repudiation; otherwise OTel-export to WORM storage is the pragmatic 80%.

  1. Write-access failure modes. Layered native answers: HITL interrupts pause before irreversible tools (approve/edit/reject), Deep Agents path permissions (allow/deny/interrupt) bound the blast radius, and sandboxes isolate code execution. One trap worth flagging: “path-based restrictions alone cannot prevent filesystem access through shell commands” - permissions don’t cover the sandbox execute tool or custom/MCP tools (permissions). Mitigate with sandboxes + backend policy hooks there.

  2. Standardized layer? Middleware is already the de-facto standard extension point - so the highest-leverage form for a governance layer is middleware + a LangSmith export sink, not a parallel AST/sandbox/relay runtime that risks drifting from how the graph actually executes. Ride the existing enforcement/audit rails; add only what’s truly missing (tamper-evidence + regulation mapping).

On @VLSiddarth’s time-blind retrieval point - spot on. A decision log records what the agent did, not whether what it read was current. There’s no built-in freshness score, but since retrieval runs inside the graph, retrieved doc IDs + effective_date/superseded_by metadata land in the checkpointed state and trace - so “what was read and how current it was” becomes a recorded, replayable fact, provided you version the chunks. Adjudicating which version was in-force at decision time (Canon’s job) is genuinely outside the framework’s scope, so the two are complementary, not redundant.

7

@pawel-twardziak nailed the gap: the checkpointed state records what was retrieved, but not whether it was current at decision time. That’s the piece that’s genuinely outside the framework’s scope and it’s the one that matters most for regulated pipelines.

The problem isn’t just auditing that a document was retrieved. It’s that the same chunk ID can represent different states of knowledge depending on when it was read. A RAG pipeline that ingested a Stack Overflow answer in 2024 and a LangChain repo README last week are referencing information that aged at completely different rates and no amount of LangSmith tracing tells you which one was still authoritative at decision time.

We built Knowledge Universe (https://api.knowledgeuniverse.tech/) specifically around this. Every source gets a Knowledge Decay Score computed from publication date and platform-specific half-life constants, GitHub at 180 days, arXiv at 3 years, HuggingFace at 120 days. The score travels with the source through the entire pipeline so the audit record contains not just what was retrieved but how fresh it was when it was used.

The response also now includes a Knowledge Velocity field that answers a question Anchor and LangSmith can’t: is this domain evolving fast enough that my 4-hour Redis cache is already wrong? A freshness score on a static chunk doesn’t tell you that. Velocity does.

@Tanishq1030, the Canon model of versioned evidence packages with tamper-evident approval records is exactly the right frame for governance sources. The provenance gap I’d add: Canon tracks when a rule changed, but the RAG layer that reads those rules also needs to know when the interpretation of that rule - the SO answers, the GitHub issues, the forum threads - has drifted out of sync with the canonical version. That’s where decay scoring on the retrieval layer and Canon’s governance layer become directly complementary rather than parallel efforts.

Happy to share the decay formula and half-life constants if useful for Canon’s evidence freshness model.

8

Thanks for taking the time to write such a detailed response, @pawel-twardziak.

I think your distinction between existing framework primitives and the remaining governance gaps is exactly the clarification I was hoping to get from this discussion.

I completely agree that middleware is the natural enforcement point. Anchor isn’t intended to replace LangGraph or become another orchestration framework it’s intended to integrate with existing execution frameworks and provide deterministic governance, cryptographic decision integrity, and regulation-aware policy enforcement around them.

Your point about riding the existing middleware and audit rails rather than duplicating them is particularly valuable, and it’s something I’ll incorporate as Anchor Runtime evolves. The focus should be adding what’s missing rather than rebuilding what’s already working well.

Thanks again for the thoughtful feedback.

9

Thanks, @VLSiddarth. I think you’ve identified a really interesting complementary problem.

Canon’s current focus is deterministic governance provenance tracking authoritative governance sources, producing versioned evidence packages, and recording tamper-evident approval records before updates reach runtime enforcement.

What you’re describing addresses a different but equally important layer: the freshness and evolution of the knowledge an agent reasons over. I like the distinction between what was retrieved and whether it was still authoritative at decision time.

Rather than trying to make Canon estimate knowledge decay itself, I think a better architecture is for Canon to consume externally produced freshness metadata as evidence alongside governance provenance. That keeps the responsibilities clean while allowing both systems to complement each other.

I’d definitely be interested in reading more about your decay model and the half-life constants you’re using. Thanks for sharing the idea.

10

@Tanishq1030, Glad the distinction landed clearly. The “externally produced freshness metadata as evidence” framing is exactly right and maps cleanly to what we built.

The decay model uses platform-calibrated half-lives rather than a single universal constant, because source type is a stronger predictor of relevance decay than age alone:

  • GitHub repo: 180 days - code goes stale as dependencies update

  • Stack Overflow answer: 365 days - tied to library versions

  • arXiv preprint: 1,095 days - research ages slowly

  • Hugging Face model card: 120 days - ML moves fast

  • Regulatory document: domain-specific, 30–90 days for active rulemaking

The formula is deterministic: decay = 1 - 0.5^(age_days / half_life_days)

Every retrieved document gets a decay_score (0.0 = fresh, 1.0 = fully stale), a knowledge_velocity label, days_until_stale as an integer, and a conflict_detection flag when two sources in the same retrieval set contradict each other.

For Canon’s evidence package specifically, the point-in-time snapshot is already included — each scored document carries the timestamp of what was knowable at retrieval time, which makes the evidence auditable rather than inferred after the fact.

The live API is at api.knowledgeuniverse.tech — free tier, 500 calls, no card. Happy to discuss how the metadata schema could map into Canon’s evidence format if that would be useful.

11

Thanks for sharing the implementation details. I appreciate you taking the time to explain the model and expose the API.

I like the distinction between governance provenance and retrieval freshness they answer different questions but compose well together.

Canon is intentionally focused on authoritative governance artifacts and deterministic evidence packages, while your decay model addresses the evolution of the broader knowledge an agent reasons over. I can definitely see value in exploring how these metadata models could complement one another rather than overlap.

I’ll spend some time looking through the documentation and API. Thanks again for sharing it.

12

Really useful thread. One distinction I’d separate explicitly: a production agent usually needs two audit records, not one.

  1. A decision/action record: what did the runtime authorize before the side effect, under which policy, with what input scope, and what happened afterward?

  2. A source/freshness record: what information was the agent allowed to reason over, and was that information still authoritative at decision time?

LangGraph middleware, checkpoints, and LangSmith traces cover a lot of the runtime shape. Anchor-style tamper-evident chains seem complementary when the decision record needs independent verification. Freshness metadata is complementary again: a perfect decision log can still be wrong if the agent reasoned from stale policy or stale retrieved context.

The portable interface I’d want to compare is a minimal pair of receipts: decision_receipt and evidence_freshness_receipt. That keeps execution, governance, and provenance modular instead of forcing all three into one framework layer.