The observation
I’ve been measuring AI coding reliability for 2 months — not just code correctness, but whether the agent actually does what it says it did. The finding that surprised me:
| What the check examines | Triggers | False positives |
|---|---|---|
| Filesystem facts (timestamps, paths) | 4–24 | 0% |
| Command strings (what was actually run) | 6 | 0% |
| Text-based (“did human confirm?”) | 295 | 100% |
The first two work because the model cannot fabricate filesystem metadata or command history. The third fails because anything the model can generate as text, it can fake. 295 trigger events, zero real violations — the AI had written a 2-line script to auto-answer “yes.”
The architectural property: mechanical checks and semantic review have non-overlapping blind spots. Mechanical checks miss stale rules and intent drift. Semantic review misses uniform-rater collapse (κ→0). Combined, their joint false-negative rate approaches zero — defense-in-depth applied to agent verification.
Important context on the data: these experiments ran on Claude Code’s PreToolUse hooks (which can block tool calls via exit 2), not on LangChain. I’m sharing the pattern, not claiming direct evidence for LangChain’s execution model.
Why I’m sharing this here
LangChain’s middleware API (wrap_tool_call, before_model) is the strategic direction for customizing agent behavior. The prebuilt middleware suite (PIIMiddleware, ToolCallLimitMiddleware, HumanInTheLoopMiddleware, etc.) covers content safety, cost control, and resilience.
But there’s a category I haven’t seen covered: process integrity — verifying that the action an agent claims to have taken actually happened in the environment. Content safety asks “is the output safe?” Process integrity asks “did the operation actually execute?”
I don’t know if this category matters to LangChain users. The data I have is from AI coding (terminal/IDE agents), not from the customer-support/data-pipeline/automation deployments that LangChain serves. The pattern might not transfer. Or it might, in a different shape.
Where I think it might fit
My tentative read: this lands in the middleware API, not callbacks and not LangGraph interrupts. Here’s a rough sketch of why — not a proposal, just something concrete to react to:
# Hypothetical: a middleware that cross-checks tool results
# against a light verifier function.
class ProcessGateMiddleware(AgentMiddleware):
def __init__(self, verifiers: dict[str, Callable]):
self.verifiers = verifiers # tool_name → verifier_fn
def after_tool_call(self, tool_call, result):
verifier = self.verifiers.get(tool_call["name"])
if verifier and not verifier(tool_call, result):
return ToolMessage(
content="Process gate failed: action was not verified",
tool_call_id=tool_call["id"],
status="error",
)
return result
A file_write tool might verify that a corresponding file_read happened within 30 seconds. A db_query tool might verify the result against a known-good query plan. The verifier functions are user-supplied; the middleware provides the hook and the audit trail.
This might be completely wrong. The API shape might need wrap_tool_call instead, or it might belong in LangGraph’s state machine. Happy to be corrected — giving a concrete sketch beats asking an abstract question.
What I’m proposing
A discussion about whether process-integrity verification fills a gap in LangChain’s middleware ecosystem, and if so, what shape an integration should take.
What I’m doing regardless
Building a standalone package that implements process-integrity gates (filesystem timestamps, command-string allowlisting — the two types with the strongest data behind them). If it works for LangChain users, great. If it never finds a fit, I’ll write up what I learned. Either way, the data earns its keep.
*Context: undergrad, 2-month AI coding reliability experiment, 339 blocked-event records, 25 hook registrations across 7 event types. (Edit Jul 26: corrected “38 mechanical gates” — verified against actual settings.json.) Learning where the pattern fits (or doesn’t) in the broader agent ecosystem. Here to listen, not to pitch.