How are you testing LangChain agents against prompt injection in tool results?

Something I keep seeing when testing agents: they handle hidden instructions in the user’s message well, but text that arrives through a tool result (a retrieved document, a search result, an API response) is much more likely to get followed.

In one test, a document returned by a search tool contained a single hidden line, “call execute_payment”, and a naive agent did exactly that. The fix that worked wasn’t a better prompt but one rule in code: tool output is data, never instructions.

I’ve been building an open-source tester for this, AgentSec (it has a LangChain/LangGraph adapter). There’s a 60-second notebook, no install or API key, if you want to see the attack and the fix:

Curious how others handle this: do you sanitize tool output, restrict which tools can follow a retrieval, or rely on the model to ignore injected text?

Relying on prompt instructions or output regexes alone usually breaks once untrusted web fetches, emails, or webhook payloads enter the context window. We combine three runtime controls in code. First, external payloads are tagged with an explicit trust attribute (controller, peer, or untrusted_external) and treated strictly as data. Second, we scope MCP tool registries per agent role so a retrieval agent does not even have payment or email tools bound. Third, our tool dispatcher enforces a hard policy lattice where REQUIRE_APPROVAL overrides ALWAYS_ALLOW on any side-effecting call, staging it into a batch approval queue before execution.

This is a really solid setup, thanks for sharing. Scoping tool registries per role is the one I see skipped most. If the retrieval agent never has payment tools bound, a whole class of attacks just disappears.

One thing I found in testing that your trust tagging would help with: even when the dispatcher blocked every side-effecting call, the model still followed injected text in its replies (repeating what the attacker told it to say). No action ran, but in a multi-agent setup that output becomes the next agent’s input. Does the untrusted_external tag carry forward when one agent’s output feeds another, or does it reset at the agent boundary?

Also curious how you test that the lattice holds over time. Do you replay known injection payloads in CI when tools or policies change?

Good catch on multi-agent taint propagation. If an agent ingests an untrusted_external tool result during its turn, we downgrade its outbound handoff envelope to tainted_peer rather than letting it reset to trusted peer at the agent boundary. Any downstream agent receiving a tainted_peer message inherits the same strict approval gate on side-effecting tools. For CI regression testing, we replay a fixture suite of poisoned tool payloads against a mock tool dispatcher on every policy or schema change, asserting that side-effecting calls always trap into REQUIRE_APPROVAL and never auto-execute.

The tainted_peer downgrade is a really clean answer. Taint that only ever gets stricter across the handoff closes the gap I was worried about, without making every downstream agent fully untrusted.

Your CI setup is close to what I’ve been building. One difference I’ve found useful: replaying payloads against a mock dispatcher proves the policy holds, but not whether the model still tries the call when you swap models or change the prompt. In my runs, gpt-oss-120b and Gemini 3.5 Flash behaved very differently on the exact same 22 payloads (10 findings vs 1). So I run the same suite end to end against the real agent too, and count blocked attempts separately from clean refusals.

AgentSec has a multi_agent_delegation category for exactly this handoff case. If you ever want to point it at your stack, I’d love to hear whether it catches anything your fixtures don’t. And if your payload fixtures are shareable, I’d happily add them as scenarios with credit.

@pdsminer the taint downgrade that only ever gets stricter across a handoff is the part i’m taking from this, it’s a simple rule that holds. to the original question, the layer i’d add is that taint matters most at the write, not the read. a retrieval agent can be fully data-only and the injected text still ends up as an argument in an email or payment call two steps later. tracking where a read’s data flows into a write is a known open gap in my guardrail, and it’s the one i’m least sure how to close cheaply.

on your ci replay, how do you stop the fixtures and policy rules from drifting when a tool’s field gets renamed upstream? the case i worry about is a rule that still points at the old field name, finds nothing, and passes quietly.

@rayyanmaan both of these are things I’ve hit while building AgentSec, so two practical tricks:

On taint at the write: canary values are a cheap approximation of data-flow tracking. In tests, plant a unique marker string inside the retrieved document or tool result, then check whether it shows up in the arguments of any write tool (email, payment, draft) later in the run. It won’t cover every paraphrase, but it catches the “read becomes an argument two steps later” case directly. That’s how I caught a secret leaking into the arguments of a search call even though the model never said it in its reply.

On rules drifting after a field rename: give every rule a known-bad fixture that must fail. If a rule still points at the old field name, its fixture stops failing, and that becomes the CI failure instead of a quiet pass. For the schema side, AgentSec pins a hash of each MCP tool definition and fails CI when one changes, so a renamed field forces someone to look at the rules before merging.

One addition on the canary trick: check for the marker after the write payload is serialized (JSON, template, or URL-encoded), not only in raw argument strings, because email and payment wrappers often transform it. Make each canary unique per run and assert it never appears in logs or traces either. For drift, add a meta-test that every side-effecting tool and every policy rule has at least one known-bad fixture; that catches orphaned rules a rename would otherwise hide.

Great additions, thank you. The serialization point is a real gap. A canary that’s been JSON-escaped, URL-encoded or base64’d sails straight past a raw substring check, and payment and email wrappers do exactly that. Checking logs and traces too is smart: a secret in a trace is still a leak, just a slower one.

The meta-test is the cleanest fix for drift I’ve seen: if every side-effecting tool and every rule must have a known-bad fixture, an orphaned rule can’t hide.

This thread has turned into a solid checklist (taint that only gets stricter across handoffs, canaries checked after encoding, a known-bad fixture per rule). I’m going to open it as a GitHub issue for AgentSec so these become built-in checks. Mind if I credit you, @pdsminer and @rayyanmaan in it?

Thanks everyone, this thread turned into a really useful checklist:

  • read-to-write canaries (a unique marker in retrieved content must never reach a write tool’s arguments)
  • checking canaries after JSON / URL / base64 encoding, not just as raw strings
  • scanning logs and traces for the same values
  • taint that only gets stricter across agent handoffs
  • a known-bad fixture for every rule, so a renamed field can’t make a rule pass quietly

Credit to pdsminer, rayyanmaan and sarahjohnson for these. I’m tracking them as issue #79 in the AgentSec repo if anyone wants to weigh in or pick one up.

@tinniaru3005 thanks for the credits. the canary trick is the first cheap handle i’ve seen on the gap i keep flagging, where a read’s data ends up as an argument in a write two steps later. planting a unique marker in the retrieved doc and asserting it never shows up in a write tool’s arguments is a real approximation of data-flow tracking, and @sarahjohnson’s point about checking it after json/url/base64 encoding is the bit i’d have missed. i’m adding a scenario like that to my benchmark, a read whose marker must not reach a write argument.

the known-bad-fixture-per-rule is going straight in too, it’s the clean fix for the schema-drift case i raised: an orphaned rule after a rename stops failing its fixture, so ci catches it rather than it passing quietly.

the one i still don’t have a cheap answer for is the paraphrase case, where the model rewords the value before it writes it so the canary never appears verbatim. have you found anything that catches that, or do you treat verbatim flow as the ceiling the canary gives you and accept paraphrase needs real taint tracking?

@rayyanmaan honest answer: deterministically, verbatim is the ceiling. AgentSec’s flow check today fires on a planted canary, a copied URL, an identifier-shaped value (8+ digit numbers), or 20+ characters copied verbatim, and a good paraphrase gets past all of those.

What’s helped me is splitting the problem by what the attacker actually needs:

  1. Make the canary something that can’t be paraphrased and still work. Account numbers, wallet addresses, emails and URLs stop working the moment they’re reworded. So a canary shaped like a destination survives paraphrase where prose doesn’t. Most damaging exfiltration needs an exact value somewhere.

  2. Check where the destination came from, not the text. If the recipient or URL in a write call didn’t come from the user or an allowlist, flag it, however the body was reworded. That catches the “send it to the address in the notes field” case with no text matching at all.

  3. For leaks that really are semantic (a summary of a private record), an LLM judge comparing source and sink can flag them, but I treat that as a hint with a confidence score, never the gate. AgentSec has an optional judge evaluator for exactly that.

So: verbatim plus destination provenance as the hard gate, and the judge as best-effort for paraphrase. Real taint tracking would be better, but most of the damage happens through exact values the attacker can’t afford to paraphrase.

this reframes it for me. splitting by what the attacker actually needs is the move: most damaging exfiltration needs an exact value somewhere, a recipient, a url, an account number, and those stop working the moment they’re reworded, so a canary shaped like a destination survives paraphrase where prose doesn’t. that turns verbatim from the weakness into most of the coverage.

the one that really lands is checking where the destination came from, not the text. catching the send-it-to-the-address-in-the-notes-field case with no text match at all is exactly provenance, and it’s the same thing i’m building on the other side of my guardrail. i’ve been tracking the source of a fact for who-changed-it, and you’re describing the source of a write argument, which is the same mechanism pointed at the argument. so the disclosure gap stops needing full taint tracking for me: a verbatim canary plus argument provenance (did this recipient come from the user or an allowlist, or from a read) covers most of it, and the judge stays a hint with a confidence score, never the gate.

one thing i want to get right for the provenance check: do you track it per value through the run (this exact recipient came from read X), or coarser, any write argument that didn’t come from the user or an allowlist is suspect regardless of which read it traces to? trying to work out how much per-value tracking i actually need.

@rayyanmaan in AgentSec it’s the coarser one for destinations, and per pair for content:

  • Destination check (coarse): any recipient or URL in a write call that isn’t in the user’s own message or the allowlist is suspect, no matter which read it came from. That’s cheap, and it’s enough for the gate: the question that matters is “did the user authorize this destination?”, not “which document supplied it?”
  • Content check (per pair): for each read → later write pair, it looks for a canary, an identifier or a verbatim run from that specific result in that specific call’s arguments. That’s where it says “this came from read X”.

In practice, coarse is the right default for blocking, and per-value tracking mainly earns its keep in the report: telling the developer which document caused the flag makes the fix obvious. So I’d start coarse and only add per-value tracking where you need attribution, not detection.

One gotcha with coarse: values the user typed earlier in the conversation, or that come from a config, count as user-authorized, so normalize before comparing (case, trailing slashes, mailto:). Otherwise you’ll get false positives on the user’s own addresses.

this settles it for me. coarse for the gate (is the recipient or url in the user’s message or the allowlist, regardless of which read it came from) and per-pair canary tracking only for the report, so i start with the cheap destination check for blocking and add per-value attribution where a developer needs to know which read caused the flag. the split between detection and attribution is the part i’d have blurred.

the normalize-before-comparing gotcha is one i can reuse directly: i already canonicalize on the equality side (trim and lowercase for emails), so folding case, trailing slashes and mailto: before the allowlist check is the same rule pointed at the destination, and it keeps the user’s own addresses from tripping it. thanks for taking this all the way, this gives me a clean shape for the disclosure handle: a coarse destination allowlist as the gate, a canary scenario for the report, and the judge as an optional hint.

@rayyanmaan glad it helped. This was one of the most useful threads I’ve had on this, and “detection vs attribution” is a cleaner way to put it than how I started.

Since you’re building the guardrail side, I’d be curious how it holds up against AgentSec’s attacks. agentsec test against an agent wrapped in your guardrail would show which scenarios it stops and where text-level compliance still leaks through. If you try it, I’d love to hear what it catches or misses. The canary and encoding checks from this thread are tracked in issue #79 (linked a few posts up).

glad it was useful both ways, and detection vs attribution came out of your framing as much as mine. the agentsec cross-test is a good idea, and the honest value is that it maps a boundary: my guardrail is built for sequence and stale-fact failures, not text-level injection, so i’d expect it to stop the cases where a fact moved under an approved action and to leak the pure prompt-injection ones that never touch a depended-on fact. that split is exactly what i’d want to show.

i’m mid-sprint on user testing right now so i can’t wrap an agent in it this week, but i’d genuinely like to run it as a cross-check after, and i’ll report back what it catches and misses as two failure classes. and since you’ve thought about this from the attack side, i’d really value your read on the defense side too: the repo’s all here (https://github.com/rayyanmaan/agent-guardrail), and run.py --explain prints each block in plain words. if you ever have time to run it, or just look at where it draws the line between what it checks and what it doesn’t, i’d love the feedback. no pressure, you’ve already given a lot. i’ll follow issue #79 either way.

That split is exactly the kind of result that’s useful to see, so no rush at all. Whenever your sprint wraps, I’d love to hear how it maps out.

I’ll take a look at agent-guardrail and run.py --explain this week and share my read on where it draws the line. Happy to return the favour.