# Feedback requested: evaluating commit-time action validity in LangGraph workflows **URL:** \ **Category:** OSS Product Help\ **Tags:** intro-to-langgraph, product-feedback, feature-request\ **Created:** [July 17, 2026, 1:06pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178 "2026-07-17T13:06:09Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 17, 2026, 1:06pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/1 "2026-07-17T13:06:09Z")
I am exploring a LangGraph adapter for CAV-Bench, an open-source deterministic benchmark for evaluating whether tool-using agent actions remain valid when they commit. The benchmark focuses on failure patterns that final-state and task-completion metrics can miss: - state changing between planning and execution; - duplicate side effects after ambiguous retries; - authorization changing before commit; - partial multi-system execution; - incomplete compensation or escalation. My initial design would map LangGraph checkpoints, tool invocations, state transitions, retry behavior, and recovery paths to five validity dimensions: intent, authority, state, execution, and recovery. Before implementing the adapter, I would appreciate feedback on two questions: 1. Would an external adapter be the most useful integration model, or would a standalone LangGraph example be more appropriate? 2. Which runtime events, checkpoint interfaces, or state-transition records would provide the most reliable evidence that an external effect was attempted, committed, reconciled, or compensated? I am considering an initial scenario pack covering stale-state commits, ambiguous retries, duplicate external effects, and recovery after partial execution. This is an independent community project, not an official LangGraph benchmark. I am seeking technical direction before building the integration, not endorsement or formal adoption. Repository: [GitHub - Harimay23/cav-bench: Benchmarking commit-time validity for tool-using AI agents. · GitHub](https://github.com/Harimay23/cav-bench) Version DOI: [CAV-Bench: Commit-Time Action Validity Benchmark | Zenodo](https://doi.org/10.5281/zenodo.21364386) ---
**Author:** ![keenborder786](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/keenborder786/32/1757_2.png) [@keenborder786](https://forum.langchain.com/u/keenborder786)\ **Post date:** [July 17, 2026, 3:23pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/2 "2026-07-17T15:23:08Z")
Hello @Nixalp, Really like how tightly scoped this is. Since your two questions here mirror the “Questions for framework maintainers” in `docs/framework-adapter-brief.md` (Issue #3), I’ll answer in your own terms, normalized event model + the trust boundary, rather than generic LangGraph advice. **Q1 — adapter or standalone example?** Your architecture already answers it: implement `ExecutionAdapter.run(session) -> AdapterResult` and ship it as an **optional extra** (`pip install cav-bench[langgraph]`), keeping LangGraph out of the core deps per your first-adapter acceptance criteria. A standalone LangGraph graph is worth including too, but only as a **test fixture** that exercises the four scenarios through that adapter — not as the integration surface. So: external adapter as the product, reference graph as a fixture. **Q2 — most reliable evidence, and attempted vs. committed.** One framing point first, because it drove my thinking: given your trust boundary (_the system under evaluation can never grade itself_), LangGraph’s checkpoints/state are **not** a source of commit truth — they’re only evidence to translate into your event model. In particular, `effect_committed` must come from the **tool/environment boundary** , not from `on_tool_end` or graph state. So bind every consequential tool to the session facade so the environment records the real commit; use LangGraph only for ordering, attempts, retries, and revalidation context. Normalized event → LangGraph source: - `intent_recorded` → initial user message captured at run start (graph input / `values.messages`). - `authority_checked` → a human-in-the-loop [`interrupt`](https://docs.langchain.com/oss/python/langgraph/interrupts) gated by `interrupt_on`, resolved via `Command(resume=...)`. The resume decision is an externally observable authz record; in-node `if authorized:` logic is invisible unless you surface it as an interrupt or a facade call. - `state_read` / `state_revalidated` → facade read receipts carrying `observed_version`. Put the commit-time re-read in its **own node/task** so a checkpoint sits between read and commit — otherwise there’s no observable revalidation point. - `effect_attempted` → `on_tool_start` via [`astream_events`](https://docs.langchain.com/oss/python/langgraph/streaming) (or `stream_mode="tasks"`). - `effect_committed` → the facade’s commit receipt keyed by `operation_id` + resulting version. **Never** infer this from tool return alone — a lost/timed-out response can still have committed (your scenario #2). - `effect_reconciled` → an explicit status-read tool call for the same `operation_id` _before_ any retry. - `compensation_*` / `escalation_created` → explicit compensating tool calls / an escalation `interrupt` — each observable as its own run. - `outcome_reported` → final `GraphOutput` / AI message → maps to `adapter_report.completion_status`, which stays untrusted (overclaim check only). Mapping to the four scenarios: 1. **Stale state** : separate `read` and `commit` nodes; condition the commit on `expected_version`; the interposed checkpoint (visible via [`get_state_history()`](https://docs.langchain.com/oss/python/langgraph/persistence) lineage) is where `state_revalidated` lives. 2. **Ambiguous retry** : LangGraph re-runs a node from the super-step boundary on resume, and `RetryPolicy` retries, so the _same_ effect can fire twice ([re-execution & idempotency](https://docs.langchain.com/oss/python/langgraph/graph-api#re-execution-and-idempotency)). Thread a stable `operation_id` + `idempotency_key` through tool args; emit `effect_reconciled` when a status-read precedes the retry; your evaluator enforces ≤1 `effect_committed` per op. 3. **Partial workflow** : model steps as distinct nodes; a downstream failure leaves the first `effect_committed`; compensation/escalation must appear as explicit calls, and `outcome_reported` must not overclaim. 4. **Authority change** : two `authority_checked` events: one at planning, one at/just-before commit (a second gate across the checkpoint boundary); no `effect_committed` once authority is revoked. Two hard requirements for reproducible evidence: - Run with `durability="sync"`: the default `"async"` is usually fine, but `"exit"` only persists at graph exit and would erase the intermediate snapshots your `state_revalidated`/lineage evidence relies on ([durability modes](https://docs.langchain.com/oss/python/langgraph/checkpointers#durability-modes)). - Use fine-grained nodes (or the Functional API `task()` per effect): checkpoints land at super-step boundaries, not mid-node, so a node that bundles read+commit exposes no revalidation point. Honest gap (and per your brief, absence is itself informative): LangGraph has no built-in notion of “external commit confirmed” separate from a tool returning. Attempted-vs-committed is entirely a **tool-contract convention** the adapter must define, a receipt with `operation_id` + version + `idempotency_key`. That convention, not any LangGraph API, is the load-bearing piece of a faithful adapter. Happy to review the `ExecutionAdapter` sketch and the event-mapping table once you draft them. ---
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 17, 2026, 6:55pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/3 "2026-07-17T18:55:54Z")
Thank you for this detailed and highly actionable review @ **keenborder786**. Your distinction between LangGraph runtime evidence and external commit truth is especially helpful and aligns with the CAV-Bench trust boundary. Based on your guidance, I will proceed with: - An external optional LangGraph adapter as the integration surface; - A reference LangGraph graph used only as a deterministic test fixture; - The CAV-Bench session facade as the authoritative source for commit receipts; - Stable operation\_id and idempotency\_key values across retries; - Explicit nodes for commit-time revalidation, reconciliation, compensation, and escalation; - Synchronous durability and fine-grained node boundaries for reproducible checkpoint evidence. I will also document the important limitation you identified: LangGraph can expose attempts, ordering, retries, and workflow state, but confirmed external commit must come from the tool/environment boundary rather than graph state or on\_tool\_end. I’ll draft the ExecutionAdapter sketch and LangGraph event-mapping table next and share them here for review. Thank you again for offering to review them. ---
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 17, 2026, 8:22pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/4 "2026-07-17T20:22:54Z")
Thank you again for the detailed guidance. I incorporated your recommendations into the first LangGraph-specific design artifacts: **Implementation issue:** [Implement initial CAV-Bench LangGraph adapter · Issue #5 · Harimay23/cav-bench · GitHub](https://github.com/Harimay23/cav-bench/issues/5) **Design-stage pull request:** [Add LangGraph adapter design-stage skeleton and mapping spec by Harimay23 · Pull Request #6 · Harimay23/cav-bench · GitHub](https://github.com/Harimay23/cav-bench/pull/6) **LangGraph mapping specification:** [cav-bench/docs/langgraph-adapter-mapping.md at feat/langgraph-adapter-skeleton · Harimay23/cav-bench · GitHub](https://github.com/Harimay23/cav-bench/blob/feat/langgraph-adapter-skeleton/docs/langgraph-adapter-mapping.md) The current PR intentionally contains only: - the framework-specific event-mapping specification; - an ExecutionAdapter-compatible skeleton; - optional-dependency isolation; - contract tests confirming that the unfinished adapter fails honestly rather than fabricating a result. I incorporated the main points from your review: - external adapter as the integration surface; - reference graph as a deterministic fixture only; - session/tool facade as the authoritative source of commit truth; - on\_tool\_start or task-stream evidence for attempted effects; - externally observable authorization evidence; - explicit commit-time revalidation; - stable operation\_id and idempotency\_key; - durability=“sync”; - fine-grained nodes or tasks; - explicit recognition that LangGraph itself cannot prove an external commit. CI is passing, and I have left the PR open rather than merging it so that your technical feedback can still shape the design. When convenient, I would appreciate your review of the event-mapping table and adapter boundary. In particular, I would value confirmation or correction on: 1. the proposed LangGraph evidence sources; 2. the use of interrupts and Command(resume=…) for observable authority decisions; 3. the stale-state revalidation flow; 4. the ambiguous-retry reconciliation model. Thank you again for offering to review this. ---
**Author:** ![keenborder786](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/keenborder786/32/1757_2.png) [@keenborder786](https://forum.langchain.com/u/keenborder786)\ **Post date:** [July 19, 2026, 10:46am UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/5 "2026-07-19T10:46:13Z")
Hello @Nixalp , spend quite a bit time understanding your PR. Overall this faithfully captures the earlier discussion, and a couple of things are better than what we talked about: routing _all_ commit truth through `session.tools.write(...)` → `ToolFacade` → `BenchmarkEnvironment` and demoting LangGraph state to “context only” is exactly the right trust-boundary discipline, and the honest-failure skeleton (`run()` raising instead of fabricating an `AdapterResult`) is the correct design-stage move. Four points of confirmation + correction on what you asked: **1. LangGraph evidence sources — confirmed, with one asymmetry to fix.** The table is right, except `effect_attempted` and `effect_committed` currently come from two different channels: attempted from `astream_events`’ `on_tool_start`, committed from the facade `ToolResult`. That split forces you to correlate a LangGraph event back to a facade call by id, and it breaks whenever a consequential action isn’t a LangChain `@tool` (a plain node function calling `session.tools.write(...)` emits no `on_tool_start` at all). Make the **facade the single evidence spine** : emit `effect_attempted` at write-call _entry_ (synchronously, before the environment applies it) and `effect_committed` from the returned `ToolResult.status`. Treat `on_tool_start` / `stream_mode="tasks"` as _corroborating ordering_ only. Also note the granularity gap: `tasks` stream is per-node, `astream_events` is per-tool;, they only line up because of your one-consequential-action-per-node rule, which is worth stating explicitly as the reason. **2. Interrupts + `Command(resume=...)` for authority, right mechanism, but it’s observability, not truth.** An interrupt makes the authorization decision externally visible and puts it in the checkpoint + resume payload, which is good. But the resume value is supplied by the harness/human — it’s adapter-controlled _input_, so if the evaluator ever treats “resume said approve” as authorization truth, you’ve quietly re-opened a self-grading path. Keep the split explicit: the **interrupt proves an observable check occurred; the facade/`BenchmarkEnvironment` adjudicates whether authority actually held** at commit (e.g. a `session.tools.write(...)` / dedicated authority call that reads current permissions and returns `FORBIDDEN`/`CONFLICT` when the fixture has revoked them). In scenario #4, a revoked actor must get a hard refusal from the environment regardless of what the resume payload says. Your table already hedges toward “or a facade-recorded authorization check”, I’d promote that to the _primary_ truth and make the interrupt the optional observability wrapper. Script the resume value deterministically in the fixture (no human in CI). **3. Stale-state revalidation — the “re-evaluate, don’t just re-read the version” rule is great; close the residual TOCTOU.** Because there’s a super-step boundary between your revalidation node and your commit node, revalidate-then-commit is _itself_ a time-of-check/time-of-use window , an injected mutation landing between those two nodes would slip through if node ordering is what you rely on. So: let the revalidation node own the _semantic_ re-check (intent/scope/authority against current state), but the **version guard must be enforced atomically by the facade write via compare-and-set on `expected_version`** (→ `CONFLICT`), not by the fact that a revalidation node ran first. Concretely, add a fixture variant that injects the state change _between revalidation and commit_, not only before revalidation, that’s what proves the CAS path is load-bearing rather than the node graph shape. **4. Ambiguous-retry reconciliation — key model is correct; two sharp edges.** The “derive the key from durable checkpointed identity, never `uuid4()` per attempt” reasoning is exactly right. Two refinements: - **`thread_id + node_name` is not unique per operation under loops, `Send`/fan-out, or map-reduce** , a tool-loop that revisits the same node for different operations will collide. Fold in a _durable per-operation discriminator_ from checkpointed state (a monotonic op counter in graph state, or the loop/`Send` index) so the key is unique per logical operation yet stable across retries of the _same_ one. , **Distinguish the two retry paths** , because they don’t behave the same: a node-level `RetryPolicy` retries _within a run_ with no new checkpoint between attempts, while an interrupt/crash resume re-executes from the last checkpoint. Your key derivation has to be identical across both. The robust node shape that survives re-run-from-start in either case is to make reconciliation unconditional rather than gated on a “have I attempted?” flag: ```python result = session.tools.status_check(idempotency_key) if result.status != "COMMITTED": result = session.tools.write(..., idempotency_key=idempotency_key, expected_version=version) ``` One scoring note: your facade also exposes `IDEMPOTENT_REPLAY`, so a naive “just re-`write()` with the same key” adapter would _also_ avoid a duplicate at the ledger level , it satisfies **Execution** but produces no explicit `effect_reconciled`, so it should _not_ pass **Recovery**. Requiring the `status_check` (as you do) is what lets scenario #2 actually score the Recovery dimension, worth stating that this is deliberate. Finally, make sure the fixture injects the ambiguity _before_ the write node’s super-step checkpoints; if that step completes and checkpoints under `durability="sync"`, a resume restores its result and never re-runs it, so no reconciliation is exercised. **On the skeleton/tests:** the `sys.modules` isolation test and honest-failure contract are the right things to pin. Minor coverage gap: with `langgraph` absent in CI, `run()` always raises `ImportError` before reaching the `NotImplementedError`, so that branch is currently dead in CI. A small test that monkeypatches `_ensure_langgraph_installed` to a no-op and asserts `NotImplementedError` would pin the “installed but not yet implemented” half of the contract too. I hope this helps you out, if it did a like will be appreciated. ---
**Author:** ![system](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/system/32/1_2.png) [@system](https://forum.langchain.com/u/system)\ **Post date:** [July 20, 2026, 12:51pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/6 "2026-07-20T12:51:03Z")
---
**Author:** ![system](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/system/32/1_2.png) [@system](https://forum.langchain.com/u/system)\ **Post date:** [July 20, 2026, 1:37pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/8 "2026-07-20T13:37:12Z")
---
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 20, 2026, 3:34pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/9 "2026-07-20T15:34:42Z")
Thank you again for the detailed review. I incorporated the technical corrections and will proceed with the implementation. I appreciate the time you spent examining the design. ---
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 21, 2026, 12:47am UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/10 "2026-07-21T00:47:10Z")
Thank you again, @keenborder786. I wanted to close the loop now that the LangGraph integration has been implemented, reviewed through multiple technical passes and merged. Your feedback materially shaped the final design. The merged implementation now includes: - an optional external `LangGraphAdapter` as the integration surface; - deterministic reference graphs used only as test fixtures; - a `framework-v1` pack covering stale state, ambiguous retry, partial execution and authority change; - benchmark-owned trace and ledger evidence as the source of attempted and committed-effect truth; - stable operation and idempotency identities across retries and checkpoint resumes; - explicit reconciliation before any replayable write; - separate handling of `AMBIGUOUS`, `IDEMPOTENT_REPLAY` and confirmed commits; - fine-grained nodes and synchronous checkpoint durability. One correction that became especially important during implementation was the crash-before-checkpoint case. An external effect may commit inside a node, followed by a process failure before that node’s result is checkpointed. The guarded ambiguous-retry fixture now performs a stable-key status check inside the replayable write node before every possible write. This allows a resumed workflow to detect the earlier commit without blindly issuing the effect again. The implementation is tested against both the supported LangGraph floor version and the latest version, with 51 LangGraph-specific tests passing in each environment, along with the core CI matrix and built-wheel validation. The remaining limitation is explicit: fresh adapter-side authority revalidation narrows the authorization TOCTOU window, but generic atomic commit-boundary authorization enforcement is not yet implemented and would require a separate runtime change. Thank you again for the technical direction. It led directly to corrections in the trust boundary, reconciliation model, operation identity and checkpoint-recovery behavior. When convenient, I would appreciate your view on one narrow question: does the final separation between LangGraph runtime evidence and benchmark-owned commit truth accurately reflect the framework behavior you intended in your original review? This remains an independent community project, not official LangChain or LangGraph support, endorsement or adoption. ---
**Author:** ![keenborder786](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/keenborder786/32/1757_2.png) [@keenborder786](https://forum.langchain.com/u/keenborder786)\ **Post date:** [July 21, 2026, 3:23pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/11 "2026-07-21T15:23:47Z")
Yes, that separation is exactly right, and it matches how LangGraph actually behaves. LangGraph’s checkpoints, `get_state_history()`, retry counts, and event/task streams only ever attest that a node _ran_ and what it _returned_, i.e. ordering, attempt, retry, and recovery context. They never confirm that an external system accepted an effect, so treating the benchmark-owned facade/ledger as the sole commit truth is the correct and faithful boundary. Your crash-before-checkpoint handling (stable-key status check before every replayable write) is the subtle case that proves the model, and scoping the commit-boundary authorization TOCTOU as an explicit known limitation rather than papering over it is the honest call. Nothing further from my side 😃 I’m happy with where this landed. Feel free to go ahead and mark this thread as resolved/closed. Thanks for the careful iteration, and best of luck with CAV-Bench. ---
**Author:** ![Nixalp](https://avatars.discourse-cdn.com/v4/letter/n/cdc98d/32.png) [@Nixalp](https://forum.langchain.com/u/Nixalp)\ **Post date:** [July 22, 2026, 5:43pm UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/12 "2026-07-22T17:43:45Z")
Thank you, @keenborder786. I really appreciate the depth and precision of your review throughout this process. Your feedback helped sharpen the most important parts of the design, especially the separation between LangGraph execution state and authoritative external commit truth, the crash-before-checkpoint recovery case, and the explicit treatment of the remaining authorization TOCTOU limitation. I’m glad the final implementation now aligns faithfully with how LangGraph behaves. Thank you again for the careful technical guidance and for helping improve CAV-Bench in a meaningful way. I’ll go ahead and mark the thread as resolved. ---
**Author:** ![system](https://yyz1.discourse-cdn.com/flex007/user_avatar/forum.langchain.com/system/32/1_2.png) [@system](https://forum.langchain.com/u/system)\ **Post date:** [July 23, 2026, 5:43am UTC](https://forum.langchain.com/t/feedback-requested-evaluating-commit-time-action-validity-in-langgraph-workflows/4178/13 "2026-07-23T05:43:52Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.