How are people making multi-step LLM workflows inspectable and rerunnable?

I keep running into the same problem with LLM workflows: the useful part is no longer just the prompt, but the trail around it.

For one-off exploration, chat is fine. For anything repeatable, I want to know:

  • which model/provider ran each step
  • what files or upstream outputs were used
  • what constraints were active
  • what artifact was produced
  • why the next step ran

I am experimenting with a small syntax for this called ICC DSL (Intent-Cell Coding). The shape is a local-first notebook where each step is an intent cell: readable prompt text plus explicit routing, constraints, references, and output contracts.

Example:

c1 Collect
> openai.max
@file -markdown facts.md

c2 Review
> claude.max
%from c1
@file -json risks.json

c3 Final
> (openai + claude).best
%from c1
%from c2
@file -docx final_brief.docx

I am curious how others here think about this boundary.

Do you keep this structure in LangGraph/LangChain code, YAML/config, notebooks, execution traces, or application logs?

The design question I am trying to answer is: what is the minimum useful state record for an LLM workflow to be genuinely rerunnable?

hi @Levitan-factory

Your ICC DSL is trying to hold two very different things in one file, and that is where the friction comes from:

  1. The workflow definition - structure, routing, references, output contracts (c1 to c2 to c3). This is code. It belongs in a StateGraph, or the Functional API, in git.
  2. The execution record - what actually ran, with which inputs and outputs, in what order and why. This is data. It belongs in a checkpointer for rerun, plus LangSmith traces for inspection.

Once you split them, both goals fall out of primitives LangGraph already ships, so you stop reinventing persistence and replay.

On where to keep it - it is not an either/or menu. YAML is fine for constraints and knobs but terrible for routing; the moment it grows conditionals it is a programming language with no debugger. Notebooks are fine for authoring, hostile to reproducibility (hidden state, out-of-order cells). Logs are neither inspectable at scale nor rerunnable. So: definition in code, constraints in config passed at invoke time, rerun state in a durable checkpointer, inspection trail in LangSmith. Your c1/c2/c3 map almost 1:1 onto nodes; %from is edges plus state channels; > openai.max is the model bound in that node; @file is a typed output channel. If you want something that reads like intent cells, the Functional API (one @task per cell) is the closest fit - each task result is checkpointed and restored on resume.

On the minimum state record - LangGraph has effectively answered this. A checkpointer saves a snapshot of graph state at each super-step, keyed by thread_id and checkpoint_id. The Checkpoint record is channel_values (the artifacts), channel_versions plus versions_seen (which node consumed which upstream version - this is your %from, recorded rather than declared, and it is literally how LangGraph decides what runs next), and next (pending nodes), with metadata step, parents, run_id. That maps straight onto your five wants: artifact produced is channel_values; upstream inputs used is channel_values plus versions_seen; why the next step ran is versions_seen plus next; you read it back via get_state_history and replay by invoking with a past checkpoint config.

Two things the checkpoint does not capture for free, and your DSL is right to make them explicit: which model ran and what constraints were active. Put those in state and in trace metadata/tags, so which model each cell used becomes a LangSmith query instead of a grep.

The critical caveat, which most homegrown DSLs get wrong: rerunnable is not the same as reproducible. Replaying a step re-executes it, it does not return a cached result - LLM calls fire again and can return different output. So resume is deterministic (completed steps are restored from the checkpoint), replay is a fresh answer, and the exact past artifact is reproduced by reading the checkpoint, not by re-running the model. Corollary: > openai.max is a moving target, max resolves to a different model next quarter, so pin the exact model id and params (temperature, seed, max_tokens). The checkpoint freezes your state; you must freeze the model contract.

So the genuine minimum record is: one checkpoint per step (channel_values, versions_seen, next, metadata) keyed by thread_id/checkpoint_id, plus the pinned model id and params and the active constraints per step. Keep the DSL if you like it, but make it a thin front-end that compiles to a graph, and let the checkpointer plus LangSmith be the actual trail - you get durable execution, time travel and queryable traces for free.

Docs: Checkpointers Checkpointers - Docs by LangChain - Replay/time travel Use time-travel - Docs by LangChain - Functional API determinism Functional API overview - Docs by LangChain - Metadata and tags Add metadata and tags to traces - Docs by LangChain

This is a very helpful breakdown, thank you.

I agree with the definition vs execution-record split. That is probably the cleanest way to say what I was circling around.

The way I am starting to think about ICC DSL after reading your comment is:

  • the DSL/cell source is an authoring surface
  • the workflow definition should compile to a real graph
  • the actual execution trail should be data: checkpoints, traces, artifacts, model metadata, input identity

So yes, ICC DSL should not try to be the persistence layer itself. At most it should make the human-authored intent readable and portable, then let the backend produce the durable run record.

Your mapping is also useful:

  • c1/c2/c3 → nodes / tasks
  • %from → edges + state channels
  • @file → typed output channel / artifact contract
  • > openai.max → model route, but ideally resolved to pinned model id + params at execution time

The “rerunnable is not reproducible” distinction is especially important. I think I need to be more precise there. A rerun may be a fresh execution; the exact past artifact comes from the checkpoint/artifact store, not from asking the model again and hoping it matches.

The local-first part is still something I care about, so the open question for me is what the minimal local equivalent of the LangGraph/LangSmith trail should be.

It sounds like the useful minimum is roughly:

  • checkpoint per step
  • channel/artifact values
  • versions_seen / consumed upstream versions
  • pending next nodes
  • run metadata
  • pinned model id + params
  • active constraints
  • input file hashes or snapshots

One follow-up question: if you were treating a notebook/cell syntax as a thin front-end over LangGraph, would you store the original cell source alongside the graph/checkpoint metadata, or keep it purely as compile-time input?