Memory Preservation with ToolNode

OSS Product Help LangGraph
,
1

Hi there! I have noticed Langgraph.prebuilt.ToolNode doesn’t preserve the chat history, do you know why?

Context: I was trying to create a React-fashioned agent to save some text in a .txt file and I got the error

Error code: 400 - {'error': {'message': "Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.", 'type': 'invalid_request_error', 'param': 'messages.[0].role', 'code': None}}

When I checked out the traces with langfuse I noticed the ToolNode “forgets” the chat history and just returns the tools message

image
image1914×1075 138 KB

So that leads me to ask you

  1. Am I interpreting / using something wrongly?
  2. Is this a common error?
  3. When is the ToolNode tipically used? Are there best practices for tool nodes implementations?

Thanks for any comment!

2

Hi @santiagoahl

what is the definition of state in your graph? And what is the graph itself? Could you share as many code snippets as possible?

The chat history is preserved by your graph state and reducers (for example, using add_messages) or by a checkpointer. If your state replaces the message list with only the ToolMessage output, the next LLM call will start with a tool role message and OpenAI will reject it with the 400 error you saw.

1) Am I using it wrongly?

Likely yes - it seems that your state is not accumulating messages. ToolNode expects a state containing a list of messages (default key is "messages") where the last message is an AIMessage with tool_calls. It runs the tools and emits ToolMessage(s). If your state reducer overwrote messages with those ToolMessage(s) (instead of appending), the next model call begins with a tool message and violates OpenAI’s rule that tool outputs must respond to a preceding assistant message with tool_calls.

A minimal, correct state shape uses a message-accumulating reducer:

from typing import Annotated, List
from langchain_core.messages import BaseMessage
from langgraph.graph import StateGraph
from langgraph.graph.message import add_messages

class State(TypedDict):

messages: Annotated[List[BaseMessage], add_messages]

Make sure your graph uses the same key that ToolNode reads/writes (default messages, configurable via messages_key).

2) Is this a common error?

Yes. It happens whenever the state loses the preceding assistant message with tool_calls and only forwards the tool output to the next LLM invocation. OpenAI enforces: a message with role tool must be a response to a preceding message with tool_calls. If the history isn’t accumulated or persisted (e.g., no checkpointer/threaded memory), you’ll see this error.

3) When to use ToolNode and best practices

  • When to use: In agent graphs where the model decides to call tools; route to ToolNode only when the last AIMessage has tool_calls (use tools_condition). ToolNode executes tools (in parallel if multiple), returning ToolMessage(s) to be added to your state.
  • Best practices:
    • Use a message-accumulating reducer for state (add_messages) so history is appended, not replaced.
    • Keep the key consistent with ToolNode(messages_key=...) if you customize it.
    • Persist conversation across turns with a checkpointer (e.g., MemorySaver) and a thread_id.
    • Use tools_condition to branch only when tool calls exist.
    • Configure error handling with handle_tool_errors to return a ToolMessage on exceptions instead of crashing the graph.
    • If tools need graph context or a store, use injected arguments (e.g., InjectedState, InjectedStore).

Why the error appears

  • ToolNode emits ToolMessage(s). If your state reducer replaced the message list with just those ToolMessage(s), the next LLM call starts with a tool message. OpenAI requires a preceding assistant message with tool_calls that those tool outputs are replying to. Without it, the API returns:

Invalid parameter: messages with role ‘tool’ must be a response to a preceding message with ‘tool_calls’.

Fix by accumulating messages (append, don’t replace) and/or persisting the thread state.

References (official docs and source)

1 Like
3

Fully agreed with @pawel-twardziak response. The state defined for the graph will influence if the node returns an accumulated history or simply overwrite the state with the outcome of the tool call.

1 Like