Dynamic, large "Variables" in LangGraph / ReAct Agent (without polluting chat history)?

OSS Product Help LangGraph
1

Hey guys! Still new to LangGraph, so sorry if this question has an obvious answer.

We’re building a chatbot whose top level graph is a ReAct agent. It has a few tools, web search, etc, but the main tool is a custom queryDatabase() tool.

It’s working great, but I have one concern. Depending on the user’s query, sometimes the returned data can be absolutely enormous.

Is there a preferred strategy to hold this returned data in some state somewhere that isn’t specifically in the chat history for the conversation? I don’t want a massive JSON array being thrown into the context window if it’s avoidable. We’re manually limiting all results to 25 rows, but even so, it can get really really big depending on any joins that were done.

I DO want the agent to know that data was/wasn’t returned. We’re planning on implementing some sort of custom component that can render a representation of the data, and ideally the Agent might just know some metadata, like array length/ type of data, and maybe have a way for it to access parts of the data in the response via tool call, if the user references it, maybe via fuzzy text search or via index.

I’d also like this data to persist across the whole chat, so if the user sends a message that triggers additional tool calls to queryDatabase(), I’d like to be able to persist all of those return values separately (again, without being thrown into the context window).

Again, sorry if this has an obvious answer, still very new to Langchain / LangGraph.

  • Joe
2

Store the large data in your graph state outside of the messages array, and only include metadata in the chat history. Create a state field like database_results that holds the actual data, while your tool returns a summary message to the agent:

// State schema
const GraphState = {
  messages: [],
  database_results: {} // Store actual data here, keyed by query ID
}

// In your queryDatabase tool
async function queryDatabase(query) {
  const data = await runQuery(query);
  const queryId = generateId();
  
  // Store full data in state
  state.database_results[queryId] = data;
  
  // Return only metadata to agent
  return `Found ${data.length} results (ID: ${queryId}). Data types: ${getDataTypes(data)}`;
}

Add a companion tool like accessStoredData(queryId, operation) that lets the agent retrieve specific parts of stored data when needed. This keeps your context window clean while maintaining access to the full dataset across the conversation. The agent knows what data exists via metadata but only pulls specific portions when referenced by the user.

3

Thanks so much Abdul, this is exactly how I was imagining it might work.

I just wanted to make sure there wasn’t any “out of the box” LangGraph way to do this that I was missing.

1 Like