Issues about the entry points of RAG knowledge

OSS Product Help LangGraph
,
1 
class MessageState(TypedDict):
    messages: Annotated[list[AnyMessage], add_messages]
    knowledge: list[AIMessage]

As follows, am I supposed to directly concatenate the messages in state with my knowledge and send them?

But after the model receives my information, it always assumes that the knowledge was actively told to it by me, just like user input. The model always says, ‘According to the information you provided…’ but I don’t know the relevant knowledge. Even if I wrapped it with AImessage.

async def model_with_tools_node(state:dict):
    model_list_get = await model_list()
    return {
        "messages":[
            model_list_get[1].invoke([SystemMessage(content=model_with_tools_prompt)]+state["messages"]+state["knowledge"])
        ]
    }

Or maybe it’s just because of my prompt?

1 Like
2

What confuses me is that, clearly it’s AI messages, yet the model still thinks the knowledge is provided by the user?

1 Like
3

Hi @Huimin-station

the root mistake is storing retrieved knowledge as AIMessage.

AIMessage semantically means “the LLM said this in a prior turn.” When you store retrieved knowledge as AIMessage and then append it after state[“messages”], you construct this message sequence for the model:

SystemMessage → “You are a helpful assistant…”
HumanMessage → “What is X?”
AIMessage → [retrieved knowledge chunk 1]
AIMessage → [retrieved knowledge chunk 2]

From the model’s perspective, the conversation is already complete - a Human turn followed, by AI responses. It cannot tell why it’s being invoked again, so it tries to reconcile the context by reattributing those AIMessage chunks to the user. That’s exactly why it says “According to the information you provided…” - it’s trying to make narrative sense of content it found in an unexpected position.

LangChain message roles:

type - role - meaning
SystemMessage - system - Application-level instructions/context
HumanMessage - user - What the human said
AIMessage - assistant - What the model previously said
ToolMessage - tool - Output of a tool call

Solution 1: Inject into SystemMessage

Models universally understand system role as application context, not user-provided data:

async def model_with_tools_node(state: dict):
    model_list_get = await model_list()
  
    knowledge_text = "\n\n".join(
        msg.content for msg in state.get("knowledge", [])
    )
  
    system_content = model_with_tools_prompt
    if knowledge_text:
        system_content += (
            "\n\nUse the following retrieved context to answer the user's question:\n\n"
            + knowledge_text
        )
  
    return {
        "messages": [
            model_list_get[1].invoke(
                [SystemMessage(content=system_content)] + state["messages"]
            )
        ]
    }

The model will never say “you provided” because SystemMessage is clearly application-injected. See the LangSmith RAG quickstart.

Solution 2: Use the Official Agentic RAG Pattern with ToolMessage

If you’re building an agent (which the node-based LangGraph structure suggests), use the proper tool call flow. Retrieved content should arrive as ToolMessage it’s the correct semantic type for “something a tool returned”:

SystemMessage → “You are a helpful assistant…”
HumanMessage → “What is X?”
AIMessage → [tool_call: retrieve(“X”)]
ToolMessage → [retrieved content] ← unambiguously a tool result

Aalso in the official LangGraph RAG tutorial:

retriever_tool = create_retriever_tool(retriever, "retrieve_docs", "...")
retrieve = ToolNode([retriever_tool])
workflow.add_node("retrieve", retrieve)

ToolNode handles wrapping results in ToolMessage automatically. The model correctly understands this as tool-sourced knowledge.

Solution 3: Middleware Pattern (augment the HumanMessage)

The LangChain RAG chains docs show a before_model middleware that augments the user’s message directly with the retrieved context:

class RetrieveDocumentsMiddleware(AgentMiddleware[State]):
    def before_model(self, state):
        last_message = state["messages"][-1]
        docs = vector_store.similarity_search(last_message.text)
        augmented = f"{last_message.text}\n\nContext:\n{'\n\n'.join(d.page_content for d in
 docs)}"
        return {"messages": [last_message.model_copy(update={"content": augmented})]}

agent = create_agent(model, tools=[], middleware=[RetrieveDocumentsMiddleware()])

Summary

Symptom Root cause Fix
“According to information you provided…” AIMessage placement creates a fake completed exchange; model reattributes content to the user Inject into SystemMessage
Knowledge placed after state["messages"] Appended after the HumanMessage, looks like prior AI turns Keep only state["messages"] in the call; merge knowledge into SystemMessage
knowledge: list[AIMessage] Wrong message type for retrieved context Store as plain text and inject into system prompt, or use ToolMessage via ToolNode

References