I have been trying to figure out that is it possible that a model response can be a image block or any other block other than the text block.
So can a model response give an image block like this-
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_str}",
},
Yes, a model response can contain image blocks. LangChain has no separate class for image-output models. Images come back in the regular AIMessage. Read them from msg.content_blocks, where they’re normalized to {“type”: “image”, “base64”: …, “mime_type”: …}. msg.content keeps the provider’s raw format.
ChatOpenAI(model=“gpt-image-2”) won’t work. ChatOpenAI only calls the Chat Completions and Responses endpoints, and GPT Image models are served from the Images API (/v1/images/generations). The model profile shipped with langchain-openai says the same thing: text_outputs False, max_output_tokens 0.
There are two ways that do work:
Use a chat model with OpenAI’s built-in image_generation tool – ChatOpenAI(model=“gpt-5.5”).bind_tools([{“type”: “image_generation”}], tool_choice=“image_generation”). ChatOpenAI switches to the Responses API automatically, and the image_generation_call output is converted into a standard image block. Multi-turn editing works if you pass the previous AIMessage back in. Streaming partial images („partial_images”) isn’t supported yet and raises NotImplementedError.
Call the Images API yourself – wrap client.images.generate(model=“gpt-image-2”, prompt=…) from the OpenAI SDK in a @tool. Have it return [{“type”: “image”, “base64”: result.data[0].b64_json, “mime_type”: “image/png”}]. Tools can return standard content blocks. Don’t use DallEAPIWrapper from langchain-community for this. It only reads image URLs, and GPT Image models return base64.
The exact block from your example, {“type”: “image_url”, “image_url”: {“url”: “data:image/png;base64,…”}}, is what ChatGoogleGenerativeAI returns. Use a Gemini image model (e.g. gemini-2.5-flash-image) with response_modalities=[Modality.TEXT, Modality.IMAGE].
Docs: Models - Docs by LangChain and Tools - Docs by LangChain
Thanks for the earlier response. Could you also help me with this?
I’m currently working on a proof-of-concept where I’m using a RAG subagent as part of my Deep Agent (using the Deep Agents library). The retrieved chunks can contain text, images, tables, etc. Ideally, I’d like to pass the complete retrieved content, including images, to the model and have it generate the final response.
My current model (gemini-3.1-flash-lite) is text-output only, so whatever it generates is represented as text content. Even if the model outputs JSON describing an image, that JSON is still a text block and therefore isn’t rendered as an image by LangSmith Studio.
As a workaround, I keep the image reference IDs in the agent state and use @after_agent middleware to resolve them and add image_url blocks to the final AIMessage.
The images in my canonical representation are stored as base64-encoded data, and I have been able to render them in LangSmith Studio using:
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{base64_str}",
},
}
Is there a native/recommended Deep Agents pattern where the retrieved multimodal content can be passed to the model and the resulting final AIMessage can contain both the generated text and the retrieved image blocks, without the @after_agent post-processing?
Passing the images to the model may be less efficient with a text-output model, but I’d prefer this architecture since it would also work naturally with a multimodal output model later.
I’m using LangGraph Dev/Agent Server with LangSmith Studio, and I’m not manually constructing the graph with LangGraph nodes.
afaik there’s no native Deep Agents pattern for this, and your after_agent approach is a reasonable one.
An AIMessage only contains what the model actually generated. A text-output model like gemini-3.1-flash-lite can’t emit image blocks. Even a multimodal-output model would generate new images rather than return your retrieved ones byte-for-byte. So if the final message must contain the original retrieved images, some deterministic post-processing is always needed. Middleware (after_agent, after_model, wrap_model_call) is the supported place for it.
A few things from the deepagents source that matter for your architecture:
The subagent result is text-only. The task tool builds its ToolMessage from the last non-empty AIMessage.text of the subagent, or from structured_response serialized to JSON (_return_command_with_state_update in deepagents/middleware/subagents.py). Image blocks your RAG subagent retrieves never reach the main agent’s model.
Custom state keys do flow back. The same function merges every subagent state key into the parent, except messages, todos, structured_response, skills_metadata and a few internal ones. So a key like „retrieved_images” set inside the subagent ends up in the main agent’s state. This is basically what you’re doing already, and it’s the supported route. Declare the key in both state schemas (e.g. via a middleware state_schema). Give it a reducer if the main agent can call the subagent several times.
If you want the main model to actually see the images, make retrieval a tool on the main agent instead of a subagent. Have it return standard content blocks, i.e. text plus {“type”: “image”, “base64”: …, “mime_type”: “image/png”}. Deep Agents passes those to the model as a multimodal ToolMessage: Multimodal inputs and outputs - Docs by LangChain. Your model supports this, since its profile has image_inputs and image_tool_message set to True.
One small improvement to your post-processing: ask the model to cite image IDs inline (e.g. [img:abc123]). Then have the middleware split the text at those markers and interleave the image blocks, so each image appears where it’s referenced instead of all at the end.
Image blocks you add to the final AIMessage are sent back to the model on the next turn as part of the history. Gemini accepts that. Some providers don’t - for example, OpenAI Chat Completions rejects images in assistant messages. Deepagents’ unsupported-content middleware only sanitizes HumanMessage and ToolMessage. If you might switch providers later, strip those blocks in wrap_model_call, or keep display-only images in state.
Thanks, that’s very helpful.