Structured output for stateGraph

OSS Product Help LangGraph
1

I built the langchain agent using stateGraph, below is the original code.

I defined the response_format(pydantic model), it works fine for openAI provider, but it doesn’t work for gemini provider

llm_with_tools = (
            model
            .bind_tools(tools, strict=True)
            .bind(response_format=response_format)
        )

        response = llm_with_tools.invoke(final_messages, config=config)

So I tried to update the code like this
But second option works fine for both providers, but it doesnt triger any tool, because when I use with_structured_output, langchain tells the model this as one tool, so ignoring other tools

llm = model.bind_tools(tools, strict=True)
llm_chain = llm.with_structured_output(response_format)
response = llm_chain.invoke(final_messages, config=config)

I am trying to resolve the structured output problem by creating the separated node for structuring the output, it works functionally. but it doesn’t feel okay to me,
any other production solution for it?

Thanks in advance

2

@Dev which gemini model are you using?

3

I tried with gemini-2.5-pro, and gemini-3.0-flash-preview

4

Hi @Dev
which version of VartexAI/GoogleGenAI are you using?

5

I am using Init_chat_model

model = init_chat_model(
ai_model_name, temperature=temperature, model_provider=provider_name
)
1 Like
6

@Dev Structured outputs with tools is only available since gemini 3 series according to Google’s documentation: https://ai.google.dev/gemini-api/docs/structured-output?example=recipe#structured_outputs_with_tools
So any model from 2.5 series won’t work.

However, you tried it with gemini 3.0 flash, but what happened is:

response_format=… in LangChain is not a cross-provider contract. It works for OpenAI because OpenAI supports response_format natively. Gemini’s native mechanism is response_mime_type=“application/json” + response_json_schema=… (JSON Schema / Pydantic).

For Gemini, “tools + native structured output” is still constrained by the provider. Google’s docs i provided above only explicitly advertise combining structured outputs with built-in tools (Google Search, URL context, code execution, file search) as a Gemini 3 preview feature, and do not show custom function tools + JSON schema together in one call.

So my recommendation would be keep a two-node StateGraph:

  1. Tool node: model bound only with your tools (planning + tool calling).
  2. Formatting node: a separate model call that formats the final answer into your Pydantic schema (Gemini JSON schema mode if you want strictness).

This is actually a very common production pattern when providers can’t reliably do tool calling AND hard JSON-schema constrained final output in one request.

1 Like
7

Hi @Dev

@simon.budziak is right. Right now many models do not fully support both the tools and the structured output in the same request.

Anyway, have you tried create_agent pattern instead of using bind_tools and response_format=response_format directly on LLM model?

1 Like
8

within the create_agent, I think it works fine for both gemini and openai agent.

But Regarding the @simon.budziak ‘s solution, I don’t want to create the separated node for structuring output.

If I don’t use the init_chat_model, and use the VartexAI/GoogleGenAI, is there any other solution??

9

Thanks for your solution @simon.budziak

the separated node for formatting the response will be additional call to LLM, right?

10

Then it might be an implementation issue (I’m actually expecting that…) then. I’ll look into the implementation.

Can’t you use create_agent if it works for Gemini and OpenAI?

11

since create_agent has several limitation, we switched to use stateGraph, so I have to find the solution with stateGraph

12

Yes, that’s correct - the separated formatting node does require an additional LLM call.

@Dev Here is a sample how it may look like:

def call_model(state: AgentState):
    """Node that produces the AI tool-call message."""
    return {"messages": [planner_model.invoke(state["messages"])]}


def format_output(state: AgentState):
    """Node that turns the final text into structured output."""
    message = formatter_model.invoke(state["messages"])
    # Parse the model's JSON string into the Pydantic object.
    structured = Answer.model_validate_json(message.content)
    return {"final": structured, "messages": state["messages"]}

The flow requires 2 LLM calls. However, unfortunately be aware of cost/token Impact:

  • Latency: around 2x slower due to sequential calls
  • Cost: around 2x more tokens (you pay for both conversations)

But if you use a simple model, this is worth it for production systems where structured output reliability is important. Since OpenAi supports single call with structured output the two-call approach is the production-ready solution for Gemini + structured output only.

13

I see, I can’t create additional LLM call.

I need to find the other solution for that.

14

Just curious - what are the limitations? I’m pretty sure create_agent is flexible enough with its middlewares - you can control almost entire behaviour of the agent with it.

15

I am using stateGraph for two features, but we will add more features to it.

  1. toollimit middleware
  2. using the state Field parameters to interact the toolResult more specifically.(state management)
16

Then I still don’t get why you can’t use create_agent? Those two features can be achieved with middlewares of create_agent.

Nonetheless, I’ll look at the implementation of the area of the issue in question in GoogleGenAI