I’m implementing a workflow where one of the nodes invokes an LLM that:
- Returns a structured output (written to state), and
- Conditionally calls a tool via a conditional edge, based on that output, and comes back to the same node in a React-like architecture.
I wrote something like this:
llm_with_tools = llm.bind_tools(tools)
structured_llm = llm_with_tools.with_structured_output(A_given_class)
The issue:
When the LLM tries to call a tool, it mistakenly uses fields from the structured output class (A_given_class) as tool arguments—instead of using the correct arguments expected by the tool.
How can I ensure the LLM uses the proper tool input schema rather than mixing it with the structured output class?
Thanks!
Hey Ignacio! Its best not to combine structured output and tool calling simultaneously.
So combining bind_tools() and with_structured_output() creates schema confusion - the LLM sees multiple competing function schemas and gets confused about which one to use for which purpose. The best approach would be to use separate nodes - one with llm.with_structured_output(A_given_class) for structured data, another with llm.bind_tools(tools) for tool calling, connected via conditional edges.
Sequential processing in one node would also work (which I feel is what you’re aiming for), first call llm.with_structured_output(), then conditionally call llm.bind_tools() based on the structured result but this would still require two LLM calls to achieve reliable results.
Hey niilooy,
It’s actually the other way around, I have a node that first conditionally calls a tools, and then that data needs to be passed to the next node in a structured format. The red arrow below depicts where the structure output is expected.
