I’ve been trying to do this for a while, and I think at this point I’m realising it’s a feature request and not actually doable
I’m currently trying to track token count of my different LLM calls in a swarm application, however an important detail is to know what model was used with these tokens. Understandably the llm itself isn’t present in all callbacks, however sadly the custom metadata of the llm is also missing. Other callbacks expose this metadata, as it is available inside the CallbackManagerForLLMRun class under this.metadata, however handleLLMEnd does not actually pass this on. Only the generation output and other params that are of no use
I’m using this callback because the Gemini llm responds with token output under output.llmOutput.usage_metadata, but I have no way of accessing the llm or metadata here
Being able to access this data would be great
I’m opening this message in the langchain section, as I see that this class is defined in the @langchain/core package
You are right, this is not currently supported. handleLLMEnd does not expose the LLM metadata that’s available in CallbackManagerForLLMRun, so you cannot reliably associate usage data with the originating model. That’s a real gap especially for multi model or swarm setups where cost attribution depends on knowing which model produced the tokens.
Workarounds today include:
Using handleLLMStart to capture the model info and correlating it with handleLLMEnd via run_id.
Wrapping your LLM to inject model metadata into the response before it hits callbacks.
Falling back on provider specific SDKs or callbacks when they expose usage data.
The long term fix is a change in @langchain/core so that handleLLMEnd passes through the model metadata along with the generation output. This would align it with other callbacks and make usage tracking much more robust. It’s worth opening or upvoting a GitHub issue so the core team prioritizes it.
Ah thank you, I wasn’t sure if run_id was specific to the model or the entire graph run. I ended up doing something with llm specific callback handlers, one for each model, but it’s still a pain
Where would I do this feature request? I initially went to the langchain-ai/langchainjs repository but when trying to create an issue it says that feature requests belong on the forum