As far as I can tell you cannot use the google genai or openai file uploads to pass multimodal input into the google or openai langchain chat models. It looks like only base64 is supported in most cases. Am I wrong about this, I looked at the source code and still don’t see anything.
This is a huge blocker as the base64 method takes up a TON of tokens
I think it depends on how the LLM/provider handles files.
As you can see here, there are three possible ways to do this. I don’t think LangChain adds any special handling here. If token count is a concern, you could host the images somewhere and send a static URL instead, or use the OpenAI Files API . There are, of course, pros and cons to each approach.
Unfortunately, for Google GenAI, the File API is the only option. This means you would need to update the LangChain source code to support it.