feat(core): forward document metadata to embedding implementations to enable multimodal providers

OSS Product Help LangChain
1

Background

Several embedding providers now support multimodal inputs — text, images, audio, and video — in the same embedding space:

  • Gemini Embedding 2 (gemini-embedding-2-preview) — text, images, video, audio, PDFs
  • Voyage AI (voyage-multimodal-3) — text and images
  • Cohere (embed-v4.0) — text and images

These models are production-ready. The current LangChain architecture makes them impossible to use correctly without workarounds.

Current Limitation

VectorStore.add_documents() strips documents down to strings before calling embed_documents():

def add_documents(self, documents: list[Document], **kwargs):
    texts = [doc.page_content for doc in documents]
    metadatas = [doc.metadata for doc in documents]
    return self.add_texts(texts, metadatas, **kwargs)

add_texts() then calls the embedding implementation with only the text:

def add_texts(self, texts, metadatas=None, **kwargs):
    embeddings = self.embedding.embed_documents(texts)  # metadata discarded
    ...

As a result, embedding implementations have no access to the original Document or its metadata.

Current Workaround

To work around this, multimodal content must be serialized directly into page_content:

Document(
    page_content="___IMAGE___|A cat sitting on a sofa|gs://bucket/cat.png"
)

A custom embedding implementation must then parse this string to reconstruct the multimodal input — standard embedding implementations have no awareness of this format.

This works but has real costs: page_content becomes a transport format rather than a readable text representation. This breaks BM25 retrieval, re-ranking pipelines, and any UI that displays page_content directly to users.

Proposed Change

Forward metadatas from VectorStore into embed_documents() as an optional keyword argument.

1. Embedding interface — additive, optional parameter

class Embeddings(ABC):

    @abstractmethod
    def embed_documents(
        self,
        texts: List[str],
        metadatas: Optional[List[dict]] = None,  # new, optional
    ) -> List[List[float]]:
        ...

    @abstractmethod
    def embed_query(
        self,
        text: str,
        metadata: Optional[dict] = None,  # new, optional
    ) -> List[float]:
        ...

2. VectorStore — stop discarding metadata before embedding

def add_texts(self, texts, metadatas=None, **kwargs):
    embeddings = self.embedding.embed_documents(texts, metadatas=metadatas)
    ...

metadatas is already computed in add_documents() — it just isn’t forwarded today.

3. Multimodal provider — opt in by using the metadata

class GoogleGenerativeAIEmbeddings(Embeddings):

    def embed_documents(
        self,
        texts: List[str],
        metadatas: Optional[List[dict]] = None,
    ) -> List[List[float]]:
        if metadatas is None:
            return self._embed_texts(texts)

        requests = []
        for text, meta in zip(texts, metadatas or [{}] * len(texts)):
            modality = meta.get("modality", "text")
            if modality == "image":
                requests.append({
                    "text": text,
                    "image_uri": meta["image_uri"],
                })
            else:
                requests.append({"text": text})
        return self._embed_multimodal(requests)

With this in place, documents can be represented cleanly:

Document(
    page_content="A cat sitting on a sofa",
    metadata={
        "image_uri": "gs://bucket/cat.png",
        "modality": "image",
        "source": "product_catalog.pdf",
        "page": 3,
    }
)

Backward Compatibility

All existing Embeddings subclasses are unaffected. Because metadatas is an optional keyword argument with a default of None, any implementation that does not override the signature continues to work without modification. Implementations that do opt in must handle metadatas=None gracefully to remain safe when embed_documents() is called directly outside a VectorStore context.

Scenario Behavior
Existing embedding implementation (unchanged) Ignores metadatas kwarg — no regression
Multimodal provider (e.g. Gemini, Voyage, Cohere) Reads metadatas to construct multimodal requests
embed_documents(texts) called directly metadatas defaults to None — implementations handle gracefully
VectorStore using add_texts() without metadata Passes metadatas=None — safe fallback to text-only path

Scope

This proposal touches two things:

  1. The Embeddings base class in langchain-core — add optional metadatas kwarg to embed_documents and embed_query
  2. VectorStore implementations — forward metadatas into embed_documents instead of discarding it

No new abstractions, no new base classes, no migration required for existing integrations.

Test Cases

  • Calling embed_documents(texts) without metadatas on any existing
    implementation must return the same result as before — no regression
  • Calling embed_documents(texts, metadatas=None) explicitly must behave
    identically to the above
  • A multimodal implementation receiving metadatas with "modality": "image"
    must produce embeddings in the same vector space as text embeddings from
    the same model
  • A multimodal implementation receiving metadatas=None must fall back to
    the text-only path without raising an error
  • embed_query(text, metadata=None) with no metadata must return the same
    result as the current embed_query(text)
2

Hello @VedantSonani welcome to our community! Please feel free to create an issue here

3

Thanks! I have already opened an issue for this here: #38186
I’ve also been investigating a prototype and posted some implementation details there.