Great question — this is a common multi-tenant pattern and there are two supported paths depending on how much you want to own vs. offload. Short version: keep per-tenant config/keys in your own secret store (or LangSmith Provider Secrets), resolve the model per-request in a wrap_model_call middleware, and never let keys touch graph state or traces.
On your Q1 — is there a built-in mechanism? Two options:
- DIY (GA today): your backend/secret store is the source of truth; the graph fetches the active config at runtime. Most flexible, works on any deployment.
- LLM Gateway (private beta): a proxy that sits between your agents and providers. Keys are stored once in LangSmith as Provider Secrets; clients authenticate with a LangSmith API key, and the gateway resolves the real provider key, enforces spend limits + PII/secrets redaction, and traces every call. This is likely the cleanest fit for BYOK + cost attribution if you’re open to the beta. → LLM Gateway, custom / OpenAI-compatible providers, governance & spend policies. Gateway spend-cap policies support per-subject matchers with
current_spend_usd, which gives you per-tenant cost attribution out of the box. (It’s private beta — worth joining the waitlist.)
Q2 — own backend + fetch at runtime: yes, this is the recommended DIY pattern. Use a wrap_model_call middleware that reads the org from runtime context, looks up that org’s config + key (from your secret store, cached), and overrides the model per-invocation:
from dataclasses import dataclass
from typing import Callable
from langchain.agents import create_agent
from langchain.agents.middleware import wrap_model_call, ModelRequest, ModelResponse
from langchain.chat_models import init_chat_model
@dataclass
class Context:
org_id: str
@wrap_model_call
def per_tenant_model(request: ModelRequest, handler: Callable[[ModelRequest], ModelResponse]) -> ModelResponse:
org_id = request.runtime.context.org_id
cfg = get_tenant_config(org_id) # your backend/secret store (cache this)
model = init_chat_model(
cfg.model, # e.g. "openai:gpt-...", "anthropic:...", or an OpenAI-compatible base_url
api_key=cfg.api_key, # BYOK, fetched per-request, never persisted
)
return handler(request.override(model=model))
agent = create_agent(model="openai:gpt-4o", middleware=[per_tenant_model], context_schema=Context)
Runtime context is the right home for this: it’s per-run, read-only, and not persisted between invocations — the docs explicitly call out “tenant/workspace context” and “API keys” as its intended use.
Q3 — passing authenticated org context into the run: on LangSmith Deployment, use custom auth. Whatever your @auth.authenticate handler returns is attached to the run and readable in the graph via config["configurable"]["langgraph_auth_user"] — so you resolve the org from the validated token, not from client-supplied input. See Auth overview and the end-to-end multi-tenant auth tutorial (Q5’s public example). You can then map that identity to context for the middleware above.
Q4 — the leakage gotchas (most important part):
Q5 — cost attribution: tag each run with org_id via metadata/tags so cost tracking rolls up per tenant — or lean on Gateway spend policies for hard per-tenant caps.
TL;DR: one deployment + custom auth to establish org identity → runtime context carries org_id → wrap_model_call resolves the tenant’s model/key from your secret store (or the LLM Gateway) → keep keys out of state/checkpoints and redact them from traces.
Happy to sketch the custom-auth handler side if useful.