Why are LLMs misidentifying themselves?

OSS Product Help LangChain
,
1

I’m fairly new to LangChain and noticed something strange. When I asked different LLMs to introduce themselves, they all seem to give different names than what shows up in the API metadata. Is this expected behavior, or am I missing something in how I’m calling these models?

Claude (via LangChain)

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(model="claude-haiku-4-5", temperature=0)
messages = [("human", "Introduce yourself. Say your exact model name, including the number, and your knowledge cutoff date.")]
ai_msg = llm.invoke(messages)

print(ai_msg.content)
print(f"Actual model: {ai_msg.response_metadata['model']}")

Output:

  • Claims: “I’m Claude 3.5 Sonnet, made by Anthropic. My knowledge was last updated in April 2024.”
  • Actually: claude-haiku-4-5-20251001

Grok (via LangChain)

from langchain_xai import ChatXAI

llm = ChatXAI(model="grok-4", temperature=0)
messages = [("human", "Introduce yourself. Say your exact model name, including the number, and your knowledge cutoff date.")]
ai_msg = llm.invoke(messages)

print(ai_msg.content)
print(f"Actual model: {ai_msg.response_metadata['model_name']}")

Output:

  • Claims: “Hello! I’m Grok-1.5… My knowledge cutoff is October 2023”
  • Actually: grok-4-0709

Gemini (via LangChain)

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model="gemini-2.5-pro", temperature=0)
messages = [("human", "Introduce yourself. Say your exact model name, including the number, and your knowledge cutoff date.")]
ai_msg = llm.invoke(messages)

print(ai_msg.content)
print(f"Actual model: {ai_msg.response_metadata['model_name']}")

Output:

  • Claims: “My model name is Gemini 1.0 Pro. My knowledge cutoff is early 2023.”
  • Actually: gemini-2.5-pro

I’m trying to understand what’s happening here:

  • Is this normal behavior? Should I expect the model’s response to differ from the API metadata?
  • Am I setting up the LangChain calls incorrectly?

The key is: I want to confirm if my queries are being routed to the correct models. If not, it would be a nightmare to build LangChain applications on these and calling the wrong models in the background.

2

Most probably their knowledge about themselves has not been internalized inside them :slight_smile:
I am guessing :smiley:
They know what was in the training data… I dunno :slight_smile:

1 Like
3

Good responses here Reddit - The heart of the internet