Error trying to add an LLM-as-judge Evaluator

Product Help LangSmith
1

When trying to create an LLM-as-judge evaluator on an experiment, I get the following error message pop up in the bottom-right of the window, after clicking the “Save” button:

”Failed to save evaluator: Manifest type langchain.schema.runnable.RunnableSequence is not supported”

Any ideas what might be wrong or how to fix it? I’m just trying to do a ‘hello world’ style experiment.

2

Hi Daniel! Do you mind adding the steps you took to get this error to help us reproduce the issue?

3

I’m getting the same issue

  1. Create a new evaluator by selecting any prebuilt evaluator like “Correctness”
  2. Change the model configuration
    1. Provider: Amazon Bedrock
    2. Model: eu.anthropic.claude-sonnet-4-20250514-v1:0
    3. API: InvokeModel
    4. Region: eu-west-1
  3. Try to save the evaluator
4

We’ve identified a fix for this and it will be live soon. Thanks for reporting!

5

The fix is live!