Feature Request: Add a pluggable binary classification evaluator for System 1 models

Description

I would like to propose adding a lightweight binary classification evaluator to OpenEvals.

Many evaluation workflows today use LLM-as-a-judge, where a larger model performs a more deliberative, rubric-based evaluation. However, some use cases only need a fast binary judgment such as good / bad, pass / fail, safe / unsafe, or 1 / 0.

With the growing interest in smaller, faster System 1 style models such as Laya and Jev, it would be useful for OpenEvals to support a simple interface where users can attach their own binary classifier models as evaluators.

The goal is not to replace LLM-as-a-judge and not to ship any specific model inside OpenEvals. Instead, this feature would provide a model-agnostic interface where users can plug in their own classifier logic.

Proposed Solution

Add a binary classification evaluator that accepts a user-provided classifier callable or adapter.

The classifier could be:

  • deterministic rule logic
  • a local Python function
  • a call to an external classification API
  • a wrapper around a user’s own fine-tuned System 1 model such as Laya or Jev

For example:

from openevals.evaluators import create_binary_classifier_evaluator

def my_classifier(inputs=None, outputs=None, reference_outputs=None, **kwargs):
    # Could call a user-owned Laya/Jev-style classifier endpoint here
    return "good"

evaluator = create_binary_classifier_evaluator(
    classifier=my_classifier,
)

result = evaluator(
    inputs="What is the capital of France?",
    outputs="Paris",
    reference_outputs="Paris",
)

Conceptually, the evaluator would normalize labels like:

good / bad
pass / fail
true / false
1 / 0
safe / unsafe
acceptable / unacceptable

and return a standard OpenEvals result format.

For example:

{
    "key": "binary_classification",
    "score": True,
    "comment": "Classified as good."
}

The exact return shape should follow existing OpenEvals evaluator conventions.

Motivation

This would be useful for cases where users already have a fast binary classifier and do not need a generative judge model.

Examples:

  • evaluating whether an answer is acceptable or unacceptable
  • classifying outputs as safe or unsafe
  • checking if a response meets a task-specific quality threshold
  • using a fine-tuned System 1 model such as Laya or Jev as an evaluator
  • replacing expensive judge calls with cheaper deterministic or classifier-based evaluation
  • running high-throughput evaluations where LLM-as-a-judge is too slow or expensive

This keeps OpenEvals flexible while avoiding the need to bundle, train, host, or maintain any specific model.

Non-goals

This proposal does not aim to:

  • fine-tune a model inside OpenEvals
  • ship Laya, Jev, or any other built-in classifier model
  • replace LLM-as-a-judge evaluators
  • add heavy ML dependencies
  • assume any specific model provider
  • require users to use System 1 models specifically

Possible API

evaluator = create_binary_classifier_evaluator(
    classifier=my_classifier,
    positive_labels=["good", "pass", "true", "1", True, 1, "safe", "acceptable"],
    negative_labels=["bad", "fail", "false", "0", False, 0, "unsafe", "unacceptable"],
)

Acceptance Criteria

  • Add a binary classification evaluator following existing OpenEvals conventions.

  • Support user-provided classifier callables/adapters.

  • Support classifiers backed by local logic, APIs, or user-owned fine-tuned models.

  • Normalize common positive and negative labels.

  • Return results using the standard evaluator result schema.

  • Include unit tests for:

    • positive labels
    • negative labels
    • custom labels
    • invalid labels
    • classifier callable behavior
  • Avoid adding unnecessary dependencies.

Additional Context

System 1 style models such as Laya and Jev are increasingly useful for fast, specialized judgments. A model-agnostic binary classification evaluator would let users connect those classifiers to OpenEvals without requiring OpenEvals to own the training, hosting, or maintenance of those models.

References

This proposal is motivated by recent work on System 1 / typed-decision models that return structured decisions or calibrated probabilities without generating free-form text.