Description
I would like to propose adding a lightweight binary classification evaluator to OpenEvals.
Many evaluation workflows today use LLM-as-a-judge, where a larger model performs a more deliberative, rubric-based evaluation. However, some use cases only need a fast binary judgment such as good / bad, pass / fail, safe / unsafe, or 1 / 0.
With the growing interest in smaller, faster System 1 style models such as Laya and Jev, it would be useful for OpenEvals to support a simple interface where users can attach their own binary classifier models as evaluators.
The goal is not to replace LLM-as-a-judge and not to ship any specific model inside OpenEvals. Instead, this feature would provide a model-agnostic interface where users can plug in their own classifier logic.
Proposed Solution
Add a binary classification evaluator that accepts a user-provided classifier callable or adapter.
The classifier could be:
- deterministic rule logic
- a local Python function
- a call to an external classification API
- a wrapper around a user’s own fine-tuned System 1 model such as Laya or Jev
For example:
from openevals.evaluators import create_binary_classifier_evaluator
def my_classifier(inputs=None, outputs=None, reference_outputs=None, **kwargs):
# Could call a user-owned Laya/Jev-style classifier endpoint here
return "good"
evaluator = create_binary_classifier_evaluator(
classifier=my_classifier,
)
result = evaluator(
inputs="What is the capital of France?",
outputs="Paris",
reference_outputs="Paris",
)
Conceptually, the evaluator would normalize labels like:
good / bad
pass / fail
true / false
1 / 0
safe / unsafe
acceptable / unacceptable
and return a standard OpenEvals result format.
For example:
{
"key": "binary_classification",
"score": True,
"comment": "Classified as good."
}
The exact return shape should follow existing OpenEvals evaluator conventions.
Motivation
This would be useful for cases where users already have a fast binary classifier and do not need a generative judge model.
Examples:
- evaluating whether an answer is acceptable or unacceptable
- classifying outputs as safe or unsafe
- checking if a response meets a task-specific quality threshold
- using a fine-tuned System 1 model such as Laya or Jev as an evaluator
- replacing expensive judge calls with cheaper deterministic or classifier-based evaluation
- running high-throughput evaluations where LLM-as-a-judge is too slow or expensive
This keeps OpenEvals flexible while avoiding the need to bundle, train, host, or maintain any specific model.
Non-goals
This proposal does not aim to:
- fine-tune a model inside OpenEvals
- ship Laya, Jev, or any other built-in classifier model
- replace LLM-as-a-judge evaluators
- add heavy ML dependencies
- assume any specific model provider
- require users to use System 1 models specifically
Possible API
evaluator = create_binary_classifier_evaluator(
classifier=my_classifier,
positive_labels=["good", "pass", "true", "1", True, 1, "safe", "acceptable"],
negative_labels=["bad", "fail", "false", "0", False, 0, "unsafe", "unacceptable"],
)
Acceptance Criteria
-
Add a binary classification evaluator following existing OpenEvals conventions.
-
Support user-provided classifier callables/adapters.
-
Support classifiers backed by local logic, APIs, or user-owned fine-tuned models.
-
Normalize common positive and negative labels.
-
Return results using the standard evaluator result schema.
-
Include unit tests for:
- positive labels
- negative labels
- custom labels
- invalid labels
- classifier callable behavior
-
Avoid adding unnecessary dependencies.
Additional Context
System 1 style models such as Laya and Jev are increasingly useful for fast, specialized judgments. A model-agnostic binary classification evaluator would let users connect those classifiers to OpenEvals without requiring OpenEvals to own the training, hosting, or maintenance of those models.
References
This proposal is motivated by recent work on System 1 / typed-decision models that return structured decisions or calibrated probabilities without generating free-form text.
-
Laya: an open-weight, Jev-compatible System 1 decision model for typed decisions over text/JSON-like state, returning labels/probabilities without free-form generation.
GitHub - NandhaKishorM/laya: Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request. · GitHub