Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, Sambit Sahu
Read on arXiv →
Key claim
BINEVAL improves LLM evaluation with interpretable, binary feedback.
In plain English
Imagine you're trying to assess how well a language model generates text. Traditionally, you might rely on human evaluators, which can be slow and costly, or use automated metrics that often don't align with human opinions. This leads to problems like poor feedback on what specifically went wrong in the generated text, making it hard to improve the model. This is what's called the evaluation bottleneck in NLP.
The current methods often fail because they either oversimplify the evaluation process or produce scores that are hard to interpret. For instance, a single score might not tell you why a model's output was good or bad, leaving you in the dark about how to make improvements. This is where BINEVAL comes in. Instead of giving a single score, it breaks down the evaluation into smaller, binary questions that can be answered independently. This way, you get a clearer picture of the model's performance across different aspects.
With BINEVAL, you can generate specific questions based on the task at hand, and the model answers these questions to provide detailed feedback. This not only makes the evaluation process more transparent but also allows for better diagnostics and iterative improvements in prompts. In practice, BINEVAL has shown to perform as well as or better than existing evaluation methods across various benchmarks, particularly excelling in areas like factual consistency. This means that if you're building systems that rely on LLMs, using BINEVAL can help you get more actionable insights and improve your models more effectively.
BINEVAL introduces a novel framework for LLM evaluation that decomposes criteria into binary questions, enhancing interpretability.
The framework is validated across multiple datasets and outperforms existing baselines, supporting its claims with empirical evidence.
Deep reliability assessment
The results support that decomposed binary judging can improve correlation and diagnostic usefulness on the tested summarization, dialogue, and factual-consistency benchmarks, especially where errors are claim-level and concrete. The broader claim of task-agnostic reliability is less established because question generation quality, aggregation calibration, evaluator model choice, and test-set-selected prompt updates could strongly affect outcomes in real deployments.
Reproducibility
No code repository is mentioned in the provided paper text. The evaluation uses public datasets including SummEval, Topical-Chat, QAGS, and IFBench, but exact prompts, generated binary questions, model versions, and inference settings would be needed for full reproduction.
Key figure
Figure 1 compares per-dimension score distributions on SummEval, showing that BINEVAL, especially with Claude, more closely follows human score distributions and avoids some ceiling effects seen in other LLM judge baselines.