Artificial intelligence / news brief
Ai2 releases XYBench to test whether assistants catch the wrong question
- Event
- Published

Ai2 released XYBench, an 8,115-question benchmark for whether assistants recognize when a user's requested method does not address the underlying goal. The associated paper predates the release and most examples are synthetic; reported model gaps rely on an automated Claude judge whose emphasis labels had 74.2%–79.8% human agreement in the authors' checks.
Why it matters
The benchmark targets a practical failure mode that conventional factuality and instruction-following scores often miss: answering the stated method instead of solving the user's problem.
Sources
- XYBench: Helping with the problem behind the question (external site)Ai2 · primary source ·
- XYBench examines misguided requests (external site)BIG CHANGE · coverage ·