Skip to main content
An evaluator judges each answer of a benchmark against the baseline’s: better, same or worse. Your library keeps them to reuse across benchmarks.

​
Types

  • AI judge: a model reads both answers and follows your instructions, e.g. “The answer names the policy clause it relies on”.
  • Deterministic: a Python check, e.g. at most 120 words. Describe it and Generate code writes it for you.
  • Built-in: ready-made judges (Overall quality, Context fidelity, Instruction fidelity, Completeness, Tone fidelity).

​
Test before you save

Test on a real call runs the evaluator on one of your calls, with your production answer and another model’s, so you can check its verdict before using it in a benchmark.