> ## Documentation Index
> Fetch the complete documentation index at: https://docs.weflayr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarking

> Candidate models are replayed on your own past calls, and each answer is judged against your current model's: better, same or worse.

> [!TIP]
>
> Start your [benchmark](https://docs.weflayr.com/product/model-benchmarks).

The benchmark evaluates candidate models across **all reasoning levels**. The candidates are chosen by price, to meet your cost objective. From these results, it tries different combinations of models through the [router](https://docs.weflayr.com/concepts/routing.md) to find the mix that gives the best quality for your objective.

## The steps of a benchmark

A running benchmark goes through seven steps:

1. **Testing every candidate on a few examples:** each answer is judged against your current model's.
2. **Searching for each model's optimal prompt caching configuration:** every call of the period is replayed through a simulation of the model's cache, and the cheapest configuration is kept.
3. **Selecting which models will run the full benchmark:** for each optimization objective, the candidates most likely to reach it go on. The others stop here.
4. **Testing every reasoning effort of the selected models:** on part of the examples.
5. **Finishing the full benchmark on the best performing model-effort combinations:** on the rest of the examples.
6. **Computing the optimal model mix:** for each optimization objective, through the router.
7. **Testing the Weflayr router on the optimal model mix:** on the validation examples, to check its quality and cost on new calls.

## Judging the quality of an answer

Each candidate's answer is judged against the answer of your current model (the baseline), in three steps.

![Judge instructions are generated from the agent's structure, then the baseline's and the candidate's answers are compared side by side by an OpenAI and an Anthropic judge, with a Google judge when they disagree, giving a worse, same or better verdict.](https://mintcdn.com/weflayr/G_EStOCPuaN7N6tQ/images/concepts/side-by-side-evaluation-judges.png?fit=max\&auto=format\&n=G_EStOCPuaN7N6tQ\&q=85\&s=054bb7899bacbf5b94b3d4a92de6dae8)

### Evaluators specific to your use case

Weflayr writes [evaluators](https://docs.weflayr.com/product/evaluators.md) about your business context automatically (through the [API](https://docs.weflayr.com/api/management/generate-evaluators.md) or the **Generate evaluators for this benchmark** button of the [benchmark creation page](https://docs.weflayr.com/product/create-a-benchmark.md)) by reading your calls to find their instructions, tools, response formats and recurring patterns.

### Side-by-side relative evaluation

**We compare candidate model answers side by side with the baseline's answer, so every result is relative.** For each benchmark call, the judge sees the request, the baseline model's answer and the candidate model's answer, then decides whether the candidate is **better, same or worse** than the baseline.

This is a deliberate choice. **Absolute scoring** ("is this answer correct?") is often **unreliable** in real-world applications, **even when compared against a ground truth.** Answers can be partially correct, omit information or include unnecessary details, making even human experts disagree on how to grade them. **Side-by-side comparison makes evaluation more reliable for LLMs and more relevant to routing:** it tells us whether switching models improves or degrades the result, regardless of whether the baseline answer itself is good or bad.

![To 'Do you ship to Canada?' with a system prompt asking for relevant extra information, the baseline answers 'Yes, we ship to Canada.' and the candidate 'Yes, in 5-8 days.'. Graded alone, both are correct, so they look the same. Compared, the candidate is better: it adds the delivery time.](https://mintcdn.com/weflayr/G_EStOCPuaN7N6tQ/images/concepts/side-by-side-vs-absolute-example.png?fit=max\&auto=format\&n=G_EStOCPuaN7N6tQ\&q=85\&s=d8cc03e9480c9d481515ab1e12351d53)

Side-by-side evaluation also shows where a candidate improves or degrades your production behavior, which tells you whether switching is worth it.

### A panel of judges

A single judge can favor answers from models of its own provider. Each verdict comes from judges of three different providers: an OpenAI model and a Claude model each compare the two answers, and when they disagree, a Gemini model settles it.

## Next

> [card] [Routing](https://docs.weflayr.com/concepts/routing.md)
>
> How the verdicts decide which model answers each new call.

> [card] [Create a benchmark](https://docs.weflayr.com/product/create-a-benchmark.md)
>
> Pick a call, the models to try and how to judge them.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.