Skip to main content
Start your benchmark.
The benchmark evaluates candidate models across all reasoning levels. The candidates are chosen by price, to meet your cost objective. From these results, it tries different combinations of models through the router to find the mix that gives the best quality for your objective.

​
The steps of a benchmark

A running benchmark goes through seven steps:
  1. Testing every candidate on a few examples: each answer is judged against your current model’s.
  2. Searching for each model’s optimal prompt caching configuration: every call of the period is replayed through a simulation of the model’s cache, and the cheapest configuration is kept.
  3. Selecting which models will run the full benchmark: for each optimization objective, the candidates most likely to reach it go on. The others stop here.
  4. Testing every reasoning effort of the selected models: on part of the examples.
  5. Finishing the full benchmark on the best performing model-effort combinations: on the rest of the examples.
  6. Computing the optimal model mix: for each optimization objective, through the router.
  7. Testing the Weflayr router on the optimal model mix: on the validation examples, to check its quality and cost on new calls.

​
Judging the quality of an answer

Each candidate’s answer is judged against the answer of your current model (the baseline), in three steps.
Judge instructions are generated from the agent's structure, then the baseline's and the candidate's answers are compared side by side by an OpenAI and an Anthropic judge, with a Google judge when they disagree, giving a worse, same or better verdict.

​
Evaluators specific to your use case

Weflayr writes evaluators about your business context automatically (through the API or the Generate evaluators for this benchmark button of the benchmark creation page) by reading your calls to find their instructions, tools, response formats and recurring patterns.

​
Side-by-side relative evaluation

We compare candidate model answers side by side with the baseline’s answer, so every result is relative. For each benchmark call, the judge sees the request, the baseline model’s answer and the candidate model’s answer, then decides whether the candidate is better, same or worse than the baseline. This is a deliberate choice. Absolute scoring (“is this answer correct?”) is often unreliable in real-world applications, even when compared against a ground truth. Answers can be partially correct, omit information or include unnecessary details, making even human experts disagree on how to grade them. Side-by-side comparison makes evaluation more reliable for LLMs and more relevant to routing: it tells us whether switching models improves or degrades the result, regardless of whether the baseline answer itself is good or bad.
To 'Do you ship to Canada?' with a system prompt asking for relevant extra information, the baseline answers 'Yes, we ship to Canada.' and the candidate 'Yes, in 5-8 days.'. Graded alone, both are correct, so they look the same. Compared, the candidate is better: it adds the delivery time.
Side-by-side evaluation also shows where a candidate improves or degrades your production behavior, which tells you whether switching is worth it.

​
A panel of judges

A single judge can favor answers from models of its own provider. Each verdict comes from judges of three different providers: an OpenAI model and a Claude model each compare the two answers, and when they disagree, a Gemini model settles it.

​
Next

Routing

How the verdicts decide which model answers each new call.

Create a benchmark

Pick a call, the models to try and how to judge them.