> ## Documentation Index
> Fetch the complete documentation index at: https://docs.weflayr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Start a benchmark

> Benchmark models on your past calls with an API key or an agent: pick the calls, pick evaluators, estimate the cost, start the benchmark and follow it to completion.

> [!NOTE]
>
> An agent, or you, can follow this page by calling the API routes directly with an API key that has the **Management** scope. The [MCP server](https://docs.weflayr.com/api/management/mcp-server.md) is optional: it exposes each route below as a tool with the same name.

Give your agent the sentence "benchmark my `summarize_profile` calls". It follows the steps below, using the `blockers` of each response. Once the benchmark completes, [Create a routing strategy from a benchmark](https://docs.weflayr.com/agents/create-routing-strategy-from-a-benchmark.md) takes over.

## Before you start: the API key

Check this first, before any other step. Every step needs an API key with the **Management** scope, sent in the `Authorization` header, or set in the MCP server's configuration if the agent uses it. If the agent has none, it asks you to create one at `https://app.weflayr.com/api-keys/`. It then calls [Get API key info](https://docs.weflayr.com/api/management/get-api-key-info.md) (`get_api_key_info`, which works with a key of any scope) and, if `scopes` lacks `management` or the key has expired, asks you for one that has it before going on.

## What costs money

The agent runs every step on its own except these, where it asks you first:

| Step | Why it asks |
| - | - |
| [Start or resume benchmark](https://docs.weflayr.com/api/management/start-or-resume-benchmark.md) | The benchmark spends credit. You accept its estimated cost, which the agent passes back as `accepted_estimated_cost`. [Stopping it](https://docs.weflayr.com/api/management/stop-benchmark.md) gives back the credit it has not spent yet. |
| [Generate evaluators](https://docs.weflayr.com/api/management/generate-evaluators.md) and [Test evaluator](https://docs.weflayr.com/api/management/test-evaluator.md) | They spend a little credit. |

## What an agent must do and tell you

* Tell you about any `blockers` it cannot clear itself, and give you the `url` of everything it creates.
* List the evaluators it would use, including the ones it would create, and ask you which to keep. An evaluator created through the API stays out of your library until you add it from the dashboard.
* Poll a running benchmark on demand, or at most once every 5 minutes: you receive an email when it completes.

## Who it is for

An API key belongs to a project, not to a person. What an agent creates is recorded for the person who created the API key, unless the request names another member of your organization with `user_email`. The response says which email was recorded (`recorded_for_email`): if it is not yours, the agent asks you for the email of a member of your organization and never guesses it.

## Technical questions

When the person asks how something works (how models are judged, how the router picks a model, the baseline, prompt caching, fallbacks), answer from the **How it works** pages rather than guessing: [Adaptive routing](https://docs.weflayr.com/concepts/adaptive-routing.md), [Benchmarking](https://docs.weflayr.com/concepts/benchmarking.md), [Routing](https://docs.weflayr.com/concepts/routing.md), [Prompt caching](https://docs.weflayr.com/concepts/prompt-caching.md), [Self-heal](https://docs.weflayr.com/concepts/self-heal.md) and [Fallback and model health](https://docs.weflayr.com/concepts/fallback-and-model-health.md). Give the link of the page you answered from.

## Continue a recent draft

Before step 1, [Get benchmarks](https://docs.weflayr.com/api/management/get-benchmarks.md) with `status` `draft` lists the saved setups. If one was updated less than 24 hours ago (`updated_at`), show its name and its `url` and ask the person whether to continue from it. If they do, read it with [Get benchmark](https://docs.weflayr.com/api/management/get-benchmark.md) and go on from the first step its setup does not cover yet, saving onto it (its `benchmark_uuid`).

## Save the draft after every change

A session can drop at any time, so save the setup with [Set benchmark](https://docs.weflayr.com/api/management/set-benchmark.md) as soon as the calls are picked, then again after every change, passing the draft's `benchmark_uuid` so each save replaces it rather than creating another. Saving spends nothing.

The agent asks the person twice only: once to pick the calls (step 1), once to accept the whole setup and its cost (step 3). The one exception is a production model with no price (step 2). Everything in between uses the defaults of the benchmark creation page, in its order: the data and models, then the evaluators.

## 1. Pick the calls, the objectives and the evaluators to write

A benchmark replays calls that carry a **call name**. List them with [Get calls](https://docs.weflayr.com/api/management/get-calls.md) over the past month (`date_from` 30 days ago), then ask the person, in one message:

* **Which call(s) to benchmark:** each call name with its number of calls.

* **Which optimization objectives to run**, the first two by default:
  * Same quality, lowest cost: the cheapest models that answer as well as the baseline.
  * Same cost, best quality: better answers for about the baseline's cost.
  * Best quality: the best answers, whatever they cost.

* **Whether AI should write evaluators for these calls** ([Generate evaluators](https://docs.weflayr.com/api/management/generate-evaluators.md), the API side of the **Generate evaluators for this benchmark** button, see [Evaluators specific to your use case](https://docs.weflayr.com/concepts/benchmarking.md#evaluators-specific-to-your-use-case)), which spends a little credit.

* **Several calls in one benchmark:** only when they are very similar, for example when they share a system prompt. Otherwise, set up one benchmark per call.

* **A call the person expects is missing, or has few calls:** offer to look at earlier months (an earlier `date_from`).

* **`unnamed_call_count` above 0:** those calls carry no call name, so no benchmark can replay them. Say so, and offer to name them with [Integrate Weflayr](https://docs.weflayr.com/agents/integration-playbook.md).

* **No call at all:** there is nothing to benchmark yet. Offer two ways to get calls:
  * Integrate Weflayr in their code with [Integrate Weflayr](https://docs.weflayr.com/agents/integration-playbook.md), then come back once calls are logged.
  * Import past calls now, if they can give you an export of them (a CSV, a JSON or a tracing tool's export): [Import past data](https://docs.weflayr.com/api/management/import-past-data.md) imports them, and they show in Get calls within a minute. See [Import past calls](https://docs.weflayr.com/product/import-past-calls.md) for the fields.

## 2. Build the setup with the defaults

Without asking anything, in the benchmark creation page's order:

1. **Data:** the last 30 days, 300 calls benchmarked and 50 more kept aside to check the recommended model mix on calls it never saw. When the period has fewer than 350 calls (add up the `call_count` of the picked calls), use them all, about one sixth kept aside: 120 calls make 100 benchmarked and 20 kept aside. Smart sampling, one replay per call. The baseline is the answers production gave to those calls: every model is judged against them.
2. **Models:** [Get benchmark candidates](https://docs.weflayr.com/api/management/get-benchmark-candidates.md), with the objectives picked, lists the models the calls can be replayed on. Put in `deployments` the deployments marked `recommended` whose `blocked_from_exploration` is empty, the ones the benchmark creation page picks. Use `forced_deployments` (straight to the full benchmark) only for a model the person asks to benchmark whatever its exploration, and only with a deployment whose `blocked_from_always_benchmark` is empty. Then:

   * If `production.models_to_price` is not empty, nothing prices those production models, so the objectives have no baseline price to compare with: ask the person for the input and output price per 1M tokens of each model listed, save them in `config.model_prices_prompted_by_user`, and call Get benchmark candidates again with them as `model_prices_prompted_by_user`, since the blocks based on price depend on them.
   * If `production.default_replay_deployment` is set (production served by one model), save it as `config.production_model_replay_deployment`, as the benchmark creation page does. Without it, production's model cannot be part of the recommended mix.
   * If `baseline_unavailable_message` is set, the gateway no longer serves the baseline: tell the person and ask them for another baseline (`config.baseline_deployment`).

   Save the setup with [Set benchmark](https://docs.weflayr.com/api/management/set-benchmark.md).
3. **Evaluators:** start from the always-used evaluators of [Get evaluators](https://docs.weflayr.com/api/management/get-evaluators.md), as the benchmark creation page does. Then look through the rest of the library (the built-in evaluators and the project's own) and add the ones relevant to what the calls do, judged from the call name and, when you can read the codebase, from the code making the call (its prompt, its tools, the shape of answer it expects). Add the evaluators AI writes when the person asked for them in step 1 (they need the setup saved above). Every evaluator added counts for the recommendation (`evaluators_impacting_benchmark`). Save the setup again.
4. **Estimate:** the estimate of [Get benchmark](https://docs.weflayr.com/api/management/get-benchmark.md) prices the saved setup and checks the credit.

Always leave `config.auto_disallow_too_costly_reasoning_efforts` on (its default) and do not mention it, unless the person asks to turn it off.

## 3. Show the whole setup and ask the person to accept it

In one message, give the person:

* the calls and their number: benchmarked and kept aside, over which period,
* the baseline (production's answers) and the objectives,
* the models: how many, with an offer to list them all, and to apply a restriction (for example EU or US deployments only, each deployment naming its `region`, or some providers left out),
* every evaluator, each with one line on what it checks, and whether it counts for the recommendation or is only shown for reference,
* the estimated cost, and whether their credit covers it,
* the draft's `url`, which opens the whole setup on the benchmark creation page so they can check it there.

Then ask them to accept, or to say what to change: other settings, another baseline (`config.baseline_deployment`), other models, evaluators to add, write ([Set evaluator](https://docs.weflayr.com/api/management/set-evaluator.md)), test ([Test evaluator](https://docs.weflayr.com/api/management/test-evaluator.md)), remove or show for reference only (`evaluators_for_reference`). After a change, save the setup, estimate it again and ask again. Never start the benchmark without their acceptance of the estimated cost.

## 4. Start it with the cost you accept

[Start or resume benchmark](https://docs.weflayr.com/api/management/start-or-resume-benchmark.md) with the estimate as `accepted_estimated_cost` starts the run. It is refused when the benchmark now costs more than the cost you accepted, and when your available credit is below the estimate plus 30%, the amount it reserves.

## 5. Poll it to completion

[Get benchmark](https://docs.weflayr.com/api/management/get-benchmark.md) with the benchmark's id gives its status, its cost so far and the credit set aside. A running benchmark takes a while: its creator receives an email when it completes.

## Next: create a routing strategy

Once the status is `completed`, follow [Create a routing strategy from a benchmark](https://docs.weflayr.com/agents/create-routing-strategy-from-a-benchmark.md) to read the report and route your calls with it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.