Start a benchmark

Give your agent the sentence “benchmark my summarize_profile calls”. It follows the steps below, using the blockers of each response. Once the benchmark completes, Create a routing strategy from a benchmark takes over.

​

Before you start: the API key

Check this first, before any other step. Every step needs an API key with the Management scope, sent in the Authorization header, or set in the MCP server’s configuration if the agent uses it. If the agent has none, it asks you to create one at https://app.weflayr.com/api-keys/. It then calls Get API key info (get_api_key_info, which works with a key of any scope) and, if scopes lacks management or the key has expired, asks you for one that has it before going on.

​

What costs money

The agent runs every step on its own except these, where it asks you first:

​

What an agent must do and tell you

• Tell you about any blockers it cannot clear itself, and give you the url of everything it creates.
• List the evaluators it would use, including the ones it would create, and ask you which to keep. An evaluator created through the API stays out of your library until you add it from the dashboard.
• Poll a running benchmark on demand, or at most once every 5 minutes: you receive an email when it completes.

​

Who it is for

An API key belongs to a project, not to a person. What an agent creates is recorded for the person who created the API key, unless the request names another member of your organization with user_email. The response says which email was recorded (recorded_for_email): if it is not yours, the agent asks you for the email of a member of your organization and never guesses it.

​

Technical questions

When the person asks how something works (how models are judged, how the router picks a model, the baseline, prompt caching, fallbacks), answer from the How it works pages rather than guessing: Benchmark-driven routing, Benchmarking, Routing, Prompt caching, Self-heal and Fallback and model health. Give the link of the page you answered from.

​

Continue a recent draft

Before step 1, Get benchmarks with status draft lists the saved setups. If one was updated less than 24 hours ago (updated_at), show its name and its url and ask the person whether to continue from it. If they do, read it with Get benchmark and go on from the first step its setup does not cover yet, saving onto it (its benchmark_uuid).

​

Save the draft after every change

A session can drop at any time, so save the setup with Set benchmark as soon as the calls are picked, then again after every change, passing the draft’s benchmark_uuid so each save replaces it rather than creating another. Saving spends nothing. The agent asks the person twice only: once to pick the calls (step 1), once to accept the whole setup and its cost (step 3). The one exception is a production model with no price (step 2). Everything in between uses the defaults of the benchmark creation page, in its order: the data and models, then the evaluators.

​

1. Pick the calls, the objectives and the evaluators to write

A benchmark replays calls that carry a call name. List them with Get calls over the past month (date_from 30 days ago), then ask the person, in one message:

• Which call(s) to benchmark: each call name with its number of calls.
• Which optimization objectives to run, the first two by default:
  • Same quality, lowest cost: the cheapest models that answer as well as the baseline.
  • Same cost, best quality: better answers for about the baseline’s cost.
  • Best quality: the best answers, whatever they cost.
• Whether AI should write evaluators for these calls (Generate evaluators, the API side of the Generate evaluators for this benchmark button, see Evaluators specific to your use case), which spends a little credit.
• Several calls in one benchmark: only when they are very similar, for example when they share a system prompt. Otherwise, set up one benchmark per call.
• A call the person expects is missing, or has few calls: offer to look at earlier months (an earlier date_from).
• unnamed_call_count above 0: those calls carry no call name, so no benchmark can replay them. Say so, and offer to name them with Integrate Weflayr.
• No call at all: there is nothing to benchmark yet. Offer two ways to get calls:
  • Integrate Weflayr in their code with Integrate Weflayr, then come back once calls are logged.
  • Import past calls now, if they can give you an export of them (a CSV, a JSON or a tracing tool’s export): Import past data imports them, and they show in Get calls within a minute. See Import past calls for the fields.

​

2. Build the setup with the defaults

Without asking anything, in the benchmark creation page’s order:

1. Data: the last 30 days, 300 calls benchmarked and 50 more kept aside to check the recommended model mix on calls it never saw. When the period has fewer than 350 calls (add up the call_count of the picked calls), use them all, about one sixth kept aside: 120 calls make 100 benchmarked and 20 kept aside. Smart sampling, one replay per call. The baseline is the answers production gave to those calls: every model is judged against them.
2. Models: Get benchmark candidates, with the objectives picked, lists the models the calls can be replayed on. Put in deployments the deployments marked recommended whose blocked_from_exploration is empty, the ones the benchmark creation page picks. Use forced_deployments (straight to the full benchmark) only for a model the person asks to benchmark whatever its exploration, and only with a deployment whose blocked_from_always_benchmark is empty. Then:
   • If production.models_to_price is not empty, nothing prices those production models, so the objectives have no baseline price to compare with: ask the person for the input and output price per 1M tokens of each model listed, save them in config.model_prices_prompted_by_user, and call Get benchmark candidates again with them as model_prices_prompted_by_user, since the blocks based on price depend on them.
   • If production.default_replay_deployment is set (production served by one model), save it as config.production_model_replay_deployment, as the benchmark creation page does. Without it, production’s model cannot be part of the recommended mix.
   • If baseline_unavailable_message is set, the gateway no longer serves the baseline: tell the person and ask them for another baseline (config.baseline_deployment).
   Save the setup with Set benchmark.
3. Evaluators: start from the always-used evaluators of Get evaluators, as the benchmark creation page does. Then look through the rest of the library (the built-in evaluators and the project’s own) and add the ones relevant to what the calls do, judged from the call name and, when you can read the codebase, from the code making the call (its prompt, its tools, the shape of answer it expects). Add the evaluators AI writes when the person asked for them in step 1 (they need the setup saved above). Every evaluator added counts for the recommendation (evaluators_impacting_benchmark). Save the setup again.
4. Estimate: the estimate of Get benchmark prices the saved setup and checks the credit.

Always leave config.auto_disallow_too_costly_reasoning_efforts on (its default) and do not mention it, unless the person asks to turn it off.

​

3. Show the whole setup and ask the person to accept it

In one message, give the person:

• the calls and their number: benchmarked and kept aside, over which period,
• the baseline (production’s answers) and the objectives,
• the models: how many, with an offer to list them all, and to apply a restriction (for example EU or US deployments only, each deployment naming its region, or some providers left out),
• every evaluator, each with one line on what it checks, and whether it counts for the recommendation or is only shown for reference,
• the estimated cost, and whether their credit covers it,
• the draft’s url, which opens the whole setup on the benchmark creation page so they can check it there.

Then ask them to accept, or to say what to change: other settings, another baseline (config.baseline_deployment), other models, evaluators to add, write (Set evaluator), test (Test evaluator), remove or show for reference only (evaluators_for_reference). After a change, save the setup, estimate it again and ask again. Never start the benchmark without their acceptance of the estimated cost.

​

4. Start it with the cost you accept

Start or resume benchmark with the estimate as accepted_estimated_cost starts the run. It is refused when the benchmark now costs more than the cost you accepted, and when your available credit is below the estimate plus 30%, the amount it reserves.

​

5. Poll it to completion

Get benchmark with the benchmark’s id gives its status, its cost so far and the credit set aside. A running benchmark takes a while: its creator receives an email when it completes.

​

Next: create a routing strategy

Once the status is completed, follow Create a routing strategy from a benchmark to read the report and route your calls with it.