Create a routing strategy from a benchmark

Give your agent the sentence “route my summarize_profile calls with the results of my last benchmark”. It starts from a completed benchmark: if there is none yet, follow Start a benchmark first.

​

Before you start: the API key

Every step needs an API key with the Management scope, sent in the Authorization header, or set in the MCP server’s configuration if the agent uses it. If the agent has none, it asks you to create one at https://app.weflayr.com/api-keys/. It then calls Get API key info (get_api_key_info, which works with a key of any scope) and, if scopes lacks management or the key has expired, asks you for one that has it before going on.

​

What changes live routing

Set routing strategy, Archive routing strategy and Restore routing strategy change how live calls are routed at once, so the agent asks you before each of them. Every other step only reads.

​

Who it is for

An API key belongs to a project, not to a person. What an agent creates is recorded for the person who created the API key, unless the request names another member of your organization with user_email. The response says which email was recorded (recorded_for_email): if it is not yours, the agent asks you for the email of a member of your organization and never guesses it.

​

Technical questions

When the person asks how something works (how models are judged, how the router picks a model, the baseline, prompt caching, fallbacks), answer from the How it works pages rather than guessing: Benchmark-driven routing, Benchmarking, Routing, Prompt caching, Self-heal and Fallback and model health. Give the link of the page you answered from. The agent asks the person twice only: once to confirm the benchmark and the objective (step 1), once to accept the strategy before it changes live routing (step 3). Going through the report is optional.

​

1. Confirm the benchmark and the objective

Use the benchmark the person names. Otherwise Get benchmarks with status completed lists them, newest first, without their calls: read each one’s call_names with Get benchmark and propose the latest for their call. Only a completed or archived benchmark has a report. A draft, running or stopped one has none yet: tell the person, point to Start a benchmark to follow it, and stop. Read its report and its suggested_routing_strategies with Get benchmark. When its blockers say that no optimization objective recommends a model, or that the benchmark offers no model a strategy can route to, tell the person: a strategy made from it would keep their calls on the baseline and save nothing. Then stop. Otherwise ask the person, in one message:

• Which benchmark: the one proposed, with its report’s url.
• Which objective to route for: the objectives the routing strategy page offers, in one line each with its recommended mix, its quality against the baseline and its savings per month. An objective of suggested_routing_strategies whose offered_objective is another one is not offered (Best quality, when it would route as Same cost, best quality). An objective whose document is null cannot be routed for either: none of the benchmark’s deployments can be routed to for it (no longer served, too expensive at today’s prices, or lacking a capability the calls use). Propose the benchmark’s first objective, as the page does, or the one it is offered as.
• Whether to go through the results first, or straight to the strategy.

​

2. Walk through the report (optional)

Only when the person asked for it. The report gives, for each optimization objective, the recommended mix of models with each one’s share, the quality against the baseline, the cost and the savings per month. Get benchmark examples shows the calls behind the figures. Present the report as it measures it: the baseline is production (the model or models that served the calls) or the model picked as baseline, and the judges are the evaluators picked. Do not guess why a result looks good or bad, and show the examples if the person wants to read the answers. A model that matches or beats the baseline at a lower cost is what a benchmark is meant to find.

​

3. Show the strategy and ask the person to accept it

Get benchmark with suggested_routing_strategies returns, for the objective picked, a strategy document made from the benchmark and ready to push: keep it as it is, named after the call. Its slug is empty, so it is the name made into a slug (lowercase, spaces turned into -) once saved. A slug stays taken by an archived strategy too, and saving a new strategy with a taken slug is refused (409 slug_taken), so first list the strategies with Get routing strategies and status all. When one already has that slug (usually an earlier strategy for the same call), ask the person whether to save the next version of it (its slug in the document, and its live version as expected_version) or to give the new one another name. Saving a version of an archived strategy makes it live again: say so when asking. In one message, give the person:

• its name and the slug the calls will send as their model (strategy/<slug>),
• the models it routes to,
• what it costs per month and how good its answers are against the baseline, worked out as below. The suggested document leaves out the mix’s models that the gateway no longer serves, that cost too much at today’s prices or that lack a capability the calls use, so work it out from its actual picks rather than assuming it is the objective’s recommendation,
• that saving it changes live routing at once for the calls that send strategy/<slug>.
• what to do with the cache breakpoints their code sends (cache_control or prompt_cache_breakpoint), when the document applies the benchmark’s caching (apply_auto_caching_from_benchmark, on in the suggested document). Read user_cache_breakpoints in suggested_routing_strategies.routable_models for each model the strategy routes to, overrides included:
  • any kept: tell them to keep their breakpoints, naming those models. The cheapest caching the benchmark found on them uses those breakpoints as they are, so removing them makes those models’ calls cost more.
  • every non-null one replaced: tell them the gateway replaces their breakpoints with its own on every model, so they can remove them from their code. They must add them back if they later turn the caching off or route to a model whose value is kept.
  • all null (none of the benchmark’s calls carried breakpoints): say nothing about them.
• when the project’s customers carry tags (Get tags, customer_tags), a proposal of override rules for the segments they define, for example the customers tagged plan:enterprise routed for Best quality while the others route for the cheapest. Such a rule is a customer_selection override rule whose included_customers are the customers Get customers returns for that tag. Say that the list is the one of today: a customer tagged later is not in it until a new version is saved, and the predicted cost covers the global rule only. Route each override rule for an objective that is offered (its offered_objective is itself, so often not Best quality), give it the global rule’s switch_from_cached_model_or_cheaper_better_model_condition, and take its allowed deployments from that objective’s document.

Then ask them to accept, or to say what to change (other models, another objective, a name). After a change, predict again and ask again. For each change:

• Another objective: start from that objective’s own document, since each objective can route to different deployments.
• Other models: a rule routing for an objective cannot allow a deployment listed in that objective’s not_selectable, which gives the reason (it costs too much against the baseline, or lacks a capability the calls use). Tell the person why instead of adding it.
• Fixed order: a fixed_order rule takes null as its switch_from_cached_model_or_cheaper_better_model_condition, and a rule routing for an objective needs one (same_or_better_quality_is_unlikely in the suggested document).

When a save is refused with a 422 naming deployments (a price changed since the benchmark can make a deployment not_selectable), remove them or switch that rule to fixed_order, and tell the person why with the reason. Never save it without their acceptance. Once accepted, Set routing strategy saves it: give the person its url.

​

Predict what a strategy costs

The agent works the prediction out from the report of Get benchmark, with no other call. It reads the picks from the strategy’s global_rule and its allowed deployments, counting the deployments of one model once. The benchmark’s figures are per model: objectives[].models[] gives each model’s cost_per_run (mean USD per call) and quality (against the baseline, null for the baseline, which counts as 0% better, 100% same, 0% worse and 100% same or better). A month is calls_per_month calls. The report names a production baseline after the model its calls logged (gpt-4.1-2025-04-14), while the strategy names it after the gateway’s catalog (gpt-4.1): match the strategy’s model whose suggested_routing_strategies entry has baseline true to the report’s model with baseline true. A model’s own figures are read from an objective whose entry for it has full_run true (or baseline true), since the other objectives explored it on a few calls only.

• Baseline: its monthly cost is the cost_per_run of the model with baseline true, times calls_per_month.
• fixed_order: every call goes to the first model, so the prediction is that model’s own cost_per_run times calls_per_month, and its own quality. The calls that fall back to the next models are left out. A first model the report does not list cannot be predicted.
• An optimization objective: read the objective of objectives[] with that optimization_objective. Every picked model must be listed in its models, since a strategy routing dynamically only routes to the benchmark’s models. Nothing can be predicted when the objective is not in the report, when its recommendation is null, or when a picked model has a null mix (it is outside the recommended mix, and the gateway never sends it a call under that rule). A null recommendation has two causes: no model held the baseline’s quality (no model has a mix), where the gateway keeps every call on the production model, so the strategy saves nothing and the agent suggests not creating it; or the mix was not validated (models have a mix but no call was held back to check it), where the gateway still routes with the mix.
  • The picks are exactly the models with a mix: the prediction is the recommendation, its monthly_cost and its quality.
  • The picks are part of the mix: keep the picked models whose mix.num_calls is above 0 (none left: nothing can be predicted). The cost per call is the mean of their mix.cost_per_run weighted by their mix.num_calls, and the quality is the mean of each model’s own quality (not mix.quality) weighted the same way, each percentage rounded to one decimal. The monthly cost is that cost per call times calls_per_month. Use mix.cost_per_run, not the model’s cost_per_run: the baseline’s holds its model at other reasoning efforts at their own cost. The prediction assumes the router sends each kept model the calls it sent it in the benchmark, which is an approximation.

When nothing can be predicted, the agent tells you why instead of giving a figure.

​

4. Send the calls to the strategy

A strategy routes only the calls sent through the Weflayr gateway with strategy/<slug> as their model. Offer to make that change in the code making the call, keeping or removing its cache breakpoints as step 3 told the person. When the call sends a reasoning effort (reasoning_effort or reasoning.effort), check it is in the routed_reasoning_efforts of a model the rule routes to, in the objective’s models of suggested_routing_strategies: otherwise the gateway refuses every such call (400 reasoning_effort_not_benchmarked). Offer to remove the effort from the call then, so the router picks the effort with the model. If the call does not go through the gateway yet (it is logged with the telemetry SDK, or imported), say so and point to Integrate Weflayr to move it there first. End with the strategy’s url, which Set routing strategy returns: it opens the strategy on the routing strategies page, so the person can check it there.

​

5. Read it back and save a new version

Get routing strategy returns the live version’s document and its version, and Get routing strategy results the calls it routed by rule and by route. To change it, save the next version with Set routing strategy, passing the version you read as expected_version. Saving a version of an archived strategy makes it live again, so ask the person first.