Skip to main content
GET
Get one benchmark: its setup, status and cost, and each part it has reached. A part it has not reached is null.
  • Status: a running benchmark takes a while, and its creator receives an email when it completes. One stopped because it spent its credit reservation lists the reasoning efforts that cost far more than its model’s exploration effort in overpriced_reasoning_efforts: resume it with disallow_overpriced_reasoning_efforts to leave them out.
  • Estimate, for a saved setup: the cost of the candidates’ replays, the baseline’s and the judges, the credit it needs while it runs and the credit you have. Pass its estimated_cost as accepted_estimated_cost to start the benchmark.
  • Report, once finished: the benchmark report. For each optimization objective, the recommended mix of models with each one’s share of the calls, its quality against the baseline, its cost and savings per month, and every model’s figures.
  • Suggested routing strategies, once finished: for each optimization objective, the models a routing strategy should route to, ranked with the benchmark’s figures, and the deployments it cannot route to and why. Its document routes to the recommended mix: name it, edit it if needed and push it with set_routing_strategy. It is null when no deployment can be routed to.

​
Parameters

​
string
required
The benchmark’s id.
​
boolean
default:"true"
Include what a saved setup costs, with the credit it needs. Null once the benchmark has started. Defaults to true.
​
boolean
default:"true"
Include the report of a completed or archived benchmark. Null before it is finished. Defaults to true.
​
boolean
default:"false"
Include the routing strategy the benchmark suggests for each optimization objective. Null before it is finished. Defaults to false.
The response also carries blockers (what prevents the usual next step), in plain words. It is empty when the usual workflow applies.