Skip to main content
Create your routing strategy.
For every incoming call, the router finds the benchmark calls most similar to it and looks at how each candidate model did on those calls. By combining their verdicts, weighted by similarity, it estimates which models are most likely to match or beat your current model’s quality on that specific call.
The new request is encoded as one vector, its nearest benchmark requests are found, and each model at each reasoning level gets a quality score from those requests' verdicts, weighted by distance, which ranks the models.

​
Finding similar calls

The router encodes each call from two kinds of signals:
  • Semantic: embeddings of what the call is about, such as its latest message or its tool definitions.
  • Structural: the shape of the call, such as its length and its number of turns and tool calls.
It then finds the benchmark calls closest to it.

​
Predicting each model’s quality against the baseline

For each candidate model, at each reasoning level, the router reads its verdicts on those similar calls, giving more weight to the calls closest to the incoming one. The result is a quality score per model: how likely its answer is to be as good as or better than your current model’s.

​
Choosing which model to call

The router ranks the models by the objective of your routing strategy:
  • Best quality: the model with the highest quality score, the cheapest one on a tie.
  • Same quality, lowest cost: the cheapest model whose quality score reaches your current model’s.
  • Same cost, best quality: the best model at about your current model’s cost.
The gateway calls the first model at the reasoning level it ranked best at. If that call fails, it moves on to the next model of the ranking (see Fallback and model health). The ranking also accounts for each model’s prompt cache: a model that already holds the start of the prompt costs less to call.

​
Next

Routing strategies

Pick the objective and the models in the dashboard.

Benchmarking

How the verdicts the router reads are produced.