Skip to main content
A call through the gateway rarely depends on one provider. The same model is often served by several providers and regions, and a routing strategy lists several models.

​
How it works

When a call fails, the gateway sends it to the next option instead of returning the error. A model that is down or unhealthy is tried after the other options, so most calls never reach it.

​
What is tried next

  • A routing strategy tries its models in the order of its rule: the router’s ranking under dynamic routing, your order under fixed order. Each model is tried on its deployments, then the next model.
  • A call naming a model tries every provider and region serving it, or the ones you list in provider and region. model can also be a list of models, tried in order.
A timeout, a network failure or a provider server error can retry the same model first (Retry on error in a routing strategy, retry_on_error on a call). Any other failure moves the call on.