Skip to main content
Providers bill the start of a prompt they already hold in cache at a fraction of the input price. On agents and long conversations, where each call repeats the previous ones, caching is often the largest cost saving after the model choice. But each provider caches differently, and routing between models can lose the cache at every switch. Weflayr ensures your use case runs with optimal caching in three steps.

​
1. The benchmark finds the best caching for each model

Each provider has its own rules: whether it caches implicitly or only where a breakpoint is marked, how long a cached prompt lives, the minimum length it caches, where it looks for a cached prefix, how many cache breakpoints a request takes, what a cache write costs, etc. The best setup depends on those rules and on your traffic: how long your prompts are, how they grow from one call to the next, and how much time passes between calls. So during the benchmark, Weflayr replays your calls of the same period in time order against a simulation of each candidate model’s cache, and tries different breakpoint placements and cache lifetimes. It keeps, for each model, the setup that costs the least on your calls, cache writes included. The benchmark report prices each model both with the caching your calls use today and with this setup.

​
2. Your routing strategy applies it on every call

In a routing strategy made from a benchmark, you can choose to apply recommended caching from benchmark to make the gateway place the recommended breakpoints for the model each call goes to, with no code to change. Your app does not need to know which model answers: a call routed to Claude gets Claude’s best breakpoints, the same call routed to GPT gets GPT’s. When it is on, the gateway keeps your app’s breakpoints on the models where they cache better than its own setup and replaces them on the others. Mark them with Anthropic’s cache_control or OpenAI’s prompt_cache_breakpoint, whatever the model.

​
3. You decide when quality is worth losing the cache

The gateway keeps track of which prompt prefixes each provider is likely to hold in cache for your organization. When the router ranks the models for a call, it prices each one with the tokens it would read from its cache, and by default it keeps the call on the model whose cache is warm: switching would lose the cheaper cache reads. Staying on the cached model is cheaper, but it is not always the right model for the next call: a conversation can start with easy questions a cheap model answers well, then ask a hard one it answers badly. How much quality is worth giving up for the cheaper cache reads depends on your use case, so you choose it. Weflayr scores every call separately, and When to switch from a model with warm cache sets, for each rule of your routing strategy, when a call leaves the cached model even if that costs more: Whatever the setting, a model with a higher quality score that stays cheaper than the cached model, its cache discount included, is always called first.

​
Next

Routing strategies

Turn on the benchmark’s caching and choose when to switch.

Benchmark-driven routing

How the benchmark and the router work together.