> ## Documentation Index
> Fetch the complete documentation index at: https://docs.weflayr.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Benchmarking

> The Model Benchmarking tries to find cheaper model that performs as well as the one you run today.

Weflayr replays your real production prompts through candidate models and compares their outputs with the current model, and prices the swap against your actual usage.

Benchmarks run on their own: once Weflayr has seen enough traffic on a feature, it replays that feature in the background and publishes the result here. A feature is the `feature_name` you stamp on your calls when [propagating metadata](/quickstart#add-metadata-to-your-llm-calls-for-finer-analysis). Like [prompt caching optimisation](/product/prompt-caching), it needs `capture message content` on (see [Cost Optimisation](/product/cost-optimisation)).

<Frame>
  <video src="https://mintcdn.com/weflayr/_Mt_r8ZiFzCJL_Dd/images/product/model-benchmark-front.mp4?fit=max&auto=format&n=_Mt_r8ZiFzCJL_Dd&q=85&s=20fe45c9104fb464eb7e0958553c2da8" poster="/images/product/model-benchmark-front-poster.jpg" aria-label="Model benchmarking page with the savings summary, one row per feature, and the all-ship impact simulation" controls loop muted playsInline preload="metadata" className="w-full rounded-xl" data-path="images/product/model-benchmark-front.mp4" />
</Frame>

Open a benchmarked feature to see its result.

## Model Benchmark detail

High level view of the results from the different model's results, and the retrospective impact on your unit economics.

<Frame>
  <video src="https://mintcdn.com/weflayr/kH6f2e9L10ZBJ0nT/images/product/model-benchmark-detail.mp4?fit=max&auto=format&n=kH6f2e9L10ZBJ0nT&q=85&s=74e5af1f65536d7a54515060f46aa850" poster="/images/product/model-benchmark-detail-poster.jpg" aria-label="Candidates side by side matrix with the five scores, then the impact simulation" controls loop muted playsInline preload="metadata" className="w-full rounded-xl" data-path="images/product/model-benchmark-detail.mp4" />
</Frame>

For every replayed request, an LLM judge compares your current model's answer and the candidate's answer, and rates the candidate `better`, `same` or `worse` on five scores:

| Score                | What the judge rates                                                                                                  |
| -------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Output quality       | The main score. Overall quality of the answer for the task                                                            |
| Context fidelity     | Every claim is supported by the context given in the request (documents, conversation history), with nothing invented |
| Instruction fidelity | Every constraint the system prompt sets (format, rules, forbidden content) is respected                               |
| Completeness         | Every piece of information the request asks for is present                                                            |
| Tone fidelity        | The tone and register match what the prompt asks for                                                                  |

## Per-example results

Better understand the benchmarked model behavior.

<Frame>
  <video src="https://mintcdn.com/weflayr/kH6f2e9L10ZBJ0nT/images/product/model-benchmark-examples.mp4?fit=max&auto=format&n=kH6f2e9L10ZBJ0nT&q=85&s=f5adb5052b95ab4cc853b2e47ecca295" poster="/images/product/model-benchmark-examples-poster.jpg" aria-label="Per-example results with score columns and an expanded row showing the judge scorecard" controls loop muted playsInline preload="metadata" className="w-full rounded-xl" data-path="images/product/model-benchmark-examples.mp4" />
</Frame>
