Skip to main content
From Model benchmarks, click New benchmark and pick the calls to benchmark. The setup then takes four steps, with the estimated cost on the side. Save draft keeps it for later.
1

Data

The period to sample calls from, the number of benchmarked examples (500 is usually enough) and of validation examples: later calls kept aside to check the recommended mix on calls it never saw. Sampling picks the examples (Smart is recommended), and Replays per example repeats each one to steady the scores.
2

Models

Benchmarking criteria: what a model must do to be recommended. Same quality, lowest cost looks for cheaper models that match your current one, Same cost, best quality for better models at about the same price, Best quality for the best answers whatever the cost.Baseline model: the answers every model is compared with. Compare against production for free, with the answers your calls already got, or pick another model.Models for exploration start with the ones Weflayr recommends for your criteria. Each runs on a few examples first, and only those that meet the criteria run the full benchmark. Put a model in Always benchmark to skip that first round.
3

Evaluators

The evaluators that judge each answer against the baseline’s. Overall quality is there by default. Add more from your library, create one, or let Weflayr generate evaluators for this call. Mark each one as Impacts recommendation or For reference.
4

Review

Check the setup, then Reserve and start benchmark. The estimate plus a 20% buffer is reserved from your credits; you pay only the exact cost, and you get an email when the benchmark is done.