import weflayr
from weflayr.openapi.models import GenerateEvaluatorsInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.generate_evaluators(body=GenerateEvaluatorsInput.from_dict({
"description": "The answer has fewer than 50 words.",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.generateEvaluators({
client,
throwOnError: true,
body: {
description: 'The answer has fewer than 50 words.',
},
});
curl -X POST https://app.weflayr.com/api/evaluator-generation/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"description": "The answer has fewer than 50 words."
}'
{
"evaluators": [],
"instruction": "def check(input: str, baseline_output: str, evaluated_output: str, tags: dict[str, str]) -> bool:\n return len(evaluated_output.split()) < 50",
"url": null,
"recorded_for_email": null,
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "Your organization has no credit left. Top up on the Credits page, then call this again.", "code": "insufficient_credits"}
{"error": "No saved setup of this project has the id 5c1d6a52-3f5f-4c53-9a0e-6f2b9c7d8e01. Save one with set_benchmark, or list the benchmarks with get_benchmarks."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, set either \"benchmark_uuid\" or \"description\", not both"}]}
Evaluators
Generate evaluators
Have AI write evaluators.
POST
/
api
/
evaluator-generation
/
import weflayr
from weflayr.openapi.models import GenerateEvaluatorsInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.generate_evaluators(body=GenerateEvaluatorsInput.from_dict({
"description": "The answer has fewer than 50 words.",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.generateEvaluators({
client,
throwOnError: true,
body: {
description: 'The answer has fewer than 50 words.',
},
});
curl -X POST https://app.weflayr.com/api/evaluator-generation/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"description": "The answer has fewer than 50 words."
}'
{
"evaluators": [],
"instruction": "def check(input: str, baseline_output: str, evaluated_output: str, tags: dict[str, str]) -> bool:\n return len(evaluated_output.split()) < 50",
"url": null,
"recorded_for_email": null,
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "Your organization has no credit left. Top up on the Credits page, then call this again.", "code": "insufficient_credits"}
{"error": "No saved setup of this project has the id 5c1d6a52-3f5f-4c53-9a0e-6f2b9c7d8e01. Save one with set_benchmark, or list the benchmarks with get_benchmarks."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, set either \"benchmark_uuid\" or \"description\", not both"}]}
Have AI write evaluators (how). With a benchmark_uuid, it writes judge evaluators from what the calls of that saved setup share (a prompt text, a tool, an answer schema) and saves them to your project, out of your library until the person adds them from the dashboard. With a description, it writes the Python code of a deterministic evaluator from a rule in words.
The response also carries
Spends a little credit.
Parameters
Passed in the request body.string
Write judge evaluators for the calls this saved setup sampled. They are saved to your project.
string
Write the Python code of a deterministic evaluator from this rule in words.
string
The email of the member of your organization this is for, recorded as the author of the evaluators written for a benchmark_uuid. Defaults to the email of the person who created the API key, or for a key without one of the project’s creator. The response’s recorded_for_email gives the email recorded: tell the person, and send this to record another.
blockers (what prevents the usual next step), in plain words. It is empty when the usual workflow applies.
import weflayr
from weflayr.openapi.models import GenerateEvaluatorsInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.generate_evaluators(body=GenerateEvaluatorsInput.from_dict({
"description": "The answer has fewer than 50 words.",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.generateEvaluators({
client,
throwOnError: true,
body: {
description: 'The answer has fewer than 50 words.',
},
});
curl -X POST https://app.weflayr.com/api/evaluator-generation/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"description": "The answer has fewer than 50 words."
}'
{
"evaluators": [],
"instruction": "def check(input: str, baseline_output: str, evaluated_output: str, tags: dict[str, str]) -> bool:\n return len(evaluated_output.split()) < 50",
"url": null,
"recorded_for_email": null,
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "Your organization has no credit left. Top up on the Credits page, then call this again.", "code": "insufficient_credits"}
{"error": "No saved setup of this project has the id 5c1d6a52-3f5f-4c53-9a0e-6f2b9c7d8e01. Save one with set_benchmark, or list the benchmarks with get_benchmarks."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, set either \"benchmark_uuid\" or \"description\", not both"}]}