import weflayr
from weflayr.openapi.models import SetEvaluatorInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.set_evaluator(body=SetEvaluatorInput.from_dict({
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.setEvaluator({
client,
throwOnError: true,
body: {
name: 'Polite tone',
kind: 'judge',
instruction: 'The answer is polite and never blames the customer.',
userEmail: 'alex@acme.com',
},
});
curl -X POST https://app.weflayr.com/api/evaluators/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com"
}'
{
"evaluator_id": 14,
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"builtin": false,
"always_used": false,
"in_library": false,
"generated": false,
"url": "https://app.weflayr.com/dashboard/optimization/benchmark/?evaluators=1",
"recorded_for_email": "alex@acme.com",
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "No evaluator of this project has the id 14, or it was deleted. List the evaluators with get_evaluators."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, to create an evaluator, set kind, instruction"}]}
Evaluators
Set evaluator
Create an evaluator, or change one of the project’s.
POST
/
api
/
evaluators
/
import weflayr
from weflayr.openapi.models import SetEvaluatorInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.set_evaluator(body=SetEvaluatorInput.from_dict({
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.setEvaluator({
client,
throwOnError: true,
body: {
name: 'Polite tone',
kind: 'judge',
instruction: 'The answer is polite and never blames the customer.',
userEmail: 'alex@acme.com',
},
});
curl -X POST https://app.weflayr.com/api/evaluators/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com"
}'
{
"evaluator_id": 14,
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"builtin": false,
"always_used": false,
"in_library": false,
"generated": false,
"url": "https://app.weflayr.com/dashboard/optimization/benchmark/?evaluators=1",
"recorded_for_email": "alex@acme.com",
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "No evaluator of this project has the id 14, or it was deleted. List the evaluators with get_evaluators."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, to create an evaluator, set kind, instruction"}]}
Create an evaluator, or change one of your project’s. Try it on a logged call with test_evaluator, then use its id in evaluators_impacting_benchmark or evaluators_for_reference of set_benchmark.
The response also carries
Parameters
Passed in the request body.integer
The evaluator to change. Omit it to create one. A built-in evaluator cannot be changed.
string
The evaluator’s name. Required to create one.
string
“judge” (an AI compares the candidate’s answer with the baseline’s by following your instruction in words) or “deterministic” (Python code checks the answers). Required to create one.
string
A judge’s instruction in words, or a deterministic check’s Python code. Required to create one.
boolean
Add the evaluator to your library, which the Evaluators page lists and every benchmark can pick from. Defaults to false on creation: the person adds it from the dashboard once they have checked it.
string
The email of the member of your organization this is for, recorded as the evaluator’s author when it is created. Defaults to the email of the person who created the API key, or for a key without one of the project’s creator. The response’s recorded_for_email gives the email recorded: tell the person, and send this to record another.
blockers (what prevents the usual next step), in plain words. It is empty when the usual workflow applies.
import weflayr
from weflayr.openapi.models import SetEvaluatorInput
client = weflayr.client() # reads WEFLAYR_API_KEY
client.set_evaluator(body=SetEvaluatorInput.from_dict({
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com",
}))
const weflayr = require('weflayr');
const client = weflayr.client(); // reads WEFLAYR_API_KEY
await weflayr.api.setEvaluator({
client,
throwOnError: true,
body: {
name: 'Polite tone',
kind: 'judge',
instruction: 'The answer is polite and never blames the customer.',
userEmail: 'alex@acme.com',
},
});
curl -X POST https://app.weflayr.com/api/evaluators/ \
-H "Authorization: Bearer $WEFLAYR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"user_email": "alex@acme.com"
}'
{
"evaluator_id": 14,
"name": "Polite tone",
"kind": "judge",
"instruction": "The answer is polite and never blames the customer.",
"builtin": false,
"always_used": false,
"in_library": false,
"generated": false,
"url": "https://app.weflayr.com/dashboard/optimization/benchmark/?evaluators=1",
"recorded_for_email": "alex@acme.com",
"blockers": []
}
{"error": "Invalid or missing API key. Send it as \"Authorization: Bearer <your key>\"."}
{"error": "No member of your organization has the email sam@acme.com. Ask the person for the email they use in Weflayr, and send it again.", "code": "unknown_user"}
{"error": "No evaluator of this project has the id 14, or it was deleted. List the evaluators with get_evaluators."}
{"error": "Invalid request payload.", "detail": [{"field": "", "error": "Value error, to create an evaluator, set kind, instruction"}]}