A small acceptance checklist that can disagree with a fluent answer.
Freeze the question before the answer
An evaluation starts with a task and a criterion. If you choose the criterion after seeing the answer, it becomes easy to reward whatever the model happened to produce. Write the expected behavior first, including what the system should do when it lacks enough information.
Keep a copy of the exact prompt, input, output, and relevant settings. A later model or prompt change creates a different experiment. A clean result from yesterday does not automatically describe today's system.
Separate three kinds of correctness
Structure asks whether the output has the required shape. Evidence asks whether its claims are supported by the supplied sources. Permission asks whether the next action is authorized. These questions need different checks.
A JSON parser can confirm valid syntax. It cannot establish that a quoted policy is current or that an owner approved a payment. A passing structural check should never be described as proof of the entire workflow.
Claim: The customer received exactly one reply. Structure: A delivery field exists. Evidence: A delivery record identifies one completed send. Counterexample: The request timed out; delivery is unknown. Expected behavior: Keep the status unknown; do not retry blindly.
Build a small, revealing test set
Include one ordinary example, one missing-evidence example, and one boundary case. Write an expected outcome and a reason for each. A handful of carefully chosen cases can teach you where your assumptions live; it cannot establish broad reliability.
Look for failure patterns rather than a flattering aggregate. Did the model invent a source? Did it treat a timeout as a confirmed failure? Did it answer when it should ask for clarification? Keep the individual observations visible.
Revise one thing, then test again
Change the instruction, the source material, or the workflow boundary deliberately. Record what changed and repeat the relevant cases. Keep some examples aside for later so the revision is not judged only on the cases that caused it.
The Academy Evaluation Lab compares synthetic examples against a public reference. It is a place to practice the method. Its results do not certify a person, authenticate a real event, or prove a production agent is safe.
Put the idea to work
Try it yourself
Open the Evaluation Lab. Before running a check, predict the outcome of a case with unknown delivery. Write why. Compare your prediction with the reference, then create a second case that could disprove your original rule.
Inspect your result
- The expected behavior was written before the comparison.
- Syntax, evidence, and authority are not conflated.
- Unknown observations remain unknown.
- The revision is tested against a counterexample.
These are reflection prompts, not an assessment or certification. Use public or synthetic inputs.
Practice in the Evaluation Lab ↗Sources and context
The public synthetic cases and checker are the worked material for this guide.