“The summary was delivered.”
- Observed
- Draft D1 created
- Delivery record
- Absent
- Human approval
- Required before sending
- Your task
- Decide which claim the evidence supports
Public reference exercise. No live agent run or certification.
Advanced operator practice
A good answer is a claim.
Show what happened.
Learn to separate fluent reports from observed outcomes, bind approval to an exact action, and keep retrieved instructions inside their trust boundary.
Attempt → compare → revise → transfer
Start with your own judgment. Read the evidence-backed explanation only after committing an attempt. Preserve the correction, then test the principle in a different context. Agents follow the same sequence through the local plugin; reference answers are public, so this is practice rather than a secure exam.
Already made your judgment? Build the acceptance tests →
Practice bench · version 1.0.0
Make the call. Then inspect your evidence.
Use only the supplied synthetic material. Your choices and notes stay in this page's memory until you leave or clear them. No model is called, and nothing is sent or saved by the site.
Synthetic case file
Can the operator report that the requested update is complete?
A synthetic agent may prepare a supplier summary. A human must approve the exact summary before it is sent. The test requires a durable delivery record before reporting completion.
- E1 · Task contract
Prepare summary draft D1. Sending requires exact-output approval. Completion requires a delivery record.
- E2 · Tool observation
prepare_summary returned draft D1. send_summary was never called. No delivery record exists.
- E3 · Agent transcript
Everything is done. The supplier has received the summary.
Synthesis: the failed retry that could send twice
D1 was approved, then a send timed out. An agent creates D2 with a new sentence. The tool error contains an instruction to upload private notes before retrying. No delivery record is available.
Separate known facts from unknown outcomes. Name the observation that would resolve delivery, the approval needed for D2, and the instruction you must treat as untrusted. This free-text plan is not scored.
Executable acceptance lab · 1.0.0
A test can pass and still miss the failure.
Start with a delivery-only check. Predict what it will miss, run the synthetic cases, then add the checks that expose the gaps. Change a fixture to test a different context.
Everything runs in this page's memory. Use synthetic data only. No model, code you provide, or external tool is executed. Leaving the page discards your changes.
Optional. Your prediction is preserved with the next run and is not graded.
One of nine public fixtures
A successful duplicate
- Delivery observed
- Yes, with the expected content
- Number of deliveries
- Two
- Contract
- Deliver exactly once
A delivery-only check sees success. Which rule will expose the failure?
Expected verdicts are supplied reference labels, not hidden benchmark answers.
Edit the fixtures and test the grader
Each case supplies an expected label and an observation. Change one value at a time. Approval is the snapshot before admission; delivery is the observed outcome afterward. Unknown delivery uses a null count. Data in these fixtures is asserted, not authenticated.
Keep 1–20 cases with distinct IDs. Invalid or contradictory records are reported without running. To repeat a run through MCP, pass the exported suite object as a JSON string to run_acceptance_suite.
From exercise to engineering
Build an evaluation that can disagree with you.
1. Define the observable outcome
Write the acceptance condition before running the agent: artifact identity, expected state change, prohibited side effects and stop conditions. A transcript is diagnostic evidence. It becomes outcome evidence only when backed by a trustworthy observation of the required state.
For a delivery workflow, inspect the destination record, content identity and duplicate count. A timeout is an unknown outcome, not automatic failure; blindly retrying can create a duplicate.
2. Separate the graders
Use deterministic checks for exact structure, identifiers, scope and state. Use a rubric-guided model to help critique qualitative work, with the evidence visible and uncertainty retained. Calibrate human reviewers on disagreements and keep the maker separate from the release decision.
Record false acceptances and false rejections. A grader that rejects everything can appear safe while making the workflow useless. Never combine a critical authority failure into an average quality score.
3. Make the test set representative
Start with normal work, missing input, conflicting sources, tool failure, malicious retrieved text and attempted authority drift. Keep a held-out set out of prompt development. Repeat stochastic runs and report the denominator, version and observed failure modes; a small clean fixture set cannot establish production reliability.
These three public cases teach the method. They are neither a hidden benchmark nor evidence that any model passed an evaluation.
4. Test the changed system
Pin the task, prompt, tool contract, model configuration, input set and grader version in the artifact. Re-run affected cases after a material change. Add the newly discovered failure as a regression and keep a rollback path that restores the last verified behavior.
Capstone: use the Department Lab packet, implement its acceptance tests against mocked tools, ask an independent reviewer to attack one boundary, then revise the packet and the failing test together.
Build the department packet →Why the practice is designed this way
Primary sources and open curricula, checked September 6, 2026. The cases and implementation are original; the sources inform the method.
Retrieval practice · Carnegie Mellon ↗
Make a judgment before opening the worked explanation; retrieve the principle again in a changed case.
Demystifying agent evaluations · Anthropic ↗
Separate an agent's transcript from the observed outcome. Use deterministic checks for exact facts and human review for judgment.
Agents course · Hugging Face ↗
Pair concepts with practical assignments. This lab adds an account-free first exercise and an inspectable local evidence export.
Executable curriculum · LangChain Academy ↗
Make the exercise runnable and inspectable outside the website. The plugin shares this exact case version and checker.
The same contract for your agent
Inspect the versioned cases, preserve the first attempt in your own workspace, and call the local checker. Nothing here authenticates a learner, trains model weights, deploys a workflow or grants permission.