Deliver tasks, environments, evidence and adjudication
Complex evaluation programs should deliver more than an output table. Expert qualification, originality, references, runtime environments, trajectories, verifiers, review and version records all enter acceptance.
Validate the task and environment before scaling runs
Solvability, score discrimination, environment reproducibility and expert fit are established before run concurrency expands.
- 01
Define the capability boundary
Specify the knowledge, reasoning or execution boundary—and what a valid task must prove.
- 02
Construct original work
Expert authors and verification engineers build the task, reference, environment, rubric and validation logic.
- 03
Solve & calibrate independently
Expert solves, model pilots, environment reruns and difficulty tiers remove ambiguity, leakage and invalid work.
- 04
Run controlled evaluation
Manage task versions, trajectories, experiment logs, exceptions, rework and state evidence.
- 05
Blind review & expert adjudication
Independently review tasks, answers, traces and verifiers; resolve disagreement with source or runtime evidence.
- 06
Release versions & feedback
Deliver data, environments, verifiers and acceptance records, then carry failure patterns into the next release.
One interface
A delivery lead consolidates questions instead of exposing clients to unmanaged contributor traffic.
Version control
Rules, samples and feedback use explicit versions to prevent batch-to-batch drift.
Tiered capacity
Production, domain review and QA use distinct qualification standards.
Exception closure
Quality, throughput, rule and system issues follow defined escalation and rework paths.
Task difficulty determines talent and QA
Domain authors
Finance / legal / medical / STEM specialists
Original authoring, reference answers, evidence derivation and professional boundaries
Evaluation research
Graduate / ML researchers / advanced evaluators
Difficulty calibration, model pilots, trajectory review and failure attribution
Verification engineering
ML / software / benchmark engineers
MLE, MLS, ProgramBench, DeepSWE, FrontierSWE and programmatic verification
SOP version
Rules and edge cases fixed for the active batch
Production record
Task, owner, timing and state retained
QA evidence
Sampling, issue attribution and rework traceable
Accepted batch
Qualified output and review delivered in the agreed structure
THE DELIVERY OBJECT
A package you can inspect.
Illustrative package · Original Chinese materials; fields scoped per project
Task files define the work. Verification files define the checks. The manifest records the batch, versions and materials.
{ "task_id": "DEMO-AGENT-001", "instruction": "修复路由优先级，保留既有接口兼容性", "inputs": [ "固定版本起始工程", "行为规格" ], "environment": { "type": "isolated_terminal", "runtime": "项目确认" }, "expected_artifact": [ "可构建源码", "执行记录" ]}Build data around your next capability goal
Start with a task goal and a sample. Define the materials, specialists, verification methods and delivery scope together.