Quality lives across the production chain
Qualification, calibration, versioning, sampling, cross-review, rework and adjudication jointly determine the result.
SIX DIMENSIONS OF DATA QUALITY
Quality, defined as concrete checks.
Align methods, evidence and project acceptance.
SOLVABILITY
Prove the task can be completed.
Run the reference solution under agreed conditions. Check that inputs, dependencies and verification form a reproducible task.
Designed to reduce delivery variance
Expert qualification & originality
Authors pass domain challenges and rule tests; STEM tasks also receive similarity and searchability screening.
Task validity
Independent checks confirm complete conditions, answer uniqueness, reproducible environments and score discrimination.
Source, derivation & runtime evidence
Professional claims retain sources; scientific tasks retain derivations; agent and code work retains traces, logs, patches and tests.
Author–verifier separation
Task authoring, reference resolution, verification engineering and adjudication remain distinct responsibilities.
Blind review & expert adjudication
Disputes are re-reviewed without exposing the original judgment, with evidence, version and ruling documented.
Version & contamination control
Tasks, data, environments, verifiers and model feedback are versioned, with leakage risks managed per project.
Four-stage review of complex AI work
Review not only the answer, but the task, process and reliability of the original evaluation.
Input quality
Whether instructions, environment and task are understandable, executable and evaluable.
Process & trajectory
Whether planning, tool use, iteration and self-check are reasonable.
Output quality
Whether the result is complete, accurate, professional and usable.
Evaluation reliability
Whether original scoring, failure attribution and judge behavior can be trusted.
Keep judgment in a reviewable record
A quality system connects rules, actions, rationales and adjudication instead of leaving only a final score.
A difficult prompt is not automatically a valid task
Independent solves, answer uniqueness, reference derivations, verifiers and model calibration establish task validity.
A correct outcome does not prove a reliable process
Critical decisions, tool calls, observations, recovery and corrections enter node-level review.
Engineering conclusions must survive environment verification
Patches, tests, logs, verifiers and reruns support the ruling together.
Security, confidentiality, access and retention requirements are agreed before production and implemented as project controls.
Build data around your next capability goal
Start with a task goal and a sample. Define the materials, specialists, verification methods and delivery scope together.