Expert-authored STEM challenge sets
Original tasks that combine advanced knowledge, multi-step derivation and verifiable answers beyond direct retrieval.
Metric context: model versions, inference budgets and task slices are not public here. This range describes task difficulty in the existing case; it is not a cross-model ranking.
Task understanding before scale
A frontier-reasoning evaluation required original problems that could not be answered through search or template transfer. We defined the target capability and expected failure mode before specialists across mathematics, physics, chemistry, biology and computer science submitted prompts, complete derivations, rubrics and validation logic. A second specialist solved each task independently; model pilots, similarity search and programmatic checks removed ambiguity, contamination and accidental guessability. Volumes and pass rates are banded from a representative sanitized batch. This capability aligns to frontier scientific-reasoning methods and does not imply official affiliation with a public benchmark.
Mathematics, physics, chemistry, biology and computer science
Problem, reference, rubric and verifier delivered together
Independent solve, model pilot and contamination-risk screen
One structure aligns the questions asked by model, data, procurement and security teams.
Original tasks that combine advanced knowledge, multi-step derivation and verifiable answers beyond direct retrieval.
Original, non-searchable STEM problems aimed at a specified capability gap
Expert authoring, independent solve, verifier construction, model pilots and difficulty tiers
Representative batch of approximately 1,200 original tasks across five STEM disciplines
Original authoring × reference derivation × verifier-led calibration; Independent solve, model pilot and contamination-risk screen
Mathematics, physics, chemistry, biology and computer science; Problem, reference, rubric and verifier delivered together; Target-model pass rates held between 6% and 18%; ambiguous and searchable items were removed pre-release
Client identity, original material and selected volumes are removed or banded. This brief explains delivery method and does not guarantee future results.
A case file a technical team can interrogate
Difficulty is not obscurity. A valid challenge must isolate the intended capability gap while keeping the prompt clear, the answer determinate and the derivation independently reproducible.
Capability slice
Work backward from model error clusters to the knowledge and reasoning boundary that needs isolation.
Original authoring
The specialist submits the problem, full reference, scoring rubric and validation logic as one unit.
Independent solve
A second specialist verifies interpretation, answer and critical intermediate results without seeing the first solution.
Model stress test
Repeated pilots expose shortcuts, ambiguity, accidental success and difficulty collapse.
Version release
Only tasks that clear originality, verifiability and difficulty gates enter the accepted batch.
What does not enter the accepted batch
- Multiple valid answers or unstated assumptions
- Reference derivation cannot be independently reproduced
- Public retrieval or surface rewriting reveals the answer
- Verifier checks formatting without covering the core reasoning
The handoff is more than an answer column
- Problem and capability labels
- Expert reference and critical steps
- Scoring rubric and programmatic verifier
- Independent solve, model-pilot and version records
Accepted tasks can support a private challenge set, capability regression testing or targeted post-training design. The final use depends on model stage and the agreed data rights.
Review samples. Then define the task.
Carry this direction into your enquiry to confirm delivery, verification and sample scope.
Build data around your next capability goal
Start with a task goal and a sample. Define the materials, specialists, verification methods and delivery scope together.