Executable program-reconstruction evaluation
Models receive behavioral specifications and a black-box reference, then rebuild a complete compatible program against hidden tests.
Task understanding before scale
A coding-model evaluation moved beyond short function completion into installable, interactive programs with state and error semantics. Task authors defined public behavior and locked a black-box reference. Verification engineers built hidden tests across interface, files, errors, state, boundaries and concurrency. Candidate submissions were installed, executed and retested in isolation, while independent reviewers checked grader stability.
Reconstruct complete programs from behavior rather than source
Interface, state, error and boundary tests combined
Author, verifier and grader-replication roles separated
One structure aligns the questions asked by model, data, procurement and security teams.
Models receive behavioral specifications and a black-box reference, then rebuild a complete compatible program against hidden tests.
Complete programs with CLI, file I/O, persistent state and error semantics
Black-box specification, reference lock, hidden-test construction and isolated execution
Representative batch of 360+ tasks with 26–140 hidden behavioral tests each
Black-box specification × hidden tests × grader replication; Author, verifier and grader-replication roles separated
Reconstruct complete programs from behavior rather than source; Interface, state, error and boundary tests combined; Cross-environment grader agreement above 98%; accidental passes and leakage removed pre-acceptance
Client identity, original material and selected volumes are removed or banded. This brief explains delivery method and does not guarantee future results.
Review samples. Then define the task.
Carry this direction into your enquiry to confirm delivery, verification and sample scope.
Build data around your next capability goal
Start with a task goal and a sample. Define the materials, specialists, verification methods and delivery scope together.