Long-horizon repository engineering evaluation
Repository exploration, multi-file implementation, regression protection and environment verification organized as adjudicable long-horizon work.
Task understanding before scale
A software-intelligence program needed to measure whether models could understand an unfamiliar repository, locate cross-module defects and submit complete patches without breaking existing behavior. Tasks were sanitized from authorized repository snapshots with dependencies, containers and base commits fixed. Candidate runs retained search, edit, test and recovery traces; programmatic verifiers and independent engineers jointly reviewed correctness, regression risk and completion.
Unfamiliar repository exploration and multi-file implementation
Patch, tests, logs and recovery trace accepted together
Programmatic verifiers plus independent engineering review
One structure aligns the questions asked by model, data, procurement and security teams.
Repository exploration, multi-file implementation, regression protection and environment verification organized as adjudicable long-horizon work.
Authorized, sanitized repository snapshots with cross-module engineering issues
Original task construction, container lock, long-horizon run, regression tests and engineering review
Representative batch of 310+ repository tasks with 2–14 files changed per task
Repository snapshot × long-horizon trace × CI verification; Programmatic verifiers plus independent engineering review
Unfamiliar repository exploration and multi-file implementation; Patch, tests, logs and recovery trace accepted together; CI reproduction above 96%; environment failures separated from model implementation failures
Client identity, original material and selected volumes are removed or banded. This brief explains delivery method and does not guarantee future results.
Review samples. Then define the task.
Carry this direction into your enquiry to confirm delivery, verification and sample scope.
Build data around your next capability goal
Start with a task goal and a sample. Define the materials, specialists, verification methods and delivery scope together.