Agent capability boundary benchmark

ReproBoundary

An Agent Capability Boundary Benchmark for Complex Paper-to-Code Reproduction

Code Hugging Face Tasks Submit

47 tasks · 9 model-agent settings · 423 trajectories · 15 domains

Introduction

Reproducing a machine-learning paper requires more than generating plausible code. An agent must recover the target claim, respect the experimental contract, complete every required stage, and deliver artifacts whose reported metric can be independently checked. ReproBoundary evaluates this end-to-end capability across 47 frozen paper-to-code tasks.

Each task fixes the source repository and revision, data scope, required stages, compute budget, target metric, reference value, and evaluator before the run begins. Twenty tasks retain the paper-selected experimental scale; 27 reduce training, search, or repetition while keeping the complete target evaluation set.

Scope. Each task reproduces one selected result under a fixed resource limit. It does not reproduce every result or support every conclusion in the source paper.

Benchmark Overview

A frozen task contract connects the paper claim, agent execution, delivered artifacts, and independent verification.

47paper tasks
20paper-scale
27resource-bounded
423execution logs
15research domains
Distribution of the 47 ReproBoundary tasks across research domains
Distribution of the frozen task pool across research domains.

What ReproBoundary Checks

01

Understand and plan

Recover the selected claim and translate the frozen specification into a feasible experimental pipeline.

02

Execute the contract

Complete every required stage within the declared compute, time, memory, and storage envelope.

03

Deliver evidence

Retain traceable outputs and process evidence rather than treating a plausible final report as success.

04

Verify independently

Check artifact completeness and metric agreement separately with the frozen task-specific evaluator.

Task Explorer

Browse all frozen tasks by paper, domain, target metric, dataset, or experimental scale.

Loading task inventory...

Paper and target taskDomainTarget metricScaleBudgetSource repository

Data and Verification

GitHub hosts the lightweight task definitions, evaluator records, protocol templates, and verification code. Hugging Face hosts 423 cleaned, compressed execution logs with a central provenance index and SHA-256 fingerprints.

git clone https://github.com/ReproBoundary/ReproBoundary.git
cd ReproBoundary
python3 scripts/verify_release.py

# verification OK: tasks=47 evaluators=47 trajectory_records=423 bundled_logs=0

Evidence boundary. The logs support chronology, tool-use, failure, and release-integrity analysis. They do not include checkpoints, predictions, model weights, or training states and cannot by themselves replay most scientific metrics.

Results

The public leaderboard is pending publication of frozen, provenance-checked aggregates. No provisional result is presented as a verified benchmark score.

Submit a Run

Send the evaluated configuration, result files, and a link to supporting artifacts for verification.