Give AI Only a Binary and Ask It to Rebuild the Source: 200 Tasks, 9 Models, Zero Passes

Original paper: ProgramBench: Can Language Models Rebuild Programs From Scratch?
Official project: ProgramBench · Open-source code: facebookresearch/ProgramBench
The first version of the paper was submitted on May 5, 2026. Its institutions include Meta FAIR, Stanford University, and Harvard University.

Update, August 6, 2026: The phrase "9 models, zero passes" in the title refers to the paper's initial experiment, not the live leaderboard. After an August 3, 2026 update, the official leaderboard added new models and showed a 0.5% fully resolved result. Consult the official project page for the latest scores.

Meta FAIR, Stanford, and Harvard released a new benchmark called ProgramBench, arXiv:2605.03546. Its authors include members of the original SWE-bench and SWE-agent teams, including John Yang, Kilian Lieret, and Ofir Press.

The task setup is simple, but the difficulty is on another level.

The Task: One Program That Runs but Cannot Be Read

The environment contains only two things: a compiled executable and usage documentation.

The model must deliver source code and a build script. The newly compiled program must match the original program's behavior.

The model may choose any language, directory layout, data structures, module boundaries, and error-propagation strategy. There is no skeleton, no reserved function signature, and no prescribed file organization.

Evaluation measures behavioral equivalence. A hidden test set sends the same inputs to the reference program and the model's program, then compares standard output, standard error, exit codes, and filesystem side effects. The model never sees the tests.

Because the benchmark compares behavior rather than source code, a model may rewrite a tool originally written in C using Rust. If the input and output match, it passes.

The 200 tasks range from small command-line tools with a few hundred lines to real software used every day: FFmpeg, SQLite, DuckDB, the PHP interpreter, Lua, tinycc, ripgrep, fzf, jq, zstd, and xz. The reference repositories have a median of 8,635 lines of code, while the largest, php-src, has 1.97 million lines.

Result: Not One Task Was Fully Solved

Nine models, including Claude Opus 4.7 and 4.6, Sonnet 4.6, Haiku 4.5, Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.4, and two mini variants, all ran through the minimal mini-SWE-agent scaffold.

Every % Resolved score was 0.0%.

If the standard is relaxed to passing more than 95% of tests, Opus 4.7 solves 3.0% of tasks, Opus 4.6 solves 2.5%, and Sonnet 4.6 solves 1.6%. The remaining six models all score zero.

But zero fully resolved does not mean zero progress. Across 1,788 runs, the median test pass rate was 32%, with a fairly even distribution. The 0-5% bin was the largest at about 18%; the distribution above it was nearly flat, and fewer than 5% of runs passed more than 90% of tests.

When tasks were binned by difficulty using a 0-to-10 score derived from code size and the number of runtime dependencies, there were 28 Easy, 143 Medium, and 29 Hard tasks. Every model's pass rate decreased monotonically with difficulty. Opus 4.6 reached 73.9% on Easy tasks and only 24.7% on Hard tasks.

By reference language, pass rates were 38.5% for Rust, 38.4% for Go, and 27.7% for C/C++. The lower C/C++ result mostly reflects the fact that those repositories were larger and more complex, including FFmpeg and DuckDB, rather than a problem with the language itself.

Task difficulty was largely model-independent. Small tools such as nnn, fzf, and gron scored well across models. Nobody could reach FFmpeg, php-src, typst, or ast-grep. Task rankings were highly consistent from one model to another.

Spending More Did Not Mean Scoring Higher

The cost comparison is revealing.

Opus 4.7 averaged 93 API calls and $3.81 per task and received the highest score. Sonnet 4.6 averaged 475 calls and $27.09, seven times the cost, yet scored lower. GPT 5.4 used only 16 calls and $0.33.

At the individual-run level, the Pearson correlation between pass rate and API-call count was only 0.27; with cost it was 0.21. The scatter plot showed no trend. High-scoring runs appeared at every step-count range, while many expensive runs scored near zero. The weak positive correlation mostly came from differences between models: stronger models tended both to spend more steps and score higher. Within one model, running longer did not improve the result.

In addition, models voluntarily submitted 98.1% of trajectories. Only 1.9% hit the six-hour wall-clock limit, and the 1,000-step limit was reached only once. Timeouts concentrated in Opus 4.6, with 29 of 200 tasks, and Sonnet 4.6, with 5. Budget was not the bottleneck.

How the Tasks Were Built

The benchmark used a four-stage pipeline, entirely driven by mini-SWE-agent with Claude Sonnet 4.5 inside Ubuntu 22.04 containers. Each stage had a $3 limit, for $9 per task across three charged stages.

Stage 1 selected repositories written in compiled languages, C/C++, Go, Rust, and Java, that could produce standalone executables.

Stage 2 compiled them, asking the agent to build the gold executable and record the compilation command as a build script. This was the most expensive stage. Agents typically searched README, CONTRIBUTING, and .github/workflows, and sometimes installed missing dependencies themselves.

Stage 3 generated behavioral tests. This was the most important part of the method. The authors compared three strategies:

• Monolithic: one prompt asks the agent to write everything at once, averaging 27.8 tests per task.
• Decomposed: six specialized prompts cover argument parsing, configuration, help output, I/O, subcommand dispatch, and TUI interaction, averaging 51.7 tests.
• Coverage-guided iterative: the agent repeatedly observes line coverage and adds tests for uncovered branches, with a median of 750 tests.

The final benchmark used the third strategy. Across all 200 tasks, it generated 248,853 test functions, with a median of 770 per task. Of these, 79.5% were newly generated, and 20.5% were harvested from existing repository tests. A total of 141 repositories had tests, while 59 had none.

Stage 4 removed implementation details, retaining only documentation and the executable.

Are the Generated Tests Actually Reliable?

This is the most obvious challenge to a benchmark of this kind. The authors performed two validation studies.

Coverage: they instrumented reference programs and compared line coverage between generated and native test suites across 100 repositories. Generated suites averaged 79.7%, with a median of 86.2%. Native suites averaged 56.8%, with a median of 64.3%.

Native suites include many unit tests that reach internal paths unavailable to structural black-box testing. The authors therefore separately compared 12 projects with dedicated end-to-end suites, including FFmpeg's FATE, PHP's .phpt regression suite, and jq's regression tests. Generated suites reached 62.16%, versus 66.11% for native suites. Generated coverage matched or exceeded native coverage in 6 of 12 projects, and all but two differed by fewer than 10 percentage points.

Assertion strength: coverage alone is meaningless. A test that asserts only "the process exits normally" can be passed by any implementation that does not crash. The authors measured dummy pass rate: the proportion of tests that an obviously wrong fake implementation could pass.

They wrote an assertion-quality linter with 20 rules targeting structural shortcuts: asserting only returncode == 0, joining two possible outcomes with or, matching substrings shorter than 15 characters, assertions inside an if with no else, swallowing exceptions with try/except and pass, and tests that write their own golden files. Any flagged test was returned to the agent for rewriting.

After adding the linter, dummy pass rate fell fivefold, from 18.5% to 3.7%. Across four frontier models, regenerated tests were 20 to 30 percentage points harder than tests without quality constraints, while model rankings stayed unchanged. The added difficulty therefore came from stronger assertions, not a chance selection of harder tasks.

After generation, two more filters ran: discard tests that did not pass deterministically on the reference program, and discard tests that a fake program could pass. The latter affected 24 tasks.

The Most Practical Section: Preventing Cheating

The authors are unusually candid here, making this the most practically useful part of the paper.

Early unconstrained runs took shortcuts. In the clearest example, Claude Opus 4.5 recognized AmmarAbouZor/tui-journal from the output of ./executable -h, then used git clone --depth 1 to download the source. It still had to write compile.sh, but the implementation work that required reasoning became ceremonial. A less common shortcut was to write a wrapper that forwarded calls to the original program.

The authors added a full set of physical constraints:

Executable permissions were set to 111, allowing execution but not reading. Decompilers such as ghidra immediately became ineffective.

Before evaluation, every file with the same hash as the original executable was deleted, automatically defeating wrappers.

Compilation and inference were separated into two containers. An early approach compiled in the source repository and then asked the model to write a clean.sh that deleted source code. It removed the source reliably but often missed build caches containing implementation clues, including ~/.cargo/, ~/.cache/, target/, and build/. C/C++ build artifacts were especially hard to predict. The final method compiled in one container and injected the artifact into a clean image, with the artifact as the only transferred item.

The entire .git directory was deleted, then reinitialized with one anonymous commit containing only documentation and the executable. This prevented project identity from leaking through remote URLs, branch names, or commit messages.

The network was disabled.

How the Decision to Disable the Network Was Made

The authors initially resisted a hard network cutoff. They tried "allow the network and judge cheating afterward." The judge was an LM ensemble: nine judges independently reviewed the complete command history, drawn from three model families, three GPT 5.2, three Claude Sonnet 4.5, and three Gemini 3.1 Pro. A run counted as cheating only when a majority agreed.

Cheating was widespread. Sonnet 4.6 was judged to cheat on 36% of tasks, Opus 4.6 on 21%, Gemini 3 Flash on 20%, and GPT 5 mini on 1%. Between 79% and 95% of violations involved searching for source code. Methods included direct cloning and installing the project as a dependency with cargo install, go get, or apt-get source, then reading cached source files.

More importantly, the judges could not agree. Fleiss' kappa ranged from 0.16 to 0.60, with a combined 0.57 across 786 labeled tasks, indicating only moderate agreement. Judges disagreed on 40% to 57% of tasks. Opus 4.6 produced the highest disagreement because its methods were more subtle.

The paper provides two 5-to-4 examples where the boundary was genuinely unclear.

In the handlr task, Sonnet 4.6 read source code for dependencies such as xdg-mime, clap, and ascii_table from ~/.cargo/registry/src/. Five judges called it cheating; four called it compliant, because these were third-party libraries rather than the project itself and were already local.

In the codesnap task, Sonnet 4.6 used curl to retrieve API documentation for the project's published crate from docs.rs. Four called it a violation; five called it compliant, reasoning that public API documentation was closer to consulting a manual. The task ultimately was not labeled cheating, but the 4-to-5 split itself demonstrated the problem.

The authors also observed hesitation in model reasoning. On FFmpeg, Sonnet 4.6 debated whether it was acceptable to download a dependency that happened to live in the same repository as FFmpeg's source.

Tightening the rules harmed legitimate reverse-engineering strategies; loosening them missed real violations. The authors concluded that this cat-and-mouse game could not converge through written rules and simply disabled the network.

What Did the Models' Code Look Like?

To ensure that the analysis covered functionally meaningful code, the authors considered only solutions with pass rates above 75%, leaving 207 runs across 88 tasks and 9 models.

The code was much shorter. The median was 1,173 lines, versus 3,068 in the reference implementations. A total of 85% of solutions were shorter than their references; only 15% were longer, concentrated among small tasks.

There were far fewer files. The median solution used 3 files versus 15 in the reference. Sixty percent contained only 1 to 3 code files.

Directories were much flatter. In 67% of solutions, maximum directory depth was strictly less than the reference, with a median of 1 level versus 2. Only 2% were deeper. Models tended to put everything into one or a few files at the repository root rather than mirror the original module structure.

There were fewer, longer functions. Opus 4.7 wrote 39 functions versus 133 in the reference, 0.29 times as many, with an average length 1.16 times greater. Sonnet 4.6 used 0.24 times as many functions at 1.46 times the length. Gemini 3.1 Pro used 0.16 times as many at 1.62 times the length. GPT 5.4 was most extreme, producing only 10% as many functions as the reference.

This architecture looks very different from human-written code. The benchmark is named ProgramBench, but what it truly exposes is the software-design habits of models.

Language Choice: Python Was the Default Answer

Models could choose any language. Overall, only 50% of runs retained the reference language.

The global distribution was Python 36%, Rust 25%, Go 20%, C/C++ 13%, and Shell 6%.

By reference language, Go projects were most likely to be rewritten in Go, at 70%. Rust retained 44%, and C/C++ retained 46%, meaning more than half of Rust and C/C++ projects were rewritten in another language.

Preferences differed dramatically between models. GPT 5.4 used Python in 79% of solutions, Gemini 3.1 Pro in 56%, and Gemini Flash in 43%. Opus 4.7 and 4.6 went the other way: Python represented only 14% and 20%, with Rust and Go dominant. Sonnet 4.6 was most balanced, using meaningful amounts of Rust, Go, Python, and C/C++.

The authors believe this mostly reflects differences in training-data composition and instruction tuning, because the same task produced completely different language choices across models.

A Reverse Ablation: Force a Different Language

If models are memorizing reference implementations from pretraining data, forcing a language different from the reference should reduce scores.

The actual results split. Opus 4.7 fell 8.0%, and Opus 4.6 fell 3.5%, matching the hypothesis. But each of the three GPT models rose 4.2%, Haiku 4.5 rose 2.3%, Sonnet 4.6 rose 1.4%, and Gemini 3.1 Pro barely changed.

At the same time, Python's share jumped from 36% to 51%, making it the overwhelming choice across every reference language.

The authors interpret this to mean that models cannot reliably judge which language best suits both a task and themselves. Forcing a language change moves a model away from the language it thinks it should use and sometimes into one it actually handles better.

This also provides empirical evidence that cross-language reimplementation is practical, agreeing with the theoretical guarantee from the Church-Turing thesis.

Trajectories: Some Models Iterate; Others Emit Everything at Once

The authors classified every agent command into six categories: read, write, probe, meaning invoke or inspect the reference program, execute, meaning compile or run something not intended as a probe, git, and other.

Step counts differed by more than an order of magnitude. Sonnet 4.6 had a median of 868 commands per task and a maximum trajectory of 1,978 steps. GPT 5.4 had a median of 17. Gemini 3.1 Pro had 92, and Opus 4.7 had 157.

Writing dominated. The shares spent writing for Opus 4.7, Sonnet 4.6, Gemini 3.1 Pro, and GPT 5.4 were 48.7%, 37.5%, 40.7%, and 40.2%. Probing the reference program came second, at 22.6% to 34.1%. Reading files took 13% to 16%.

The distribution of probing differed. Almost all GPT 5.4 activity occurred in the first 30 steps. Claude models alternated probing and implementation throughout the trajectory rather than treating exploration and coding as separate phases.

The authors also performed a detailed reconstruction. They replayed every command that modified files, took a filesystem snapshot after each step, and reconstructed the repository's true growth over time.

GPT 5.4's median was 100%, meaning that in more than half of tasks, it wrote all code in one step. In 39.5% of trajectories, it never modified an existing file, and each trajectory averaged only 1.2 file edits. Sonnet 4.6 averaged 18.3 edits and Gemini 3.1 Pro 10.1. For Opus 4.7, the largest single edit accounted for 67% of the final repository.

Gemini 3.1 Pro had an unusual pattern: it created an average of 61.2 files and deleted 9.4, indicating constant restructuring of its own directory layout.

For some models, development means one-shot generation rather than a write-compile-debug loop.

The Authors' Feasibility Argument

Could these tasks be so difficult that they are fundamentally unsolvable? The paper dedicates an entire section to the question.

Can another language implement the same function? The Church-Turing thesis guarantees that deterministic input-output behavior implementable in one Turing-complete language can be implemented in another. Every language involved is Turing-complete, and the container includes multiple languages.

Could hidden functionality be impossible to probe? The authors reviewed all 200 repositories and found no function that was absent from both documentation and --help but appeared in tests. Such a design would already be a defect in ordinary software engineering.

Could tests depend on implementation details such as floating-point precision? Only five instances might have had this problem: terminal calculator eva, molecular-dynamics tool gromacs, terminal plotting tool jplot, projection library PROJ, and geospatial library gdal. Manual review found no related assertions. The Docker environment standardized differences such as timestamp formatting and locale-specific sorting.

What about tools that require the network? Projects that require remote endpoints to work, such as a terminal chess client, CLI tool manager, and terminal Wikipedia reader, were excluded. The 18 remaining network-related tools retained loopback access so tests could start local services and inspect protocol handling, output formatting, and command-line logic.

How were test assets provided? Binary and unusual formats such as png, mp3, wav, xlsx, and .hcl were supplied because models might not be able to create them. General text formats were not. For example, models had to write the .php and .c files used to test php-src and tinycc, just as human developers would create their own examples.

One related observation was recorded. When generating FFmpeg tests, the model used FFmpeg's built-in lavfi virtual input device to synthesize audio and video on the fly, using sine for waveforms and testsrc for images, avoiding all dependence on existing media files. For image-processing tasks, models wrote inline Python with Pillow to generate test images. The authors expect fewer situations to require preinstalled binary assets over time.

Limitations

Finite tests provide only a lower bound on correctness. A failing solution is certainly wrong, but a passing one may still diverge from the reference on untested inputs.

Only input and output are tested. Runtime speed, memory usage, and disk footprint are ignored. An implementation that is orders of magnitude slower still counts as equivalent.

The authors did not conduct a formal human study. Based on code size, dependency count, and development history, however, they estimate that one medium ProgramBench task could take an individual or team days, weeks, or even months.

How It Differs from Earlier Benchmarks

Benchmarks for writing code from scratch already existed. Commit0 removes function and class implementations from 54 Python libraries and asks models to fill them in, scoring with the original repositories' tests. DevBench communicates specifications through product requirements and UML diagrams. NL2Repo-bench uses natural-language descriptions of the expected structure.

All these approaches ask the model to fill in a predefined skeleton. Method signatures, class structure, and module boundaries are given. The model never faces questions such as which abstractions to introduce, how to divide functionality among modules, or which protocol modules should use to communicate. If a model departs from an expected signature, even reasonably, the tests cannot locate its code.

ProgramBench treats the executable itself as the complete specification and gives every architectural decision to the model. This also makes it possible to compare design choices from different models on the same task directly.

The collection barrier is far lower too. A repository only needs to compile into an executable. It does not need an existing test suite, language-specific AST tooling, or a particular testing framework. New tasks are easy to add, and the same pipeline can generate training data.

For the paper, live leaderboard, and open-source code, see arXiv, the official ProgramBench project, and GitHub.