What We Learned by Reproducing 2,200 papers from ICML

In a massive, community-driven experiment that pushed the boundaries of scientific rigor, over 1,200 researchers and developers recently spent 19 days attempting to verify the claims of ICML 2026 papers. Utilizing a fleet of autonomous coding agents, the participants generated 6,816 detailed logbooks, successfully auditing 2,226 papers—roughly one-third of the entire conference proceedings.

This unprecedented initiative, which ran from July 15 to August 2, offers a sobering and illuminating look at the state of AI research. As the volume of academic submissions continues to explode, this hackathon provides a blueprint for how we might maintain scientific integrity in an era where AI agents can write papers faster than humans can review them.

The Scaling Crisis in Academic Review

The challenges surrounding the reproducibility of AI research are not new, but they have reached a breaking point due to the sheer scale of modern output. ICML 2026 saw a staggering 23,918 submissions, with 6,352 papers accepted. This exponential growth is fueled in part by the very AI agents that are now being used to draft experiments and write manuscripts.

Unfortunately, the human capacity for peer review has not kept pace. Reviewers are typically volunteers, often balancing these duties with full-time research or industry roles. The result is a system where deep, line-by-line verification is increasingly rare. Consider this candid remark from a reviewer of an accepted ICML 2026 spotlight paper:

"My low confidence score is because I did not check all the proofs carefully."

Despite this admission, the paper received high marks and a prestigious spotlight slot. This incident serves as a primary case study for the necessity of automated, scalable auditing. While a human reviewer might struggle to find a weekend to verify complex proofs, modern coding agents—such as Claude Code, Codex, and Cursor—can ingest a paper, write the necessary code, execute experiments, and report findings in a single afternoon.

The Mechanics of the Hackathon

Rather than centralizing the audit, the organizers opened the challenge to the global community, embracing a diverse array of agent frameworks and compute strategies. The workflow was designed to be both transparent and rigorous:

• Targeted Audits: Every accepted ICML 2026 paper was indexed, with core scientific claims extracted into a machine-readable format. This allowed agents to focus on specific, checkable hypotheses rather than parsing 40-page PDFs.
• Standardized Reporting: Every experiment resulted in a Trackio logbook—a public Hugging Face Space containing the code, the artifacts, and the full execution trace.
• Automated Adjudication: A "Logbook Judge" powered by the GLM-5.2 model reviewed every submission, assigning a verdict of verified, falsified, toy, or inconclusive. Crucially, the judge was instructed to treat the participants' own self-assessments with skepticism.
• Compute Support: Participants were provided with $20 in Hugging Face compute credits, leading to 2,962 cloud-based jobs.

By the end of the event, the numbers were staggering: 1,221 participants, 6,816 published logbooks, and 35,908 individual claims judged.

What the Data Revealed

The results of the audit paint a complex picture of modern AI research. Reproducibility is rarely a binary state; it is an adversarial process.

• Verified Successes: 51% of the examined papers (1,103) saw at least one claim independently verified. Of these, 266 were fully reproduced, with every claim confirmed.
• Contested Claims: 23% of the papers (496) had at least one claim falsified or contested. This included 49 papers where every claim failed verification, and 242 instances where different teams reached opposite conclusions on the same paper.
• The Middle Ground: 502 papers provided only "toy-scale" evidence, while 280 remained inconclusive, often due to missing datasets or proprietary code.

Some papers stood out for their robustness. For instance, the paper "Flat Minima and Generalization: Insights from Stochastic Convex Optimization" was independently reproduced by 20 different teams, 12 of which verified every claim. Similarly, a paper titled "A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness" held up under the scrutiny of 14 out of 17 independent agent-led audits—a poetic victory for a paper about the reliability of AI judges.

Anatomy of a Falsification

When 35 participants reported successful falsifications, the organizers performed an adversarial re-verification of every claim. The findings were revealing:

1. The Paging Paper Revisited

Returning to the spotlight paper mentioned earlier, the participant who actually checked the proofs found that the algorithm’s robustness did not hold as claimed. While the paper promised a specific additive term, the agent-led audit confirmed that the error grew logarithmically as the scale increased. The "low confidence" reviewer was right to be concerned; the math simply did not hold up at scale.

2. The "Finite Horizon" Trap

In the paper "Attention's forward pass and Frank-Wolfe," the authors proved that token particles collapse to the origin. However, three independent teams found counterexamples. The issue? Most verification attempts stopped too early. The collapse only failed at specific, high-step counts (e.g., 224, 3,800, and 6,416 steps). Because previous checks were too short, the error remained hidden.

3. Theory vs. Implementation

In "Self-Distillation Enables Continual Learning," the theory section analyzed reverse KL divergence, but the released code defaulted to forward KL. The discrepancy was caught by a participant who noticed the headline results could not be replicated using the provided code. The authors have since uploaded a corrected version to arXiv.

4. Diluted Evaluations

In "Do Transformers Need Three Projections?" an audit revealed that 66% of the evaluated label positions were simply padding tokens. Once these were removed, the performance gains claimed by the authors dropped significantly, from a 3.1% quality cost to roughly 9.4%.

The Enduring Role of Humans

Perhaps the most significant takeaway from this hackathon is that human oversight is not obsolete—it is more critical than ever. While agents are powerful, they are prone to specific failures: they can get stuck in infinite loops, misinterpret scale-dependent behaviors, or fall victim to unit mismatches.

The most reliable results during the challenge came from "human-in-the-loop" workflows. In one notable case, an agent was tasked with evaluating image generation quality under quantization. While numerical metrics suggested no degradation, the human participant built a custom UI, manually reviewed 128 image pairs, and had the agent validate the consistency of those human annotations.

"The challenge's most reliable results came from workflows where a human was steering: re-pointing the agent, questioning an assumption, or deciding that an experiment's premise was wrong before burning a week of compute on it."

Moving Forward

The organizers have begun contacting the authors of the papers where errors were confirmed. The response has been overwhelmingly constructive, with several authors already submitting corrections to arXiv.

This event proves that we have the tools to audit the flood of AI research, but we must be intentional about how we use them. The role of the human researcher is evolving into that of a "Principal Investigator" for AI agents—setting the environment, defining the parameters, and providing the critical judgment that machines cannot yet replicate.

As the largest open, claim-by-claim audit of a machine learning conference to date, this hackathon sets a new standard for transparency. The organizers hope this record is broken soon, signaling a future where every scientific claim is subject to the same level of rigorous, automated, and community-led verification.