For seven years, if you wanted to know the actual state of the art on a machine learning task — not what a survey paper claimed six months ago, but what was winning right now — there was one place to look. You typed in “image classification,” or “question answering,” or whatever corner of ML you worked in, and Papers with Code handed you a live leaderboard: methods ranked by reported performance, each one linked straight to its paper, its code, and the dataset it ran on. No searching through conference proceedings. No stitching together GitHub stars and citation counts to guess at relevance. Just the leaderboard.
Then, on July 24–25, 2025, one of the most influential pieces of open ML research infrastructure was gone. Meta pulled the plug with no announcement, no transition window, and no warning to the researchers who had built parts of their workflow around it. The first anyone heard about it was 502 errors and garbled dataset pages. GitHub issues went unanswered. There was no blog post explaining what happened, or why.
This is not really a story about nostalgia. It is about understanding what got built, what it actually changed in how ML research works, and what the landscape looks like now that the thing everyone relied on is switched off. If you used Papers with Code for state-of-the-art (SOTA) tracking, benchmark comparison, or simply finding a working implementation of a paper, that context matters for how you navigate what comes next.
What Papers with Code Actually Was
Founded in 2018 by Cambridge graduates Robert Stojnic and Ross Taylor, Papers with Code set out to solve a simple but consequential problem: connecting machine learning papers to the code needed to reproduce them.
Rather than acting as another paper repository, it let the community maintain the links between four things that had always lived apart — research papers, open-source implementations, benchmark leaderboards, and datasets. The platform grew quickly precisely because contributions came directly from the research community, which made it one of the most current resources for SOTA tracking in machine learning.
When Meta acquired Papers with Code in 2019, the founders promised it would remain “open, neutral, and free,” and for nearly six years that held. Stojnic and Taylor went on to lead development on Llama 2 and Llama 3 at Meta AI — its own small irony, given how the platform they built eventually ended.
The structure underneath it was simple and effective. Tasks sat at the center, each with a leaderboard, and each leaderboard entry linked out to its paper, its code, and its dataset. At its peak that meant over 1,000 tasks organized, 9,327 leaderboards, 79,817 paper-to-code linkages, and 5,628 cataloged datasets, all released under CC-BY-SA. Crucially, none of it was editorially gatekept — researchers submitted and linked their own work directly. That kept the platform current, but it also meant coverage tracked who showed up to contribute, not what actually existed in the field.
What It Got Right
Making reproducibility the default expectation. Before Papers with Code, plenty of ML papers reported impressive results without shipping the code behind them, and reproducing an experiment often meant rebuilding a model from scratch. By putting the official GitHub repository next to the paper, Papers with Code turned code sharing into an expectation rather than a bonus. Papers with accessible, well-maintained implementations became more visible and easier to trust.
Real-time SOTA tracking. The benchmark leaderboards were the platform’s biggest single strength. Instead of relying on review papers that went stale within months, researchers could see immediately which models performed best on a specific task. Whether the task was image classification, question answering, or text generation, Papers with Code was the fastest way to track state-of-the-art progress.
Bridging research and practice. A paper is valuable; usable code makes it practical. Papers with Code connected publications to official implementations and community reimplementations, frequently across multiple frameworks such as PyTorch and TensorFlow. That saved researchers, students, and engineers an enormous amount of time and made adopting new methods substantially easier.
Better dataset discovery. The platform simplified dataset exploration in the same way. Rather than searching across multiple sources, researchers could find datasets alongside the papers that introduced or used them, plus the benchmarks built around them. It was never exhaustive, but it became one of the most useful directories of machine learning datasets available.
Faster research cycles. The dataset catalog extended the same logic to data: 5,628 datasets across vision, NLP, audio, and reinforcement learning, each tied to the papers and benchmarks that used them. Researchers could compare methods, identify performance gaps, and build on existing work instead of starting over. In many ways it became the backbone of open ML research infrastructure, helping the community move faster while making research more transparent.
Where It Fell Short
None of this makes Papers with Code beyond criticism, and it is worth being honest about where it did not hold up. Self-reported results on a community-maintained leaderboard are not the same thing as independently verified benchmark performance. The platform made that gap visible to anyone paying attention, but it never closed it. Leaderboards reflected what researchers claimed on standard test sets, with no audit layer behind them. Cherry-picked evaluation conditions, metrics optimized specifically for the leaderboard, unflattering baselines quietly left out — all of it was possible, and all of it happened.
Coverage also skewed hard toward high-visibility work. Computer vision, NLP, and reinforcement learning tasks were tracked in real depth. Computational biology benchmarks, ecological ML tasks, and materials science prediction challenges existed on the platform, but thinly. For researchers working in ML-adjacent scientific fields rather than core ML, Papers with Code was useful but partial.
And then there is the limitation that turned out to matter most: infrastructure that lives entirely inside one company is not community infrastructure, no matter what commitments are made at acquisition. Meta said in 2019 that Papers with Code would stay open. In July 2025 it did not ask anyone, and shut the platform down. Seven years of benchmark history, paper linkages, and dataset metadata became unreachable because one company made one decision.
The Shutdown, and What’s Actually Gone
The mechanics of the shutdown were almost aggressively quiet. On July 24–25, 2025, the platform went dark with no announcement. Users found out through “Bad Gateway” errors and dataset pages rendering as garbled text. GitHub issues piled up unanswered. There was no blog post, no timeline for what would happen to the data, and no export notice giving people a chance to pull what they needed first.
What was lost is not fully recoverable. The live benchmark tables that tracked reported results over time. The links connecting papers to multiple independent implementations across different frameworks. The historical view of how SOTA on a given task actually progressed, entry by entry, year by year. And the thing that tied it all together — a single searchable interface across papers, code, benchmarks, and datasets — simply gone from its original location.
The historical data does still exist, archived at paperswithcode/paperswithcode-data on GitHub under the original CC-BY-SA license. But it is frozen as of the shutdown date, with no updates and no maintenance, and it requires someone willing to work with raw data rather than a web interface. A 2025 paper on post-publication code verification noted that the platform’s disappearance meant the loss of information that could not be fully reconstructed — including the historical benchmark tables and the cross-framework implementation links that made it genuinely distinctive.
What’s Filling the Gap
No direct replacement offers the same experience, but several tools now cover different parts of what Papers with Code did.
Hugging Face Paper Pages. Launched immediately after the shutdown, this has become the closest alternative. It helps researchers discover new papers, reach linked code repositories, and engage in community discussion around them. Its benchmark tracking, however, is still considerably less comprehensive than what Papers with Code offered.
Semantic Scholar and Connected Papers. For literature discovery, both remain excellent. They make it easy to explore citation networks, related research, and influential papers. What they do not provide is the tight integration between papers, code, datasets, and benchmark leaderboards that made Papers with Code unique.
CodeSOTA. CodeSOTA takes a different approach, independently verifying benchmark results instead of relying on self-reported scores. Its coverage is currently limited to selected domains, but it addresses one of the platform’s biggest weaknesses: benchmark reliability.
Domain-specific resources. Researchers in computational biology, materials science, and other specialized fields increasingly rely on dedicated platforms such as MoleculeNet, Therapeutics Data Commons, and DGL-LifeSci, alongside Hugging Face’s Dataset Hub. These are valuable, but they remain fragmented — researchers end up switching between platforms instead of working from a single connected knowledge base.
Where That Leaves Things
In seven years, Papers with Code built something the ML community had talked about wanting for a long time but never quite coordinated on its own: one open, queryable map of papers, code, benchmarks, and data, all connected. It changed how people found methods, how practitioners located working implementations, and how the field kept track of its own progress without waiting for a survey paper to catch up.
Its shutdown is now the clearest case study the field has for what happens when open infrastructure lives inside a single company’s decision-making. The data survived. The platform did not. And the researchers who built real workflows around it are now assembling a set of replacements that, together, still do less than the original did on its own.
What the disappearance made obvious is that this kind of infrastructure mattered enough for people to notice the moment it vanished. The open question is whether the field builds a version of it that is not one executive decision away from disappearing again.
Call to Action
Research infrastructure is only as durable as its ownership. Papers with Code showed how much value a connected map of papers, code, benchmarks, and datasets unlocks — and how quickly that value evaporates when the map belongs to one company. Axy is built with that fragility in mind: a scientific knowledge graph where every link is auditable, conflicting evidence is preserved rather than flattened, and every change passes human review.
If you want to help build the shared, connected map of science rather than just use another tool, apply to join below.
