Open transcript
So OpenAI just published this, PaperBench, evaluating AI's ability to replicate AI research. OpenAI has been publishing more research and open sourcing a lot more projects recently. And this one's very interesting for a number of reasons. As they've tweeted from their account, they're saying that they're releasing PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. As part of our preparedness framework, and a lot of the Frontier AI labs, they have their own version of the preparedness framework. Basically, it's how we track the potential AI risk. As these AI models get better and better, we want to kind of see the escalating threats that they potentially could pose. And we're sort of tracking it on this, at least with OpenAI, we're tracking in this scale of low, medium, high, and critical. OpenAI tracks the AI risk across these four categories, cybersecurity, CBRN, which is chemical, biological, nuclear, and radiological threats. Persuasion and model autonomy, right? And sort of like the highest thing, any model scores across these categories, that sort of is taken as the risk score. So if it's a low on cybersecurity, but critical in persuasion, for example, we'll still call that a critical model risk. Now, specifically today, we're talking about the model autonomy. Of course, model autonomy is kind of the promise of AI agents, how well they're able to execute long horizon tasks. And of course, that will provide a great many benefits. But also, we have to be a little bit careful about how we kind of put that out there in the world, because there could be a number of bad unintended consequences. And one specific thing that a lot of people are kind of uneasy about is this idea of recursively self-improving AI agents. Underneath this post by OpenAI, AI safety memes posted this, like, I will not build recursively self-improving AI. I will not build recursively self-improving AI. And of course, we saw this idea covered in the situational awareness by Leopold Aschenbrenner. So he's kind of talking about a potential inflection point where automated AI research takes over. So in other words, these AI agents will be better than human scientists at improving AI. It will be able to improve itself better than we were able to improve it. And theoretically, that would lead to this sort of intelligence explosion, right? Recursive exponential growth of AI intelligence. And this is why this paper, this benchmark and the open source code that OpenAI published, that's why it's so interesting to read about, because OpenAI is tasking various AI agents, not just OpenAI, but also you'll see Anthropic in there, for example, they're trying to get them to replicate the top machine learning papers of 2024. So the ICML, that's the International Conference on Machine Learning. So we're taking cutting edge machine learning AI papers, and we're creating these AI agents tasked with replicating them. And we're sort of testing them on their ability to number one, read and understand the paper. Are they able to sort of grasp what the point of the paper is, the finer details? Two, they need to build a full code base from scratch to conduct the experiment in that paper, for example. We'll take a look at exactly what that looks like in a second. But after that, we test it on if it's able to run and reproduce that experiment. And of course, does it match the reported results by the paper? Now, not that long ago, just a few weeks ago, we covered this paper out of Sakana AI, or this blog post rather, the AI scientists. So this is the creation of Sakana AI. They used a large language model capable of producing scientific papers on machine learning. And a few weeks ago, they submitted it to a peer review workshop to see if it has the muster, so to speak, that all the human-ridden, you know, top machine learning papers have to pass in order to be accepted. Now, they did notify the human reviewers that maybe some of the papers they'll be reviewing could be machine-generated, AI-generated. But the important point to understand here is that these AI-generated papers were created entirely end-to-end by AI. The only sort of human contribution was they had to pick kind of the broad subject so that it would fit the workshop that they were being submitted to. So they said, for example, you know, this has to be machine learning papers based on sort of this subtopic. And then the AI scientist came up with a scientific hypothesis, proposed the experiments to test that hypothesis. It wrote and refined the code to conduct those experiments. It ran the experiments. It analyzed the data. It visualized the data in figures, so charts, figures, visuals, etc. And it wrote every word of the entire scientific manuscript, from title to the final reference, including placing all the figures, all the formatting, I mean everything, right? So it created the paper. So they submitted three AI-generated papers to this workshop, and one of them got a score that was enough to be sort of accepted. What that means is that these scores are higher than many other accepted human-ridden papers. Now, this was just an experiment, so they pulled the paper out after that. They didn't continue on the track, but basically they wanted to see if it could pass the peer review process. It did, or rather it would have if it was allowed to continue, most likely. And it was the first time a fully AI-generated paper was good enough to pass a standard scientific peer review process. And with OpenAI's paper bench, we're doing something a little bit different. We're taking human-ridden papers, so state-of-the-art, sort of some of the best papers produced, you know, in 2024, the previous year. And we're trying to see if the machines can figure out what it's talking about to reproduce it. Now, in science, if you're publishing a paper, it's very important that other people are able to reproduce that paper, right? You might recall a while back, the world was in shock when it appeared that somebody proposed how to create a superconductor, right? LK99. In July 2023, they published a paper claiming that LK99 is a superconductor. One problem, you know, during the replication, right, when other scientists all around the world were trying to replicate those results, they couldn't. They were not able to replicate those results. So replicating other results in science is extremely important. Number one, it verifies whether it's true or not. Now, when we look at sort of all the published research, we know that there's some issues with how that research and what kind of research gets published. So as you can see from this chart, for example, the papers that get published, generally we see kind of a big result that they found. So if we try some medicine to treat depression, for example, and it works really, really well, like we publish those results. If it doesn't work too well, it's not statistically significant. People tend to not publish it as much. So you can see here, we really like the strong positive effects. We'll take the strong negative effects because it has some meaning. We don't like this kind of like lukewarm results here in the middle. What happened to all the papers with these lukewarm results? It seems like somebody just kind of like took a big bite out of them. Who knows? The point is people either don't publish them because they're not as interesting, or maybe some people fudge numbers here and there to get certain things published. And this is where AI can actually help out quite a bit. So for example, with the AI scientist, the results were negative, right? So that first paper that was ever peer-reviewed, that was fully AI-generated, the AI came up with a hypothesis. It tested that hypothesis, but as it says right here, contrary to our expectations, our experiments on whatever, they did not achieve significant improvements, right? So it said, let's see if this approach works. And they tested it and they're like, this approach doesn't really work or it doesn't seem to work. So a human being might be tempted to go, okay, then I'm not putting in the work of typing all this up and publishing it because like, what's the point? But the reality is that work is important because it shows other scientists what doesn't work. So for example, one of the things that this AI did is actually one of the first things it does when it's coming up with a hypothesis is it sees if there's any other papers that are on that subject that already have been published. When it realized that there weren't, it did that experiment. It's very possible that several teams did this research, did not find a statistically significant improvement and just decided not to publish that research. Again, it kind of fell here. It was not statistically significant. So people were like, ah, why bother? But AI scientists still did it, still added that knowledge to kind of like the global knowledge about machine learning. So it is useful. It is helpful. And AI might be great at doing this type of work that's seen as like unexciting or unimportant. Paper Bench on the other hand is able to replicate the work to see if those results are able to be replicated. Again, to see if it's real or not, to make sure it's not a coincidence, to make sure it's not cherry pick data, to make sure there's no faulty methods or faulty data. Ideally, you'd be able to sort of make sure that it's real, right? So if somebody anywhere in the world produces a paper, it can double check and see, okay, this, this looks legit. We're able to reproduce those results. We're able to test it. And then other scientists would be able to build on top of it, knowing that this is sort of reproducible and legitimate and valuable. So let's take a look at their findings. So Paper Bench, evaluating AI's ability to replicate AI research. Can AI agents replicate state-of-the-art AI research? They're taking 20 papers from the ICML 2024, Spotlight and Oral papers. It's trying to replicate it from scratch, including understanding paper contributions, developing a code base, and successfully executing experiments. So they took those 20 papers and they've carefully, meticulously constructed sort of a grading rubric. Interestingly, there's over 8,000 individually gradable tasks. So you can see here, the rubric kind of breaks down the replication tasks into a tree of increasingly granular requirements. So these leaf nodes are graded on a kind of a binary pass-fail criteria. And the sort of the parent node is the weighted average of the children. So the leaves. So weighted just means that some of them are more important than others. And these rubrics were co-developed with the authors of each of these papers for accuracy and realism. So this is kind of one of the big limitations because for each one, they had to take the author of the paper, sit down, and create this very intricate grading rubric. So in that sense, it's not very scalable, right? So it requires a state-of-the-art paper. It requires the author to kind of code develop the rubric and stuff like that. But within those 20 papers, we've created sort of a very good benchmark for testing the various AI agents on how they're able to perform. And also they have an LLM-based judge that automatically grades the replication temps against the rubric. And they also have a separate benchmark for those judges to make sure that the judges are good. And they tested several frontier models on this benchmark, finding that the best performing is CLAW 3.5 Sonnet with open source scaffolding. So open source scaffolding. Scaffolding is kind of attaching some code to it and some tools, kind of giving it some additional functionality. So it's like a little vehicle. You plug in the AI inside of it and it does the stuff, right? So it's usually kind of helps guide it along and it feeds it a series of prompts so that it's able to walk through the entire process. All right. So CLAW 3.5 Sonnet gets a 21, right? It's able to replicate 21% of those papers. Now, interestingly, on a three-paper subset, right? So they took three papers that were a subset of those 20, and they took human machine learning PhDs. This is kind of fascinating, right? So we're not comparing this to the average person off the street. We're taking machine learning PhDs. How well are they able to replicate these papers, right? So they give them 48 hours of effort, not 48 hours to get done, like 48 hours of actually, you know, working on the thing. And it looks like best of three attempts. So the machine learning PhDs, right, they achieved a 41.4%, right? The O1 achieved 26.6 on that same subset of three papers. And a lighter sort of judging rubric, a more lightweight evaluation, the O1 achieves a score of 43.4. So what does that mean? Well, the finding that models do not yet outperform the human baseline. Now, interestingly, as far as judging goes, what they find is that the best LLM-based judge, which uses the O3 mini high with custom scaffolding, achieves a score of 0.83, suggesting that this judge is a reasonable stand-in for a human judge. So the models are able to act like a human judge, right? As a human grader, they can sit there and kind of go through it and check what's working, what's not, to make sure that the replication of the paper is correct, right? So like somebody grading the student's papers, they're a reasonable stand-in for a human judge. But the important thing to me here is that, yes, the models don't yet outperform the human baseline. And again, the human baseline is machine learning PhDs. That's the baseline. Are the models right now better than humans with a PhD in machine learning? No, they're not. And maybe thank God, they're not quite there yet, right? Maybe we're not quite ready for that. But don't dismiss all of this because they're not quite outperforming the human level PhDs, because these agents do exhibit non-trivial capabilities in replicating machine learning research papers. All right, so really fast, let's kind of briefly go over like what actually happens when they try to reproduce these papers. So the agent, right? So it's the model, whether that's Anthropics or Opening Eyes or any other model you choose. And it's got some scaffolding, right? Some tools and some things that give it some extra functionality to make sure that it goes through and completes everything. And that model needs to produce a submission, right? So it tries something and it gets graded on that submission. And that submission, it's a repository, including all the code required to reproduce the paper's empirical results. And it needs the specific file, reproduce.sh, which is like the entry point for executing all the necessary code. So basically, it's just like when you're writing an essay, the teacher gives you some requirements. It's got to have this, it's got to have this, like pretty self-explanatory. And a submission successfully replicates the paper if that file, running that file, reproduces the empirical results reported in the paper. Now, you might recall Dr. Kyle Kavasaris, we've talked about him on this channel before. He actually jumped on one of the live streams. And then when we were testing out the 03 mini high, or no, it was the Grok, I think. And then he picked up his live stream where my fell off because I had to go to sleep. So it was kind of an interesting crossover. But last year, he was covering the release of the 01, 01 preview. And I apologize if I'm getting some of the details, if I'm not remembering all the details, but he wrote his PhD dissertation based on some research into black holes. So there's some sort of an astronomical data that was used and you needed a code base to run through that simulation. I don't remember all the details. I apologize, but it took him something like 10 months to create that code base for that paper. So this is a person that got his PhD. It took him 10 months to write that code. And he, on a whim, decided to ask the 01 model if it could like replicate that code for him. And it did, kind of. He had to go back and forth for a little bit. He had to give it some hints. It used synthetic data that he asked it to create, I believe. But the point is, so this is somebody that has a PhD. It takes them, let's say, one year to create that code base. 01 preview plus 01 mini knocks that out in about an hour. Obviously, there's a huge potential here. There's a lot of power here. Now with Dr. Kyle's paper, a lot of people were asking, well, if you posted the paper, isn't the code base out there on GitHub somewhere or something like that? It sounds like it wasn't. So the GitHub was private. Again, I don't know all the details. Whatever the case, here in this paper, they, of course, disallow agents from using or viewing the paper author's original code bases. So again, just so people kind of understand what we're talking about here. So this is that paper by Dr. Kyle, Black Hole Mass Measurement of Early Type Galaxies, NGC. I'm not going to try to read that. Observations and Gas Dynamical Modeling. But the point is, there's the paper where they meticulously describe the scientific experiment, the hypothesis, and how they're going to build the code to test that hypothesis. So here in English, they described the method for doing that thing, right? So here he takes a prompt and he's saying, please read the following method section of this research paper and recreate the Python code described. And he just copies and pastes that in there from that PDF of the paper. And the model reproduces the code, not the same exact code that he wrote. Again, it creates a slightly, you know, like if you ask 10 different people to code something up, they might produce 10 different versions of it. It does the same thing. So it's not a copy and paste. It's not like it found the code somewhere. It thought through how to do it and it created that itself. Now, if you watch the video, it doesn't do it one shot, right? They had to go back and forth a little bit. But the point is it gets it. It gets it in under an hour of work. So here in the paper, the models reproduce the code. They're not allowed to look at the researcher's code that they made. They have to do it from scratch, which is important because if there's some problems with the code, with mistakes or some hidden things, some faulty methods or faulty code or whatever, we don't want to just like assuming it's right and just copy and basting it. We want it to create it from scratch based on the paper, on the, you know, the English language or whatever language it's in, the natural language, figure out how to build the code base for that. And how do we grade it? Well, number one, the results match. Did the results of the code base, did they match that of the research paper? Also the execution, right? Since the results match are very challenging to achieve, having several execution nodes allow submissions to receive credit for marking partial progress towards a result, even if the corresponding result match node isn't achieved. So it gets credit for showing its work. Like if it's able to reproduce some portion of the code base, if we're able to run it, like it still gets credit for that. And the code development, does the source code appear to contain a correct implementation of some requirement. So to test these models against the human baseline, they recruited eight participants who are currently enrolled in or have completed a PhD in machine learning. The participants have similar conditions to the AI agents. They get the paper and PDF and a markdown format. They get the addendum, the instructions that are as close as possible to those used with the AI agents. They have a single NVIDIA A100 GPU. And of course they're able to use any AI assistants such as JATGPT and GitHub co-pilot. There are some blacklisted websites, right? So they can't look at the code for it seems like. And interestingly, so initially the AI agents outperform the human performance. O1 initially outperforms the human baseline during the early stages of the replication attempt, but humans start outperforming the AI after 24 hours. This trend of AI agents, you know, initially outperforming humans, but falling behind at longer time horizons is consistent with previous results, right? So the O1 here is fast out of the gate. It writes a lot of code quickly at the beginning, but fails to effectively work beyond a certain time horizon. Humans are slow in the initial hours, right? But then start getting better because perhaps as humans spend time digesting the paper, they improve. So incredibly exciting. So we're beginning to see the very, very early stages of these AI models beginning to contribute to science. We have one passing the peer review process. We have these beginning to replicate some of these machine learning papers. Already they are effective judges, right? They're able to score these submissions fairly well. And already we're beginning to see it creating those code bases for papers, which of course is exciting for many reasons, maybe also a little bit scary for many reasons. Doesn't mean that we could potentially be approaching this mythical intelligence explosion as these models get better, as we develop ways to incorporate into the scientific discovery better. I would point to they become better than humans at developing AI, contributing to the machine learning field, creating code better and faster than humans can. We certainly went from, you know, when they couldn't code or do any of this to here where we are now pretty quickly in a number of several years, let's say. But maybe there's some plateau that is going to hit to where this progress won't continue, or maybe it's going to blow past this point and keeps going. Let me know what you think. Is this exciting? Is this scary? Do you think we're going to see an intelligence explosion within our lifetimes? And what do you think about the idea of these large language models meaningfully contributing to scientific discovery, scientific research? Are you excited about it? Let me know in the comments. If you made it this far, thank you so much for watching. My name is Wes Roth and I'll see you next time. Since I have, I'm. Thank you. Thank you.
