Open transcript
So we have yet another paper about an autonomous, self-improving AI agent. This one is learning to play settlers of Qatar. What's not to love? So I know not everyone loves this term AI agents, and I totally get that. It's probably not the best term for it, but we don't currently have a better one. So we're just going to use it here. The paper uses it. These are large language models based agent architecture. So we're taking a large language model, creating some scaffolding around it, some architecture that allows it to play the game, to write code, to take notes, etc. And this kind of approach is very popular. That's basically what Alpha Evolve is. The big breakthrough out of Google DeepMind. The Darwin Godel machine is also very, very similar. It's a self-improving coding agent. You can think of it as a large language model with scaffolding. Same thing with any of those LMs that play Pokemon, for example. Minecraft Voyager, the AI game, GPT-4 that played Minecraft and got better, wrote its own code to improve its abilities, etc. That was kind of the same thing. LLM plus scaffolding. Where scaffolding is basically stuff that we give it to make it better. We give it tools, the ability to write code, maybe some documentation, etc. so that it knows what it's supposed to be doing. And the results for some of these things have been very good. And here we're looking at an approach of if we can get these AI agents to get good at playing the Settlers of Catan. Here's kind of what that game board looks like. It's a fairly complicated game. There's a little bit of strategy, math, negotiations, etc. I actually haven't played it, but I've seen people play it. And it seems very interesting. Seems like there's a lot of depth to it. So this paper is called Agents of Change, Self-Evolving LLM Agents for Strategic Planning. So they're saying that recent advances in large language models have enabled their use as autonomous agents across a range of tasks. This is true. We're seeing more and more of that. But there's a problem. They continue to struggle with formulating and adhering to coherent long-term strategies. We often see papers where these AI agents are really good right out of the gate. Superhuman level, better than humans, faster. But over a long enough time period, their ability to kind of, you know, stick with the plot, it kind of falls apart. It's that idea of long-term coherence. It kind of falls apart. So this is a very interesting approach because they're trying to figure out if they're able to get these large language model agents to self-improve when placed in environments that explicitly challenge the strategic planning abilities, right? So that long-term coherence. Can we create a scaffolding where they get better with time as opposed to getting worse? And we're going to be playing Assetless of Catan using the open-source Catanatron framework, right? So some sort of an open-source way to play this game. I believe it's a Python-based open-source thing that allows you to simulate the game, to play the game, etc. Here's kind of what that looks like, or at least the terminal, right? So you're able to simulate these games. They run pretty quickly, it looks like. You got four players, it looks like, and it quickly runs it through the game. So you can have bots playing it. And so the idea here is to benchmark the progression of LLM-based agents from a simple game-playing agent to systems capable of autonomously rewriting their own prompts and the player's agent code. They introduce a multi-agent structure. So we got analyzer, researcher, coder, and player. And these work together to analyze gameplay, research new strategies, and modify the agent's logic or prompt. So interestingly, NVIDIA's Minecraft Voyager was similar to this. It was guided by, at the time, GBT-4. This was really early on, one of the more mind-blowing papers. Way before it was apparent where stuff was going, NVIDIA released this, their team of AI researchers. And it really was kind of ahead of its time, because now, a few years later now, we're getting a lot of the stuff that's basically similar to this framework, but more advanced, with more advanced models. What's interesting here is this was before the vision models, right? So this was when large language models, I mean, they were text only. There was not a visual component. They couldn't see images. So this whole thing, it was basically kind of like the matrix, where it could just like see the matrix, see the code. That's kind of what it was doing. It would work with something called Mine Flare, which allowed you to write little JavaScript code to interact with the game. It would only give you the abilities that a player would be able to do. It wasn't like a god mode or anything. But instead of having to click to mine something or chop down wood, it can create a little skill in JavaScript that would do that for you. So for example, here was the code to fight a zombie, right? And if it worked, it was tested in the game world. If it worked, then it would be added to the skill library. And this thing did really, really well, much better. And it didn't plateau. Notice here, it keeps going, keeps improving. So right off the bat, this seems very, very promising because we know that approaches like this can work. And they're saying by comparing manually crafted agents to those evolved entirely by LLMs, we evaluate how effectively these systems can diagnose failure and adapt over time. Our results show that self-evolving agents, particularly when powered by models like CLOD 3.7 and GPT-4.0, outperform static baselines by autonomously adopting their strategies, passing along sample behavior to game playing agents and demonstrate adaptive reasoning over multiple iterations. So this was similar to the Darwin-Godel machine where they compared it to, you know, there was some AI coding agents that were made by humans through human innovation. Somebody sat there and kept adding code to improve how well this thing was able to help with coding tasks. And some researchers tried to see if they took a baseline AI coding agent and they tasked it with just sort of improving itself with this evolutionary search. Each time it would, you know, add some modification, it would be tested against a coding benchmark. If it improved on the benchmark, that modification would be added to its code base. So it was like attempting to prove itself, testing if that piece of code would improve its abilities to code. And if it did, it would add it to its own sort of code base, improving itself. This is very similar, but with Settlers of Catan and also multiple sort of agents similar to NVIDIA's Voyager. So Settlers of Catan is a prime example of a multi-agent strategy game requiring players to plan resource management, expansion, and negotiations over many turns. And here's kind of the important thing. So traditional game AI methods, right? So things that were done with just reinforcement learning, not ones that were LLM based. Well, they have achieved superhuman performance in perfect information games such as chess and Go. So in chess and Go, you see everything there is to see. There's no chance. There's no probability. There's just the entire board and you're able to kind of observe where all the pieces are, et cetera. With Catan, you have dice rolls, right? So you have probability. You have partial observability. So you don't have all of the data at a glance. There's things that are hidden from you. You have to kind of guess at what's there. So this can be a little bit more difficult for kind of this reinforcement learning approach. It can be done, of course, but it's not as easy as these perfect information games like chess. So here we have four agent architectures. One is the base agent. So it maps an unstructured game state description directly to an action. We'll break that down what it means in a second. We have a structured agent that receives a representation of the game state, available actions, and a basic strategy and natural language for better parsing and guidance. So here's a prompt example, right? So you are playing Settlers of Catan, right? It gives you the rules, the board understanding guide. So it kind of spells out some ideas behind this game. And by, you know, the game state, the board state, this is what that looks like, right? So game status, longest road, largest army, you know, the various resources, et cetera. This, by the way, seems to be a big deal because when these papers provide something like this, they kind of keep reminding the agent with every prompt kind of its state, that seems to work really, really well. Minecraft Voyager did that and it never sort of lost track of what it was doing because it kept being reminded. Papers like Vending Bench, right? Where it was tasked with running a vending machine business. It broke down over time, but it didn't have the sort of reminder. It didn't keep telling the state of, you know, the game that it was playing. It didn't update it every turn. So over time, the large language models often just lost the plot. So this seems like a big deal. Like this is a key to making stuff like this work. Three, we have a prompt Evolver. So it's an Evolver agent and a player agent interact for up to 10 iterations to refine, test, and evaluate a prompt for an LLM to play Catan. Then we have the agent Evolver comprising of Evolver, Analyzer, Researcher, Coder, and Player roles, which can autonomously rewrite its game playing code between games. So again, this is very similar to Minecraft Voyager in that you have multiple agents kind of taking care of different parts of the game. One is sort of playing the game. One is trying to improve it. The other one's testing it. One is, you know, writing descriptions. It's, this is a proven approach. And I would guess that it works incredibly well. If you've ever thought about building something like an AI agent for business purposes or something else, I mean, these are really, it seems like these are the recipes for how to do it right. And, you know, also some papers show you how to do it wrong. This so far, it seems like it's doing everything right. And how they test these models as they compare it to Catanatron's strongest heuristic based bot in head-to-head games. So there's a logic based bot, kind of like a simple scripted bot, or here they're saying an alpha beta search AI, but it's not AI in a sense, like with large language models, kind of not like this, right? So it's a, I mean, I guess you could say like a more basic version of it. And we're tracking relevant metrics, such as average victory points, numbers of settlement and cities, largest army, and other development indicators. So with Catan, victory goes to the first player to reach 10 points, and you get points for settlements, cities, etc. Every turn starts with a dice roll that produces resources, and the active player may trade and build. If a seven is rolled, the robber is activated, blocking a tile and stealing a resource. Yikes. All right, so the kind of self-evolving part of this whole thing is the agent evolver. It starts with a blank template and over time evolves the ability. So this is the self-improving aspect. And so we have the evolver agent. This is the central coordinator. It reads the analyzer reports, right? So the analyzer agent evaluates the player's gameplay, identify weaknesses, summarize key areas for improvement, right? The evolver agent can read those reports. So this is almost like a tool for the evolver agent, right? So this one analyzes the game and hands it over to the evolver for the evolver to read and figure out what to do next. The research agent, that's interesting. So it handles specific queries about the Canatron and a broader Catan strategy, utilizing local file access and web search, right? So this is kind of like a deep search. And again, it provides its deep search results to the evolver agent. That's interesting. So they're combining not just one agent that analyzes the actual gameplay. There's one that can do deep research to maybe find new strategies to attempt. So it does a web search and, you know, it goes on Reddit. It's like, what's the best strategy for Catan? And then based on that, writes up a report, hands it to the evolver agent. This, I gotta say, is kind of brilliant. We have the strategizer. Suggests high-level gameplay strategies or critiques past strategic choices. The coder agent translates proposed changes into concrete code modifications. So every time it gets the existing code and, you know, writes, adds changes to it, et cetera, and then outputs a new version along with a summary of what it did. And then the player agent is the actual AI player that actually plays the game, right? So, and this is the thing that gets improved over time. So this data was collected on a MacBook Pro 2019 16 gigabytes and a MacBook M1 Max 2021 with 32 gigabytes over a total time of 60 hours. So this is fairly accessible if you wanted to run an experiment like this. This isn't out of reach, really. I mean, MacBooks aren't cheap, but they're also not tens of thousands of dollars. And the models that we're trying is GPT-40, CLOD 3.7, and Mistral Large Latest. So that's the open source large language model, Mistral. I would love to see this played. We'll see what the results are, but I wonder if you take the latest like O3 Pro. So obviously it's going to be a lot more expensive to run it because you're also, you have a lot of different agents running each one, you know, having an API bill, unless you're doing it locally with something open source. They're probably not. They're using Mistral AI API. I wonder why they wouldn't use a DeepSea car one. It's very, very cheap. So the baseline agents, they just play 10 full games against alpha beta. So that's that kind of more simple AI approach to playing this game. And random seeds are fixed for initial board state generations to ensure consistency across runs. So they're basically minimizing how variable the games will be between runs, trying to keep everything, you know, as consistent as possible. So it sounds like they're mainly for the initial board state generation, just so you're not completely rolling a brand new world every time. That could be very different. Like where you start could have a massive impact. Then the prompt evolver runs. So across 10 evolutions, and then the agent evolver is playing similarly to the prompt evolver. So one is trying to improve the prompt. One is trying to improve the actual agent. And finally, the results. How well do you think our LLM agents did? Did they beat the heuristic based alpha beta kind of the baseline? Were they able to improve? I'm assuming they started out worse, but over time, did they get better? That's the question. And here are the results. So agent performance across models, right? So GPT-4O, CLOD, and Mistral Large. So in all cases, the base agent is here, right around 3.6. So that's kind of the unevolved, unimproved agent. Structured agent improves 6% for GPT-4O, 11 for CLOD, 3.7. 31% drop for Mistral Large. I'm not sure why that is. So this seemed like this was the worst performer, the Mistral. CLOD 3.7 was the best performer. Go CLOD. Prompt Evolver, 22% increase. And for GPT-4O, 95% increase for CLOD 3.7. This seems huge, right? And then agent Evolver, 36% improvement for GPT-4O, 40% improvement for CLOD 3.7. So the point here is this system successfully improved agent performance by autonomously iterating on strategic prompts, although the degree of success heavily depended on the model used. Keep in mind, we have probably better performing models now, or, you know, if you're willing to pay more. CLOD 3.7 exhibited the most significant strategic advancements, systematically developing detailed strategic prompts that outline clear short-term and long-term plans, including precise settlement placement, resource prioritization, development card usage, and robust response strategies against opponent actions. So CLOD 3.7 did great, right? Up to a 95% improvement over the base. GPT-4 was mid, and Mistral. Large was the least effective. The biggest limitation is the underlying LLM, right? So the better the LLM, the better the outcome is. And this is kind of that whole thing when Sam Altman was talking, like if you're building a startup, you don't want to be trying to fix something that's currently wrong with AI. You want to have some sort of a system where the better the models get, the better the performance of your entire business. And it's the same thing with papers like this, with systems like this, right? So imagine the improvements we're seeing now with these models. Now, fast forward a few years, imagine kind of those next generation models. If you just plug those in and keep everything else the same, it's likely that those improvements that we've seen, right, in 95% improvement, we're probably going to see even better ones if we just improve, you know, the model, the underlying model. That's an important point to grok, so to speak, right? These models will get better at self-improvement just, you know, as they're improving, as they're growing, as they're getting better, they will also get better at self-improvement or improving systems like these autonomously. One of the things is that they've only did 10 steps of evolution. And as you can see here, you know, especially with Cloud 3.7, our best model, right, it keeps improving. Even, let's see, what is this? This is, you know, evolution 7. There's still improvements here and here. So it plateaus after 7, although it might get better, you know, if you keep going and if you project a 20, 30 evolutions forward. But as you can see, it's not like all of the improvements are early on and then just nothing. Even towards the end of the evolutionary steps, it keeps improving. I wonder how well it would do if we allowed it more time, more evolutionary steps. Like, is it just going to plateau from 7 to, you know, 100? Or is there going to be, you know, some modest improvements if you just keep going? Obviously, the researchers likely had limitations in terms of funding, right? I don't know how much this would cost, but it could be somewhat expensive. So absolutely phenomenal work. I love this project. I wonder if it's open source somewhere because I'd love to play it. Now, the Cataran, I forgot what's called Catanatron. That is open source. That's on GitHub. So you can download it and use it. You can probably attach your own API, your own large language model to it. But this is yet another example of what we probably would call an AI agent that's able to recursively self-improve. And we're seeing more and more of these. So kind of like the recipe for building these things. We're seeing more examples of it. We're beginning to understand what works, what doesn't. Really exciting time to be alive. And I love the fact that they're using games to demonstrate some of these things. And I can't wait to see where this goes. Thank you so much for watching.
