Open transcript
So today, OpenAI introduced Codex. This is not to be confused with the 10 other things that they have named Codex. And Codex is powered by Codex 1. Now, Sam promised that he would name it better this time, just in case it takes off. At this point, it's clear he's just trolling. But the direction of the project, kind of where this is going, is very, very exciting. Here's why. So first and foremost, Google I.O. is in a couple of days, and they're planning to unveil their very own software AI agent. So this is, of course, OpenAI in their own fashion, as they have a habit of doing, trying to take the wind out of Google's sails by rolling out their thing before Google has a chance to announce it at the conference. And Google is kind of going after developing a software development lifecycle agent. So something that kind of from the beginning to end helps software developers build whatever they're building. This thing kind of helps them along every step of the way. Of course, we saw Cursor valued at $9 billion. Windsurf got purchased by OpenAI for $3 billion. Google is rapidly building their Firebase Studio, which is another very interesting project. The whole point is to get as many developers kind of building stuff on your platform using your AI chatbot, your AI models, so that these companies are able to kind of see the entire sort of life cycle, right? Because if you're asking for some code in chatgpt, then you're copying, pasting that code into VS Code or PyCharm or whatever else you're doing. So, you know, whether that's Google or OpenAI, they don't kind of get to see that. They don't see where, whether the code works, what issues it's running into. By keeping that entire software development process within one thing, it kind of allows them to collect a lot more data and better able to create better code and better models that are able to take care of, as Google puts it, the entire life cycle of the development. The subreddit chatgpt had an AMA, ask me anything, with the OpenAI Codex team. So tons of goodies here. I managed to sneak in a few questions and we'll take a look at the answers in just a second. But I kind of thought this was hilarious where this girl here kind of says that, you know, how she trusts the codex more than she does her coworkers when it comes to, you know, code. And everybody kind of nervously laughs. But as this person puts it, that's how AGI arrived and not with a bang, but with a stunned pause. In Codex to be like as trustworthy, if not more trustworthy than our own coworkers. I don't have, I don't have this kind of access to like what Andre did on any given day in terms of like the logs or the actual test outputs. And I think as we move towards this world where AI writes more and more code, this kind of verifiability is going to be really important. So now in order to fully sort of grok what everybody's trying to build, whether that's Google or OpenAI or Anthropic, we can go over the benchmarks and the tests. But I think really to truly understand what's happening and where this thing will lead, I think one of the most interesting kind of use cases for this is on the YouTube channel Sentdex. I feel like a lot of you probably know this channel, so I'll skip the introduction, but he basically got a Unitry G1 EDU robot, the humanoid robot. Here it is. He's kind of like getting it piece by piece out of the box that it got shipped with. And here it is being kind of suspended on a gantry. I'm being informed that this is called the gantry. Thank you to everyone that joined me for the live stream earlier. That was a lot of fun. One thing that I can't help but notice there's like a million swords here on the wall, right? Katanas and various other swords. So in case this robot goes rogue, the newest weapon is not that far away. But I digress. The interesting thing here is that the Unitry, the kind of the code base for G1, all the documentations, Unitry provides a lot of it open source. But there's one issue. A lot of it is in C++, which is kind of a notoriously difficult, demanding language, especially if you're sort of not familiar with it, obviously. So I believe Sendex, what he was saying is he doesn't have too much time invested in learning C++, at least not as much as he has with some of the other languages. So what does he do? Because he wants to be able to control the robot. He wants to be able to teach it new skills, to interact with it. Now, just a few years ago, your choices would have been to either learn that language from scratch or just kind of abandon the project. But what he does is he actually uses codecs. So he uses the local installation of codecs. That's what you see here running off of O3, the O3 OpenAI's model. And the codecs acts as almost like an operating system for him to interact with the code base. He can ask questions and have things explained to him. He can add various things. He can even program in other languages to almost build like a layer on top of it to be able to interact with a language that he prefers to write in, to code in. So I encourage people to watch the video because I might be getting some details wrong, but basically you can see here, right? So this is the O3 model and you can say things like explain this code base to me or check for bugs, etc. So as an example, there's this one function where you're trying to get the robot to walk that wasn't working. So notice here, the user, right? He says, okay, you think R&D walk should work now? Codecs, the O3 model, things for six seconds and then answers. It should. This last change that it's implemented should now work for this continuous gate. There is some command that was missing before that it added. And here's what happens after you press enter. Like this is, it explains to you step-by-step what's going to happen. So again, it read the manual, it read the code base and figured out what it meant. And now it's like walking you through step-by-step with infinite patience and being able to break it down on any level that you want. It's going to walk you through it, explain to you what's working, what's not working, why it wasn't working before, and what fixes it implemented on your behalf to make sure that it works. That says, give a try if it still doesn't walk. And it gives you some probably more steps to continue troubleshooting in case this approach didn't work. And here in this next video, you can see the thing walking around the house. So the top left camera, that's the, uh, it's vision. So here's a camera pointed out so you can see where it is. And I'm guessing this is some sort of a radar, LIDAR that sort of kind of maps the house so they can better understand where it is in the house, uh, some sort of a depth map here in the center. And if I zoom over here, you can see the, what I assume to be, you know, codecs running in the background, kind of giving you a printout of what's happening. So I hope you understand what I'm saying here. This is when they say that AI will be an operating system. This is what they're talking about because before you sort of, you, you had the code base and you had to interact with it by typing in the code and clicking on things and knowing how to do all of the things yourself. And now more and more the codecs or whatever other program that you're using is almost like a little layer between you and the computer, the code base that interacts with it, adds functionality, does the coding, explains stuff to you. You're interacting with it in English, in the natural language, whatever language we want you want to use. And then its job is to do the coding, the checking, the unit tests, et cetera, et cetera, et cetera. What can you do with it? Tons of stuff. You can ask it to kind of refactor your code. So if it's getting too clunky, complicated, you can have it break it apart, kind of split up the functions, isolate certain functionalities. It can help you walk through the database to kind of get a deep understanding of what it's doing. It can look for security vulnerabilities. It can do code review. It can add various like unit tests to tests, to test the functionality of the code to make sure there's no issues, bug fixing, product and UI fixes, et cetera. And what's really interesting about this sort of iteration of codecs, because again, we had it sort of a version of it that we can run locally. I think I've had a few videos where I showcased some of the stuff that it can do. It's very good, similar to cloud code. So it's able to run commands through the terminal, install whatever packages that you want. So you can kind of clone a GitHub repo, ask it to explain it to you, kind of have it get familiarized with all the stuff in there. And then you can start changing code, adding whatever stuff you want, checking for bugs, whatever. One interesting thing is that it's pretty good at, let's say there's some GitHub project that you want to install, but there's tons of complicated dependencies. I found both codecs and cloud code to be fairly good at installing all of that stuff and figuring out how to troubleshoot the issues that will invariably come up. So to give you an example, there's this factorial learning environment. So there's, you know, the installation isn't rocket science, but it's not easy, right? It's not the easiest thing to install. I think that's fair to say. I tested this with cloud code. This was before codecs was out, but I tested with cloud code and cloud code was, I would say 90% able to set everything up so that I was able to run it. So it's not quite one shotting it. And it did take like 20 or 30 minutes and quite a bit of credits that you have to pay for it. That's true. But I would say it got me 90% of the way there. One very annoying thing about it is you have to kind of sit there as it sort of like just thinks through, thinks through stuff and does stuff. And it's a little bit frustrating because you can't really go away from the keyboard, you know, AFK and do something else. Or, you know, it would be great to be able to not be tied down to whatever local machine that you're using. And this is where kind of this new wave of stuff is a lot more interesting. So with Google's Firebase Studio, that runs in the cloud, that runs in your browser. So you can connect remotely. It can be running on its own while you can be somewhere else doing your own thing. You don't have to sit there in front of the keyboard, kind of babysitting it, waiting for when it stops and interacting with it, etc. Now, I think it's pretty obvious kind of where this is going. The future of this is going to be, you know, your AI agent, some sort of like, think of, think of chat GPT in voice mode, advanced voice mode. That's talking into your ear, you know, you're going about your business, going to the store and goes, oh, you know, your AI software developer agent, you know, finished the last thing you asked it. So it installed all of this stuff. Like, what do you want to do next? You can just verbally say, you know, okay, continue doing this next function. It goes, okay, I'll notify you when it's done. And that thing keeps running while you're going around doing your business. Because again, it takes time for it to, you know, physically download the packages and install it and check it and write the code. That takes time. And if you're in front of the computer, like waiting for it, you can't walk away because you don't know when that's going to end. That can be frustrating. If you're texting back and forth with it, or just able to chat with it, if you're able to interact with it remotely somehow, I mean, that's the next step. It seems pretty obvious. That's where things are going. Obviously getting it into the cloud, like what they're doing with Codex and Google's Firebase Studio, that's kind of the next intermediary step towards that future. That's my guess, but that's kind of where I see things going. So I was joking earlier about like how many things are named Codex. So we have the Codex CLI. So that's a lightweight coding agent that runs in your terminal. So that thing that was trying to install that factorial, well, that was cloud code, but Codex is very, very similar. And it's open source. It's local. Once you install, you open it on your, I'm on Windows, but whatever you're running it on, run Codex, and it looks something like this, right? So this is the OpenAI Codex Research Preview, the version running on 04 mini, but you're able to change it to the 03 or whatever you want. And you're just chatting with it through this window. Now, by default, if it tries to install something at first, it's going to suggest that you have to approve it. There's also ways of just putting it on full auto so it just can go wild and kind of do whatever it wants. And this thing will ask you like, are you sure it's dangerous? Don't, don't do this, but you are able to if you so choose. And then this is Codex. This is the thing that got released today. So this is the thing that runs in your browser. So I'm at chatgpt.com slash codex slash onboarding. And so I haven't set it up yet. This is kind of the onboarding. You basically, if I understand correctly, you've got to attach it to your GitHub. And it seems like it does require multi-factor authentication, but I'll be testing this very, very soon. So come back and check out my testing video because again, this I think is the future of how these things will be operated. You know, I was very impressed with the Manus AI because at its core, it's an open source operating system. In that case, like a Linux distribution called Ubuntu. And so it's, it's sitting on its own kind of virtual machine where it can install whatever it wants. It can search the web. Like it can do whatever pretty much a human being that knows how to operate that computer can do. And so it's sitting on its own computer doing the research or writing the code or whatever you, you ask it to do, and then just telling you the output. And that to me seems like the future at, at some point, we're not going to be actually interacting with the operating system more and more. We'll be operating with something that resembles a chat bot that does a lot of the stuff for us. But let's take a look at how good this thing is compared to just running the AI model through a chat bot. So again, codex CLI, that's the local thing. Codex is what they're referring to as this thing that we just looked at, where you can delegate tasks to a software engineering agent in the cloud. And codex takes on many tasks in parallel, right? So you can just tell it to do 10 things. It starts running them in the cloud. You don't have to sit there. You don't have to do it in sequence and say, continue, continue, or whatever. All right. So you can write features, answer questions about the code base, run tests, and it can actually commit code, but they're probably going to have some sort of a warning. Like it's not going to do it without your explicit permission. Like I said, I haven't had a chance to play around with it quite yet, but in some of the documentations, they're pretty clear about that. It's not just going to do whatever it wants. You have to approve it. So we'll see how that works. One feature that's new as far as I can tell that some of the other things haven't used is there's two separate ways of interacting with it. There's code and there's ask. So each one is processed independently in a separate isolated environment, which probably is so that if you just wanted to ask about the code base and have zero chance of it messing with anything, then you just ask, you don't code. And once codex completes the task, it commits its changes in its environment. So it kind of does everything, let's say like on its own computer, so to speak. And then you're able to, if you so choose, you can review the results, request revisions, or you can go ahead and just integrate that code wherever you want to, your local environment, for example, and you can configure it so that it's as closely matching your environment as possible. So however, whatever you're using to develop the code, it can kind of try to match that environment as closely as possible. Codex is guided by the agents.md file. And, you know, it works better if you have configured dev environments, reliable testing setups, and clear documentation. And here's kind of where it's sitting in terms of the accuracy on OpenAI's internal software engineering tasks, right? So 04 mini high at 67, 03 high at 70, and Codex 1 at 75. So there's a very interesting paper that we covered recently, the absolute zero reasoner. So the idea is we're using two models. One is the proposer and one is the solver to kind of almost train itself. So it's kind of this idea of self-play. And at the Sequoia Capital AI Summit, one of the team members from OpenAI was talking about kind of the next big wave of scaling, which is going to be the reinforcement learning compute. So putting more and more compute towards reinforcement learning. So with this absolute zero, they're calling it reinforced self-play reasoning with zero data. So moving away from human data or human label data and having the models themselves kind of generate synthetic data and get trained on it. So similar to what we've seen with from AlphaGo to AlphaZero, the DeepSeq R1-0 kind of had that same, some of the same ideas. So basically, I've been just harassing everybody at OpenAI. They get asked questions of like, are they doing anything with the ideas in this absolute zero paper? So one of the researchers at OpenAI that worked on Codex, this is them. So they are aware of the paper. They're probably incorporating it somehow. So it's not a clear answer, but it sounds like they're excited about these potential approaches. And in the different question that I asked, another member of the OpenAI Codex team did point me to this idea. They said that we have some longer term research bets, like multiple agents working together to watch out for. So, and that link was to Noam Brown, who's at OpenAI, ex-Meta. And this was actually from September 19, 2024, but they're saying they're hiring machine learning engineers for a new multi-agent research team. We view multi-agent as a path to even better AI reasoning. So Noam Brown was the Cicero diplomacy AI of Meta, if you remember that. That was a fascinating research project. He worked on some sort of a superhuman poker AIs as well as OpenAI's O1. So it seems like he's working on some sort of multi-agent self-play or however they're sort of referring to that. So I think the kind of the big point here is that, and take this with a grain of salt, because we don't know where this is going, right? We're kind of like just connecting the dots. Obviously, a lot of people that are working on this, the researchers, they can't share a lot of information. So by the time we see something come out, it's usually pretty late. It's after they've already discovered it. Or I should say, it might be a long time after they discovered it. So like one of the things that I found with Alpha Evolved that was so fascinating is the results, which were incredibly exciting and kind of seems like a pretty big step forward. Well, those findings were from running that model on Gemini 2.0. So not the latest generation of models. So for example, the public has access to Gemini 2.5 Pro. And I think it's very likely that at the Google I.O. we'll see kind of some next level model being announced. That's, I'm just guessing, but maybe Gemini 2.5 Ultra or maybe something else. And the results from Alpha Evolved were implemented in the various Google infrastructure projects over a year ago. So, you know, we don't know what's coming down the road, but, you know, if we were to guess, it seems like both Google and OpenAI and the other companies, they're putting together all the pieces that they needed to create these superhuman coding agents. So one of the pieces for OpenAI, they talked to Cursor. They attempted to buy Cursor. I guess that didn't happen for one reason or another. They ended up purchasing Windsor for $3 billion. And the point of that in part was to create this flywheel of users using the platform, you know, working with code and improving code that gives data to OpenAI about how it's being used, what's working, what's not. So that allows them to improve the model further. And that, of course, hopefully continues to improve the usability for users, improves the models. And it's kind of like this flywheel effect, right? Which again, doesn't really exist if you're just, you know, chatting with the LLM through the chatbot interface and then copying and pasting that code somewhere else. Because then it's not, it's sort of not the lifecycle software development environment. It's kind of like you're just like pulling out bits and pieces, but it's not integrated throughout the entire flow. So both OpenAI and Google wants to capture that entire flow from beginning to end. And so things like Cursor, things like Codex, things like Windsor and Google's Firebase Studio and all that stuff, that's an attempt to kind of build that end-to-end platform. So I think that's one part of it. The second part of it is having that kind of things that run in the cloud as in parallel, as they say, right? So you can run many tasks in parallel, right? So again, this kind of idea that if you kind of zoom out, you might have, it might be like playing a game of Factorio where you kind of zoom out and you have all these little agents running around doing your things. You don't have to sit there and babysit each one. There can be 100, 200 of them running in parallel, doing all the little tasks that need to be done. Sam, I'll never post it. This will depuse saying, I do think the future of work is like StarCraft or Age of Empires. You have 200 micro agents you're directing to fix problems, gather information, reach out to people, design new systems, etc. By the way, it also seems obvious that the deep research functionality kind of gets built into this, right? Because some of us, I'm sure, have tried, you know, sometimes when the project is more complicated, you start by doing deep research. You're having the model do deep research for how to build a particular software or something, something. And then you're taking the research that it does and you're posting it into the thing that's going to code it up for you. And this might be the same model, right? The O3 or whatever, or you're doing a deep research with Google. You're taking that, although I haven't tried that. I don't know how that works, but I've done deep research through OpenAI for, you know, how to code something up. And then you're asking O3 to code it up, right? So, but I think that's another sort of piece that they're trying to build and implement to have there, right? This idea of being able to run a lot of stuff in parallel that doesn't depend on you being in front of your computer in which it's running. And I'm pretty sure that a lot of them are also working on something like this, the idea of doing reinforcement learning, of scaling it up, of doing self-play. The absolute zero paper, definitely, it seems like there's a lot of potential there. One of the interesting findings there that when these LLMs are trained to solve coding tasks, for example, they also get better at math, right? So even though they're not doing any math-specific tasks, just the exercise of going through and solving coding tasks generalizes to other areas like math. This is a presentation done at the Sequoia Capitol. This is Dan Roberts from OpenAI kind of saying that we used to have trained time compute, so how much hardware resources we're throwing to train these models. Later, we had our test time compute, so how much hardware resources we're throwing at these models to help them think, right? So after they're trained, when they're answering the question, we're like, hey, think harder about this. And both those, as you can see, help improve the accuracy on various benchmarks. And one point during his talk that really jumped out at me is he's saying, so before this white circle, well, that was the pre-training compute, so how much hardware resources we're throwing at training the models. And the red was the reinforcement learning compute, right? So us teaching it to do all the tricks that we wanted to do, as you can see, it's much smaller. What might the future look like? Well, it might look like we're scaling RL, we're scaling reinforcement learning. And now, as you can see, reinforcement learning compute dwarves how much sort of resources we've spent on pre-training. Why do they believe that? Well, we don't know, but I think a good guess is that some of these ideas in Absolute Zero Reasoner and Self-Play and all that stuff, the multi-agent learning, all that stuff will allow us to scale reinforcement learning. And again, so my question here was, how are they thinking about solving kind of the long-term coherence of the AI agents? Because they seem to do really well right out of the gate, but kind of tend to break down over time. And so this person that answered, he's on that Codex team, right? That's Hanson Wang, the researcher on the OpenAI Codex team. So they're saying, yes, we are sort of working on direction. There's some long-term research bets. The fact that this was posted in September of last year, where, what, six, seven months later, they're referring to this as like, it's an ongoing research. I don't know. To me, seems like that's exactly kind of the direction that they're going in. There was this expression, like, if you want to know what the future looks like, kind of like where technology would take us, what you have to look at is what are all the tech nerds doing with their free time on the weekends? I think what Centex is doing certainly qualifies for that. And of course, I mean nerds, as really as all of us, you are included. And of course, I mean that in the best possible way, the term of endearment. But what you're seeing here is that you're using something like the Codex, which by the way, it seems to be like that's going to keep getting better. And there's tons of money and research being thrown at making sure that it gets really, really, really good. But we're using these open source software to train these humanoid robots to perform various tasks in our own households. This is nowhere near mainstream yet. This is as niche as it gets, I think. But, you know, if you think forward a year or two years, again, we're seeing somebody working with C++ without knowing the language by using an AI assistant like Codex in this case to interact with the code base. A lot of the NVIDIA stuff, all their kind of training robots and simulation, a lot of that is available. It's open source. I think it's very likely that let's say two years from now, you're going to have kids that are going to be able to take their household robot that's going to be a lot more available, hopefully by that point, and just train them to do some task in a perfectly simulated 3D rendition of their home. You might have seen Meta's open source partner program where you're able to kind of create these 3D scenes of various environments of homes. And then something like this gets trained to go around and pick up various objects. And it's kind of like a partner. So it kind of collaborates with you to do cleanup or whatever you want it to do. So I think it's very realistic that within two years, maybe longer, this is going to be a lot more accessible. A lot more people are going to be doing stuff like this. Custom trained robots to do the dishes and the laundry and the cleanup and the garden work. As I'm saying it, can you imagine having this thing go walk your dog around the block? I feel like I have to do that just for the lulls. I feel like seeing that for the first time will blow somebody's mind. But let me know what you think. What do you think about Codex? What do you think about kind of the direction that it seems like everything is going in? Leave me a comment. Make sure you're subscribed. If you made it this far, thank you so much for watching. My name is Wes Roth and I'll see you next time.
