Airdroplet AI summary

Claude 4 is not what you think...

May 22, 2025Matthew BermanAI score 9885,726 views

AI-generated summary

Here's a summary of the video about Anthropic's Claude 4 models:

Anthropic has launched Claude 4, available in two versions: Opus and Sonnet. The big news is that Anthropic seems to be making a strategic pivot, stepping back from the general chatbot competition dominated by players like OpenAI and Google. Instead, they are focusing heavily on becoming a leader in building sophisticated AI agents, particularly for complex tasks and coding, introducing new features and API capabilities to support this direction.

Here are the key topics and details discussed:

Anthropic has released Claude 4, available in two sizes: Opus and Sonnet.
Opus is being claimed as the "world's best coding model," which hints strongly at Anthropic's new focus.
A major emphasis is placed on the models' ability to handle "long horizon tasks," meaning complex jobs that can take tens of minutes or even hours to complete without losing context or failing.
Both Claude 4 models are "hybrid models," offering the option of near-instant responses or an "extended thinking" mode for more complex problems.
The extended thinking mode incorporates tool use, which is considered a standard but necessary feature today.
Available tools currently include web search, Google Drive search, Gmail search, and Calendar search. Anthropic has also integrated their MCP framework more deeply into the API for connecting other tools.
A unique and cool feature is that both Opus and Sonnet can use tools in parallel, sending multiple requests simultaneously for potentially much greater efficiency than sequential processing.
The models are reportedly much better at managing their internal memory, crucial for handling complex, multi-step tasks.
Anthropic's new API includes four key features aimed at enabling agent development: a code execution tool (seems to be Python initially), an MCP connector to link external tools, a Files API to easily give Claude access to local code files and repositories, and prompt caching for up to an hour, which is noted as the way to achieve the most efficient and cheapest usage.
The presenter hit his rate limit quickly with only a few prompts, indicating he'll need to subscribe to a higher tier for more thorough testing.
The core strategic shift is that Anthropic has "given up on the chatbot race," believing OpenAI, Google, and Microsoft have won that battle.
Anthropic is instead repositioning itself as an "infrastructure company," providing the tools and models necessary to build the best coding and general agents. The presenter feels this focus is "required to win" in a specific niche.
Claude 4 Sonnet is already available in GitHub Copilot and is their default option, which the presenter calls "incredible."
Early evaluations in GitHub Copilot show Sonnet 4 delivering up to a 10% improvement over the previous generation in "agentic scenarios," driven by sharper tool use, better instruction following, and stronger coding instincts.
Claude 4 is also available in other major coding platforms like Cursor and Windsurf.
The video sponsor, Box AI, is highlighted as a good partner for Claude 4 due to its ability to handle long horizon tasks and memory.
Box AI helps extract metadata from documents (contracts, invoices, etc.), automate workflows, and allows users to ask questions and do deep dives into their company's data.
For developers, Box AI handles the complex RAG (Retrieval Augmented Generation) pipeline automatically, so you don't need to manage things like vector databases or chunking. It also offers enterprise-level security and compliance.
Using Claude Code with Box SDKs is easy, as Claude can process the Box developer documentation to understand how to build with it. A demo was shown of building a backend contract generation tool using Box, DocGen, and Claude Code.
Benchmark results are presented but should be taken "with a grain of salt." On SWE bench verified (software engineering), Opus 4 and Sonnet 4 show significant improvement, with Sonnet 4 scoring slightly higher than Opus 4 with parallel compute.
The presenter's initial, anecdotal usage found Opus to be faster than Sonnet at outputting code, despite the benchmark showing Sonnet slightly ahead on SWE bench. He plans to test this further.
On TerminalBench, Opus 4 wins. Gemini 1.5 Pro is mentioned as the presenter's favorite coding model to date before testing Claude 4 extensively.
Other benchmarks (GPQA Diamond, Multilingual Q&A, High School Math) also show improvements, especially with agentic tool use. Visual reasoning scores were similar to the previous model.
A surprising point is raised based on analysis by John Shoneth: comparing Sonnet 4 to Sonnet 3.7 on Anthropic's own provided benchmarks, about half actually showed a decrease in performance, while others improved or stayed the same. The presenter finds this "kind of nuts" and "very interesting," noting benchmarks are often the most favorable view before real-world usage ("vibe checks").
Anthropic claims they've addressed previous issues where models were either too "lazy" or "tried too hard" with coding, believing they've now "dialed it in."
Safety is a focus: Claude 4 models are significantly less likely (65%) to use shortcuts or loopholes compared to Sonnet 3.7 on tasks susceptible to such behavior.
Improved memory capabilities are highlighted again, stating Opus 4 dramatically outperforms previous models. Memory for agents is seen as the "key ingredient" to making them "hyper-personal."
Claude 4 is designed to get better the more you use it, learning and developing a shorthand with the user through creating and maintaining memory files for better long-term awareness and coherence.
Thinking processes are now summarized by a smaller model, but the raw, detailed "chains of thought" are not publicly available unless you contact sales, implying you'll likely have to "pay up" to access them.
Claude Code is now generally available with new extensions for VS Code and JetBrains for direct IDE integration, allowing inline edits.
An SDK for Claude Code is also being released, further supporting the infrastructure/agent building focus.
An example shows how Claude Code can be tagged in a GitHub pull request comment to automatically address reviewer feedback, fix errors, and prepare the PR for review, which the presenter thinks is very cool.
Pricing for Claude 4 Opus is provided: $15 per million input tokens and $75 per million output tokens, with a 200k context window and a 50% discount for batch processing. (Sonnet pricing was not mentioned in the transcript's pricing section).
The presenter plans to conduct thorough testing and release a video soon.

Video transcript

Open transcript

Claude 4 is finally here. It comes in two sizes, Sonnet and Opus, and it seems Anthropic has pivoted in a completely new direction. I'll explain that in a moment. Let me give you all of the details. Right away, they claim Claude 4 Opus is the world's best coding model, which is a hint in the direction that they are heading. And what seems to make it really special is its ability to complete long horizon tasks. That is tasks over tens of minutes up to hours without losing the thread and actually being able to complete real world tasks. All right, so a few details about both of these models, and then I'm going to get into the benchmarks. First, you have extended thinking with both of them, and they are both hybrid models, which means they can give you instant responses with no thinking, or you can turn on thinking for those more complex tasks. And during the thinking, you have tool use, which is of course really nice, but kind of table stakes at this point. And now I've already been playing around with it and hit my rate limit until 2pm today, which is a few hours away. And really, I only submitted a few prompts. So I think I'm going to have to subscribe to Max and put together a thorough test for you all. So you can see right here, we have Claude 4 Opus, Claude 4 Sonnet. If you click right here on search and tools, you can see the different tools available. You can select the style, you can turn on and off extended thinking. It has web search, drive search, Gmail search, and calendar search. Those are the available tools for now, but they have more deeply integrated the MCP framework into their API. And remember, Anthropik is the company that created the MCP framework that now OpenAI, Microsoft, Google, and so many other companies have adopted. One unique thing that I really haven't seen elsewhere is that both models can use tools in parallel, which means it can send off requests to multiple tools at the same time. That seems really cool and much more efficient than doing everything sequentially. And it also seems to be much better at handling its own memory. All of this stuff is available in Claude code, which is also now generally available and has the Claude 4 models available. During the keynote that live streamed this morning, the chief product officer of Anthropik spent a lot of time talking about long horizon tasks and how they were able to accomplish this. Even giving an example of a company that was using Claude 4 that was able to do a task over seven hours. And as part of Claude's new API, they have four new features, including code execution tool, MCP connector, a files API, and the ability to cache prompts for up to one hour. Here's what the code execution tool looks like. You simply type in a prompt, Claude will start thinking, and write code, and of course, execute that code. And I believe it needs to be Python for it to execute. The MCP connector allows you to connect any MCP server to the Claude API. So now your Claude API has access to all of the MCP tools throughout the world. They also have the files API. So giving access to Claude to your local files, specifically your code files, your repositories just became a lot easier. And then prompt caching. So of course, you want to get the most efficient usage, you want to get the cheapest price and caching is the way to go. Now with all of these, you probably can guess where this is going. Claude has basically given up on the chatbots race. It is clear that OpenAI and the major tech companies, Google, Microsoft, and unfortunately not Apple have all won the chatbot race, the personal assistant race. So now Anthropic has transitioned into being an infrastructure company. They are providing the tools necessary to have the best coding agent. They are building the best agents. They're building the best coding agents and they are plugging it into everyone. Thomas Domke, the CEO of GitHub announced Claude Force Onnit is here. So it's available in GitHub Copilot and it's their default option. By the way, I interviewed Thomas at Microsoft Build. I'll drop that interview soon. So be sure to subscribe to this channel so you can get updated when that video drops. It is incredible. But look at this. In early evaluations, the model soared in agentic scenarios. That's the key. That is what we keep hearing. Memory, tools, long horizon tasks, all done by these agents, powered by Claude 4, delivering up to a 10% improvement over the previous generation, driven by sharper tool use, tighter instruction following and stronger coding instincts. And of course, it's also available in Cursor and Windsurf and basically all of the major coding platforms out there. Now that Claude 4 is especially good at long horizon tasks, has excellent memory, built in parallel tool usage. It's going to be especially good at pairing with Box AI. And that's the sponsor of today's video. I'm really excited to tell you about them. You're going to be able to build on Box AI using the new Claude 4 models soon. With Box AI, you can use artificial intelligence to extract key metadata fields from contracts, invoices, financial documents, resumes, and more. And you can automate workflows super easily. And not just metadata. You can ask questions about it. You can really do deep dives into your company's own data. And again, if you're a developer building on Box AI is easy. It handles the entire RAG pipeline for you. So you don't need to think about vector databases. You don't need to think about chunking. It's just done and it works. And of course, because it's Box, they have enterprise level security, governance, and compliance. And with the launch of Claude Code, if you want to use Claude Code with Box SDKs, it could not be easier. Simply give Claude Code links to the Box developer docs, and it just knows how to build with it. Check out Box's blog post about the Claude Code launch to see a demo of them building a backend contract generation tool using Box, DocGen, and Claude Code. I'll drop all of the links in the description below. So unlock the power of your documents and data with Box and Box AI. Thanks again to Box for sponsoring this video. All right, so back to the announcement blog post. Claude Opus 4 and Sonnet 4, by the way, they kind of switched the name, right? It was Claude 3.5 Opus, Claude 3.5 Sonnet, and now it's the opposite way. Claude Opus 4 and Sonnet 4. Anyways, are hybrid models offering two modes, near instant responses, and extended thinking for deeper reasoning? All right, I know you want to see the benchmarks. Benchmarks only mean so much, so take it with a grain of salt, but here they are. So software engineering, SWE bench verified, yep, Claude 4 is the by far winner. So here's OpenAI Codex 1, which was just announced about a week ago at 72% on the SWE bench verified, compared to Sonnet 3.7, which was at 62.3%, and with parallel test time compute, 70.3, but now we have a big jump all the way up to 80.2 with parallel test time compute for Sonnet 4, and 72.5 and 79.4 with parallel test time compute for Opus 4. And by the way, for those of you who weren't sure what parallel test time compute is, it basically just means they sampled a few test time compute solutions to a prompt and chose the best one. Now, if you're looking at this, you're probably thinking the same thing I am. Did Sonnet just score better than Opus? Well, yeah, it did. And with my initial usage, I actually found Opus to be faster than Sonnet. Now, that's just anecdotal, me using it a couple of times, so I'm going to need to test it a lot more, but it does seem to output code much faster. Now, here are some more benchmarks. Here's TerminalBench, Claude Opus 4 winning at 43.2% compared to Sonnet 4 35%. Here is the O3 model at 30%. GPT 4.1 at 30%. Gemini 2.5 Pro at 25%, which to date, Gemini 2.5 Pro is still my favorite coding model. Here's GPQA Diamond, which is graduate level reasoning. We have agentic tool use doing quite well compared to the other models. Now, you're probably noticing one other thing. Sonnet 3.7 is still doing quite well. I'm going to show you that in a second. We have multilingual Q&A, again, getting a nice bump. Visual reasoning, getting about the same score. And then high school math competition, Amy 2025, getting a very nice bump over Claude 3.7. Now, I'm going to pause for a second and show you something. This is a post by John Shoneth, and he actually points out the green boxes are around benchmarks, which Claude Sonnet 4 did better than Claude Sonnet 3.7. The yellow ones are where it did about the same, and red is where it actually got a decrease in performance, which is kind of nuts. So of all of these benchmarks that they submitted, half actually went down. So I don't really know what to think about that. They're saying it was a huge bump, but the benchmarks don't actually reflect that. And the benchmarks tend to be the nicest view of these models until people start doing the vibe checks of them. So very interesting. And of course, I'm going to be testing it thoroughly. We'll see. Now, one other thing that they called out during the keynote today is that when Claude 3 came out, it was kind of lazy with coding. And then Claude 3.5 and 3.7 kind of went the other way. It tried too hard and did things it shouldn't and outputted way too much code. And they think they really dialed it in with Claude 4. They also, being anthropic, focused a lot on safety. So we've significantly reduced behavior where the models use shortcuts or loopholes to complete tasks. And of course, they're using the Pokemon example here. Both models are 65% less likely to engage in this behavior than Sonic 3.7 on agentic tasks that are particularly susceptible to shortcuts or loopholes. Claude Opus 4 also dramatically outperforms all previous models on memory capabilities, which I've mentioned already. But I have said memory for agents is really the key ingredient to making them hyper-personal. And they called out in the keynote today, the hundredth time you use Claude 4 should be much better, much more efficient, and much more concise than the first time you use Claude 4. That's because it's learning and it's understanding what you want. It's developing a shorthand with you as the user. Opus 4 becomes skilled at creating and maintaining memory files to store key information. This unlocks better long-term task awareness, coherence, and performance on agent tasks. And here's the example of the Pokemon benchmark. They've also introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. I would love to see the thought process, but you basically see nothing now. Now here's the key. Users requiring raw chains of thought for advanced prompt engineering can contact sales. So if you want to see the raw chains of thought, you're probably going to have to pay up. All right, the next big announcement, I touched on it. Let's get into more detail. Claude code is now generally available. They have new extensions for VS code and JetBrains that integrate Claude code directly into your IDE, which is nice. This is a direct competition to all of the coding tools out there. Claude's proposed edits appear inline in your files, streamlining review and tracking with the familiar editor interface. And they're releasing a Claude code SDK so you can build your own coding agent. So again, they're really building out the infrastructure layer of agentic coding. So Claude code on GitHub now available. And that's an example of what's possible with the SDK tag Claude code on PRs to respond to reviewer feedback, fix CI errors, or modify code. So here's an example. Here's a PR right here. You're going to come into a comment. You're going to tag Claude. Could you please address this feedback comment? And it's going to jump in and start doing it right away. Gather issue and comment context, address the feedback, create a pull request, verify lint, make the tests, and so on. And then you have a PR ready to review. Now, chief science officer at Anthropic has said, according to TechMean, Anthropic's Jared Kaplan says the company stopped investing in chatbots at the end of 2024 and instead focused on improving Claude's ability to do complex tasks. And this makes sense. Claude is just not achieving the mindshare necessary to win at the chatbot game. That's ChatGPT. That's Gemini. Hopefully Siri in the future. So they gave up on that and went in and focused on agentic capabilities. And you know what? Good for them. Focus is what is required to win. And how about the pricing? Let's check it out. So Claude 4 Opus, the most intelligent model for complex tasks. It has a 200k context window, which is still relatively small. And you get a 50% discount with batch processing. $15 per million tokens input and $75 per million tokens output. So that's it. I'm going to be testing it out. Expect a testing video soon. If you enjoyed this video, please consider giving a like and subscribe. And I'll see you in the next one.