Apple: “AI Can’t Think” — Are They Wrong?

Apple dropped this paper called The Illusion of Thinking and it has gone absolutely viral. In it, they claim that the large language models capable of thinking aren't really thinking and in fact aren't really that much better than regular models. They make claims of data contamination in the models, meaning they are essentially cheating to pass the benchmarks. They claim lack of generalization ability, which is really the core thesis for why these thinking models are so powerful. And all of this is being seen through the lens of Apple being way behind in the race to AI supremacy. So I'm going to show you the paper. I'm going to break it down. It actually has a ton of fascinating insights for how these thinking models operate, regardless of whether you agree with its assertions or not. And I want to say thank you to Retool for sponsoring this video. More about them later in the video. Here's the paper, The Illusion of Thinking, Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity by Apple. The core assertion of this paper is that we are vastly overestimating the capabilities of large reasoning models, LRMs. Those are the models that have the ability to think and to churn through tokens during that thinking time. Those are models like OpenAI's 03, 04, DeepSeq R1, Cloud 3.7 thinking, Cloud 4.0 thinking. These are all of the state-of-the-art models that are absolutely saturating today's current benchmarks. And Apple is calling them out. Current evaluations primarily focus on established mathematical and coding benchmarks, emphasizing final answer accuracy. However, this evaluation paradigm often suffers from data contamination. Data contamination means that the models were trained on the benchmarks. That's another way of saying basically cheating, whether intentionally or unintentionally. And does not provide insights into the reasoning, traces, structure, and quality. So they are questioning all of today's major benchmarks. And they propose a new benchmark, which is puzzles. Puzzles of varying degrees of difficulty. And I'm going to show you what those puzzles are. I'll show you how these models performed. And I'm also going to show you some of the insights this paper surfaces about how these thinking models actually get worse with additional complexity. And if a puzzle benchmark sounds familiar, it is very similar to the Arc AGI benchmark. And quick reminder, the Arc AGI benchmark is essentially a puzzle slash generalization problem for large language models. It takes an example of how to manipulate these squares and then gives it a new version of it to solve. And a lot of these models aren't doing all that well. So let's get into the nitty gritty of the paper. Large language models have recently evolved to include specialized variants explicitly designed for reasoning tasks, large reasoning models, that is thinking models, such as OpenAI's 0103, DeepSeq R1, Cloud 3.7, Sonnet Thinking, and Gemini Thinking. These models are new artifacts characterized by their thinking mechanisms such as long chain of thought, that is how they actually execute the thinking portion, with self-reflection and have demonstrated promising results across various reasoning benchmarks. And here's the counterclaim. Are these models capable of generalizable reasoning or are they leveraging different forms of pattern matching? How does their performance scale with increasing problem complexity? How do they compare to their non-thinking standard LLM counterparts when provided with the same inference token compute? Most importantly, what are the inherent limitations of current reasoning approaches and what improvements might be necessary to advance toward more robust reasoning capabilities? Apple is questioning all of the state-of-the-art models and benchmarks right now. Apple goes on to talk about current benchmarks. So existing evaluations predominantly focus on established mathematical encoding benchmarks, which, while valuable, often suffer from data contamination issues and do not allow for controlled experimental conditions across different settings and complexities. So again, calling these model providers out for having data contamination issues. Essentially, the model is being trained on the benchmarks. So if they're trained on the benchmarks, of course they're going to do well on them. These evaluations do not provide insights into the structure and quality of reasoning traces. So this paper is not only claiming that the top models are contaminated, but the benchmarks aren't really testing the important part, which is the structure and quality of the reasoning traces, the thinking portion of the model. And you know what scored really high on my own personal benchmark? The sponsor of today's video, Retool. Retool is the leading platform for building AI applications. And they just launched their enterprise-grade agentic development platform. There's this huge gap between really cool AI demos and actually building stuff of value for your business. And that is where Retool comes in. Retool helps you build better applications more quickly by connecting to your internal business applications, logic, data, everything. And that allows the applications you build to just be so much more effective. Plus, Retool is especially powerful because it allows you to use basically any model, including top models from OpenAI, Anthropic, Azure, and Google. You can give your agents access to all of your internal business tools, allowing them to take real actions across all of your business systems. And all of this is locked down with enterprise-grade access control. So build autonomous AI agents and workflows easily with Retool. It has already been used by companies like AWS and Databricks to automate a hundred million hours of work just in the last 12 months alone. So give Retool a try, check it out, let them know I sent you by going to retool.com slash Matthew. And let me know what you think. I'll drop links down in the description below. Thanks again to Retool and now back to the video. And here is their proposal. Rather than standard benchmarks like math problems, we adopt controllable puzzle environments that let us vary complexity systematically by adjusting puzzle elements while preserving the core logic. Here are the four puzzles that this paper used as the evaluation metric for these models. So we have Tower of Hanoi and this might look familiar. Essentially every kid has played with something like this. Essentially there are three columns and varying size plates. And your goal is to essentially move the plates from one side to the other with the rule that you can only move the topmost plate on any stack. And the stack must always be sorted top to bottom from smallest to largest. And so here is the beginning state and the task is to reach the end state as shown here. They also have Checkers Jumping, River Crossing, and Blocks World. All of these are puzzle games. They take logic and reasoning to solve and they can actually vary the complexity. The way that they vary the complexity here is by adding more plates. For the Checkers Jumping game, they can add more of the checkers. For the River Crossing puzzle, they can add more entities to cross the river. And for Blocks World, obviously add more blocks. And that increases the number of moves that it takes to complete the puzzle, thus increasing the complexity. They also suggest that these puzzles are not only better at evaluating, but often avoid contamination. So the puzzles offer fine-grained control over complexity, avoid contamination common in established benchmarks, require only the explicitly provided rules emphasizing algorithmic reasoning, and support rigorous simulator-based evaluation enabling precise solution checks and detailed failure analyses. And so the contributions of this paper are, we question the current evaluation paradigms of large reasoning models. We show that state-of-the-art LRMs still fail to develop generalizable problem-solving capabilities. We find that there exists a scaling limit in LRMs reasoning effort with respect to problem complexity. We question the current evaluation paradigm based on final accuracy and extend our evaluation to intermediate solutions of thinking traces. And we uncover surprising limitations in LRMs ability to perform exact computation. Some of these findings are extremely interesting. If it sounds confusing, don't worry, I'm going to break it all down in simple terms. So first, let's look at the AME 2024 and AME 2025 benchmarks. These are standard benchmarks. Every state-of-the-art model gets tested against them pretty much. And according to this paper, there might be data contamination with these benchmarks. So the first question that this paper poses is why these thinking models are doing better on certain benchmarks than their non-thinking versions. So it is not clear whether the performance enhancements observed in recent RL-based thinking models are attributable to increased exposure to established mathematical benchmark data. Again, that is model contamination to the significantly greater inference compute allocated to thinking tokens, which would be the reason that all of the frontier model providers give and think, or to reasoning capabilities developed by RL-based training, which again is another belief of the frontier model providers. Now here is something really interesting. They have shown that under equivalent inference token budgets, non-thinking LLMs can eventually reach performance comparable to thinking models on benchmarks like MATH 500 and AME 2024. What does that actually mean? So we have thinking models that use a lot of tokens, and then we have non-thinking models that use far fewer tokens and really just give you the first solution it comes up with. However, if you take the thinking model and the non-thinking model and you pit them head-to-head with the same inference budget, the same number of total tokens, it actually has been shown that the non-thinking models will reach parity with the thinking models. But how do they actually do that? They use something called pass at K, and that basically means generating a bunch of candidate solutions and choosing the best one. So rather than churning through a bunch of thinking tokens and then finally giving the output, like a thinking model does, the non-thinking models will generate 10, 20, 50 different candidate solutions and will choose the best, most accurate one. And so with all of those candidate solution generations, the total number of inference tokens used will be equivalent between thinking and non-thinking. And what this paper is proposing is that that actually makes their capabilities even, even though one's thinking and one's not. Very, very interesting. However, we observe that this performance gap, the gap between thinking and non-thinking, widens on the AME 2024 benchmark and widens further on AME 2025. And the gap presents an interpretive challenge. What does that actually mean? It could be attributed to either increasing complexity requiring more sophisticated reasoning processes, thus revealing genuine advantages of the thinking models for more complex problems. That is what the frontier model providers would say, or reduced data contamination in newer benchmarks, particularly AME 2025. And interestingly, human performance on AME 2025 was actually higher than AME 2024, suggesting that AME 2025 might be less complex. So although humans are doing better on 25, non-thinking models are doing worse, which is leads them to believe, or at least they're proposing it's possible that the models, the non-thinking models, really any model has data contamination with the results of AME 2024. And that is when we get to these puzzles. Even if the models have been trained on these puzzles, these puzzles can't really be memorized in the same way that a math problem can. So let's look at some of the results. So here are the four different puzzles, and we have Claude 3.7 on the top, thinking and non-thinking and DeepSeq R1 versus DeepSeq V3, that is thinking versus non-thinking on the bottom. What do we see? On the x-axis, we have the complexity of the puzzle. So remember, for the Tower of Hanoi puzzle, if you add more of those disks, it becomes more complex. And so as we increase complexity, that is what we're seeing on the x-axis. And then across the y-axis, we are seeing the accuracy. So here we see when the complexity is very low. So one through three disks, both models perform basically the same, thinking and non-thinking. However, when we start to get to mid-level complexity, so disks four through 10, what we see is a pretty substantial drop-off on the non-thinking model and a better performance for the thinking model. And then towards increasing complexity, they converge to zero, meaning they are not passing anymore. Both models are not doing well. And look at this. Across two different families of models and across four different puzzles, we are seeing the same pattern. So what does this tell us? When the task is easy, both models do well. When the task is medium complexity, the thinking models do better. But when the puzzle reaches a certain threshold of complexity, it becomes really complex. Both models fail. And so let's talk about how they set up these experiments. So first, they're using CLOD and DeepSeq. That is because those are the two models that are frontier that also allow visibility into the chain of thought. The O1 and O3 models abstract the chain of thought into summary. So they don't actually know what these models are really thinking. So that's why most of these experiments are done with CLOD and DeepSeq. For each puzzle instance, we generate 25 samples and report the average performance of each model across them. And so here's another way to look at those three tranches of complexity. We have low complexity, medium, and high complexity. And as we see here, dependent on the compute budget, the two models actually just about do the same with the non-thinking model actually doing better at fewer tokens. Then at medium complexity, the thinking model does better across the board. And then at highest complexity, they both fail. They also explore how many tokens are used by the thinking models dependent on the complexity of the puzzle. As complexity increases, reasoning models initially spend more tokens while accuracy declines gradually until a critical point where reasoning collapses, performance drops sharply, and reasoning effort decreases. So these thinking models get to a point where they actually start using less tokens even on the most complex puzzles. Does that mean they're kind of giving up? It's really interesting to think about. And it's also interesting to think about when in the thinking trace, the models actually got the answer right. Because if they get it right early on, and then continue thinking, those are all in a sense wasted tokens. Although exploring alternative paths isn't necessarily bad, but in terms of efficiency, it kind of is. So for simpler problems, reasoning models often find the correct solution early in their thinking, but then continue exploring incorrect solutions. This phenomenon referred to as overthinking leads to the waste of compute. As problems become moderately more complex, this trend reverses. Models first explore incorrect solutions and mostly later in the thought arrive at correct ones. So that's what we're seeing here. So let's look at tower of Hanoi. As we're seeing the complexity is across the x-axis and the position in the thinking where they find the correct answer is on the y-axis. So when the complexity is quite low, they find the correct answer right in the middle. Then as the complexity increases a little bit, they start to find the answer earlier, which is interesting, but then they continue to compute and find the incorrect answer later on. And you can see how this graph changes to eventually the model starts finding incorrect answers early on and the correct answer later on in the thinking. This is what we would want. We don't want the model to find the answer and then continue to compute. However, for the checker jumping up to for complexity, we see kind of the right trend, which is find the incorrect answer and then find the correct one later on. And for blocks world, we see that the incorrect and correct answers are kind of aligned. And then all of a sudden it does take more compute to start finding the correct answers. They even tested simply giving the thinking models the algorithm to solve the problems, not just saying, Hey, solve this puzzle, but here's exactly how to solve it. And what they found is interesting. Listen to this. Even when we provide the algorithm in the prompt so that the model only needs to execute the prescribed steps, performance does not improve. And the observed collapse still occurs at roughly the same point. This is noteworthy because finding and devising a solution should require substantially more computation than merely executing a given algorithm, but it doesn't. This further highlights the limitations of reasoning models and verification and in following logical steps to solve a problem. But they observe different behavior from CLOD 3.7's sonnet thinking. And the CLOD family of models is very unique in a lot of ways, which we've discussed in previous videos. So for the tower of Hanoi, the model's first error in the proposed solution often occurs much later compared to the river crossing environment. So here's what that looks like. Despite providing the solution algorithm in the prompt execution failure occurs at the same points. So here we can see the light blue versus darker blue. It's failing right at the same point. Now, this model is essentially saying that the models might not actually be thinking, or at the very least, they're not able to generalize their knowledge. However, let me play this clip of Ilya Sutskover, co-founder of OpenAI, currently working on his own startup, SSI Safe Superintelligence. And this is a recent clip from a commencement speech that he just gave. Listen to this. Sous-titled by Javier Sousa, M.D.: Slowly but surely, or maybe not so slowly, AI will keep getting better. And the day will come when AI will do all of our, all the things that we can do. Not just some of them, but all of them. Anything which I can learn, anything which any, any one of you can learn, the AI could do as well. How do we know this, by the way? How can I be so sure? How can I be so sure of that? Sousa, M.D.: The reason is that all of us have a brain, and the brain is a biological computer. That's why. We have a brain, the brain is a biological computer. So why can't a digital computer, a digital brain, do the same things? Sousa, M.D.: So why did I show that clip? Well, if this paper is trying to disprove that these models are thinking, and maybe even it's trying to disprove that the language model approach to intelligence is not sufficient to reach AGI, it seems that Ilya believes we are well on our way to achieving AGI. And specifically because we have a biological computer, our brains, and a digital computer should be sufficient in simulating our intelligence and likely much more. So let me get to the conclusion now. Our findings reveal fundamental limitations in current models. Despite sophisticated self-reflection mechanisms, these models fail to develop generalizable reasoning capabilities beyond certain complexity thresholds. But of course, they talk about the limitations of this approach. Our experiments represent a narrow slice of reasoning tasks and may not capture the diversity of real world or knowledge-intensive reasoning problems. That means that they are essentially taking a small part of intelligence, the ability to solve these puzzles, and that is what they're testing the models against. But that, again, is only a small part of intelligence. We are seeing incredible achievements out of the current generation of models. Deep thinking, the ability to generate compelling images and video, and generate code, and all of these things are intelligence. And just because it can't solve these puzzles at a certain complexity threshold, doesn't necessarily mean these models aren't thinking. They just might be better at some things than others. But here's my biggest issue with the paper. They didn't mention, even once, the ability for these models to write code to solve some of these puzzles. The closest thing they came was giving the pseudo algorithm to the model, but it still had to solve the puzzle in natural language. Now we know these models are incredibly good at writing code. So if the model can write code to solve the puzzle, isn't that the same thing? Wouldn't you give credit to the model if it can do that? Well, let's find out. So I have Claude 3.7 and I gave it a very simple prompt. Write code using HTML JavaScript to simulate the tower of Hanoi game. Allow for up to 10 disks. Add a solve button. I didn't even tell it how to solve it. Let's find out if it's able to. So this was a thinking model. It did create the tower of Hanoi game. Let's solution speed. Let's put it at the lowest because we want to get it done quickly. Up to 10. There it is. And solve. Let's see if it's able to solve it. Sure seems like it's getting there. And there we go. It solved it at a complexity of 10 disks easily. 1023 moves. No problem. And in fact, I bet it can even solve higher complexity. Let's try that. Here we go. I asked for another version with 20 disks. Now just ignore it bouncing upside down. It's just a little formatting error, but look at it solving it. It has to do over a million moves. And of course it's going to do that because there is a formula to solve it. So once the model just outputs the code to run the formula, it's going to do so easily. So let me know what you think about this. Would you give credit to the model for this kind of solution where it uses code instead of natural language to solve these puzzles? Let me know. And thanks again to Retool for sponsoring this video. Check them out. Links down below. If you enjoyed this video, please consider giving a like and subscribe and I'll see you in the next one.