A New Lens on LLM Scaling and AI Progress

This article presents a comprehensive theoretical model that reexamines how large language models (LLMs) are scaled and what that means for the future of artificial intelligence. The argument centers on the idea that, despite their impressive performance, LLMs remain far from achieving human-level intelligence because many essential cognitive functions are still missing.

Revisiting the AI Journey

The evolution of AI is marked by significant breakthroughs that have occasionally upended our expectations:

• 1958: The Perceptron Era

• Introduced by Rosenblatt, the perceptron laid the foundation for future neural architectures. Early limitations were highlighted in later theoretical work, hinting at the need for deeper hierarchical models.

• 2012: AlexNet and the CNN Revolution

• The advent of AlexNet demonstrated the practical power of convolutional neural networks (CNNs) in image classification, heralding a new age for deep learning. Despite its success, AlexNet’s impact was bound to supervised learning paradigms with clear right/wrong signals.

• The 2010s: Reinforcement Learning and Beyond

• Transitioning to more challenging decisions in complex games like Go and StarCraft II required new approaches. The progress in reinforcement learning, although hard-won, highlighted that raw scaling was not the only ingredient for achievement. Instead, conceptual insights and innovative algorithms were critical.

• The 2020s: LLMs Take Center Stage

• LLMs, powered by transformers and mostly trained through supervised (or self-supervised) learning, brought a surprising level of generality. They can converse, solve problems, and even play chess, blending multiple functions within a single system. However, a critical test remains: despite their vast stored knowledge, LLMs have yet to produce significant novel contributions or breakthroughs that matter in the long term.

Unpacking the Limits of Current LLMs

The central critique of the current scaling paradigm is that it treats intelligence as a mere volume knob. Here are some of the core observations:

• Superficial Knowledge vs. Deep Understanding

• LLMs outperform humans in the sheer amount of memorized information but fall short in producing insights, solving new problems, or achieving conceptual breakthroughs.

• Missing Cognitive Functions

• Essential abilities such as efficient in-context learning, continual learning across interactions, and long-term coherent planning appear to be absent. The underlying mechanism—next token prediction—is inherently myopic and restricts deeper deliberative processes.

• The Role of Theoretical and Conceptual Advances

• History teaches that every major leap in AI required both increased computational power and fresh conceptual insights. Just as past breakthroughs moved beyond simple supervised training, a similar shift is necessary to unlock true human-level generality.

Addressing Common Objections

Critics often simplify the discussion by suggesting that denser compute or more elaborate inference procedures (sometimes referred to as "unhobbling") could bridge the gap. However, several points are made to caution against this view:

• Oversimplifying Cognitive Processes

• The belief that a simple scratchpad or extra compute cycles would transform an LLM into an autonomous problem solver fails to recognize the complex, multi-layered nature of human cognition. The process of symbol manipulation in AI cannot fully capture the rich, dynamic nature of thought.

• Incremental Gains vs. Fundamentally New Abilities

• History suggests that while each step in deep learning improves performance, the leap to true general intelligence will require new, often unexpected, theoretical insights. LLMs may accelerate certain research and development tasks, but they are not the silver bullet for AGI.

The Long Road Ahead

Despite the excitement surrounding recent advancements, it is clear from the historical perspective and the current limitations that humanity is still far from achieving AI with full human-level generality. The overwhelming evidence indicates that more time and new conceptual breakthroughs—likely measured in decades—are necessary before the remaining puzzles of intelligence are solved.

In conclusion, while LLMs display remarkable abilities in information recall and surface-level generality, crucial elements of true intelligence remain missing. The future of AI will hinge not solely on scaling models but on developing a deeper understanding of how to replicate the full spectrum of cognitive functions.

Note: The perspectives discussed here reflect the author's theoretical background as a computer scientist and are intended as a model to spur further empirical research and debate in the field.