ARC-AGI-2: Pushing the Boundaries of AI Intelligence

The Dawn of a New AI Challenge

The Arc Prize Foundation, co-founded by renowned AI researcher François Chollet, has unveiled a groundbreaking test designed to gauge the true general intelligence of leading AI models. The newly minted ARC-AGI-2 challenges systems to face novel puzzles that go far beyond traditional tasks.

Beyond Brute Force: A New Testing Paradigm

ARC-AGI-2 presents a series of puzzle-like problems where AI models must interpret visual patterns from grids of colored squares to generate a correct answer. In doing so, the test forces these systems to adapt rapidly to unseen challenges rather than relying on vast computational power or memorized patterns:

• Rapid Adaptation: Models must interpret and react to new puzzles on the fly.
• Efficiency Focus: The test emphasizes not only problem-solving but also the cost and speed at which solutions are found.

Greg Kamradt, co-founder of the Arc Prize Foundation, explained that true intelligence is about the efficiency with which skills are acquired and deployed, not just high problem-solving rates.

Humans vs. Machines: A Stark Comparison

To establish a human baseline, over 400 participants completed the ARC-AGI-2 test, achieving an average accuracy of 60%. In contrast, current AI models struggle:

• Models such as OpenAI’s o1-pro and DeepSeek’s R1 score only between 1% and 1.3%.
• Other advanced systems, including GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash, obtain scores around 1%.

This significant gap highlights the challenge posed by ARC-AGI-2 and the urgent need for AI innovations that extend beyond sheer computational might.

Rethinking AI Progress

The tech industry is in pursuit of fresh benchmarks to truly encapsulate the essence of artificial general intelligence (AGI). ARC-AGI-2 has emerged as a vital benchmark by addressing earlier shortcomings observed in ARC-AGI-1, where models could leverage brute-force methods. The new test forces AI to learn and adapt under new constraints, such as:

• Efficiency Metrics: Evaluating not just the ability to solve tasks but doing so cost-effectively.
• Creativity and Adaptability: Ensuring models develop skills dynamically rather than simply recalling past data.

Notably, OpenAI’s breakthrough with its o3 model, which once excelled on ARC-AGI-1 by matching human performance, now sees its performance plummet to 4% on ARC-AGI-2 when measured under a strict computational budget.

A Call to Innovators

In alignment with the new benchmark, the Arc Prize Foundation has launched the Arc Prize 2025 contest. Developers are now challenged to achieve 85% accuracy on ARC-AGI-2 while keeping the cost per task at a mere $0.42. This initiative underscores the growing emphasis on efficient, scalable AI solutions that can thrive under realistic constraints.

Looking Ahead

The introduction of ARC-AGI-2 marks a transformative moment in AI evaluation. By focusing on efficiency, creativity, and adaptability, the benchmark is reshaping the industry’s understanding of intelligent behavior in machines. As AI researchers and developers rise to meet this challenge, the pursuit of truly versatile AI systems is set to enter an exciting new phase.

The ARC-AGI-2 test is more than just an academic exercise—it’s a call to innovate smarter, more agile AI that can handle the unpredictability of real-world problems.