Hello and welcome to Eye on AI. In this edition…OpenAI’s o3 model blows the AI industry’s collective mind; Trump AI policy advisor becomes a flashpoint for internecine battle over immigration policy; DoE confirms AI’s big impact on electricity demand; some predictions for 2025.
The year drew to a close with a mic drop from OpenAI. On Dec. 20, the company unveiled a new AI model called o3 that showed off-the-charts performance on a series of benchmark tests, including one specifically designed to gauge whether AI models are capable of human-like abstract reasoning and generalization. The new o3 model scored 75.7% on this ARC-AGI benchmark, when restricted to less than $10,000 in computing expense, and 87.5% with an unrestricted compute budget. OpenAI’s relatively capable GPT-4o model had scored just 5% on the same test.
The result led some AI enthusiasts to wonder out loud whether OpenAI had just achieved the field’s long-sought Holy Grail, artificial general intelligence (or AGI)—which OpenAI defines as a single software system able to perform most economically valuable cognitive tasks as well as or better than a human.
Meanwhile, some deep learning skeptics, most notably Gary Marcus, blasted the o3 result as wildly misleading, arguing that: OpenAI had specifically trained o3 to do well on the ARC-AGI benchmark (even though the benchmark was designed to make this sort of “training to the test” difficult); that the ARC-AGI benchmark was a poor judge of how well a model will perform on more open-ended and ambiguous real world tasks; and that the high cost of o3 would make it economically unfeasible to use for most practical applications.
