Hello and welcome to Eye on AI. In this edition…AI's fast-falling cost…Google goes nuclear…LLMs may be dumber than you think…and a filmmaker burned by genAI backlash.
Every year for the past seven, Nathan Benaich, the founder and solo general partner at the early-stage AI investment firm Air Street Capital, has produced a magisterial “State of AI” report. Benaich and his collaborators marshal an impressive array of data to provide a great snapshot of the technology’s evolving capabilities, the landscape of companies developing it, a survey of how AI is being deployed, and a critical examination of the challenges still facing the field.
OpenAI's lead mostly vanishes
One of the big takeaways from this year’s report, which was published late last week, is that OpenAI’s lead over other AI labs has largely eroded. Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5, X’s Grok 2, and even Meta’s open-source Llama 3.1 405 B model have equaled, or narrowly surpassed on some benchmarks, OpenAI’s GPT-4o.
But, on the other hand, OpenAI still retains an edge for the moment on reasoning tasks with the release of its o1 “Strawberry” model—which Air Street’s report rightly characterized as a weird mix of incredibly strong logical abilities for some tasks, and surprisingly weak ones for others. (For more on the fragility of o1’s reasoning abilities, see the “Research” section below.)
Inference costs fall rapidly
Another big takeaway, Benaich told me, is the extent to which the cost of using a trained AI model—an activity known as "inference"—is falling rapidly. There are several reasons for this. One is linked to that first big takeaway: With models less differentiated from one another on capabilities and performance, companies are forced to compete on price.
Another reason is that engineers for companies such as OpenAI and Anthropic—and their hyperscaler partners Microsoft and AWS, respectively—are discovering ways to optimize how the largest models run on big GPU clusters. The cost of outputs from OpenAI’s GPT-4o today is 100-times less per token (which is about equivalent to 1.5 words) than it was for GPT-4 when that model debuted in March 2023. Google’s Gemini 1.5 Pro now costs 76% less per output token than it did when that model was launched in February 2024.
AI researchers have also become good at creating small AI models that can equal the performance of larger LLMs on dialogue, summarization, or even coding, while being much cheaper to run. Taken together, these two trends mean that the economics of implementing AI-based solutions are starting to look much more attractive than they did a year ago. This may ultimately help businesses find the return on investment from generative AI that they have complained has been elusive so far.
Robotics makes a come back
Another key trend Benaich picks up on is how robotics is coming back into vogue, with robotics companies marrying LLMs and new “world models” to existing tech to make significant progress in making robots more capable and easier (as well as cheaper) to deploy and customize.
Benaich’s State of AI report always ends with some bold predictions for the year ahead (and Benaich grades himself each year on how he’s done.) Among the things he got right last year: that a Hollywood production would make use of genAI models for visual effects and that there would be limited progress on international AI governance efforts. Among those he got wrong: that a company would spend more than $1 billion training a single LLM.
This year, among the report's predictions, are that an open source alternative to OpenAI’s o1 will surpass it across a range of benchmarks and that a $10 billion investment from a sovereign state into a U.S. AI company will cause the U.S. government to institute a national security review. We’ll check back next year to see how Benaich did.
Fortune Brainstorm AI takes the pulse of a fast-changing industry
The State of AI report is not the only way to find a fantastic overview of what’s happening in AI. Another great place to gain a vantage point on AI’s rapidly evolving landscape and find out how AI is impacting business is Fortune’s upcoming Brainstorm AI conference in San Francisco. This must-attend annual event is coming up on December 9 and 10, held at the St. Regis Hotel.
This year’s conference will include conversations with, among many others: Amazon’s head scientist for artificial general intelligence, Rohit Prasad, who will update us on how the Everything Store is trying to ensure it doesn’t get left behind in the race to build superpowerful—and super useful—AI; Liz Reid, Google’s vice president of search, who will discuss the future of Google’s signature product in an AI world; Christopher Young, Microsoft’s executive vice president of business development, strategy, and ventures, who will discuss how the tech giant is trying to see around corners to what is coming next for AI; Daniela Braga, the founder and CEO of Defined.ai who will tell us what it really takes to build AI models that work for customers; and Colin Kaepernick, former Super Bowl quarterback for the San Francisco 49ers and current founder and CEO of Lumi, a company that builds AI-powered tools for content creators, who will speak about his own transformation from professional athlete to entrepreneur, and what AI may mean for influencers, brands, and beyond.
I’ll be there, of course, helping to cochair the discussion with a gaggle of ultra-talented colleagues. I hope you will all consider joining me! And I’m very excited to be able to offer Eye on AI readers a special discounted rate—20% off the regular price of attendance! Just write the code KAHN20 in the Additional Comments section of the application to secure your discount. You can click here to find out more. Follow the link on that page to apply to attend. Remember to use the discount code!
With that, here’s more AI news.
Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn
