Moments after OpenAI, Google, or Anthropic drop a major upgrade to their AI models, you'll see people already speculating on the next update's date and features. And there have been fairly regular updates to feed those rumors. However, those days may be over, according to a Bloomberg report. All three major AI developers are reportedly struggling to make their next-gen models match their ambitions for improvement over the current crop.
The report claims that OpenAI's work on the Orion model isn't going as well as the company expected. The model doesn't perform at the level the company is aiming for, especially when it comes to coding. Orion may not offer a seismic change from GPT-4 compared to how GPT-4 blew GPT-3.5 out of the water. That may be one reason OpenAI CEO Sam Altman publicly pushed back on rumors about the release date for the Orion model and an upgrade to ChatGPT.
Delays and lower expectations are also plaguing Google and Anthropic. Google's Gemini development is slower than hoped, according to Bloomberg. Anthropic has already pushed back releasing its Claude 3.5 Opus model for similar reasons despite teasing it earlier this year.
All of the AI developers are running into the same ceilings in growing their model's abilities. The biggest is likely training data. The companies have leveraged enormous datasets to train their AI models, but even the internet is not infinite, and that goes even more when discussing high-quality data useful for training AI. Finding previously unused, accessible information is becoming tricky. That's partly because of growing awareness and consideration for ethical and legal rights to use some data, but that's only part of the explanation. At some point, there aren't enough human examples for the AI models to absorb and improve upon. Even if the companies find enough raw data, processing it and incorporating it into an AI model is expensive in terms of money and computing power. If the data cannot make more than slight improvements, then upgrading the AI model might not be worth the price.
Fuel or fumes?
The report describes how OpenAI and its rivals are looking to other ways of upgrading their models, like post-training Orion with human feedback. That's a slow way to improve an AI model and raises questions about whether AI has reached the limits of rapid scaling in size and functions. Brute computing power and avalanches of data may not be enough to make the dreams of AI developers real anymore. They'll need to get more creative in how they iterate on their models without throwing the entire internet at it.
For us, we should expect somewhat slower releases of new and improved AI features. That might not be terrible if it gives everyone a chance to catch their breath and really dig into the best ways to use all the AI tools released in the last few years. There's plenty to explore with ChatGPT-o1. And, who knows, maybe this will give OpenAI the space to work on releasing the Sora AI video creator, which has been kept highly restricted despite OpenAI teasing it with a steady drip of demos.
