The tech industry may be confronting a harsh reality as generative AI, once a beacon of progress through mere scaling, appears to be approaching its limits. Reports indicate that enhancements in large language models (LLMs) like OpenAI’s highly-anticipated Orion are showing signs of plateauing, raising doubts that increasing model size and computing power would continuously drive advancements.
For years, the tech industry has relied on scaling LLMs to achieve improvements in AI. By adding more parameters, data, and computational power, models like GPT-4 delivered significant advancements over their predecessors. However, this approach may no longer hold the key to the future of AI, with diminishing returns becoming more apparent. Cognitive scientist and AI critic Gary Marcus notes that industry valuations based on the promise of Artificial General Intelligence (AGI) are overinflated, calling them “just a fantasy.”
The clearest indication of this slowdown came from OpenAI itself. According to a report by The Information, OpenAI’s forthcoming model, code-named Orion, displayed less improvement over GPT-4 than GPT-4 did over GPT-3. For applications such as coding, often a primary draw for LLMs, the improvements may even be negligible. This sentiment is echoed by Ilya Sutskever, co-founder of OpenAI and now founder of Safe Superintelligence, who commented that scaling gains have essentially plateaued.
This shift is forcing a reconsideration of the “bigger is better” mindset that has driven the tech sector’s explosive growth. Marcus underscores the grim economic prospects, noting that training these massive models demands vast resources — tens of millions of dollars, hundreds of AI chips, and months of computation. Moreover, as companies exhaust freely available data from the internet, they face increasing costs to obtain new, relevant training data. “LLMs, as they are, will become a commodity,” Marcus predicts, leading to fierce price competition and limited profitability. He suggests that once the industry fully realizes this reality, a financial bubble may burst, sending shockwaves through the market.
Yet, all hope is not lost. OpenAI and other leading AI researchers are exploring alternatives to simple scaling. A promising approach, known as “test-time compute,” allows models to assess multiple potential solutions to complex tasks, narrowing down the most promising options rather than reaching a hasty conclusion. OpenAI’s latest “o1” model has already demonstrated early examples of these reasoning skills, hinting that new techniques may drive future advancements in AI without requiring massive scaling.
In the end, while the AI industry is far from dead, its path forward may depend on innovative approaches to tackle complex challenges beyond sheer size. If new techniques can’t produce results soon, the field could face another “AI winter” as impatient investors look elsewhere.