AI Scaling Wall

From HandWiki
Short description: Limit of AI improvements by scaling


The AI Scaling Wall is the proposition that improvements in generative artificial intelligence (AI), driven by greater training data, computing power, and model size, are reaching diminishing returns.[1][2][3] The idea proposes an imminent limit to the AI scaling laws, a term OpenAI coined in 2020 after Baidu researchers showed predictable performance gains from increasing resources such as training data or computing power.[2][4] In November 2024, reports indicated that OpenAI's GPT-4.5 model fell short of expectations, while Google's Gemini and Anthropic's Claude 3.5 Opus also faced delays or disappointing results.[5][6][7] Ilya Sutskever said that gains from scaling up pre-training had plateaued, and Gary Marcus argued that models had reached diminishing returns.[6][7] Sam Altman, Jensen Huang, and Dario Amodei disputed this.[1][5][8] Proposed causes include the exhaustion of human-made training data, which Epoch AI predicted could occur by 2028, and limits on computing power and energy.[6][7][8] Synthetic data offers a partial remedy, but a Nature study warned of model collapse from its indiscriminate use.[8][9] Amodei said training runs could cost $100 billion in coming years.[8][10] Companies have turned to test-time compute, used in OpenAI's o1 model, which Huang described as a second scaling law.[1][7] The improvement seen in GPT-5 was widely seen as underwhelming and fell below expectations.[11] One 2025 analysis estimated that each tenfold rise in compute yields only one to two percentage points of MMLU-Pro accuracy.[12] Analysts noted that United States export controls and safety policy were built on assumptions of continued growth in model size.[2]

References

  1. ↑ 1.0 1.1 1.2 The Economist 2026.
  2. ↑ 2.0 2.1 2.2 Booth 2024.
  3. ↑ Heath 2024.
  4. ↑ Morrow 2024.
  5. ↑ 5.0 5.1 Bosa & Wu 2024.
  6. ↑ 6.0 6.1 6.2 Chowdhury & Nolan 2024.
  7. ↑ 7.0 7.1 7.2 7.3 Hu & Tong 2024.
  8. ↑ 8.0 8.1 8.2 8.3 Chowdhury et al. 2024.
  9. ↑ Hashmi 2026.
  10. ↑ Metz et al. 2024.
  11. ↑ Heikkilä et al. 2025.
  12. ↑ Shukla 2025.

Works cited

  1. "Nvidia’s boss dismisses fears that AI has hit a wall" (in en). 21 Nov 2024. https://www.economist.com/business/2024/11/21/nvidias-boss-dismisses-fears-that-ai-has-hit-a-wall. }
  2. Booth, Harry (21 November 2024). "Has AI Progress Really Slowed Down?" (in en). https://time.com/7178328/is-ai-progress-slowing-down/. 
  3. Bosa, Deirdre; Wu, Jasmine (11 December 2024). "The limits of intelligence — Why AI advancement could be slowing down" (in en). https://www.cnbc.com/2024/12/11/why-ai-advancement-could-be-slowing-down.html. 
  4. Chowdhury, Hasan; Kanetkar, Riddhi; Nolan, Beatrice; Langley, Hugh (27 November 2024). "AI improvements are slowing down. Companies have a plan to break through the wall." (in en). https://www.businessinsider.com/generative-ai-wall-scaling-laws-training-data-chatgpt-gemini-claude-2024-11. 
  5. Chowdhury, Hasan; Nolan, Beatrice (11 November 2024). "OpenAI is reportedly struggling to improve its next big AI model. It's a warning for the entire AI industry." (in en). https://www.businessinsider.com/openai-orion-model-scaling-law-silicon-valley-chatgpt-2024-11. 
  6. Hashmi, Sahar (3 Mar 2026). "Data Plateau: Hit The Scaling Wall With AI Or Remain An Innovator?" (in en). https://www.forbes.com/sites/saharhashmi/2026/03/03/data-plateau-hit-the-scaling-wall-with-ai-or-remain-an-innovator/. 
  7. Heath, Alex (22 November 2024). "Is AI hitting a wall?" (in en). https://www.theverge.com/2024/11/22/24303470/ai-model-llm-progress-hitting-scaling-wall. 
  8. Heikkilä, Melissa; Bradshaw, Tim; Criddle, Cristina; Hammond, George (15 Aug 2025). "Is AI hitting a wall?" (in en). https://www.ft.com/content/d01290c9-cc92-4c1f-bd70-ac332cd40f94. 
  9. Hu, Krystal; Tong, Anna (15 November 2024). "OpenAI and others seek new path to smarter AI as current methods hit limitations" (in en). https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/. 
  10. Metz, Rachel; Ghaffary, Shirin; Bass, Dina; Love, Julia (13 November 2024). "OpenAI, Google and Anthropic Are Struggling to Build More Advanced AI" (in en). https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai. 
  11. Morrow, Allison (19 November 2024). "AI is hitting a wall just as valuations reach the stratosphere | CNN Business" (in en). https://edition.cnn.com/2024/11/19/business/ai-chatgpt-nvidia-nightcap. 
  12. Shukla, Hemant (23 Dec 2025). "The AI Scaling Wall of Diminishing Returns: Of LLMs, Electric Dogs, and General Relativity" (in en). arXiv. doi:10.48550/arXiv.2512.20264. 



Retrieved from "https://handwiki.org/wiki/index.php?title=AI_Scaling_Wall&oldid=4634104"