Huang's law

From HandWiki
Short description: Computer science observation

Huang's law is an observation in computer science and engineering that advancements in graphics processing units (GPUs) are growing at a rate much faster than with traditional central processing units (CPUs). The observation is in contrast to Moore's law that predicted the number of transistors in a dense integrated circuit (IC) doubles about every two years.[1] Huang's law states that the performance of GPUs will more than double every two years.[2] The hypothesis is subject to questions about its validity.

History

An RTX 4090, the most recent flagship card in Nvidia's GeForce series, with 82.58 TFLOPS at single precision computing (FP32) …
… and its predecessor, the RTX 3090, with 35.58 TFLOPS (FP32), showing a massive leap in raw performance in just one generation

The observation was made by Jensen Huang, then chief executive officer of Nvidia, at its 2018 GPU Technology Conference (GTC) held in San Jose, California.[3] He observed that Nvidia’s GPUs were "25 times faster than five years ago" whereas Moore's law would have expected only a ten-fold increase.[2] As microchip components become smaller, it became harder for chip advancement to meet the speed of Moore's law.[4]

In 2006 Nvidia's GPU had a 4x performance advantage over other CPUs. In 2018 the Nvidia GPU was 20 times faster than a comparable CPU node: the GPUs were 1.7x faster each year. Moore’s law would predict a doubling every two years, however Nvidia's GPU performance was more than tripled every two years, fulfilling Huang's law.[5]

Huang's law claims that a synergy between hardware, software and artificial intelligence makes the new 'law' possible.[upper-alpha 1] Huang said, "The innovation isn't just about chips," he said, "It's about the entire stack." He said that graphics processors especially are important to a new paradigm.[3] Elimination of bottlenecks can speed up the process and create advantages in getting to the goal. "Nvidia is a one trick pony," Huang has said.[7] According to Huang: "Accelerated computing is liberating, … Let’s say you have an airplane that has to deliver a package. It takes 12 hours to deliver it. Instead of making the plane go faster, concentrate on how to deliver the package faster, look at 3D printing at the destination." The object "… is to deliver the goal faster."[7]

For artificial intelligence tasks, Huang said AlexNet took six days on two of Nvidia's GTX 580 processors to complete the training process but only 18 minutes on a modern DGX-2 AI server, resulting in a speed-up factor of 500. Compared to Moore's law, which focuses purely on CPU transistors, Huang's law describes a combination of advances in architecture, interconnects, memory technology, and algorithms.[2][6]

Bharath Ramsundar wrote that deep learning is being coupled with "[i]mprovements in custom architecture". For example, machine learning systems have been implemented in the blockchain world, where Bitmain assaulted "many cryptocurrencies by designing custom mining ASICs (application-specific integrated circuits)" which had been envisioned as undoable. "Nvidia's grand achievement however is in making the case that these improvement in architectures are not merely isolated victories for specific applications but perhaps broadly applicable to all of computer science." They have suggested that broad harnessing of GPUs and the GPU stack (cf., CPU stack) can deliver "dramatic growth in deep learning architecture." "The magic" of Huang's law promise is that as nascent deep learning powered software becomes more availed, the improvements from GPU scaling and more generally from architectural improvements" will concretely improve "performance and behavior of modern software stacks."[8]

There has been criticism. Journalist Joel Hruska writing in ExtremeTech in 2020 said "there is no such thing as Huang's Law", calling it an "illusion" that rests on the gains made possible by Moore's law; and that it is too soon to determine a law exists.[9] The research nonprofit Epoch has found that, between 2006 and 2021, GPU price performance (in terms of FLOPS/$) has tended to double approximately every 2.5 years, much slower than predicted by Huang's law.[10]

See also

Notes

  1. ↑ Contrary to other reports, it is said that "Huang’s Law" … is a term coined by The Wall Street Journal journalist Christopher Mims."[2][6]

References

  1. ↑ Drum, Kevin. "Moore's Law is dead. Long live Huang's Law.". https://www.motherjones.com/kevin-drum/2020/09/moores-law-is-dead-long-live-huangs-law/. 
  2. ↑ 2.0 2.1 2.2 2.3 Mims, Christopher (September 19, 2020). "Huang's Law Is the New Moore's Law, and Explains Why Nvidia Wants Arm". Wall Street Journal. https://www.wsj.com/articles/huangs-law-is-the-new-moores-law-and-explains-why-nvidia-wants-arm-11600488001. (Subscription content?) reprinted in "Huang's Law is New Moore's Law and explains why Nvidia wants an arm". Yahoo!. September 29, 2020. https://www.yahoo.com/news/m/f74288ff-6577-327e-9e31-06b8f650f125/huang%E2%80%99s-law-is-the-new.html. 
  3. ↑ 3.0 3.1 Perry, Tekla S. (May 2018). "Move Over, Moore's Law: Make Way for Huang's Law". IEEE Spectrum (IEEE). https://spectrum.ieee.org/view-from-the-valley/computing/hardware/move-over-moores-law-make-way-for-huangs-law. Retrieved September 24, 2020. "Graphics processors are on a supercharged development path that eclipses Moore’s Law. … GPUs are also advancing more quickly than CPUs because they rely upon a parallel architecture, Jesse Clayton, an Nvidia senior manager, pointed out in another session."" 
  4. ↑ Tibken, Shara (9 January 2019). "CES 2019: Moore's Law is dead, says Nvidia's CEO". CNet. CBS Interactive. https://www.cnet.com/news/moores-law-is-dead-nvidias-ceo-jensen-huang-says-at-ces-2019/. Retrieved 24 September 2020. 
  5. ↑ Woodie, Alex (27 March 2018). "Nvidia Riding High as GPU Workloads and Capabilities Soar". HPCwire. https://www.hpcwire.com/2018/03/27/nvidia-riding-high-as-gpu-workloads-and-capabilities-soar/. Retrieved 24 September 2020. 
  6. ↑ 6.0 6.1 Goetting, Brittany (September 20, 2020). "Advancing AI And 'Huang's Law' Could Be Why NVIDIA Moved To Acquire Arm". HotHardware. https://hothardware.com/news/huangs-law-could-be-why-nvidia-moved-to-acquire-arm. Retrieved September 24, 2020. 
  7. ↑ 7.0 7.1 Hayes, Caroline (October 11, 2018). "Jensen Huang: Moore's law is dead – long live AI". Electronics Weekly (Metropolis International). https://www.electronicsweekly.com/news/products/micros/jensen-huang-moores-law-dead-long-live-ai-2018-10/. Retrieved September 24, 2020. "... there are two dynamics controlling the computing industry today – the end of Moore’s law and software that can write itself, artificial intelligence, or AI. ... We can study where bottlenecks are. New software systems make the application go faster, not just the chip.". 
  8. ↑ Ramsundar, Bharath (April 7, 2018). "The Advent of Huang's Law". https://rbharath.github.io/the-advent-of-huangs-law/. Retrieved September 24, 2020. 
  9. ↑ Hruska, Joel (September 22, 2020). "There's No Such Thing as 'Huang's Law,' Despite Nvidia's AI Lead". Extreme Tech. https://www.extremetech.com/computing/315277-theres-no-such-thing-as-huangs-law. 
  10. ↑ Marius Hobbhahn and Tamay Besiroglu (2022), "Trends in GPU price-performance". Published online at epochai.org. Retrieved from: https://epochai.org/blog/trends-in-gpu-price-performance

External links



Retrieved from "https://handwiki.org/wiki/index.php?title=Huang%27s_law&oldid=3370675"