AI accelerator

From HandWiki
Jump to: navigation, search

An AI accelerator is a class of specialized hardware accelerator[1] or computer system[2][3] designed to accelerate artificial intelligence applications, especially artificial neural networks, machine vision and machine learning. Typical applications include algorithms for robotics, internet of things and other data-intensive or sensor-driven tasks.[4] They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability.[5] As of 2018, a typical AI integrated circuit chip contains billions of MOSFET transistors.[6]

A number of vendor-specific terms exist for devices in this category, and it is an emerging technology without a dominant design.

History of AI acceleration

Computer systems have frequently complemented the CPU with special purpose accelerators for specialized tasks, known as coprocessors. Notable application-specific hardware units include video cards for graphics, sound cards, graphics processing units and digital signal processors. As deep learning and artificial intelligence workloads rose in prominence in the 2010s, specialized hardware units were developed or adapted from existing products to accelerate these tasks.

Early attempts

As early as 1993, digital signal processors were used as neural network accelerators e.g. to accelerate optical character recognition software.[7] In the 1990s, there were also attempts to create parallel high-throughput systems for workstations aimed at various applications, including neural network simulations.[8][9][10] FPGA-based accelerators were also first explored in the 1990s for both inference[11] and training.[12] ANNA was a neural net CMOS accelerator developed by Yann LeCun.[13]

Heterogeneous computing

Heterogeneous computing refers to incorporating a number of specialized processors in a single system, or even a single chip, each optimized for a specific type of task. Architectures such as the cell microprocessor[14] have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, dataflow architecture, and prioritizing 'throughput' over latency. The Cell microprocessor was subsequently applied to a number of tasks[15][16][17] including AI.[18][19][20]

In the 2000s, CPUs also gained increasingly wide SIMD units, driven by video and gaming workloads; as well as support for packed low precision data types.[21]

Use of GPU

Graphics processing units or GPUs are specialized hardware for the manipulation of images and calculation of local image properties. The mathematical basis of neural networks and image manipulation are similar, embarrassingly parallel tasks involving matrices, leading GPUs to become increasingly used for machine learning tasks.[22][23][24] As of 2016, GPUs are popular for AI work, and they continue to evolve in a direction to facilitate deep learning, both for training[25] and inference in devices such as self-driving cars.[26] GPU developers such as Nvidia NVLink are developing additional connective capability for the kind of dataflow workloads AI benefits from.[27] As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated neural network specific hardware to further accelerate these tasks.[28][29] Tensor cores are intended to speed up the training of neural networks.[29]

Use of FPGAs

Deep learning frameworks are still evolving, making it hard to design custom hardware. Reconfigurable devices such as field-programmable gate arrays (FPGA) make it easier to evolve hardware, frameworks and software alongside each other.[11][12][30]

Microsoft has used FPGA chips to accelerate inference.[31] The application of FPGAs to AI acceleration motivated Intel to acquire Altera with the aim of integrating FPGAs in server CPUs, which would be capable of accelerating AI as well as general purpose tasks.[32]

Emergence of dedicated AI accelerator ASICs

While GPUs and FPGAs perform far better than CPUs for AI related tasks, a factor of up to 10 in efficiency[33][34] may be gained with a more specific design, via an application-specific integrated circuit (ASIC). These accelerators employ strategies such as optimized memory use{{citation needed|date=November 2017} cision arithmetic]] to accelerate calculation and increase throughput of computation.[35][36] Some adopted low-precision floating-point formats used AI acceleration are half-precision and the bfloat16 floating-point format.[37][38][39][40][41][42][43]

In-memory computing architectures

In June 2017, IBM researchers announced an architecture in contrast to the Von Neumann architecture based on in-memory computing and phase-change memory arrays applied to temporal correlation detection, intending to generalize the approach to heterogeneous computing and massively parallel systems.[44] In October 2018, IBM researchers announced an architecture based on in-memory processing and modeled on the human brain's synaptic network to accelerate deep neural networks.[45] The system is based on phase-change memory arrays.[46]


As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and APIs will become the dominant design. There is no consensus on the boundary between these devices, nor the exact form they will take; however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.

In the past when consumer graphics accelerators emerged, the industry eventually adopted Nvidia's self-assigned term, "the GPU",[47] as the collective noun for "graphics accelerators", which had taken many forms before settling on an overall pipeline implementing a model presented by Direct3D.

Potential applications

See also


  1. "Intel unveils Movidius Compute Stick USB AI Accelerator". 2017-07-21. 
  2. "Inspurs unveils GX4 AI Accelerator". 2017-06-21. 
  3. Wiggers, Kyle (November 6, 2019), Neural Magic raises $15 million to boost AI inferencing speed on off-the-shelf processors,, retrieved 2020-03-14 
  4. "Google Developing AI Processors". Google using its own AI accelerators.
  5. "A Survey of ReRAM-based Architectures for Processing-in-memory and Neural Networks", S. Mittal, Machine Learning and Knowledge Extraction, 2018
  6. "13 Sextillion & Counting: The Long & Winding Road to the Most Frequently Manufactured Human Artifact in History". April 2, 2018. Retrieved 28 July 2019. 
  7. "convolutional neural network demo from 1993 featuring DSP32 accelerator". 
  8. "design of a connectionist network supercomputer". 
  9. "The end of general purpose computers (not)". This presentation covers a past attempt at neural net accelerators, notes the similarity to the modern SLI GPGPU processor setup, and argues that general purpose vector accelerators are the way forward (in relation to RISC-V hwacha project. Argues that NN's are just dense and sparse matrices, one of several recurring algorithms)
  10. Ramacher, U.; Raab, W.; Hachmann, J.A.U.; Beichter, J.; Bruls, N.; Wesseling, M.; Sicheneder, E.; Glass, J. et al. (1995). Proceedings of 9th International Parallel Processing Symposium. pp. 774–781. doi:10.1109/IPPS.1995.395862. ISBN 978-0-8186-7074-9. 
  11. 11.0 11.1 "Space Efficient Neural Net Implementation". 
  12. 12.0 12.1 Gschwind, M.; Salapura, V.; Maischberger, O. (1996). "A Generic Building Block for Hopfield Neural Networks with On-Chip Learning". 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96. pp. 49–52. doi:10.1109/ISCAS.1996.598474. ISBN 0-7803-3073-0. 
  13. Application of the ANNA Neural Network Chip to High-Speed Character Recognition
  14. Gschwind, Michael; Hofstee, H. Peter; Flachs, Brian; Hopkins, Martin; Watanabe, Yukio; Yamazaki, Takeshi (2006). "Synergistic Processing in Cell's Multicore Architecture". IEEE Micro 26 (2): 10–24. doi:10.1109/MM.2006.41. 
  15. De Fabritiis, G. (2007). "Performance of Cell processor for biomolecular simulations". Computer Physics Communications 176 (11–12): 660–664. doi:10.1016/j.cpc.2007.02.107. 
  16. Video Processing and Retrieval on Cell architecture. 
  17. Benthin, Carsten; Wald, Ingo; Scherbaum, Michael; Friedrich, Heiko (2006). 2006 IEEE Symposium on Interactive Ray Tracing. pp. 15–23. doi:10.1109/RT.2006.280210. ISBN 978-1-4244-0693-7. 
  18. "Development of an artificial neural network on a heterogeneous multicore architecture to predict a successful weight loss in obese individuals". 
  19. Kwon, Bomjun; Choi, Taiho; Chung, Heejin; Kim, Geonho (2008). 2008 5th IEEE Consumer Communications and Networking Conference. pp. 1030–1034. doi:10.1109/ccnc08.2007.235. ISBN 978-1-4244-1457-4. 
  20. Duan, Rubing; Strey, Alfred (2008). Euro-Par 2008 – Parallel Processing. Lecture Notes in Computer Science. 5168. pp. 665–675. doi:10.1007/978-3-540-85451-7_71. ISBN 978-3-540-85450-0. 
  21. "Improving the performance of video with AVX". 2012-02-08. 
  22. "microsoft research/pixel shaders/MNIST". 
  23. "how the gpu came to be used for general computation". 
  24. "imagenet classification with deep convolutional neural networks". 
  25. "nvidia driving the development of deep learning". 2016-05-17. 
  26. "nvidia introduces supercomputer for self driving cars". 2016-01-06. 
  27. "how nvlink will enable faster easier multi GPU computing". 2014-11-14. 
  28. "A Survey on Optimized Implementation of Deep Learning Models on the NVIDIA Jetson Platform", 2019
  29. 29.0 29.1 Harris, Mark (May 11, 2017). "CUDA 9 Features Revealed: Volta, Cooperative Groups and More". 
  30. "FPGA Based Deep Learning Accelerators Take on ASICs". 2016-08-23. 
  31. "Project Brainwave" (in en-US). 
  32. "A Survey of FPGA-based Accelerators for Convolutional Neural Networks", Mittal et al., NCAA, 2018
  33. "Google boosts machine learning with its Tensor Processing Unit". 2016-05-19. 
  34. "Chip could bring deep learning to mobile devices". 2016-02-03. 
  35. "Deep Learning with Limited Numerical Precision". 
  36. Rastegari, Mohammad; Ordonez, Vicente; Redmon, Joseph; Farhadi, Ali (2016). "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks". arXiv:1603.05279 [cs.CV].
  37. Khari Johnson (2018-05-23). "Intel unveils Nervana Neural Net L-1000 for accelerated AI training". VentureBeat. Retrieved 2018-05-23. "...Intel will be extending bfloat16 support across our AI product lines, including Intel Xeon processors and Intel FPGAs." 
  38. Michael Feldman (2018-05-23). "Intel Lays Out New Roadmap for AI Portfolio". TOP500 Supercomputer Sites. Retrieved 2018-05-23. "Intel plans to support this format across all their AI products, including the Xeon and FPGA lines" 
  39. Lucian Armasu (2018-05-23). "Intel To Launch Spring Crest, Its First Neural Network Processor, In 2019". Tom's Hardware.,37105.html. Retrieved 2018-05-23. "Intel said that the NNP-L1000 would also support bfloat16, a numerical format that’s being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019." 
  40. "Available TensorFlow Ops | Cloud TPU | Google Cloud". Google Cloud. Retrieved 2018-05-23. "This page lists the TensorFlow Python APIs and graph operators available on Cloud TPU." 
  41. Elmar Haußmann (2018-04-26). "Comparing Google's TPUv2 against Nvidia's V100 on ResNet-50". RiseML Blog. Retrieved 2018-05-23. "For the Cloud TPU, Google recommended we use the bfloat16 implementation from the official TPU repository with TensorFlow 1.7.0. Both the TPU and GPU implementations make use of mixed-precision computation on the respective architecture and store most tensors with half-precision." 
  42. Tensorflow Authors (2018-02-28). "ResNet-50 using BFloat16 on TPU". Google. Retrieved 2018-05-23. 
  43. Joshua V. Dillon; Ian Langmore; Dustin Tran; Eugene Brevdo; Srinivas Vasudevan; Dave Moore; Brian Patton; Alex Alemi et al. (2017-11-28). TensorFlow Distributions (Report). Accessed 2018-05-23. "All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts" 
  44. Abu Sebastian; Tomas Tuma; Nikolaos Papandreou; Manuel Le Gallo; Lukas Kull; Thomas Parnell; Evangelos Eleftheriou (2017). "Temporal correlation detection using computational phase-change memory". Nature Communications 8. doi:10.1038/s41467-017-01481-9. PMID 29062022. 
  45. "A new brain-inspired architecture could improve how computers handle data and advance AI". American Institute of Physics. 2018-10-03. 
  46. Carlos Ríos; Nathan Youngblood; Zengguang Cheng; Manuel Le Gallo; Wolfram H.P. Pernice; C David Wright; Abu Sebastian; Harish Bhaskaran (2018). "In-memory computing on a photonic platform". arXiv:1801.06228 [cs.ET].
  47. "NVIDIA launches the World's First Graphics Processing Unit, the GeForce 256". 
  48. "drive px". 
  49. "design of a machine vision system for weed control". 
  50. "qualcomm research brings server class machine learning to every data devices". October 2015. 
  51. "movidius powers worlds most intelligent drone". 2016-03-16. 

External links

Hostmonster hHosting Tier.Net hosting HandWiki ads