View source for History of artificial neural networks

{{short description|Aspect of history}}
{{Machine learning|Artificial neural network}}

Artificial neural networks (ANNs) are models created using [[Machine learning|machine learning]] to perform a number of tasks. Their creation was inspired by [[Neural circuit|neural circuitry]].<ref name="rosenblatt-1959">{{cite journal|last=Rosenblatt|first=F.|year=1958|title=The Perceptron: A Probabilistic Model For Information Storage And Organization In The Brain|journal=Psychological Review|volume=65|issue=6|pages=386–408|citeseerx=10.1.1.588.3775|doi=10.1037/h0042519|pmid=13602029|s2cid=12781225 }}</ref>{{refn|group=lower-alpha|Neurons generate an [[Biology:Action potential|action potential]]—the release of neurotransmitters that are chemical inputs to other neurons—based on the sum of its incoming chemical inputs.}} While some of the computational implementations ANNs relate to earlier discoveries in mathematics, the first implementation of ANNs was by psychologist [[Biography:Frank Rosenblatt|Frank Rosenblatt]], who developed the [[Perceptron|perceptron]].<ref name="rosenblatt-1959"/> Little research was conducted on ANNs in the 1970s and 1980s, with the [[Organization:Association for the Advancement of Artificial Intelligence|AAAI]] calling that period an "[[AI winter]]".<ref>{{Crevier 1993}}</ref> 

Later, advances in hardware and the development of the [[Backpropagation|backpropagation]] algorithm as well as recurrent neural networks and convolutional neural networks, renewed interest in ANNs. The 2010s, saw the development of a deep neural network (a neural network with many layers) called [[AlexNet]].<ref>{{Cite journal|last1=Krizhevsky|first1=Alex|last2=Sutskever|first2=Ilya|last3=Hinton|first3=Geoffrey E.|date=2017-05-24|title=ImageNet classification with deep convolutional neural networks|url=https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf|journal=Communications of the ACM|volume=60|issue=6|pages=84–90|doi=10.1145/3065386|s2cid=195908774|issn=0001-0782|doi-access=free}}</ref> It greatly outperformed other image recognition models, and is thought to have launched the ongoing AI spring, and further increasing interest in ANNs.<ref name =":1">{{Cite web|url=https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/|title=The data that transformed AI research—and possibly the world|first=Dave|last=Gershgorn|website=Quartz|date=26 July 2017 }}</ref> The transformer architecture was first described in 2017 as a method to teach ANNs grammatical dependencies in language,<ref name="2017_Attention_Is_All_You_Need">{{cite journal |last1=Vaswani |first1=Ashish |last2=Shazeer |first2=Noam |last3=Parmar |first3=Niki |last4=Uszkoreit |first4=Jakob |last5=Jones |first5=Llion |last6=Gomez |first6=Aidan N |last7=Kaiser |first7=Łukasz |last8=Polosukhin |first8=Illia |date=2017 |title=Attention is All you Need |url=https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf |journal=Advances in Neural Information Processing Systems |publisher=Curran Associates, Inc. |volume=30}}</ref> and is the predominant architecture used by large language models, such as [[GPT-4]]. Diffusion models were first described in 2015, and began to be used by image generation models such as [[Software:DALL-E|DALL-E]] in the 2020s.{{cn|date=January 2024}}

== Linear neural network ==

The simplest kind of [[Feedforward neural network|feedforward neural network]] is a linear network, which consists of a single layer of output nodes; the inputs are fed directly to the outputs via a series of weights. The sum of the products of the weights and the inputs is calculated in each node. The [[Mean squared error|mean squared error]]s between these calculated outputs and a given target values are minimized by creating an adjustment to the weights. This technique has been known for over two centuries as the method of least squares or [[Linear regression|linear regression]]. It was used as a means of finding a good rough linear fit to a set of points by [[Biography:Adrien-Marie Legendre|Legendre]] (1805) and Gauss (1795) for the prediction of planetary movement.<ref name="legendre1805">Mansfield Merriman, "A List of Writings Relating to the Method of Least Squares"</ref><ref name="gauss1795">{{cite journal |first=Stephen M. |last=Stigler |year=1981 |title=Gauss and the Invention of Least Squares |journal=Ann. Stat. |volume=9 |issue=3 |pages=465–474 |doi=10.1214/aos/1176345451 |doi-access=free }}</ref><ref name=brertscher>{{cite book |last=Bretscher |first=Otto |title=Linear Algebra With Applications |edition=3rd |publisher=Prentice Hall |year=1995 |location=Upper Saddle River, NJ}}</ref><ref name=DLhistory/><ref name=stigler>
{{cite book |last = Stigler
 |first      = Stephen M.
 |year       = 1986
 |title      = The History of Statistics: The Measurement of Uncertainty before 1900
 |location   = Cambridge
 |publisher  = Harvard
 |isbn       = 0-674-40340-1
 |url-access = registration
 |url        = https://archive.org/details/historyofstatist00stig
}}</ref>

== Perceptrons and other early neural networks ==

Warren McCulloch and [[Biography:Walter Pitts|Walter Pitts]]<ref>{{cite journal|last=McCulloch|first=Warren|author2=Walter Pitts|title=A Logical Calculus of Ideas Immanent in Nervous Activity|journal=Bulletin of Mathematical Biophysics|year=1943|volume=5|issue=4|pages=115–133|doi=10.1007/BF02478259|pmid=<!--none-->}}</ref> (1943) also considered a non-learning computational model for neural networks.<ref>{{Cite news|url=https://www.degruyter.com/view/books/9781400882618/9781400882618-002/9781400882618-002.xml|title=Representation of Events in Nerve Nets and Finite Automata|last=Kleene|first=S.C.|work=Annals of Mathematics Studies|access-date=17 June 2017|publisher=Princeton University Press|year=1956|issue=34|pages=3–41}}</ref> This model paved the way for research to split into two approaches. One approach focused on biological processes while the other focused on the application of neural networks to [[Artificial intelligence|artificial intelligence]]. This work led to work on nerve networks and their link to finite automata.<ref>{{Cite news|url=https://www.degruyter.com/view/books/9781400882618/9781400882618-002/9781400882618-002.xml|title=Representation of Events in Nerve Nets and Finite Automata|last=Kleene|first=S.C.|work=Annals of Mathematics Studies|access-date=2017-06-17|publisher=Princeton University Press|year=1956|issue=34|pages=3–41}}</ref>

In the early 1940s, [[Biography:Donald O. Hebb|D. O. Hebb]]<ref>{{cite book|url={{google books |plainurl=y |id=ddB4AgAAQBAJ}}|title=The Organization of Behavior|last=Hebb|first=Donald|publisher=Wiley|year=1949|isbn=978-1-135-63190-1|location=New York}}</ref> created a learning hypothesis based on the mechanism of [[Biology:Neuroplasticity|neural plasticity]] that became known as Hebbian learning. Hebbian learning is [[Unsupervised learning|unsupervised learning]]. This evolved into models for [[Biology:Long-term potentiation|long-term potentiation]]. Researchers started applying these ideas to computational models in 1948 with Turing's B-type machines. Farley and [[Biography:Wesley A. Clark|Clark]]<ref>{{cite journal|last=Farley|first=B.G.|author2=W.A. Clark|year=1954|title=Simulation of Self-Organizing Systems by Digital Computer|journal=IRE Transactions on Information Theory|volume=4|issue=4|pages=76–84|doi=10.1109/TIT.1954.1057468}}</ref> (1954) first used computational machines, then called "calculators", to simulate a Hebbian network. Other neural network computational machines were created by [[Biography:Nathaniel Rochester (computer scientist)|Rochester]], Holland, Habit and Duda (1956).<ref>{{cite journal|last=Rochester|first=N.|author2=J.H. Holland|author3=L.H. Habit|author4=W.L. Duda|year=1956|title=Tests on a cell assembly theory of the action of the brain, using a large digital computer|journal=IRE Transactions on Information Theory|volume=2|issue=3|pages=80–93|doi=10.1109/TIT.1956.1056810}}</ref> 
 
[[Biography:Frank Rosenblatt|Rosenblatt]]<ref name="rosenblatt-1959"/> (1958) created the [[Perceptron|perceptron]], an algorithm for pattern recognition. With mathematical notation, Rosenblatt described circuitry not in the basic perceptron, such as the exclusive-or circuit that could not be processed by neural networks at the time. In 1959, a biological model proposed by Nobel laureates Hubel and [[Biography:Torsten Wiesel|Wiesel]] was based on their discovery of two types of cells in the primary visual cortex: [[Biology:Simple cell|simple cell]]s and [[Biology:Complex cell|complex cell]]s.<ref>{{cite book|url={{google books |plainurl=y |id=8YrxWojxUA4C|page=106}}|title=Brain and visual perception: the story of a 25-year collaboration|author=David H. Hubel and Torsten N. Wiesel|publisher=Oxford University Press US|year=2005|isbn=978-0-19-517618-6|page=106}}</ref>

Some say that research stagnated following [[Biography:Marvin Minsky|Minsky]] and [[Biography:Seymour Papert|Papert]] (1969),<ref>{{cite book|url={{google books |plainurl=y |id=Ow1OAQAAIAAJ}}|title=Perceptrons: An Introduction to Computational Geometry|last1=Minsky|first1=Marvin|last2=Papert|first2=Seymour|publisher=MIT Press|year=1969|isbn=978-0-262-63022-1}}</ref> who discovered that basic perceptrons were incapable of processing the exclusive-or circuit and that computers lacked sufficient power to process useful neural networks. However, by the time this book came out, methods for training multilayer perceptrons (MLPs) by [[Deep learning|deep learning]] were already known.<ref name=DLhistory />

== First deep learning ==

The first [[Deep learning|deep learning]] MLP was published by Alexey Grigorevich Ivakhnenko and Valentin Lapa in 1965, as the [[Group method of data handling|Group Method of Data Handling]].<ref name="SCHIDHUB2">{{cite journal|last=Schmidhuber|first=J.|year=2015|title=Deep Learning in Neural Networks: An Overview|journal=Neural Networks|volume=61|pages=85–117|arxiv=1404.7828|doi=10.1016/j.neunet.2014.09.003|pmid=25462637|s2cid=11715509}}</ref><ref name="ivak1965">{{cite book|url={{google books |plainurl=y |id=FhwVNQAACAAJ}}|title=Cybernetic Predicting Devices|last=Ivakhnenko|first=A. G.|publisher=CCM Information Corporation|year=1973}}</ref><ref name="ivak1967">{{cite book|url={{google books |plainurl=y |id=rGFgAAAAMAAJ}}|title=Cybernetics and forecasting techniques|last2=Grigorʹevich Lapa|first2=Valentin|publisher=American Elsevier Pub. Co.|year=1967|first1=A. G.|last1=Ivakhnenko}}</ref> This method employs incremental layer by layer training based on [[Regression analysis|regression analysis]], where useless units in hidden layers are pruned with the help of a validation set.

The first [[Deep learning|deep learning]] MLP trained by [[Stochastic gradient descent|stochastic gradient descent]]<ref name="robbins1951">{{Cite journal | last1 = Robbins | first1 = H. |  last2 = Monro | first2 = S. | doi = 10.1214/aoms/1177729586 | title = A Stochastic Approximation Method | journal = The Annals of Mathematical Statistics | volume = 22 | issue = 3 | pages = 400 | year = 1951 | doi-access = free }}</ref> was published in 1967 by Shun'ichi Amari.<ref name="Amari1967">{{cite journal |last1=Amari |first1=Shun'ichi |author-link=Shun'ichi Amari|title=A theory of adaptive pattern classifier|journal= IEEE Transactions |date=1967 |volume=EC |issue=16 |pages=279–307}}</ref><ref name=DLhistory />
In computer experiments conducted by Amari's student Saito, a five layer MLP with two modifiable layers learned  useful internal representations to classify non-linearily separable pattern classes.<ref name=DLhistory />

== Backpropagation ==

{{main|Backpropagation}}

The [[Backpropagation|backpropagation]] algorithm is an efficient application of the [[Biography:Gottfried Wilhelm Leibniz|Leibniz]] [[Chain rule|chain rule]] (1673)<ref name="leibniz1676">{{Cite book|last=Leibniz|first=Gottfried Wilhelm Freiherr von|url=https://books.google.com/books?id=bOIGAAAAYAAJ&q=leibniz+altered+manuscripts&pg=PA90|title=The Early Mathematical Manuscripts of Leibniz: Translated from the Latin Texts Published by Carl Immanuel Gerhardt with Critical and Historical Notes (Leibniz published the chain rule in a 1676 memoir)|date=1920|publisher=Open court publishing Company|isbn=9780598818461 |language=en}}</ref> to networks of differentiable nodes.<ref name=DLhistory/> It is also known as 
the reverse mode of [[Automatic differentiation|automatic differentiation]] or reverse accumulation, due to Seppo Linnainmaa (1970).<ref name="lin1970">{{cite thesis|first=Seppo|last=Linnainmaa|year=1970|type=Masters|title=The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors|language=fi|publisher=University of Helsinki|pages=6–7}}</ref><ref name="lin1976">{{cite journal|last1=Linnainmaa|first1=Seppo|author-link=Seppo Linnainmaa|year=1976|title=Taylor expansion of the accumulated rounding error|journal=BIT Numerical Mathematics|volume=16|issue=2|pages=146–160|doi=10.1007/bf01931367|s2cid=122357351}}</ref><ref name="grie2012">{{cite book |last=Griewank |first=Andreas |year=2012 |chapter=Who Invented the Reverse Mode of Differentiation? |title=Optimization Stories |series=Documenta Matematica, Extra Volume ISMP |pages=389–400 |s2cid=15568746 }}</ref><ref name="grie2008">{{cite book|url={{google books |plainurl=y |id=xoiiLaRxcbEC}}|title=Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, Second Edition|last1=Griewank|first1=Andreas|last2=Walther|first2=Andrea|publisher=SIAM|year=2008|isbn=978-0-89871-776-1}}</ref><ref name=DLhistory /> The term "back-propagating errors" was introduced in 1962 by [[Biography:Frank Rosenblatt|Frank Rosenblatt]],<ref name="rosenblatt1962">{{cite book|last=Rosenblatt|first=Frank|author-link=Frank Rosenblatt|title=Principles of Neurodynamics|year=1962|publisher=Spartan, New York}}</ref><ref name=DLhistory /> but he did not have an implementation of this procedure, although [[Biography:Henry J. Kelley|Henry J. Kelley]] had a continuous precursor of backpropagation<ref name="kelley1960">{{cite journal|last1=Kelley|first1=Henry J.|author-link=Henry J. Kelley|year=1960|title=Gradient theory of optimal flight paths|journal=ARS Journal|volume=30|issue=10|pages=947–954|doi=10.2514/8.5282}}</ref> already in 1960 in the context of [[Control theory|control theory]].<ref name=DLhistory /> In 1982, [[Biography:Paul Werbos|Paul Werbos]] applied backpropagation to MLPs in the way that has become standard.<ref name="werbos1982">{{Cite book|title=System modeling and optimization|last=Werbos|first=Paul|publisher=Springer|year=1982|pages=762–770|chapter=Applications of advances in nonlinear sensitivity analysis|author-link=Paul Werbos|chapter-url=http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf|access-date=2 July 2017|archive-date=14 April 2016|archive-url=https://web.archive.org/web/20160414055503/http://werbos.com/Neural/SensitivityIFIPSeptember1981.pdf|url-status=live}}</ref> In 1986, David E. Rumelhart et al. published an experimental analysis of the technique.<ref name="rumelhart1986">Rumelhart, David E., Geoffrey E. Hinton, and R. J. Williams. "[https://apps.dtic.mil/dtic/tr/fulltext/u2/a164453.pdf Learning Internal Representations by Error Propagation]". David E. Rumelhart, James L. McClelland, and the PDP research group. (editors), Parallel distributed processing: Explorations in the microstructure of cognition, Volume 1: Foundation. MIT Press, 1986.</ref>

== Recurrent network architectures ==

{{main|Recurrent neural network}}

[[Biography:Wilhelm Lenz|Wilhelm Lenz]] and [[Biography:Ernst Ising|Ernst Ising]] created and analyzed the [[Physics:Ising model|Ising model]] (1925)<ref name="brush67">{{cite journal |doi=10.1103/RevModPhys.39.883|title=History of the Lenz-Ising Model|year=1967|last1=Brush|first1=Stephen G.|journal=Reviews of Modern Physics|volume=39|issue=4|pages=883–893|bibcode=1967RvMP...39..883B}}</ref> which is essentially a non-learning artificial [[Recurrent neural network|recurrent neural network]] (RNN) consisting of neuron-like threshold elements.<ref name=DLhistory>{{cite arXiv|last=Schmidhuber|first=Juergen|date=2022|title=Annotated History of Modern AI and Deep Learning |class=cs.NE|eprint=2212.11279}}</ref> In 1972, Shun'ichi Amari made this architecture adaptive.<ref name="Amari1972">{{cite journal |last1=Amari |first1=Shun-Ichi |title=Learning patterns and pattern sequences by self-organizing nets of threshold elements|journal= IEEE Transactions |date=1972 |volume=C |issue=21 |pages=1197–1206 }}</ref><ref name=DLhistory /> His learning RNN was popularised by [[Biography:John Hopfield|John Hopfield]] in 1982.<ref name="Hopfield1982">{{cite journal |last1=Hopfield |first1=J. J. |title=Neural networks and physical systems with emergent collective computational abilities |journal= Proceedings of the National Academy of Sciences|date=1982 |volume=79 |issue=8 |pages=2554–2558 |doi=10.1073/pnas.79.8.2554 |pmid=6953413 |pmc=346238 |bibcode=1982PNAS...79.2554H |doi-access=free }}</ref>

== Self-organizing maps ==

{{main|Self-organizing map}}

[[Self-organizing map]]s (SOMs) were described by [[Biography:Teuvo Kohonen|Teuvo Kohonen]] in 1982.<ref name="KohonenMap">{{cite journal |title= Kohonen Network |last1= Kohonen |first1= Teuvo |last2= Honkela |first2= Timo |year= 2007 |journal= Scholarpedia |volume= 2 |issue= 1 |pages= 1568 |doi= 10.4249/scholarpedia.1568 |bibcode= 2007SchpJ...2.1568K |doi-access= free }}</ref><ref>{{cite journal |last= Kohonen |first= Teuvo |year= 1982 |title= Self-Organized Formation of Topologically Correct Feature Maps |journal= Biological Cybernetics |volume= 43 |number= 1 |pages= 59–69 |doi= 10.1007/bf00337288|s2cid= 206775459 }}</ref> SOMs are neurophysiologically inspired<ref>{{cite journal | last1 = Von der Malsburg | first1 = C | year = 1973 | title = Self-organization of orientation sensitive cells in the striate cortex | journal = Kybernetik | volume = 14 | issue = 2| pages = 85–100 | doi=10.1007/bf00288907| pmid = 4786750 | s2cid = 3351573 }}</ref> [[Artificial neural network|artificial neural network]]s that learn [[Dimensionality reduction|low-dimensional]] representations of high-dimensional data while preserving the [[Topology|topological structure]] of the data. They are trained using [[Competitive learning|competitive learning]].

SOMs create internal representations reminiscent of the [[Cortical homunculus|cortical homunculus]],<ref>{{Cite web|title=Homunculus {{!}} Meaning & Definition in UK English {{!}} Lexico.com|url=https://www.lexico.com/definition/homunculus|archive-url=https://web.archive.org/web/20210518054743/https://www.lexico.com/definition/homunculus|url-status=dead|archive-date=May 18, 2021|access-date=6 February 2022|website=Lexico Dictionaries {{!}} English|language=en}}</ref> a distorted representation of the [[Medicine:Human body|human body]], based on a neurological "map" of the areas and proportions of the [[Biology:Human brain|human brain]] dedicated to processing [[Philosophy:Sensory processing|sensory function]]s, for different parts of the body.

== Convolutional neural networks (CNNs) ==

{{main|Convolutional neural network}}

The origin of the CNN architecture is the "[[Neocognitron|neocognitron]]"<ref name=fukuneoscholar>{{cite journal |last1=Fukushima |first1=K. |year=2007 |title=Neocognitron |journal=Scholarpedia |volume=2 |issue=1 |page=1717 |doi=10.4249/scholarpedia.1717 |bibcode=2007SchpJ...2.1717F |doi-access=free}}</ref> introduced by [[Biography:Kunihiko Fukushima|Kunihiko Fukushima]] in 1980.<ref name="intro">{{cite journal |last=Fukushima |first=Kunihiko |title=Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position |journal=Biological Cybernetics |year=1980 |volume=36 |issue=4 |pages=193–202 |url=https://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf |access-date=16 November 2013 |doi=10.1007/BF00344251 |pmid=7370364 |s2cid=206775608}}</ref><ref>{{cite journal |first1=Yann |last1=LeCun |first2=Yoshua |last2=Bengio |first3=Geoffrey |last3=Hinton |title=Deep learning |journal=Nature |volume=521 |issue=7553 |year=2015 |pages=436–444 |doi=10.1038/nature14539 |pmid=26017442 |bibcode=2015Natur.521..436L |s2cid=3074096}}</ref>
It was inspired by work of  Hubel and [[Biography:Torsten Wiesel|Wiesel]] in the 1950s and 1960s which showed that cat [[Biology:Visual cortex|visual cortices]] contain neurons that individually respond to small regions of the [[Biology:Visual field|visual field]].
The neocognitron introduced the two basic types of layers in CNNs: convolutional layers, and downsampling layers. A convolutional layer contains units whose receptive fields cover a patch of the previous layer. The weight vector (the set of adaptive parameters) of such a unit is often called a filter. Units can share filters. Downsampling layers contain units whose receptive fields cover patches of previous convolutional layers. Such a unit typically computes the average of the activations of the units in its patch. This downsampling helps to correctly classify objects in visual scenes even when the objects are shifted.

In 1969, [[Biography:Kunihiko Fukushima|Kunihiko Fukushima]] also introduced the [[Rectifier (neural networks)|ReLU]] (rectified linear unit) [[Activation function|activation function]].<ref name="Fukushima1969">{{cite journal |first1=K. |last1=Fukushima |title=Visual feature extraction by a multilayered network of analog threshold elements |journal=IEEE Transactions on Systems Science and Cybernetics |volume=5 |issue=4 |date=1969 |pages=322–333 |doi=10.1109/TSSC.1969.300225}}</ref><ref name=DLhistory /> The rectifier has become the most popular activation function for CNNs and  [[Deep learning|deep neural networks]] in general.<ref>{{cite arXiv |last1=Ramachandran |first1=Prajit |last2=Barret |first2=Zoph |last3=Quoc |first3=V. Le |date=October 16, 2017 |title=Searching for Activation Functions |eprint=1710.05941 |class=cs.NE}}</ref>

The [[Time delay neural network|time delay neural network]] (TDNN) was introduced in 1987 by [[Biography:Alex Waibel|Alex Waibel]]  and was one of the first CNNs, as it achieved shift invariance.<ref name=Waibel1987>{{cite conference |title=Phoneme Recognition Using Time-Delay Neural Networks |last1=Waibel |first1=Alex |date=December 1987 |location=Tokyo, Japan |conference=Meeting of the Institute of Electrical, Information and Communication Engineers (IEICE)}}</ref> It did so by utilizing weight sharing in combination with [[Backpropagation|backpropagation]] training.<ref name="speechsignal">[[Biography:Alex Waibel|Alexander Waibel]] et al., ''[http://www.inf.ufrgs.br/~engel/data/media/file/cmp121/waibel89_TDNN.pdf Phoneme Recognition Using Time-Delay Neural Networks]'' IEEE Transactions on Acoustics, Speech, and Signal Processing, Volume 37, No. 3, pp. 328. – 339 March 1989.</ref> Thus, while also using a pyramidal structure as in the neocognitron, it performed a global optimization of the weights instead of a local one.<ref name=Waibel1987/>

In 1988, Wei Zhang et al. applied [[Backpropagation|backpropagation]] 
to a CNN (a simplified Neocognitron with convolutional interconnections between the image feature layers and the last fully connected layer) for alphabet recognition. They also proposed an implementation of the CNN with an optical computing system.<ref name="wz1988">{{cite journal |last=Zhang |first=Wei |date=1988 |title=Shift-invariant pattern recognition neural network and its optical architecture |url=https://drive.google.com/file/d/1nN_5odSG_QVae54EsQN_qSz-0ZsX6wA0/view?usp=sharing |journal=Proceedings of Annual Conference of the Japan Society of Applied Physics}}</ref><ref name="wz1990">{{cite journal |last=Zhang |first=Wei |date=1990 |title=Parallel distributed processing model with local space-invariant interconnections and its optical architecture |url=https://drive.google.com/file/d/0B65v6Wo67Tk5ODRzZmhSR29VeDg/view?usp=sharing |journal=Applied Optics |volume=29 |issue=32 |pages=4790–7 |doi=10.1364/AO.29.004790 |pmid=20577468 |bibcode=1990ApOpt..29.4790Z}}</ref>

In 1989, [[Biography:Yann LeCun|Yann LeCun]] et al. trained a CNN with the purpose of [[Software:Handwriting recognition|recognizing handwritten ZIP code]]s on mail. While the algorithm worked, training required 3 days.<ref name="LECUN1989">LeCun ''et al.'', "Backpropagation Applied to Handwritten Zip Code Recognition," ''Neural Computation'', 1, pp. 541–551, 1989.</ref> Learning was fully automatic, performed better than manual coefficient design, and was suited to a broader range of image recognition problems and image types.
Subsequently, Wei Zhang, et al. modified their model by removing the last fully connected layer and applied it for medical image object segmentation in 1991<ref>{{cite journal |last=Zhang |first=Wei |date=1991 |title=Image processing of human corneal endothelium based on a learning network |url=https://drive.google.com/file/d/0B65v6Wo67Tk5cm5DTlNGd0NPUmM/view?usp=sharing |journal=Applied Optics |volume=30 |issue=29 |pages=4211–7 |doi=10.1364/AO.30.004211 |pmid=20706526 |bibcode=1991ApOpt..30.4211Z}}</ref> and breast cancer detection in mammograms in 1994.<ref>{{cite journal |last=Zhang |first=Wei |date=1994 |title=Computerized detection of clustered microcalcifications in digital mammograms using a shift-invariant artificial neural network |url=https://drive.google.com/file/d/0B65v6Wo67Tk5Ml9qeW5nQ3poVTQ/view?usp=sharing |journal=Medical Physics |volume=21 |issue=4 |pages=517–24 |doi=10.1118/1.597177 |pmid=8058017 |bibcode=1994MedPh..21..517Z}}</ref>

In 1990 Yamaguchi et al. introduced max-pooling, a fixed filtering operation that calculates and propagates the maximum value of a given region. They combined TDNNs with max-pooling in order to realize a speaker independent isolated word recognition system.<ref name=Yamaguchi111990>{{cite conference |title=A Neural Network for Speaker-Independent Isolated Word Recognition |last1=Yamaguchi |first1=Kouichi |last2=Sakamoto |first2=Kenji |last3=Akabane |first3=Toshio |last4=Fujimoto |first4=Yoshiji |date=November 1990 |location=Kobe, Japan |conference=First International Conference on Spoken Language Processing (ICSLP 90) |url=https://www.isca-speech.org/archive/icslp_1990/i90_1077.html |access-date=2019-09-04 |archive-date=2021-03-07 |archive-url=https://web.archive.org/web/20210307233750/https://www.isca-speech.org/archive/icslp_1990/i90_1077.html |url-status=dead }}</ref> 
In a variant of the neocognitron called the cresceptron, instead of using Fukushima's spatial averaging, J. Weng et al. also used max-pooling where a downsampling unit computes the maximum of the activations of the units in its patch.<ref name="Weng1992">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronIJCNN1992.pdf Cresceptron: a self-organizing neural network which grows adaptively]," ''Proc. International Joint Conference on Neural Networks'', Baltimore, Maryland, vol I, pp. 576–581, June, 1992.</ref><ref name="Weng19932">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronICCV1993.pdf Learning recognition and segmentation of 3-D objects from 2-D images]," ''Proc. 4th International Conf. Computer Vision'', Berlin, Germany, pp. 121–128, May, 1993.</ref><ref name="Weng1997">J. Weng, N. Ahuja and T. S. Huang, "[http://www.cse.msu.edu/~weng/research/CresceptronIJCV.pdf Learning recognition and segmentation using the Cresceptron]," ''International Journal of Computer Vision'', vol. 25, no. 2, pp. 105–139, Nov. 1997.</ref><ref name="weng1993">{{cite book |first1=J |last1=Weng |first2=N |last2=Ahuja |first3=TS |last3=Huang |title=1993 (4th) International Conference on Computer Vision |chapter=Learning recognition and segmentation of 3-D objects from 2-D images |s2cid=8619176 |journal=Proc. 4th International Conf. Computer Vision |year=1993 |pages=121–128 |doi=10.1109/ICCV.1993.378228 |isbn=0-8186-3870-2}}</ref> Max-pooling is often used in modern CNNs.<ref name="schdeepscholar">{{cite journal |last1=Schmidhuber |first1=Jürgen |title=Deep Learning |journal=Scholarpedia |url=http://www.scholarpedia.org/article/Deep_Learning |date=2015 |volume=10 |issue=11 |pages=1527–54 |pmid=16764513 |doi=10.1162/neco.2006.18.7.1527 |citeseerx=10.1.1.76.1541 |s2cid=2309950}}</ref>

LeNet-5, a 7-level CNN by [[Biography:Yann LeCun|Yann LeCun]] et al. in 1998,<ref name="lecun98">{{cite journal |last=LeCun |first=Yann |author2=Léon Bottou |author3=Yoshua Bengio |author4=Patrick Haffner |title=Gradient-based learning applied to document recognition |journal=Proceedings of the IEEE |year=1998 |volume=86 |issue=11 |pages=2278–2324 |url=http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf |access-date=October 7, 2016 |doi=10.1109/5.726791 |citeseerx=10.1.1.32.9552|s2cid=14542261 }}</ref> that classifies digits, was applied by several banks to recognize hand-written numbers on checks ({{Lang-en-GB|cheques}}) digitized in 32x32 pixel images. The ability to process higher-resolution images requires larger and more layers of CNNs, so this technique is constrained by the availability of computing resources.

In 2010, Backpropagation training through [[Convolutional neural network#Pooling layer|max-pooling]] was accelerated by GPUs and shown to perform better than other pooling variants.<ref name="Scherer2010">Dominik Scherer, Andreas C. Müller, and Sven Behnke: "[https://www.ais.uni-bonn.de/papers/icann2010_maxpool.pdf Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition]," ''In 20th International Conference Artificial Neural Networks (ICANN)'', pp. 92–101, 2010. {{doi|10.1007/978-3-642-15825-4_10}}.</ref>
Behnke (2003) relied only on the sign of the gradient ([[Rprop]])<ref>{{cite book|url=http://www.ais.uni-bonn.de/books/LNCS2766.pdf|title=Hierarchical Neural Networks for Image Interpretation.|author=Sven Behnke|publisher=Springer|year=2003|series=Lecture Notes in Computer Science|volume=2766}}</ref> on problems such as image reconstruction and face localization. Rprop is a first-order optimization [[Algorithm|algorithm]] created by Martin Riedmiller and Heinrich Braun in 1992.<ref name="riedmiller1992">Martin Riedmiller und Heinrich Braun: Rprop – A Fast Adaptive Learning Algorithm. Proceedings of the International Symposium on Computer and Information Science VII, 1992</ref>

In 2011, a deep GPU-based CNN called "DanNet" by Dan Ciresan, Ueli Meier, and Juergen Schmidhuber achieved human-competitive performance for the first time in computer vision contests.<ref name=":92" /> Subsequently, a similar GPU-based CNN by [[Biography:Alex Krizhevsky|Alex Krizhevsky]], [[Biography:Ilya Sutskever|Ilya Sutskever]], and [[Biography:Geoffrey Hinton|Geoffrey Hinton]] won the ImageNet Large Scale Visual Recognition Challenge 2012.<ref name="krizhevsky2012"/> A very deep CNN with over 100 layers by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun of [[Company:Microsoft|Microsoft]] won the ImageNet 2015 contest.<ref>{{cite book |last1=He |first1=Kaiming |last2=Zhang |first2=Xiangyu |last3=Ren |first3=Shaoqing |last4=Sun |first4=Jian |title=2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) |chapter=Deep Residual Learning for Image Recognition |pages=770–778 |date=2016 |chapter-url=https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf |doi=10.1109/CVPR.2016.90 |arxiv=1512.03385 |isbn=978-1-4673-8851-1 |s2cid=206594692}}</ref>

ANNs were able to guarantee shift invariance to deal with small and large natural objects in large cluttered scenes, only when invariance extended beyond shift, to all ANN-learned concepts, such as location, type (object class label), scale, lighting and others. This was realized in Developmental Networks (DNs)<ref name="Weng2011">J. Weng, "[http://www.cse.msu.edu/~weng/research/WhyPass-Weng-NI-2011.pdf Why Have We Passed 'Neural Networks Do not Abstract Well'?]," ''Natural Intelligence: the INNS Magazine'', vol. 1, no.1, pp. 13–22, 2011.</ref> whose embodiments are Where-What Networks, WWN-1 (2008)<ref name="Weng08">Z. Ji, J. Weng, and D. Prokhorov, "[http://www.cse.msu.edu/~weng/research/ICDL08_0077.pdf Where-What Network 1: Where and What Assist Each Other Through Top-down Connections]," ''Proc. 7th International Conference on Development and Learning (ICDL'08)'', Monterey, CA, Aug. 9–12, pp. 1–6, 2008.</ref> through WWN-7 (2013).<ref name="Weng13">X. Wu, G. Guo, and J. Weng, "[http://www.cse.msu.edu/~weng/research/WWN7-Wu-ICBM-2013.pdf Skull-closed Autonomous Development: WWN-7 Dealing with Scales]," ''Proc. International Conference on Brain-Mind'', July 27–28, East Lansing, Michigan, pp. 1–9, 2013.</ref>

== Artificial curiosity and generative adversarial networks ==

{{main|Generative adversarial network}}

In 1991, Juergen Schmidhuber published adversarial [[Neural network|neural network]]s that contest with each other in the form of a zero-sum game, where one network's gain is the other network's loss.<ref name="curiosity1991">{{cite conference| title = A possibility for implementing curiosity and boredom in model-building neural controllers | last1 = Schmidhuber | first1 = Jürgen |  date = 1991 | publisher = MIT Press/Bradford Books| book-title = Proc. SAB'1991| pages = 222–227}}</ref><ref name="fun2010">{{cite journal|last1=Schmidhuber|first1=Jürgen|author-link=Jürgen Schmidhuber|year=2010|title=Formal Theory of Creativity, Fun, and Intrinsic Motivation (1990-2010)|journal= IEEE Transactions on Autonomous Mental Development|volume=2|issue=3|pages=230–247|doi=10.1109/TAMD.2010.2056368 |s2cid=234198 }}</ref><ref name="gancurpm2020">{{Cite journal|last=Schmidhuber|first=Jürgen| author-link = Juergen Schmidhuber |date=2020|title=Generative Adversarial Networks are Special Cases of Artificial Curiosity (1990) and also Closely Related to Predictability Minimization (1991)|journal=Neural Networks |language=en|volume=127|pages=58–66|doi=10.1016/j.neunet.2020.04.008 |pmid=32334341 |arxiv=1906.04493 |s2cid=216056336 }}</ref> The first network is a [[Generative model|generative model]] that models a [[Probability distribution|probability distribution]] over output patterns. The second network learns by [[Gradient descent|gradient descent]] to predict the reactions of the environment to these patterns. This was called "artificial curiosity." Earlier adversarial machine learning systems "neither involved unsupervised neural networks nor were about modeling data nor used gradient descent."<ref name="gancurpm2020"/>

In 2014, this adversarial principle was used in a [[Generative adversarial network|generative adversarial network]] (GAN) by [[Biography:Ian Goodfellow|Ian Goodfellow]] et al.<ref name="GANnips">{{cite conference|last1=Goodfellow|first1=Ian|last2=Pouget-Abadie|first2=Jean|last3=Mirza|first3=Mehdi|last4=Xu|first4=Bing|last5=Warde-Farley|first5=David|last6=Ozair|first6=Sherjil|last7=Courville|first7=Aaron|last8=Bengio|first8=Yoshua|year=2014|title=Generative Adversarial Networks|url=https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf|conference=Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014)|pages=2672–2680|access-date=20 August 2019|archive-date=22 November 2019|archive-url=https://web.archive.org/web/20191122034612/http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf|url-status=live}}</ref> Here the environmental reaction is 1 or 0 depending on whether the first network's output is in a given set. This can be used to create realistic [[Deepfake|deepfake]]s.<ref>{{Cite web|url=https://lab.witness.org/projects/synthetic-media-and-deep-fakes/|title=Prepare, Don't Panic: Synthetic Media and Deepfakes|publisher=witness.org|access-date=25 November 2020|archive-date=2 December 2020|archive-url=https://web.archive.org/web/20201202231744/https://lab.witness.org/projects/synthetic-media-and-deep-fakes/|url-status=live}}</ref>

In 1992, Schmidhuber also published another type of gradient-based adversarial neural networks where the goal of the zero-sum game is to create disentangled representations of input patterns. This was called ''predictability minimization.''<ref name="pm1992">{{Cite journal|last=Schmidhuber|first=Jürgen | date=November 1992|title=Learning Factorial Codes by Predictability Minimization|journal=Neural Computation|language=en|volume=4|issue=6|pages=863–879|doi=10.1162/neco.1992.4.6.863|s2cid=42023620 }}</ref><ref name="pm1996">{{Cite journal|last1=Schmidhuber|first1=Jürgen|last2=Eldracher|first2=Martin|last3=Foltin|first3=Bernhard|date=1996|title=Semilinear predictability minimzation produces well-known feature detectors|journal=Neural Computation|language=en|volume=8|issue=4|pages=773–786|doi=10.1162/neco.1996.8.4.773 |s2cid=16154391 }}</ref>

[[Company:Nvidia|Nvidia]]'s [[StyleGAN]] (2018)<ref name="SyncedReview2018">{{Cite web |url=https://syncedreview.com/2018/12/14/gan-2-0-nvidias-hyperrealistic-face-generator/ |title=GAN 2.0: NVIDIA's Hyperrealistic Face Generator |date=December 14, 2018 |website=SyncedReview.com|access-date=October 3, 2019}}</ref> is based on the Progressive GAN by Tero Karras, Timo Aila, Samuli  Laine, and Jaakko Lehtinen.<ref name="progressiveGAN2017">{{Cite journal |last1=Karras |first1=Tero |last2=Aila |first2=Timo |last3=Laine |first3=Samuli |last4=Lehtinen |first4=Jaakko |date=October 1, 2017 |title=Progressive Growing of GANs for Improved Quality, Stability, and Variation |arxiv=1710.10196 |url=https://ui.adsabs.harvard.edu/abs/2017arXiv171010196K}}</ref> Here the GAN generator is grown from small to large scale in a pyramidal fashion. StyleGANs improve consistency between fine and coarse details in the generator network.

== Transformers and their variants ==

{{main|Transformer (machine learning model)}}

Many modern large language models such as [[Software:ChatGPT|ChatGPT]], [[GPT-4]], and [[BERT (language model)|BERT]] use a [[Feedforward neural network|feedforward neural network]] called [[Transformer (machine learning model)|Transformer]] by Ashish Vaswani et. al. in their 2017 paper "Attention Is All You Need."<ref name="vaswani2017">{{cite arXiv|last8=Polosukhin|first8=Illia|last7=Kaiser|first7=Lukasz|last6=Gomez|first6=Aidan N.|last5=Jones|first5=Llion|last4=Uszkoreit|first4=Jakob|last3=Parmar|first3=Niki|last2=Shazeer|first2=Noam|last1=Vaswani|first1=Ashish|date=2017-06-12|title=Attention Is All You Need|eprint=1706.03762|class=cs.CL}}</ref>
Transformers have increasingly become the model of choice for [[Natural language processing|natural language processing]] problems,<ref name="wolf2020">{{cite book|last1=Wolf|first1=Thomas|last2=Debut|first2=Lysandre|last3=Sanh|first3=Victor|last4=Chaumond|first4=Julien|last5=Delangue|first5=Clement|last6=Moi|first6=Anthony|last7=Cistac|first7=Pierric|last8=Rault|first8=Tim|last9=Louf|first9=Remi|last10=Funtowicz|first10=Morgan|last11=Davison|first11=Joe|last12=Shleifer|first12=Sam|last13=von Platen|first13=Patrick|last14=Ma|first14=Clara|last15=Jernite|first15=Yacine|last16=Plu|first16=Julien|last17=Xu|first17=Canwen|last18=Le Scao|first18=Teven|last19=Gugger|first19=Sylvain|last20=Drame|first20=Mariama|last21=Lhoest|first21=Quentin|last22=Rush|first22=Alexander|title=Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations|chapter=Transformers: State-of-the-Art Natural Language Processing|year=2020|pages=38–45|doi=10.18653/v1/2020.emnlp-demos.6|s2cid=208117506}}</ref> replacing [[Recurrent neural network|recurrent neural network]]s (RNNs) such as [[Long short-term memory|long short-term memory]] (LSTM).<ref name="lstm1997">{{Cite journal |last1=Hochreiter|first1=Sepp|last2=Schmidhuber|first2=Jürgen|s2cid=1915014|author-link2=Jürgen Schmidhuber|date=1 November 1997|title=Long Short-Term Memory|journal=Neural Computation|volume=9|issue=8 |pages=1735–1780 |doi=10.1162/neco.1997.9.8.1735|pmid=9377276|issn=0899-7667}}</ref>

Basic ideas for this go back a long way: in 1992, Juergen Schmidhuber published the Transformer with "linearized self-attention" (save for a normalization operator),<ref name="transform1992">{{Cite journal |last1=Schmidhuber|first1=Jürgen|date=1 November 1992|title=Learning to control fast-weight memories: an alternative to recurrent nets.|journal=Neural Computation|volume=4|issue=1 |pages=131–139|doi=10.1162/neco.1992.4.1.131 |s2cid=16683347 }}</ref> 
which is also called the "linear Transformer."<ref name="choromanski2020">{{cite arXiv|eprint=2009.14794|class=cs.CL|last1=Choromanski |first1=Krzysztof |last2=Likhosherstov |first2=Valerii |last3=Dohan |first3=David |last4=Song |first4=Xingyou |last5=Gane |first5=Andreea |last6=Sarlos |first6=Tamas |last7=Hawkins |first7=Peter |last8=Davis |first8=Jared |last9=Mohiuddin |first9=Afroz |last10=Kaiser |first10=Lukasz |last11=Belanger |first11=David |last12=Colwell |first12=Lucy |last13=Weller |first13=Adrian |title=Rethinking Attention with Performers |year=2020 }}</ref><ref name="schlag2021"/><ref name=DLhistory/> He advertised it as an "alternative to RNNs"<ref name="transform1992"/> that can learn "internal spotlights of attention,"<ref name="attention1993">{{Cite conference | last1=Schmidhuber|first1=Jürgen|title= Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets|publisher=Springer|date=1993 |pages=460–463 |book-title=ICANN 1993}}</ref> and experimentally applied it to problems of variable binding.<ref name="transform1992"/> Here a slow [[Feedforward neural network|feedforward neural network]] learns by [[Gradient descent|gradient descent]] to control the fast weights of another neural network through [[Outer product|outer product]]s of self-generated activation patterns called "FROM" and "TO" which in Transformer terminology are called "key" and "value" for "[[Attention (machine learning)|self-attention]]."<ref name="schlag2021">{{Cite conference | last1=Schlag|first1=Imanol| last2=Irie|first2=Kazuki| last3=Schmidhuber|first3=Jürgen|author-link1=Juergen Schmidhuber|title=  Linear Transformers Are Secretly Fast Weight Programmers|publisher=Springer|date=2021 |pages=9355–9366 |book-title= ICML 2021}}</ref> This fast weight "attention mapping" is applied to queries.  The 2017 Transformer<ref name="vaswani2017"/> combines this with a softmax operator and a projection matrix.<ref name=DLhistory/>

Transformers are also increasingly being used in [[Computer vision|computer vision]].<ref>{{cite web |last1=He |first1=Cheng |title=Transformer in CV |url=https://towardsdatascience.com/transformer-in-cv-bbdb58bf335e |website=Transformer in CV |date=31 December 2021 |publisher=Towards Data Science}}</ref>

== Deep learning with unsupervised or self-supervised pre-training ==

In the 1980s, [[Backpropagation|backpropagation]] did not work well for deep FNNs and RNNs. Here the word "deep" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial ''credit assignment path'' (CAP) depth.<ref name="SCHIDHUB">{{cite journal|last=Schmidhuber|first=J.|s2cid=11715509|year=2015|title=Deep Learning in Neural Networks: An Overview|journal=Neural Networks|volume=61|pages=85–117|arxiv=1404.7828|doi=10.1016/j.neunet.2014.09.003|pmid=25462637}}</ref> The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For an FNN, the depth of the CAPs is that of the network and is the number of hidden layers plus one (as the output layer is also parameterized). For RNNs, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited.

To overcome this problem, Juergen Schmidhuber (1992) proposed a self-supervised hierarchy of RNNs pre-trained one level at a time by [[Self-supervised learning|self-supervised learning]].<ref name="schmidhuber1992">{{cite journal |last1=Schmidhuber |first1=Jürgen |year=1992 |title=Learning complex, extended sequences using the principle of history compression |url=ftp://ftp.idsia.ch/pub/juergen/chunker.pdf |journal=Neural Computation |volume=4 |issue=2 |pages=234–242 |doi=10.1162/neco.1992.4.2.234 |s2cid=18271205 }}</ref> This "neural history compressor" uses [[Philosophy:Predictive coding|predictive coding]]  to learn internal representations at multiple self-organizing time scales.<ref name=DLhistory/> 
The deep architecture may be used to reproduce the original data from the top level feature activations.<ref name="schmidhuber1992"/>
The RNN hierarchy can be "collapsed" into a single RNN, by "distilling" a higher level "chunker" network into a lower level "automatizer" network.<ref name="schmidhuber1992" /><ref name=DLhistory/> In 1993, a chunker solved a deep learning task whose CAP depth exceeded 1000.<ref name="schmidhuber1993">{{Cite book |url=ftp://ftp.idsia.ch/pub/juergen/habilitation.pdf |title=Habilitation Thesis |last=Schmidhuber |first=Jürgen |year=1993 }}</ref>
Such history compressors can substantially facilitate downstream supervised deep learning.<ref name=DLhistory/>

[[Biography:Geoffrey Hinton|Geoffrey Hinton]] et al. (2006) proposed learning a high-level internal representation using successive layers of binary or real-valued [[Social:Latent variable|latent variable]]s with a [[Restricted Boltzmann machine|restricted Boltzmann machine]]<ref name="smolensky1986">{{cite book|title=Parallel Distributed Processing: Explorations in the Microstructure of Cognition|last1=Smolensky|first1=P.|year=1986|isbn=9780262680530|editor=D. E. Rumelhart|volume=1|pages=[https://archive.org/details/paralleldistribu00rume/page/194 194–281]|chapter=Information processing in dynamical systems: Foundations of harmony theory.|editor2=J. L. McClelland|editor3=PDP Research Group|chapter-url=http://portal.acm.org/citation.cfm?id=104290|url=https://archive.org/details/paralleldistribu00rume/page/194}}</ref> to model each layer. This RBM is a [[Generative model|generative]] [[Stochastic neural network|stochastic]] [[Feedforward neural network|feedforward neural network]] that can learn a [[Probability distribution|probability distribution]] over its set of inputs. Once sufficiently many layers have been learned, the deep architecture may be used as a [[Generative model|generative model]] by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations.<ref name="hinton2006">{{cite journal|last1=Hinton|first1=G. E.|last2=Osindero|first2=S.|last3=Teh|first3=Y.|year=2006|title=A fast learning algorithm for deep belief nets|url=http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf|journal=Neural Computation|volume=18|issue=7|pages=1527–1554|citeseerx=10.1.1.76.1541|doi=10.1162/neco.2006.18.7.1527|pmid=16764513|s2cid=2309950|author-link1=Geoffrey Hinton}}</ref><ref name="hinton2009">{{Cite journal|last=Hinton|first=Geoffrey|date=2009-05-31|title=Deep belief networks|journal=Scholarpedia|volume=4|issue=5|pages=5947|bibcode=2009SchpJ...4.5947H|doi=10.4249/scholarpedia.5947 |doi-access=free |issn=1941-6016}}</ref> In 2012, [[Biography:Andrew Ng|Andrew Ng]] and [[Biography:Jeff Dean (computer scientist)|Jeff Dean]] created an FNN that learned to recognize higher-level concepts, such as cats, only from watching unlabeled images taken from [[Company:YouTube|YouTube]] videos.<ref name="ng2012">{{cite arXiv|eprint=1112.6209|class=cs.LG|first1=Andrew|last1=Ng|first2=Jeff|last2=Dean|title=Building High-level Features Using Large Scale Unsupervised Learning|year=2012}}</ref>

== The vanishing gradient problem and its solutions==

{{main|Long short-term memory}}

[[Biography:Sepp Hochreiter|Sepp Hochreiter]]'s diploma thesis (1991)<ref name="HOCH1991">S. Hochreiter., "[http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf Untersuchungen zu dynamischen neuronalen Netzen] {{Webarchive|url=https://web.archive.org/web/20150306075401/http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidhuber.pdf |date=2015-03-06 }}," ''Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber'', 1991.</ref> was called "one of the most important documents in the history of machine learning" by his supervisor Juergen Schmidhuber.<ref name=DLhistory/> Hochreiter not only tested the neural history compressor,<ref name="schmidhuber1992"/> but also identified and analyzed the [[Vanishing gradient problem|vanishing gradient problem]].<ref name="HOCH1991"/><ref name="HOCH2001">{{cite book|chapter-url={{google books |plainurl=y |id=NWOcMVA64aAC}}|title=A Field Guide to Dynamical Recurrent Networks|last=Hochreiter|first=S.|display-authors=0|date=15 January 2001|publisher=John Wiley & Sons|isbn=978-0-7803-5369-5|chapter=Gradient flow in recurrent nets: the difficulty of learning long-term dependencies|editor-last2=Kremer|editor-first2=Stefan C.|editor-first1=John F.|editor-last1=Kolen}}</ref> He proposed recurrent [[Residual neural network|residual]] connections to solve this problem. This led to the deep learning method called [[Long short-term memory|long short-term memory]] (LSTM), published in 1997.<ref name=":0">{{Cite journal|last1=Hochreiter|first1=Sepp|last2=Schmidhuber|first2=Jürgen|s2cid=1915014|date=1 November 1997|title=Long Short-Term Memory|journal=Neural Computation|volume=9|issue=8|pages=1735–1780|doi=10.1162/neco.1997.9.8.1735|issn=0899-7667|pmid=9377276}}</ref> LSTM [[Recurrent neural network|recurrent neural network]]s can learn "very deep learning" tasks<ref name="SCHIDHUB" /> with long credit assignment paths that require memories of events that happened thousands of discrete time steps before. The "vanilla LSTM" with forget gate was introduced in 1999 by [[Biography:Felix Gers|Felix Gers]], Schmidhuber and Fred Cummins.<ref name="lstm1999">{{Cite book |doi = 10.1049/cp:19991218|chapter = Learning to forget: Continual prediction with LSTM|title = 9th International Conference on Artificial Neural Networks: ICANN '99|volume = 1999|pages = 850–855|year = 1999|last1 = Gers|first1 = Felix| last2 = Schmidhuber|first2 = Jürgen| last3 = Cummins|first3 = Fred| isbn = 0-85296-721-7}}</ref> LSTM has become the  most cited neural network of the 20th century.<ref name=DLhistory/>

In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used LSTM principles to create the [[Highway network]], a [[Feedforward neural network|feedforward neural network]] with hundreds of layers, much deeper than previous networks.<ref name="highway2015">{{cite arXiv|last1=Srivastava|first1=Rupesh Kumar|last2=Greff|first2=Klaus|last3=Schmidhuber|first3=Jürgen|title=Highway Networks|eprint=1505.00387|date=2 May 2015|class=cs.LG}}</ref><ref name="highway2015neurips">{{cite journal|last1=Srivastava|first1=Rupesh K|last2=Greff|first2=Klaus|last3=Schmidhuber|first3=Juergen|title=Training Very Deep Networks|journal=Advances in Neural Information Processing Systems |date=2015|volume=28|pages=2377–2385|url=http://papers.nips.cc/paper/5850-training-very-deep-networks|publisher=Curran Associates, Inc.}}</ref> 7 months later, Kaiming He, Xiangyu Zhang;  Shaoqing Ren, and Jian Sun won the ImageNet 2015 competition with an open-gated or gateless [[Highway network]] variant called [[Residual neural network]].<ref name="resnet2015">{{Cite conference|last1=He|first1=Kaiming|last2=Zhang|first2=Xiangyu|last3=Ren|first3=Shaoqing|last4=Sun|first4=Jian|date=2016|title=Deep Residual Learning for Image Recognition|url=https://ieeexplore.ieee.org/document/7780459|journal=2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)|location=Las Vegas, NV, USA|publisher=IEEE|pages=770–778|arxiv=1512.03385|doi=10.1109/CVPR.2016.90|isbn=978-1-4673-8851-1}}</ref> This has become the most cited neural network of the 21st century.<ref name=DLhistory />

In 2011, Xavier Glorot, Antoine Bordes and [[Biography:Yoshua Bengio|Yoshua Bengio]] found that the [[Rectifier (neural networks)|ReLU]]<ref name="Fukushima1969"/> of [[Biography:Kunihiko Fukushima|Kunihiko Fukushima]] also helps to overcome the vanishing gradient problem,<ref name="glorot2011">{{cite conference |author1=Xavier Glorot |author2=Antoine Bordes |author3=[[Biography:Yoshua Bengio|Yoshua Bengio]] |year=2011 |title=Deep sparse rectifier neural networks |url=http://jmlr.org/proceedings/papers/v15/glorot11a/glorot11a.pdf |conference=AISTATS |quote=Rectifier and softplus activation functions. The second one is a smooth version of the first. }}</ref> compared to widely used activation functions prior to 2011.

== Hardware-based designs ==
The development of metal–oxide–semiconductor (MOS) [[Engineering:Very-large-scale integration|very-large-scale integration]] (VLSI), combining millions or [[Engineering:List of Nvidia graphics processing units#Volta series|billions]] of MOS transistors onto a single chip in the form of complementary MOS (CMOS) technology, enabled the development of practical artificial neural networks in the 1980s.<ref name="Mead">{{cite book|url=http://fennetic.net/irc/Christopher%20R.%20Carroll%20Carver%20Mead%20Mohammed%20Ismail%20Analog%20VLSI%20Implementation%20of%20Neural%20Systems.pdf|title=Analog VLSI Implementation of Neural Systems|date=8 May 1989|publisher=Kluwer Academic Publishers|isbn=978-1-4613-1639-8|last1=Mead|first1=Carver A.|last2=Ismail|first2=Mohammed|series=The Kluwer International Series in Engineering and Computer Science|volume=80|location=Norwell, MA|doi=10.1007/978-1-4613-1639-8}}</ref>

Computational devices were created in [[Engineering:CMOS|CMOS]], for both biophysical simulation and neuromorphic computing inspired by the structure and function of the human brain. Nanodevices<ref>{{cite journal|last1=Yang|first1=J. J.|last2=Pickett|first2=M. D.|last3=Li|first3=X. M.|last4=Ohlberg|first4=D. A. A.|last5=Stewart|first5=D. R.|last6=Williams|first6=R. S.|year=2008|title=Memristive switching mechanism for metal/oxide/metal nanodevices|journal=Nat. Nanotechnol.|volume=3|issue=7|pages=429–433|doi=10.1038/nnano.2008.160|pmid=18654568}}</ref> for very large scale principal components analyses and [[Convolution|convolution]] may create a new class of neural computing because they are fundamentally [[Engineering:Analog signal|analog]] rather than [[Digital data|digital]] (even though the first implementations may use digital devices).<ref>{{cite journal|last1=Strukov|first1=D. B.|last2=Snider|first2=G. S.|last3=Stewart|first3=D. R.|last4=Williams|first4=R. S.|year=2008|title=The missing memristor found|journal=Nature|volume=453|issue=7191|pages=80–83|bibcode=2008Natur.453...80S|doi=10.1038/nature06932|pmid=18451858|s2cid=4367148}}</ref> Ciresan and colleagues (2010)<ref name=":3">{{Cite journal|last1=Cireşan|first1=Dan Claudiu|last2=Meier|first2=Ueli|last3=Gambardella|first3=Luca Maria|last4=Schmidhuber|first4=Jürgen|date=2010-09-21|title=Deep, Big, Simple Neural Nets for Handwritten Digit Recognition|journal=Neural Computation|volume=22|issue=12|pages=3207–3220|arxiv=1003.0358|doi=10.1162/neco_a_00052|issn=0899-7667|pmid=20858131|s2cid=1918673}}</ref> in Schmidhuber's group showed that despite the vanishing gradient problem, GPUs make backpropagation feasible for many-layered feedforward neural networks.

== Contests ==
Between 2009 and 2012, [[Recurrent neural network|recurrent neural network]]s and deep feedforward neural networks developed in [[Biography:Jürgen Schmidhuber|Schmidhuber]]'s research group won eight international competitions in [[Pattern recognition|pattern recognition]] and [[Machine learning|machine learning]].<ref>[http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions 2012 Kurzweil AI Interview] {{Webarchive|url=https://web.archive.org/web/20180831075249/http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions |date=2018-08-31 }} with [[Biography:Jürgen Schmidhuber|Jürgen Schmidhuber]] on the eight competitions won by his Deep Learning team 2009–2012</ref><ref>{{Cite web|url=http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions|title=How bio-inspired deep learning keeps winning competitions {{!}} KurzweilAI|website=www.kurzweilai.net|access-date=2017-06-16|archive-url=https://web.archive.org/web/20180831075249/http://www.kurzweilai.net/how-bio-inspired-deep-learning-keeps-winning-competitions|archive-date=2018-08-31|url-status=dead}}</ref> For example, the bi-directional and multi-dimensional [[Long short-term memory|long short-term memory]] (LSTM)<ref>Graves, Alex; and Schmidhuber, Jürgen; ''[http://www.idsia.ch/~juergen/nips2009.pdf Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks]'', in  ''Advances in Neural Information Processing Systems 22 (NIPS'22), 7–10 December 2009, Vancouver, BC'', Neural Information Processing Systems (NIPS) Foundation, 2009, pp. 545–552.</ref><ref name="graves 855">{{cite journal|last1=Graves|first1=A.|last2=Liwicki|first2=M.|last3=Fernandez|first3=S.|last4=Bertolami|first4=R.|last5=Bunke|first5=H.|last6=Schmidhuber|first6=J.|year=2009|title=A Novel Connectionist System for Improved Unconstrained Handwriting Recognition|url=http://www.idsia.ch/~juergen/tpami_2008.pdf|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=31|issue=5|pages=855–868|citeseerx=10.1.1.139.4502|doi=10.1109/tpami.2008.137|pmid=19299860|s2cid=14635907}}</ref><ref name="graves20093">{{Cite journal|last1=Graves|first1=Alex|last2=Schmidhuber|first2=Jürgen|date=2009|editor-last=Bengio|editor-first=Yoshua|title=Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks|url=https://papers.nips.cc/paper/3449-offline-handwriting-recognition-with-multidimensional-recurrent-neural-networks|journal=Neural Information Processing Systems (NIPS) Foundation|volume=21 |publisher=Curran Associates, Inc|pages=545–552|editor-last2=Schuurmans|editor-first2=Dale|editor-last3=Lafferty|editor-first3=John|editor-last4=Williams|editor-first4=Chris|editor-last5=Culotta|editor-first5=Aron}}</ref><ref>{{Cite journal|last1=Graves|first1=A.|last2=Liwicki|first2=M.|last3=Fernández|first3=S.|last4=Bertolami|first4=R.|last5=Bunke|first5=H.|last6=Schmidhuber|first6=J.|date=May 2009|title=A Novel Connectionist System for Unconstrained Handwriting Recognition|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=31|issue=5|pages=855–868|citeseerx=10.1.1.139.4502|doi=10.1109/tpami.2008.137|issn=0162-8828|pmid=19299860|s2cid=14635907}}</ref> of [[Biography:Alex Graves (computer scientist)|Graves]] et al. won three competitions in connected handwriting recognition at the 2009 International Conference on Document Analysis and Recognition (ICDAR), without any prior knowledge about the three languages to be learned.<ref name="graves20093" /><ref name="graves 855" />

Ciresan and colleagues won [[Pattern recognition|pattern recognition]] contests, including the IJCNN 2011 Traffic Sign Recognition Competition,<ref name=":72">{{Cite journal|last1=Cireşan|first1=Dan|last2=Meier|first2=Ueli|last3=Masci|first3=Jonathan|last4=Schmidhuber|first4=Jürgen|date=August 2012|title=Multi-column deep neural network for traffic sign classification|journal=Neural Networks|series=Selected Papers from IJCNN 2011|volume=32|pages=333–338|citeseerx=10.1.1.226.8219|doi=10.1016/j.neunet.2012.02.023|pmid=22386783}}</ref> the ISBI 2012 Segmentation of Neuronal Structures in Electron Microscopy Stacks challenge<ref name=":8">{{Cite book|url=http://papers.nips.cc/paper/4741-deep-neural-networks-segment-neuronal-membranes-in-electron-microscopy-images.pdf|title=Advances in Neural Information Processing Systems 25|last1=Ciresan|first1=Dan|last2=Giusti|first2=Alessandro|last3=Gambardella|first3=Luca M.|last4=Schmidhuber|first4=Juergen|date=2012|publisher=Curran Associates, Inc.|editor-last=Pereira|editor-first=F.|pages=2843–2851|editor-last2=Burges|editor-first2=C. J. C.|editor-last3=Bottou|editor-first3=L.|editor-last4=Weinberger|editor-first4=K. Q.}}</ref> and others. Their neural networks were the first pattern recognizers to achieve human-competitive/superhuman performance<ref name=":92">{{Cite book|last1=Ciresan|first1=Dan|last2=Meier|first2=U.|last3=Schmidhuber|first3=J.|title=2012 IEEE Conference on Computer Vision and Pattern Recognition |chapter=Multi-column deep neural networks for image classification |date=June 2012|isbn=978-1-4673-1228-8|pages=3642–3649|arxiv=1202.2745|bibcode=2012arXiv1202.2745C|citeseerx=10.1.1.300.3283|doi=10.1109/cvpr.2012.6248110|s2cid=2161592}}</ref> on benchmarks such as traffic sign recognition (IJCNN 2012), or the [[MNIST database|MNIST handwritten digits problem]].

Researchers demonstrated (2010) that deep neural networks interfaced to a [[Hidden Markov model|hidden Markov model]] with context-dependent states that define the neural network output layer can drastically reduce errors in large-vocabulary speech recognition tasks such as voice search.{{Citation needed|date=August 2019}}

GPU-based implementations<ref name=":6">{{Cite journal|last1=Ciresan|first1=D. C.|last2=Meier|first2=U.|last3=Masci|first3=J.|last4=Gambardella|first4=L. M.|last5=Schmidhuber|first5=J.|date=2011|title=Flexible, High Performance Convolutional Neural Networks for Image Classification|url=http://ijcai.org/papers11/Papers/IJCAI11-210.pdf|journal=International Joint Conference on Artificial Intelligence|doi=10.5591/978-1-57735-516-8/ijcai11-210}}</ref> of this approach won many pattern recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition,<ref name=":72" /> the ISBI 2012 Segmentation of neuronal structures in EM stacks challenge,<ref name=":8" /> the ImageNet Competition<ref name="krizhevsky2012">{{cite journal|last1=Krizhevsky|first1=Alex|last2=Sutskever|first2=Ilya|last3=Hinton|first3=Geoffry|date=2012|title=ImageNet Classification with Deep Convolutional Neural Networks|url=https://www.cs.toronto.edu/~kriz/imagenet_classification_with_deep_convolutional.pdf|journal=NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada}}</ref> and others.

Deep, highly nonlinear neural architectures similar to the [[Neocognitron|neocognitron]]<ref name="K. Fukushima. Neocognitron 1980">{{cite journal|author=Fukushima, K.|year=1980|title=Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position|journal=Biological Cybernetics|volume=36|issue=4|pages=93–202|doi=10.1007/BF00344251|pmid=7370364|s2cid=206775608}}</ref> and the "standard architecture of vision",<ref>{{cite journal|last1=Riesenhuber|first1=M|last2=Poggio|first2=T|year=1999|title=Hierarchical models of object recognition in cortex|journal=Nature Neuroscience|volume=2|issue=11|pages=1019–1025|doi=10.1038/14819|pmid=10526343|s2cid=8920227}}</ref> inspired by [[Biology:Simple cell|simple]] and [[Biology:Complex cell|complex cell]]s, were pre-trained with unsupervised methods by Hinton.<ref name="hinton2009" /><ref name="hinton2006" /> A team from his lab won a 2012 contest sponsored by Merck to design software to help find molecules that might identify new drugs.<ref>{{cite news|url=https://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html|title=Scientists See Promise in Deep-Learning Programs|last=Markoff|first=John|date=November 23, 2012|newspaper=New York Times}}</ref>

== Notes ==
{{notelist}}

== References ==
{{Reflist}}

== External links ==
* {{Cite web|url=https://drive.google.com/file/d/1f0sPHv7ozHafASPwIOfuvF_RvP3FDPY0|title=Lecun 2019-7-11 ACM Tech Talk|website=Google Docs|access-date=2020-02-13}}

[[Category:Computational statistics]]
[[Category:Artificial neural networks| ]]
[[Category:Computational neuroscience]]

{{Sourceattribution|History of artificial neural networks}}