View source for Softmax function

{{Short description|Smooth approximation of one-hot arg max}}
{{About|the smooth approximation of one-hot arg max|the smooth approximations of max|Smooth maximum}}
{{Redirect|Softmax|the Korean video game company|ESA (company)}}
{{Machine learning}}
The '''softmax function,''' also known as '''softargmax'''<ref>{{cite book |last1=Goodfellow |first1=Ian |url=http://www.deeplearningbook.org |title=Deep Learning |last2=Bengio |first2=Yoshua |last3=Courville |first3=Aaron |publisher=MIT Press |year=2016 |isbn=978-0-26203561-3 |pages=180–184 |section=6.2.2.3 Softmax Units for Multinoulli Output Distributions |chapter-url=https://www.deeplearningbook.org/contents/mlp.html#pf15 |authorlink2=Yoshua Bengio}}</ref>{{rp|184}} or '''normalized exponential function''',<ref name="bishop" />{{rp|198}} converts a [[Tuple|tuple]] of {{mvar|K}} real numbers into a [[Probability distribution|probability distribution]] over {{mvar|K}} possible outcomes. It is a generalization of the [[Logistic function|logistic function]] to multiple dimensions, and is used in [[Multinomial logistic regression|multinomial logistic regression]]. The softmax function is often used as the last [[Activation function|activation function]] of a [[Artificial neural network|neural network]] to normalize the output of a network to a [[Probability distribution|probability distribution]] over predicted output classes.

== Definition ==

The softmax function takes as input a tuple {{mvar|z}} of {{mvar|K}} real numbers, and normalizes it into a [[Probability distribution|probability distribution]] consisting of {{mvar|K}} probabilities proportional to the exponentials of the input numbers. That is, prior to applying softmax, some tuple components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in the [[Interval (mathematics)|interval]] <math>(0, 1)</math>, and the components will add up to 1, so that they can be interpreted as probabilities. Furthermore, the larger input components will correspond to larger probabilities.

Formally, the standard (unit) softmax function <math>\sigma\colon \R^K \to (0, 1)^K</math>, where {{tmath|K > 1}}, takes a tuple <math>\mathbf{z} = (z_1, \dotsc, z_K) \in \R^K</math> and computes each component of vector <math>\sigma(\mathbf{z}) \in (0, 1)^K</math> with

<math display="block">\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}\,.</math>

In words, the softmax applies the standard [[Exponential function|exponential function]] to each element <math>z_i</math> of the input tuple <math>\mathbf z</math> (consisting of <math>K</math> real numbers), and normalizes these values by dividing by the sum of all these exponentials. The normalization ensures that the sum of the components of the output vector <math>\sigma(\mathbf z)</math> is 1. The term "softmax" derives from the amplifying effects of the exponential on any maxima in the input tuple. For example, the standard softmax of <math>(1,2,8)</math> is approximately <math>(0.001,0.002,0.997)</math>, which amounts to assigning almost all of the total unit weight in the result to the position of the tuple's maximal element (of 8).

In general, instead of {{mvar|e}} a different [[Base (exponentiation)|base]] {{math|b > 0}} can be used.  As above, if {{math|b > 1}} then larger input components will result in larger output probabilities, and increasing the value of {{mvar|b}} will create probability distributions that are more concentrated around the positions of the largest input values. Conversely, if {{math|0 < b < 1}} then smaller input components will result in larger output probabilities, and decreasing the value of {{mvar|b}} will create probability distributions that are more concentrated around the positions of the smallest input values. Writing <math>b = e^\beta</math> or <math>b = e^{-\beta}</math>{{efn|Positive {{mvar|β}} corresponds to the maximum convention, and is usual in machine learning, corresponding to the highest score having highest probability. The negative {{math|−β}} corresponds to the minimum convention, and is conventional in thermodynamics, corresponding to the lowest energy state having the highest probability; this matches the convention in the Gibbs distribution, interpreting {{mvar|β}} as coldness.}} (for real {{mvar|β}}){{efn|1=The notation {{mvar|β}} is for the [[Physics:Thermodynamic beta|thermodynamic beta]], which is inverse [[Physics:Temperature|temperature]]: <math>\beta = 1/T</math>, <math>T = 1/\beta.</math>}} yields the expressions:{{efn|For <math>\beta = 0</math> (coldness zero, infinite temperature), <math>b = e^\beta = e^0 = 1</math>, and this becomes the constant function {{tmath|(1/n, \dots, 1/n)}}, corresponding to the [[Discrete uniform distribution|discrete uniform distribution]].}}

<math display="block">\sigma(\mathbf{z})_i = \frac{e^{\beta z_i}}{\sum_{j=1}^K e^{\beta z_j}} \text{ or } \sigma(\mathbf{z})_i = \frac{e^{-\beta z_i}}{\sum_{j=1}^K e^{-\beta z_j}} \text{ for } i = 1, \dotsc , K .</math>

{{anchor|Temperature}}
A value proportional to the reciprocal of {{mvar|β}} is sometimes referred to as the ''temperature'': <math display=inline>\beta = 1 / kT</math>, where {{mvar|k}} is typically 1 or the [[Physics:Boltzmann constant|Boltzmann constant]] and {{mvar|T}} is the temperature. A higher temperature results in a more uniform output distribution (i.e. with higher [[Entropy (information theory)|entropy]]; it is "more random"), while a lower temperature results in a sharper output distribution, with one value dominating.

In some fields, the base is fixed, corresponding to a fixed scale,{{efn|In statistical mechanics, fixing {{mvar|β}} is interpreted as having coldness and temperature of 1.}} while in others the parameter {{mvar|β}} (or {{mvar|T}}) is varied.

== Interpretations ==
=== Smooth arg max ===
{{See also|Arg max}}
The Softmax function is a smooth approximation to the [[Arg max|arg max]] function: the function whose value is the ''index'' of a tuple's largest element. The name "softmax" may be misleading. Softmax is not a [[Smooth maximum|smooth maximum]] (that is, a [[Smooth approximation|smooth approximation]] to the maximum function). The term "softmax" is also used for the closely related [[LogSumExp]] function, which is a smooth maximum. For this reason, some prefer the more accurate term "softargmax", though the term "softmax" is conventional in machine learning.<ref name="sako2018"/>{{sfn|Goodfellow|Bengio|Courville|2016|pp=183–184|ps=: The name "softmax" can be somewhat confusing. The function is more closely related to the arg max function than the max function. The term "soft" derives from the fact that the softmax function is continuous and differentiable. The arg max function, with its result represented as a one-hot vector, is not continuous nor differentiable. The softmax function thus provides a "softened" version of the arg max. The corresponding soft version of the maximum function is <math>\operatorname{softmax}(\mathbf{z})^\top \mathbf{z}</math>. It would perhaps be better to call the softmax function "softargmax," but the current name is an entrenched convention.}} This section uses the term "softargmax" for clarity.

Formally, instead of considering the arg max as a function with categorical output <math>1, \dots, n</math> (corresponding to the index), consider the arg max function with [[One-hot|one-hot]] representation of the output (assuming there is a unique maximum arg):
<math display="block">\operatorname{arg\,max}(z_1,\, \dots,\, z_n) = (y_1,\, \dots,\, y_n) = (0,\, \dots,\, 0,\, 1,\, 0,\, \dots,\, 0),</math>
where the output coordinate <math>y_i = 1</math> if and only if <math>i</math> is the arg max of <math>(z_1, \dots, z_n)</math>, meaning <math>z_i</math> is the unique maximum value of <math>(z_1,\, \dots,\, z_n)</math>. For example, in this encoding <math>\operatorname{arg\,max}(1, 5, 10) = (0, 0, 1),</math> since the third argument is the maximum.

This can be generalized to multiple arg max values (multiple equal <math>z_i</math> being the maximum) by dividing the 1 between all max args; formally {{math|1/k}} where {{mvar|k}} is the number of arguments assuming the maximum. For example, <math>\operatorname{arg\,max}(1,\, 5,\, 5) = (0,\, 1/2,\, 1/2),</math> since the second and third argument are both the maximum. In case all arguments are equal, this is simply <math>\operatorname{arg\,max}(z, \dots, z) = (1/n, \dots, 1/n).</math> Points {{mvar|'''z'''}} with multiple arg max values are [[Singular point of an algebraic variety|singular points]] (or singularities, and form the singular set) – these are the points where arg max is discontinuous (with a jump discontinuity) – while points with a single arg max are known as non-singular or regular points.

With the last expression given in the introduction, softargmax is now a smooth approximation of arg max: as {{tmath|\beta \to \infty}}, softargmax converges to arg max. There are various notions of convergence of a function; softargmax converges to arg max [[Pointwise convergence|pointwise]], meaning for each fixed input {{math|'''z'''}} as {{tmath|\beta \to \infty}}, <math>\sigma_\beta(\mathbf{z}) \to \operatorname{arg\,max}(\mathbf{z}).</math> However, softargmax does not [[Uniform convergence|converge uniformly]] to arg max, meaning intuitively that different points converge at different rates, and may converge arbitrarily slowly. In fact, softargmax is continuous, but arg max is not continuous at the singular set where two coordinates are equal, while the uniform limit of continuous functions is continuous. The reason it fails to converge uniformly is that for inputs where two coordinates are almost equal (and one is the maximum), the arg max is the index of one or the other, so a small change in input yields a large change in output. For example, <math>\sigma_\beta(1,\, 1.0001) \to (0, 1),</math> but <math>\sigma_\beta(1,\, 0.9999) \to (1,\, 0),</math> and <math>\sigma_\beta(1,\, 1) = 1/2</math> for all inputs: the closer the points are to the singular set <math>(x, x)</math>, the slower they converge. However, softargmax does [[Compact convergence|converge compactly]] on the non-singular set.

Conversely, as {{tmath|\beta \to -\infty}}, softargmax converges to arg min in the same way, where here the singular set is points with two arg ''min'' values. In the language of tropical analysis, the softmax is a [[Deformation theory|deformation]] or "quantization" of arg max and arg min, corresponding to using the [[Log semiring|log semiring]] instead of the max-plus semiring (respectively min-plus semiring), and recovering the arg max or arg min by taking the limit is called "tropicalization" or "dequantization".

It is also the case that, for any fixed {{mvar|β}}, if one input {{tmath|z_i}} is much larger than the others ''relative'' to the temperature, <math>T = 1/\beta</math>, the output is approximately the arg max. For example, a difference of 10 is large relative to a temperature of 1:
<math display="block">\sigma(0,\, 10) := \sigma_1(0,\, 10) = \left(1/\left(1 + e^{10}\right),\, e^{10}/\left(1 + e^{10}\right)\right) \approx (0.00005,\, 0.99995)</math>
However, if the difference is small relative to the temperature, the value is not close to the arg max. For example, a difference of 10 is small relative to a temperature of 100:
<math display="block">\sigma_{1/100}(0,\, 10) = \left(1/\left(1 + e^{1/10}\right),\, e^{1/10}/\left(1 + e^{1/10}\right)\right) \approx (0.475,\, 0.525).</math>
As {{tmath|\beta \to \infty}}, temperature goes to zero, <math>T = 1/\beta \to 0</math>, so eventually all differences become large (relative to a shrinking temperature), which gives another interpretation for the limit behavior.

=== Statistical mechanics ===
In [[Physics:Statistical mechanics|statistical mechanics]], the softargmax function is known as the [[Physics:Boltzmann distribution|Boltzmann distribution]] (or Gibbs distribution):<ref>{{cite book
|first1=Yann |last1=LeCun
|first2=Sumit |last2=Chopra
|first3=Raia |last3=Hadsell
|first4=Marc’Aurelio |last4=Ranzato
|first5=Fu Jie |last5=Huang
|chapter=A Tutorial on Energy-Based Learning
|chapter-url=http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf
|isbn=978-0-26202617-8
|publisher=MIT Press
|year=2006
|title=Predicting Structured Data
|url=https://mitpress.mit.edu/books/predicting-structured-data
|series=Neural Information Processing series
|editor1=Gökhan Bakır
|editor2=Thomas Hofmann
|editor3=Bernhard Schölkopf
|editor4=Alexander J. Smola
|editor5=Ben Taskar
|editor6=S.V.N Vishwanathan
}}</ref>{{rp|7}} the index set <math>{1,\, \dots,\, k}</math> are the [[Physics:Microstate (statistical mechanics)|microstates]] of the system; the inputs <math>z_i</math> are the energies of that state; the denominator is known as the [[Physics:Partition function (statistical mechanics)|partition function]], often denoted by {{mvar|Z}}; and the factor {{mvar|β}} is called the coldness (or [[Physics:Thermodynamic beta|thermodynamic beta]], or inverse temperature).

== Applications ==
The softmax function is used in various [[Multiclass classification|multiclass classification]] methods, such as [[Multinomial logistic regression|multinomial logistic regression]] (also known as softmax regression),<ref name="bishop">{{cite book |first=Christopher M. |last=Bishop |year=2006 |title=Pattern Recognition and Machine Learning |publisher=Springer |isbn=0-387-31073-8 }}</ref>{{rp|206–209}}<ref>{{Cite web |title=Unsupervised Feature Learning and Deep Learning Tutorial |url=http://ufldl.stanford.edu/tutorial/supervised/SoftmaxRegression/ |access-date=2024-03-25 |website=ufldl.stanford.edu}}</ref> multiclass [[Linear discriminant analysis|linear discriminant analysis]], [[Naive Bayes classifier|naive Bayes classifier]]s, and [[Artificial neural network|artificial neural network]]s.<ref>ai-faq [http://www.faqs.org/faqs/ai-faq/neural-nets/part2/section-12.html What is a softmax activation function?]</ref> Specifically, in multinomial logistic regression and linear discriminant analysis, the input to the function is the result of {{mvar|K}} distinct [[Linear function|linear function]]s, and the predicted probability for the {{mvar|j}}th class given a sample tuple {{math|'''x'''}} and a weighting vector {{math|'''w'''}} is:

<math display="block">P(y=j\mid \mathbf{x}) = \frac{e^{\mathbf{x}^\mathsf{T}\mathbf{w}_j}}{\sum_{k=1}^K e^{\mathbf{x}^\mathsf{T}\mathbf{w}_k}}</math>

This can be seen as the [[Function composition|composition]] of {{mvar|K}} linear functions <math>\mathbf{x} \mapsto \mathbf{x}^\mathsf{T}\mathbf{w}_1, \ldots, \mathbf{x} \mapsto \mathbf{x}^\mathsf{T}\mathbf{w}_K</math> and the softmax function (where <math>\mathbf{x}^\mathsf{T}\mathbf{w}</math> denotes the inner product of <math>\mathbf{x}</math> and <math>\mathbf{w}</math>). The operation is equivalent to applying a linear operator defined by <math>\mathbf{w}</math> to tuples <math>\mathbf{x}</math>, thus transforming the original, probably highly-dimensional, input to vectors in a {{mvar|K}}-dimensional space <math>\mathbb{R}^K</math>.

=== Neural networks ===
The standard softmax function is often used in the final layer of a neural network-based classifier. Such networks are commonly trained under a [[Cross entropy|log loss]] (or [[Cross-entropy|cross-entropy]]) regime, giving a non-linear variant of multinomial logistic regression.

Since the function maps a tuple and a specific index <math>i</math> to a real value, the derivative needs to take the index into account:

<math display="block">\frac{\partial}{\partial q_k}\sigma(\textbf{q}, i) =  \sigma(\textbf{q}, i)(\delta_{ik} - \sigma(\textbf{q}, k)).</math>

This expression is symmetrical in the indexes <math>i, k</math> and thus may also be expressed as

<math display="block"> \frac{\partial}{\partial q_k}\sigma(\textbf{q}, i) =  \sigma(\textbf{q}, k)(\delta_{ik} - \sigma(\textbf{q}, i)).</math>

Here, the [[Kronecker delta]] is used for simplicity (cf. the derivative of a [[Sigmoid function|sigmoid function]], being expressed via the function itself).

To ensure stable numerical computations subtracting the maximum value from the input tuple is common. This approach, while not altering the output or the derivative theoretically, enhances stability by directly controlling the maximum exponent value computed.

If the function is scaled with the parameter <math>\beta</math>, then these expressions must be multiplied by <math>\beta</math>.

See [[Multinomial logit|multinomial logit]] for a probability model which uses the softmax activation function.

=== Reinforcement learning ===
In the field of [[Reinforcement learning|reinforcement learning]], a softmax function can be used to convert values into action probabilities. The function commonly used is:<ref>Sutton, R. S. and Barto A. G. ''Reinforcement Learning: An Introduction''. The MIT Press, Cambridge, MA, 1998. [http://incompleteideas.net/book/ebook/node17.html Softmax Action Selection]</ref>
<math display="block">P_t(a) = \frac{\exp(q_t(a)/\tau)}{\sum_{i=1}^n\exp(q_t(i)/\tau)} \text{,}</math>

where the action value <math>q_t(a)</math> corresponds to the expected reward of following action a and <math>\tau</math> is called a temperature parameter (in allusion to [[Physics:Statistical mechanics|statistical mechanics]]). For high temperatures (<math>\tau \to \infty</math>), all actions have nearly the same probability and the lower the temperature, the more expected rewards affect the probability. For a low temperature (<math>\tau \to 0^+</math>), the probability of the action with the highest expected reward tends to 1.

== Computational complexity and remedies ==
In neural network applications, the number {{mvar|K}} of possible outcomes is often large, e.g. in case of [[Language model|neural language models]] that predict the most likely outcome out of a vocabulary which might contain millions of possible words.<ref name=":0">{{Cite journal |last1=Onal |first1=Kezban Dilek |last2=Zhang |first2=Ye |last3=Altingovde |first3=Ismail Sengor |last4=Rahman |first4=Md Mustafizur |last5=Karagoz |first5=Pinar |last6=Braylan |first6=Alex |last7=Dang |first7=Brandon |last8=Chang |first8=Heng-Lu |last9=Kim |first9=Henna |last10=McNamara |first10=Quinten |last11=Angert |first11=Aaron |date=2018-06-01 |title=Neural information retrieval: at the end of the early years |journal=Information Retrieval Journal |language=en |volume=21 |issue=2 |pages=111–182 |doi=10.1007/s10791-017-9321-y |s2cid=21684923 |issn=1573-7659|doi-access=free |hdl=11245.1/008d6e8f-df13-4abf-8ae9-6ff2e17377f3 |hdl-access=free }}</ref> This can make the calculations for the softmax layer (i.e. the matrix multiplications to determine the <math>z_i</math>, followed by the application of the softmax function itself) computationally expensive.<ref name=":0" /><ref name=":1">{{Cite journal |last1=Chen |first1=Wenlin |last2=Grangier |first2=David |last3=Auli |first3=Michael |date=August 2016 |title=Strategies for Training Large Vocabulary Neural Language Models |url=https://aclanthology.org/P16-1186 |journal=Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |location=Berlin, Germany |publisher=Association for Computational Linguistics |pages=1975–1985 |doi=10.18653/v1/P16-1186|s2cid=6035643 |doi-access=free |arxiv=1512.04906 }}</ref> What's more, the [[Gradient descent|gradient descent]] [[Backpropagation|backpropagation]] method for training such a neural network involves calculating the softmax for every training example, and the number of training examples can also become large. The computational effort for the softmax became a major limiting factor in the development of larger neural language models, motivating various remedies to reduce training times.<ref name=":0" /><ref name=":1" />

Approaches that reorganize the softmax layer for more efficient calculation include the '''hierarchical softmax''' and the '''differentiated softmax'''.<ref name=":0" /> The hierarchical softmax (introduced by Morin and [[Biography:Yoshua Bengio|Bengio]] in 2005) uses a binary tree structure where the outcomes (vocabulary words) are the leaves and the intermediate nodes are suitably selected "classes" of outcomes, forming [[Social:Latent variable|latent variable]]s.<ref name=":1" /><ref name=":2">{{Cite journal |last1=Morin |first1=Frederic |last2=Bengio |first2=Yoshua |date=2005-01-06 |title=Hierarchical Probabilistic Neural Network Language Model |url=https://www.iro.umontreal.ca/~lisa/pointeurs/hierarchical-nnlm-aistats05.pdf |journal=International Workshop on Artificial Intelligence and Statistics |language=en |publisher=PMLR |pages=246–252}}</ref> The desired probability (softmax value) of a leaf (outcome) can then be calculated as the product of the probabilities of all nodes on the path from the root to that leaf.<ref name=":1" /> Ideally, when the tree is balanced, this would reduce the [[Computational complexity|computational complexity]] from <math>O(K)</math> to <math>O(\log_2 K)</math>.<ref name=":2" /> In practice, results depend on choosing a good strategy for clustering the outcomes into classes.<ref name=":1" /><ref name=":2" /> A Huffman tree was used for this in Google's [[Word2vec|word2vec]] models (introduced in 2013) to achieve scalability.<ref name=":0" />

A second kind of remedies is based on approximating the softmax (during training) with modified loss functions that avoid the calculation of the full normalization factor.<ref name=":0" /> These include methods that restrict the normalization sum to a sample of outcomes (e.g. Importance Sampling, Target Sampling).<ref name=":0" /><ref name=":1" />

== Numerical algorithms ==
The standard softmax is numerically unstable because of large exponentiations. The '''safe softmax''' method calculates instead<math display="block">\sigma(\mathbf{z})_i = \frac{e^{\beta (z_i-m)}}{\sum_{j=1}^K e^{\beta (z_j-m)}}</math>where <math>m = \max_i z_i </math> is the largest factor involved. Subtracting by it guarantees that the exponentiations result in at most 1.

The [[Attention (machine learning)|attention mechanism]] in Transformers takes three arguments: a "query vector" <math>q</math>, a list of "key vectors" <math>k_1, \dots, k_N</math>, and a list of "value vectors" <math>v_1, \dots, v_N</math>, and outputs a softmax-weighted sum over value vectors:<math display="block">o = \sum_{i=1}^N \frac{e^{q^T k_i - m}}{\sum_{j=1}^N e^{q^T k_j - m}} v_i</math>The standard softmax method involves several loops over the inputs, which would be bottlenecked by memory bandwidth. The FlashAttention method is a [[Communication-avoiding algorithm|communication-avoiding algorithm]] that fuses these operations into a single loop, increasing the arithmetic intensity. It is an online algorithm that computes the following quantities:<ref>{{cite arXiv |eprint=1805.02867 |last1=Milakov |first1=Maxim |last2=Gimelshein |first2=Natalia |title=Online normalizer calculation for softmax |date=2018 |class=cs.PF }}</ref><ref>{{Cite journal |last1=Dao |first1=Tri |last2=Fu |first2=Dan |last3=Ermon |first3=Stefano |last4=Rudra |first4=Atri |last5=Ré |first5=Christopher |date=2022-12-06 |title=FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness |url=https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html |journal=Advances in Neural Information Processing Systems |language=en |volume=35 |pages=16344–16359}}</ref><math display="block">\begin{aligned}
z_i &= q^T k_i &\\
m_i &= \max(z_1, \dots, z_i) &=& \max(m_{i-1}, z_i)\\
l_i &= e^{z_1 - m_i} + \dots + e^{z_i - m_i} &=& e^{m_{i-1}-m_i} l_{i-1}  + e^{z_i - m_i}\\
o_i &= e^{z_1 - m_i} v_1 + \dots + e^{z_i - m_i}v_i &=& e^{m_{i-1}-m_i} o_{i-1} + e^{z_i - m_i}v_i
\end{aligned}</math>and returns <math>o_N/l_N</math>. In practice, FlashAttention operates over multiple queries and keys per loop iteration, in a similar way as [[Communication-avoiding algorithm#Blocked (tiled) matrix multiplication|blocked matrix multiplication]]. If [[Backpropagation|backpropagation]] is needed, then the output vectors and the intermediate arrays <math>[m_1, \dots, m_N], [l_1, \dots, l_N]</math> are cached, and during the backward pass, attention matrices are [[Rematerialization|rematerialized]] from these, making it a form of gradient checkpointing.

== Mathematical properties ==
Geometrically the softmax function maps the [[Euclidean space]] <math>\mathbb{R}^K</math> to the [[Boundary (topology)|boundary]] of the standard <math>(K-1)</math>-simplex, cutting the dimension by one (the range is a <math>(K - 1)</math>-dimensional simplex in <math>K</math>-dimensional space), due to the linear constraint that all output sum to 1 meaning it lies on a [[Hyperplane|hyperplane]].

Along the main diagonal <math>(x,\, x,\, \dots,\, x),</math> softmax is just the uniform distribution on outputs, <math>(1/n, \dots, 1/n)</math>: equal scores yield equal probabilities.

More generally, softmax is invariant under translation by the same value in each coordinate: adding <math>\mathbf{c} = (c,\, \dots,\, c)</math> to the inputs <math>\mathbf{z}</math> yields <math>\sigma(\mathbf{z} + \mathbf{c}) = \sigma(\mathbf{z})</math>, because it multiplies each exponent by the same factor, <math>e^c</math> (because <math>e^{z_i + c} = e^{z_i} \cdot e^c</math>), so the ratios do not change:
<math display="block">\sigma(\mathbf{z} + \mathbf{c})_j = \frac{e^{z_j + c}}{\sum_{k=1}^K e^{z_k + c}} = \frac{e^{z_j} \cdot e^c}{\sum_{k=1}^K e^{z_k} \cdot e^c} = \sigma(\mathbf{z})_j.</math>

Geometrically, softmax is constant along diagonals: this is the dimension that is eliminated, and corresponds to the softmax output being independent of a translation in the input scores (a choice of 0 score). One can normalize input scores by assuming that the sum is zero (subtract the average: <math>\mathbf{c}</math> where <math display="inline">c = \frac{1}{n} \sum z_i</math>), and then the softmax takes the hyperplane of points that sum to zero, <math display="inline">\sum z_i = 0</math>, to the open simplex of positive values that sum to 1<math display="inline">\sum \sigma(\mathbf{z})_i = 1</math>, analogously to how the exponent takes 0 to 1, <math>e^0 = 1</math> and is positive.

By contrast, softmax is not invariant under scaling. For instance, <math>\sigma\bigl((0,\, 1)\bigr) = \bigl(1/(1 + e),\, e/(1 + e)\bigr)</math> but <math>\sigma\bigl((0, 2)\bigr) = \bigl(1/\left(1 + e^2\right),\, e^2/\left(1 + e^2\right)\bigr).</math>

The [[Standard logistic function|standard logistic function]] is the special case for a 1-dimensional axis in 2-dimensional space, say the ''x''-axis in the {{math|(x, y)}} plane. One variable is fixed at 0 (say <math>z_2 = 0</math>), so <math>e^0 = 1</math>, and the other variable can vary, denote it <math>z_1 = x</math>, so <math display="inline">e^{z_1}/\sum_{k=1}^2 e^{z_k} = e^x/\left(e^x + 1\right),</math> the standard logistic function, and <math display="inline">e^{z_2}/\sum_{k=1}^2 e^{z_k} = 1/\left(e^x + 1\right),</math> its complement (meaning they add up to 1). The 1-dimensional input could alternatively be expressed as the line <math>(x/2,\, -x/2)</math>, with outputs <math>e^{x/2}/\left(e^{x/2} + e^{-x/2}\right) = e^x/\left(e^x + 1\right)</math> and <math>e^{-x/2}/\left(e^{x/2} + e^{-x/2}\right) = 1/\left(e^x + 1\right).</math>

=== Gradients ===
The softmax function is also the gradient of the [[LogSumExp]] function:<math display="block">\frac{\partial}{\partial z_i} \operatorname{LSE}(\mathbf{z}) = \frac{\exp z_i}{\sum_{j=1}^{K} \exp z_j} = \sigma(\mathbf{z})_i, \quad \text{ for } i = 1, \dotsc , K, \quad \mathbf{z} = (z_1,\, \dotsc,\, z_K) \in\R^K,</math>where the LogSumExp function is defined as <math>\operatorname{LSE}(z_1,\, \dots,\, z_n) = \log\left(\exp(z_1) + \cdots + \exp(z_n)\right)</math>.

The gradient of softmax is thus <math>\partial_{z_j} \sigma_i = \sigma_i (\delta_{ij} - \sigma_j)</math>.

== History ==
The softmax function was used in [[Physics:Statistical mechanics|statistical mechanics]] as the [[Physics:Boltzmann distribution|Boltzmann distribution]] in the foundational paper {{harvtxt|Boltzmann|1868}},<ref>{{cite journal |last=Boltzmann |first=Ludwig |authorlink=Ludwig Boltzmann
|year=1868
|title=Studien über das Gleichgewicht der lebendigen Kraft zwischen bewegten materiellen Punkten
|trans-title=Studies on the balance of living force between moving material points
|journal=Wiener Berichte |volume=58 |pages=517–560
}}</ref> formalized and popularized in the influential textbook {{harvtxt|Gibbs|1902}}.<ref>{{cite journal
|year=1902
|title=Elementary Principles in Statistical Mechanics
|title-link=Elementary Principles in Statistical Mechanics
|journal=Nature |volume=66 |issue=1708 |page=291 |doi=10.1038/066291a0 |bibcode=1902Natur..66..291B }}</ref>

The use of the softmax in [[Decision theory|decision theory]] is credited to [[Biography:R. Duncan Luce|R. Duncan Luce]],<ref name="Gao">{{cite arXiv|eprint=1704.00805|last1=Gao|first1=Bolin|title=On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning|last2=Pavel|first2=Lacra|class=math.OC|year=2017}}</ref>{{rp|1}} who used the axiom of [[Social:Independence of irrelevant alternatives|independence of irrelevant alternatives]] in [[Finance:Rational choice theory|rational choice theory]] to deduce the softmax in [[Philosophy:Luce's choice axiom|Luce's choice axiom]] for relative preferences. 
In machine learning, the term "softmax" is credited to John S. Bridle in two 1989 conference papers, {{harvtxt|Bridle|1990a}}:<ref name="Gao"/>{{rp|1}} and {{harvtxt|Bridle|1990b}}:<ref name="sako2018"/>
{{blockquote
|We are concerned with feed-forward non-linear networks (multi-layer perceptrons, or MLPs) with multiple outputs. We wish to treat the outputs of the network as probabilities of alternatives (''e.g.'' pattern classes), conditioned on the inputs. We look for appropriate output non-linearities and for appropriate criteria for adaptation of the parameters of the network (''e.g.'' weights). We explain two modifications: probability scoring, which is an alternative to squared error minimisation, and a normalised exponential ('''softmax''') multi-input generalisation of the logistic non-linearity.<ref>{{cite conference
|first=John S. |last=Bridle
|year=1990a
|title=Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
|pages=227–236
|conference=Neurocomputing: Algorithms, Architectures and Applications (1989)
|editor1=Soulié F.F.
|editor2=Hérault J. 
|series=NATO ASI Series (Series F: Computer and Systems Sciences)
|volume=68
|publisher=Springer
|location=Berlin, Heidelberg
|doi=10.1007/978-3-642-76153-9_28
}}</ref>{{rp|227}}
}}
{{blockquote
|For any input, the outputs must all be positive and they must sum to unity. ...

Given a set of unconstrained values, {{tmath|V_j(x)}}, we can ensure both conditions by using a Normalised Exponential transformation:
<math display="block">Q_j(x) = \left. e^{V_j(x)} \right/ \sum_k e^{V_k(x)} </math>
This transformation can be considered a multi-input generalisation of the logistic, operating on the whole output layer. It preserves the rank order of its input values, and is a differentiable generalisation of the 'winner-take-all' operation of picking the maximum value. For this reason we like to refer to it as '''softmax'''.<ref>{{cite conference
|first=John S. |last=Bridle
|year=1990b
|title=Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters
|url=https://papers.nips.cc/paper/195-training-stochastic-model-recognition-algorithms-as-networks-can-lead-to-maximum-mutual-information-estimation-of-parameters
|conference=Advances in Neural Information Processing Systems 2 (1989)
|editor = D. S. Touretzky
|publisher = Morgan-Kaufmann
}}</ref>{{rp|213}}
}}

== Example ==
With an input of {{math|(1, 2, 3, 4, 1, 2, 3)}}, the softmax is approximately {{math|(0.024, 0.064, 0.175, 0.475, 0.024, 0.064, 0.175)}}. The output has most of its weight where the "4" was in the original input. This is what the function is normally used for: to highlight the largest values and suppress values which are significantly below the maximum value. But note: a change of ''temperature'' changes the output.  When the temperature is multiplied by 10, the inputs are effectively {{math|(0.1, 0.2, 0.3, 0.4, 0.1, 0.2, 0.3)}} and the softmax is approximately {{math|(0.125, 0.138, 0.153, 0.169, 0.125, 0.138, 0.153)}}. This shows that high temperatures de-emphasize the maximum value.

Computation of this example using [[Python (programming language)|Python]] code:

<syntaxhighlight lang="pycon">
>>> import numpy as np
>>> z = np.array([1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0])
>>> beta = 1.0
>>> np.exp(beta * z) / np.sum(np.exp(beta * z)) 
array([0.02364054, 0.06426166, 0.1746813, 0.474833, 0.02364054,
       0.06426166, 0.1746813])
</syntaxhighlight>

== Alternatives ==
The softmax function generates probability predictions densely distributed over its [[Support (mathematics)|support]]. Other functions like sparsemax or α-entmax can be used when sparse probability predictions are desired.<ref>"Speeding Up Entmax" by Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé, Zhenisbek Assylbekov, https://arxiv.org/abs/2111.06832v3</ref> Also the Gumbel-softmax reparametrization trick can be used when sampling from a discrete-discrete distribution needs to be mimicked in a differentiable manner.

== See also ==
* Softplus
* [[Multinomial logistic regression]]
* [[Dirichlet distribution]] – an alternative way to sample categorical distributions
* [[Physics:Partition function (statistical mechanics)|Partition function]]
* [[Exponential tilting]] – a generalization of Softmax to more general probability distributions

== Notes ==
{{notelist}}

== References ==
{{reflist|refs=
<ref name="sako2018">{{cite web|first=Yusaku |last=Sako
|title=Is the term "softmax" driving you nuts?
|url=https://medium.com/@u39kun/is-the-term-softmax-driving-you-nuts-ee232ab4f6bd
|date=2018-06-02
|publisher=[[Software:Medium (website)|Medium]]
}}</ref>
}}

{{Artificial intelligence navbox}}

[[Category:Computational neuroscience]]
[[Category:Logistic regression]]
[[Category:Artificial neural networks]]
[[Category:Functions and mappings]]
[[Category:Exponentials]]

{{Sourceattribution|Softmax function}}