View source for Hadamard product (matrices)

{{Short description|Matrix operation}}
[[Image:Hadamard product qtl1.svg|thumb|The Hadamard product operates on identically shaped matrices and produces a third matrix of the same dimensions.]]
In [[Mathematics|mathematics]], the '''Hadamard product''' (also known as the '''element-wise product''', '''entrywise product'''<ref name="HornJohnson" />{{rp|ch. 5}} or '''Schur product'''<ref>{{cite journal|last=Davis|first=Chandler|year=1962|title=The norm of the Schur product operation|journal=Numerische Mathematik|volume=4|pages=343–44|doi=10.1007/bf01386329|number=1|s2cid=121027182}}</ref>) is a [[Binary operation|binary operation]] that takes in two [[Matrix (mathematics)|matrices]] of the same dimensions and returns a matrix of the multiplied corresponding elements. This operation can be thought as a "naive matrix multiplication" and is different from the [[Matrix multiplication|matrix product]]. It is attributed to, and named after, either French mathematician [[Biography:Jacques Hadamard|Jacques Hadamard]] or German mathematician [[Biography:Issai Schur|Issai Schur]].

The Hadamard product is associative and [[Distributive property|distributive]]. Unlike the matrix product, it is also commutative.<ref name=":1">{{Cite web|last=Million|first=Elizabeth|date=April 12, 2007|title=The Hadamard Product|url=http://buzzard.ups.edu/courses/2007spring/projects/million-paper.pdf|access-date=September 6, 2020|website=buzzard.ups.edu}}</ref>

==Definition==
For two matrices {{math|''A''}} and {{math|''B''}} of the same dimension {{math|''m'' × ''n''}}, the Hadamard product <math>A \odot B</math> (sometimes <math>A \circ B</math><ref>{{cite web|url=https://machinelearning.wtf/terms/hadamard-product/|title=Hadamard product - Machine Learning Glossary|website=machinelearning.wtf}}</ref><ref>{{cite web|url=https://math.stackexchange.com/q/815315 |title=linear algebra - What does a dot in a circle mean?|website=Mathematics Stack Exchange}}</ref><ref>{{cite web|url=https://math.stackexchange.com/a/601545/688715|title=Element-wise (or pointwise) operations notation?|website=Mathematics Stack Exchange}}</ref>) is a matrix of the same dimension as the operands, with elements given by<ref name=":1" />
:<math>(A \odot B)_{ij} = (A)_{ij} (B)_{ij}.</math>

For matrices of different dimensions ({{math|''m'' × ''n''}} and {{math|''p'' × ''q''}}, where {{math|''m'' ≠ ''p''}} or {{math|''q''}}), the Hadamard product is undefined.

{{Anchor|Example}}For example, the Hadamard product for two arbitrary 2&nbsp;×&nbsp;3 matrices is:
:<math> 
  \begin{bmatrix}
    2 & 3 & 1 \\
    0 & 8 & -2
  \end{bmatrix} \circ \begin{bmatrix}
    3 & 1 & 4 \\
    7 & 9 & 5
  \end{bmatrix} = \begin{bmatrix}
    2 \times 3 & 3 \times 1 & 1 \times 4 \\
    0 \times 7 & 8 \times 9 & -2 \times 5
  \end{bmatrix} = \begin{bmatrix}
    6 & 3 & 4 \\
    0 & 72 & -10
  \end{bmatrix}
</math>

==Properties==
* The Hadamard product is commutative (when working with a commutative ring), associative and [[Distributive property|distributive]] over addition. That is, if ''A'', ''B'', and ''C'' are matrices of the same size, and ''k'' is a scalar: <math display="block">\begin{align}
              A \odot B &= B \odot A, \\
    A \odot (B \odot C) &= (A \odot B) \odot C, \\
        A \odot (B + C) &= A \odot B + A \odot C, \\
\left(kA\right) \odot B &= A \odot \left(kB\right) = k\left(A \odot B\right), \\
              A \odot 0 &= 0 \odot A = 0.
\end{align}</math>
* The identity matrix under Hadamard multiplication of two {{math|''m'' × ''n''}} matrices is an [[Matrix of ones|{{math|''m'' × ''n''}} matrix where all elements are equal to 1]]. This is different from the [[Identity matrix|identity matrix]] under regular matrix multiplication, where only the elements of the main diagonal are equal to 1. Furthermore, a matrix has an inverse under Hadamard multiplication if and only if none of the elements are equal to zero.<ref name="hadamardpdf1">{{cite web|last=Million| first=Elizabeth | title=The Hadamard Product|url=http://buzzard.ups.edu/courses/2007spring/projects/million-paper.pdf|access-date=2 January 2012}}</ref>
* For vectors {{math|'''x'''}} and {{math|'''y'''}}, and corresponding diagonal matrices {{math|''D''<sub>'''x'''</sub>}} and {{math|''D''<sub>'''y'''</sub>}} with these vectors as their main diagonals, the following identity holds:<ref name="HornJohnson">{{cite book|last1=Horn|first1=Roger A.|first2=Charles R.|last2=Johnson|title=Matrix analysis| publisher = Cambridge University Press|date=2012}}</ref>{{rp|479}}  where {{math|'''x'''<sup>*</sup>}} denotes the [[Conjugate transpose|conjugate transpose]] of {{math|'''x'''}}. In particular, using vectors of ones, this shows that the sum of all elements in the Hadamard product is the [[Trace (linear algebra)|trace]] of {{math|''AB''<sup>T</sup>}} where superscript T denotes the matrix transpose, that is, <math>\operatorname{tr}\left(AB^{\mathsf T}\right) = \mathbf{1}^\mathsf{T}\left(A\odot B\right)\mathbf{1}</math>. A related result for square {{mvar|A}} and {{mvar|B}}, is that the row-sums of their Hadamard product are the diagonal elements of {{math|''AB''<sup>T</sup>}}:<ref name="styan1973" /> <math display="block">\sum_i (A \circ B)_{ij} = \left(B^\mathsf{T} A\right)_{jj} = \left(AB^\mathsf{T}\right)_{ii}.</math> Similarly, <math display="block">\left(\mathbf{y}\mathbf{x}^*\right) \odot A = {D}_\mathbf{y} A {D}_\mathbf{x}^*</math> Furthermore, a Hadamard matrix-vector product can be expressed as: <math display="block">(A \odot B) \mathbf{y} = \operatorname{diag}\left( A D_\mathbf{y} B^\mathsf{T} \right)</math> where <math>\operatorname{diag}(M)</math> is the vector formed from the diagonals of matrix {{mvar|M}}.
*The Hadamard product is a principal submatrix of the [[Kronecker product]].<ref>{{cite journal |last1=Liu |first1=Shuangzhe |last2=Trenkler |first2=Götz|year=2008 
|title= Hadamard, Khatri-Rao, Kronecker and other matrix products |journal= International Journal of Information and Systems Sciences |volume=4 |issue=1 |pages=160–177}}</ref><ref>{{cite journal|last1=Liu|first1=Shuangzhe|last2=Leiva|first2=Víctor|last3=Zhuang|first3=Dan|last4=Ma|first4=Tiefeng |last5=Figueroa-Zúñiga|first5=Jorge I.|year=2022|title=Matrix differential calculus with applications in the multivariate linear model and its diagnostics|journal=Journal of Multivariate Analysis|volume=188|pages=104849|doi=10.1016/j.jmva.2021.104849|s2cid=239598156 |doi-access=free}}</ref><ref>{{Cite journal| last1=Liu|first1=Shuangzhe|
last2= Trenkler|first2=Götz| last3=Kollo|first3=Tõnu| 
last4=von Rosen|first4=Dietrich| 
last5=Baksalary|first5=Oskar Maria| 
date= 2023|
title= Professor Heinz Neudecker and matrix differential calculus| 
journal= Statistical Papers| language=en|
doi= 10.1007/s00362-023-01499-w}}</ref>
* The Hadamard product satisfies the rank inequality <math display="block">\operatorname{rank}(A \odot B) \leq \operatorname{rank}(A) \operatorname{rank}(B) </math>
* If {{math|''A''}} and {{math|''B''}} are [[Positive-definite matrix|positive-definite matrices]], then the following inequality involving the Hadamard product holds:<ref>{{cite journal|last1=Hiai|first1=Fumio|last2=Lin|first2=Minghua|title=On an eigenvalue inequality involving the Hadamard product|journal=Linear Algebra and Its Applications|date=February 2017|volume=515| pages = 313–320 | doi=10.1016/j.laa.2016.11.017|doi-access=free}}</ref> <math display="block">\prod_{i=k}^n \lambda_i(A \odot B) \ge \prod_{i=k}^n \lambda_i(A B),\quad k=1,\ldots,n,</math> where {{math|''λ<sub>i</sub>''(''A'')}} is the {{math|''i''}}th largest eigenvalue of {{math|''A''}}.
* If {{mvar|D}} and {{mvar|E}} are diagonal matrices, then<ref>{{cite web|url=http://buzzard.ups.edu/courses/2007spring/projects/million-paper.pdf |title=Project |publisher=buzzard.ups.edu |date=2007 |access-date=2019-12-18}}</ref> <math display="block">\begin{align}
  D (A \odot B) E &= (D A E) \odot B    = (D A) \odot (B E) \\
  &=  (AE) \odot (D B) =    A \odot (D B E).
\end{align}</math>
* The Hadamard product of two vectors  <math>\mathbf a</math> and <math>\mathbf b</math> is the same as matrix multiplication of the corresponding [[Diagonal matrix|diagonal matrix]] of one vector by the other vector: <math display="block">\mathbf a \odot \mathbf b = D _{\mathbf a} \mathbf b = D _{\mathbf b} \mathbf a</math>
* The vector to diagonal matrix [[Diagonal matrix#diag operator|<math>\operatorname{diag}</math> operator]] may be expressed using the Hadamard product as: <math display="block">\operatorname{diag}(\mathbf{a}) = (\mathbf{a} \mathbf{1}^T) \odot I</math> where <math>\mathbf{1}</math> is a constant vector with elements <math>1</math> and <math>I</math> is the [[Identity matrix|identity matrix]].

==The mixed-product property==

<math display="block"> (A \otimes B) \odot (C \otimes D) = (A \odot C) \otimes (B \odot D) ,</math> where <math>\otimes</math> is [[Kronecker product]], assuming <math>A</math> has the same dimensions of <math>C</math> and <math>B</math> with <math>D</math>.

<math display="block"> (A \bull B) \odot (C \bull D) = (A \odot C) \bull (B \odot D) ,</math> where <math>\bull</math> denotes [[Khatri–Rao product#Face-splitting product|face-splitting product]].<ref name="slyusar">{{Cite journal|last=Slyusar|first=V. I. |title=End products in matrices in radar applications. |url=http://slyusar.kiev.ua/en/IZV_1998_3.pdf |journal=Radioelectronics and Communications Systems |year=1998 |volume=41 |issue=3|pages=50–53}}</ref>

<math display="block">(A \bull B)(C \ast D) = (A C) \odot (B D),</math> where <math>\ast</math> is column-wise [[Khatri–Rao product]].

==Schur product theorem==
{{Main|Schur product theorem}}
The Hadamard product of two [[Positive-definite matrix|positive-semidefinite matrices]] is positive-semidefinite.<ref name=":1" /><ref name=styan1973>{{Citation | doi=10.1016/0024-3795(73)90023-2 | last=Styan | first=George P. H. | title=Hadamard Products and Multivariate Statistical Analysis | journal=Linear Algebra and Its Applications | year=1973 | volume=6 | pages=217–240| hdl=10338.dmlcz/102190 | hdl-access=free }}</ref> This is known as the Schur product theorem,<ref name=hadamardpdf1/> after Russian mathematician [[Biography:Issai Schur|Issai Schur]]. For two positive-semidefinite matrices {{mvar|A}} and {{mvar|B}}, it is also known that the [[Determinant|determinant]] of their Hadamard product is greater than or equal to the product of their respective determinants:<ref name=styan1973/><math display="block">\det({A} \odot {B}) \ge \det({A}) \det({B}).</math>

==Analogous operations==

Other Hadamard operations are also seen in the mathematical literature,<ref name="Reams">{{cite journal |last=Reams |first=Robert |year=1999 |title=Hadamard inverses, square roots and products of almost semidefinite matrices |journal=Linear Algebra and Its Applications |volume=288 |pages=35–43 |doi=10.1016/S0024-3795(98)10162-3 |doi-access=free}}</ref> namely the ''{{visible anchor|Hadamard root}}'' and ''{{visible anchor|Hadamard power}}'' (which are in effect the same thing because of fractional indices), defined for a matrix such that:

{{anchor|Root|Power}}For
<math display="block">\begin{align}
  {B} &= {A}^{\circ 2} \\
B_{ij} &= {A_{ij}}^2
\end{align}</math>

and for
<math display="block">\begin{align}
  {B} &= {A}^{\circ \frac12} \\
B_{ij} &= {A_{ij}}^\frac12
\end{align}</math>

{{anchor|Inverse}}The ''{{visible anchor|Hadamard inverse}}'' reads:<ref name="Reams" />
<math display="block">\begin{align}
  {B} &= {A}^{\circ -1} \\
B_{ij} &= {A_{ij}}^{-1}
\end{align}</math>

{{anchor|Division}}A ''{{visible anchor|Hadamard division}}'' is defined as:<ref>{{cite web |last1=Wetzstein |first1=Gordon |last2=Lanman |first2=Douglas |last3=Hirsch |first3=Matthew |last4=Raskar |first4=Ramesh |title=Supplementary Material: Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting |url=http://web.media.mit.edu/~gordonw/TensorDisplays/TensorDisplays-Supplement.pdf |work=MIT Media Lab}}</ref><ref>{{cite book |last=Cyganek |first=Boguslaw |url=https://books.google.com/books?id=upsxI3bOZvAC&pg=PT109 |title=Object Detection and Recognition in Digital Images: Theory and Practice |publisher=John Wiley & Sons |year=2013 |isbn=9781118618363 |page=109}}</ref>

<math display="block">\begin{align}
  {C} &= {A} \oslash {B} \\
      C_{ij} &= \frac{A_{ij}}{B_{ij}}
\end{align}</math>

==In programming languages==
Most scientific or numerical programming languages include the Hadamard product, under various names.

In [[Software:MATLAB|MATLAB]], the Hadamard product is expressed as "dot multiply": <code>a .* b</code>, or the function call: <code>times(a, b)</code>.<ref>{{cite web |title=MATLAB times function|url=https://www.mathworks.com/help/matlab/ref/times.html}}</ref> It also has analogous dot operators which include, for example, the operators <code>a .^ b</code> and <code>a ./ b</code> <ref>{{cite web |title=Array vs. Matrix Operations|url=https://www.mathworks.com/help/matlab/matlab_prog/array-vs-matrix-operations.html}}</ref>. Because of this mechanism, it is possible to reserve <code>*</code> and  <code>^</code> for matrix multiplication and matrix exponentials, respectively.

The programming language [[Julia (programming language)|Julia]] has similar syntax as MATLAB, where Hadamard multiplication is called ''broadcast multiplication'' and also denoted with <code>a .* b</code>, and other operators are analogously defined element-wise, for example Hadamard powers use <code>a .^ b</code><ref>{{cite web |title=Vectorized "dot" operators |url=https://docs.julialang.org/en/v1/manual/mathematical-operations/#man-dot-operators |access-date=31 January 2024}}</ref>. But unlike MATLAB, in Julia this "dot" syntax is generalized with a generic ''broadcasting operator <code>.</code>'' which can apply any function element-wise. This includes both binary operators (such as the aformentioned multiplication and exponentiation, as well as any other binary operator such as the Kronecker product), and also unary operators such as <code>!</code> and <code>√</code>. Thus, any function in prefix notation <code>f</code> can be applied as <code>f.(x)</code>.<ref>{{cite web |title=Dot Syntax for Vectorizing Functions |url=https://docs.julialang.org/en/v1/manual/functions/#man-vectorized |access-date=31 January 2024}}</ref> 

[[Python (programming language)|Python]] does not have built-in array support, leading to inconsistent/conflicting notations. The [[Software:NumPy|NumPy]] numerical library interprets <code>a*b</code> or  <code>a.multiply(b)</code> as the Hadamard product, and uses <code>a@b</code> or <code>a.matmul(b)</code> for the matrix product. With the [[Software:SymPy|SymPy]] symbolic library, multiplication of {{mono|array}} objects as either <code>a*b</code> or <code>a@b</code> will produce the matrix product. The Hadamard product can be obtained with the method call <code>a.multiply_elementwise(b)</code>.<ref>{{Cite web |title=Common Matrices — SymPy 1.9 documentation |url=https://docs.sympy.org/latest/modules/matrices/common.html?highlight=multiply_elementwise#sympy.matrices.common.MatrixCommon.multiply}}</ref> Some Python packages include support for Hadamard powers using methods like <code>np.power(a, b)</code>, or the [[Software:Pandas|Pandas]] method <code>a.pow(b)</code>.

In C++, the [[Software:Eigen (C++ library)|Eigen]] library provides a <code>cwiseProduct</code> member function for the {{mono|Matrix}} class (<code>a.cwiseProduct(b)</code>), while the [[Software:Armadillo (C++ library)|Armadillo]] library uses the operator <code>%</code> to make compact expressions (<code>a % b</code>; <code>a * b</code> is a matrix product).

In [[Software:GAUSS|GAUSS]], and [[Software:HP Prime|HP Prime]], the operation is known as ''array multiplication.''

In [[Fortran]], [[R (programming language)|R]], [[APL (programming language)|APL]], [[J (programming language)|J]] and [[Wolfram Language]] ([[Software:Mathematica|Mathematica]]), the multiplication operator <code>*</code> or <code>×</code> apply the Hadamard product, whereas the matrix product is written using <code>matmul</code>, <code>%*%</code>, <code>+.×</code>, <code>+/ .*</code> and <code>.</code>, respectively.
The R package [https://cran.r-project.org/web/packages/matrixcalc/matrixcalc.pdf matrixcalc] introduces the function <code>hadamard.prod()</code> for Hadamard Product of numeric matrices or vectors.<ref>{{cite web |date=16 May 2013 |title=Matrix multiplication |url=https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Multiplication |access-date=24 August 2013 |work=An Introduction to R |publisher=The R Project for Statistical Computing}}</ref>

==Applications==
The Hadamard product appears in [[Lossy compression|lossy compression]] algorithms such as [[JPEG]]. The decoding step involves an entry-for-entry product, in other words the Hadamard product.{{citation needed|date=November 2019}}

In [[Image processing|image processing]], the Hadamard operator can be used for enhancing, suppressing or masking image regions. One matrix represents the original image, the other acts as weight or masking matrix.

It is used in the [[Machine learning|machine learning]] literature, for example, to describe the architecture of recurrent neural networks as [[Gated recurrent unit|GRUs]] or [[Long short-term memory|LSTMs]].<ref>{{cite arXiv |last1=Sak |first1=Haşim |last2=Senior |first2=Andrew |last3=Beaufays |first3=Françoise |date=2014-02-05 |title=Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition |class=cs.NE |eprint=1402.1128 }}</ref>

It is also used to study the statistical properties of random vectors and matrices.
<ref>
{{cite journal
|last1=Neudecker|first1=Heinz|last2=Liu|first2=Shuangzhe|last3=Polasek|first3=Wolfgang
|year=1995
|title= The Hadamard product and some of its applications in statistics
|journal= Statistics |volume=26|issue=4|pages=365–373|doi=10.1080/02331889508802503 }}
</ref><ref>
{{cite journal
|last1=Neudecker|first1=Heinz|last2=Liu|first2=Shuangzhe 
|year=2001
|title= Some statistical properties of Hadamard products of random matrices
|journal= Statistical Papers|volume=42|issue=4 |pages=475–487|doi=10.1007/s003620100074 |s2cid=121385730 }}
</ref>

== The penetrating face product ==
[[File:Penetrating face product.jpg|thumb|The penetrating face product of matrices]]
According to the definition of [[Biography:Vadym Slyusar|V. Slyusar]] the penetrating face product of the ''p''×''g'' matrix <math>{A}</math> and ''n''-dimensional matrix <math>{B}</math> (''n'' > 1)  with ''p''×''g'' blocks (<math>{B} = [B_n] </math>) is a matrix of size <math>{B}</math> of the form:<ref name=slyusar2>{{Cite journal|last=Slyusar|first=V. I.|date=March 13, 1998|title=A Family of Face Products of Matrices and its properties |url=http://slyusar.kiev.ua/FACE.pdf |journal=Cybernetics and Systems Analysis C/C of Kibernetika I Sistemnyi Analiz. 1999.|volume=35|issue=3|pages=379–384|doi=10.1007/BF02733426|s2cid=119661450}}</ref>
<math display=block>
  {A} [\circ] {B} = 
  \left[\begin{array} { c | c | c | c }
{A} \circ {B}_1 & {A} \circ {B}_2 & \cdots & {A} \circ {B}_n
    \end{array}\right].
</math>

=== Example ===
If
<math display=block>{A} = 
  \begin{bmatrix}
    1 & 2 & 3 \\ 
    4 & 5 & 6 \\
    7 & 8 & 9
  \end{bmatrix},\quad
  {B} =
  \left[\begin{array} { c | c | c }
    {B}_1 & {B}_2 & {B}_3
  \end{array}\right] = 
  \left[\begin{array} { c c c | c c c | c c c }
    1 &  4 & 7 &  2 &  8 & 14 &  3 & 12 & 21 \\ 
    8 & 20 & 5 & 10 & 25 & 40 & 12 & 30 &  6 \\
    2 &  8 & 3 &  2 &  4 &  2 &  7 &  3 &  9
  \end{array}\right]
</math>

then

<math display=block>{A} [\circ] {B} =
  \left[\begin{array} { c c c | c c c | c c c }
     1 &   8 & 21 &  2 &  16 &  42 &  3 &  24 & 63 \\
    32 & 100 & 30 & 40 & 125 & 240 & 48 & 150 & 36 \\
    14 &  64 & 27 & 14 &  32 &  18 & 49 &  24 & 81
  \end{array}\right].
</math>

=== Main properties ===
:<math>{A} [\circ]  {B} = {B} [\circ] {A};</math><ref name=slyusar2 />
:<math>{M} \bull {M} = {M} [\circ] \left( {M} \otimes \mathbf {1}^\textsf{T}\right),</math>

where <math> \bull </math> denotes the face-splitting product of matrices,
:<math>\mathbf{c} \bull {M} = \mathbf {c} [\circ] {M},</math> where <math>\mathbf {c}</math> is a vector.

=== Applications ===
The penetrating face product is used in the [[Tensor|tensor]]-matrix theory of [[Digital antenna array|digital antenna array]]s.<ref name=slyusar2 /> This operation can also be used in [[Artificial neural network|artificial neural network]] models, specifically convolutional layers.<ref name=hyper>{{Cite journal|last=Ha D., Dai A.M., Le Q.V.|date= 2017|title=HyperNetworks.|journal=The International Conference on Learning Representations (ICLR) 2017. – Toulon, 2017.|page=Page 6|arxiv=1609.09106}}</ref>

==See also==
* [[Frobenius inner product]]
* [[Pointwise product]]
* [[Kronecker product]]
* [[Khatri–Rao product]]

==References==
{{reflist}}

{{Linear algebra}}

[[Category:Matrix theory]]

{{Sourceattribution|Hadamard product (matrices)}}