Siamese neural network

A Siamese neural network (sometimes called a twin neural network) is an artificial neural network that uses the same weights while working in tandem on two different input vectors to compute comparable output vectors.

• Non-negativity: [math]\displaystyle{ \delta ( x, y ) \ge 0 }[/math]
• Identity of Non-discernibles: [math]\displaystyle{ \delta ( x, y ) = 0 \iff x=y }[/math]
• Symmetry: [math]\displaystyle{ \delta ( x, y ) = \delta ( y, x ) }[/math]
• Triangle inequality: [math]\displaystyle{ \delta ( x, z ) \le \delta ( x, y ) + \delta ( y, z ) }[/math]

In particular, the triplet loss algorithm is often defined with squared Euclidean (which unlike Euclidean, does not have triangle inequality) distance at its core.

Predefined metrics, Euclidean distance metric

The common learning goal is to minimize a distance metric for similar objects and maximize for distinct ones. This gives a loss function like

[math]\displaystyle{ \begin{align} \delta(x^{(i)}, x^{(j)})= \begin {cases} \min \ \| \operatorname{f} \left ( x^{(i)} \right ) - \operatorname{f} \left ( x^{(j)} \right ) \| \, , i = j \\ \max \ \| \operatorname{f} \left ( x^{(i)} \right ) - \operatorname{f} \left ( x^{(j)} \right ) \| \, , i \neq j \end{cases} \end{align} }[/math]

[math]\displaystyle{ i,j }[/math] are indexes into a set of vectors

[math]\displaystyle{ \operatorname{f}(\cdot) }[/math] function implemented by the twin network

The most common distance metric used is Euclidean distance, in case of which the loss function can be rewritten in matrix form as

[math]\displaystyle{ \operatorname{\delta} ( \mathbf{x}^{(i)}, \mathbf{x}^{(j)} ) \approx (\mathbf{x}^{(i)} - \mathbf{x}^{(j)})^{T}(\mathbf{x}^{(i)} - \mathbf{x}^{(j)}) }[/math]

Learned metrics, nonlinear distance metric

A more general case is where the output vector from the twin network is passed through additional network layers implementing non-linear distance metrics.

[math]\displaystyle{ \begin{align} \text{if} \, i = j \, \text{then} & \, \operatorname{\delta} \left [ \operatorname{f} \left ( x^{(i)} \right ), \, \operatorname{f} \left ( x^{(j)} \right ) \right ] \, \text{is small} \\ \text{otherwise} & \, \operatorname{\delta} \left [ \operatorname{f} \left ( x^{(i)} \right ), \, \operatorname{f} \left ( x^{(j)} \right ) \right ] \, \text{is large} \end{align} }[/math]

[math]\displaystyle{ i,j }[/math] are indexes into a set of vectors

[math]\displaystyle{ \operatorname{f}(\cdot) }[/math]function implemented by the twin network

[math]\displaystyle{ \operatorname{\delta}(\cdot) }[/math]function implemented by the network joining outputs from the twin network

On a matrix form the previous is often approximated as a Mahalanobis distance for a linear space as

[math]\displaystyle{ \operatorname{\delta} ( \mathbf{x}^{(i)}, \mathbf{x}^{(j)} ) \approx (\mathbf{x}^{(i)} - \mathbf{x}^{(j)})^{T}\mathbf{M}(\mathbf{x}^{(i)} - \mathbf{x}^{(j)}) }[/math]

This can be further subdivided in at least Unsupervised learning and Supervised learning.

Learned metrics, half-twin networks

This form also allows the twin network to be more of a half-twin, implementing a slightly different functions

[math]\displaystyle{ \begin{align} \text{if} \, i = j \, \text{then} & \, \operatorname{\delta} \left [ \operatorname{f} \left ( x^{(i)} \right ), \, \operatorname{g} \left ( x^{(j)} \right ) \right ] \, \text{is small} \\ \text{otherwise} & \, \operatorname{\delta} \left [ \operatorname{f} \left ( x^{(i)} \right ), \, \operatorname{g} \left ( x^{(j)} \right ) \right ] \, \text{is large} \end{align} }[/math]

[math]\displaystyle{ i,j }[/math] are indexes into a set of vectors

[math]\displaystyle{ \operatorname{f}(\cdot), \operatorname{g}(\cdot) }[/math]function implemented by the half-twin network

[math]\displaystyle{ \operatorname{\delta}(\cdot) }[/math]function implemented by the network joining outputs from the twin network

Twin networks for object tracking

Twin networks have been used in object tracking because of its unique two tandem inputs and similarity measurement. In object tracking, one input of the twin network is user pre-selected exemplar image, the other input is a larger search image, which twin network's job is to locate exemplar inside of search image. By measuring the similarity between exemplar and each part of the search image, a map of similarity score can be given by the twin network. Furthermore, using a Fully Convolutional Network, the process of computing each sector's similarity score can be replaced with only one cross correlation layer.

After being first introduced in 2016, Twin fully convolutional network has been used in many High-performance Real-time Object Tracking Neural Networks. Like CFnet, StructSiam, SiamFC-tri, DSiam, SA-Siam, SiamRPN, DaSiamRPN, Cascaded SiamRPN, SiamMask, SiamRPN++, Deeper and Wider SiamRPN.

See also

• Artificial neural network
• Triplet loss

Further reading

• Chicco, Davide (2020), "Siamese neural networks: an overview", Artificial Neural Networks, Methods in Molecular Biology, 2190 (3rd ed.), New York City, New York, USA: Springer Protocols, Humana Press, pp. 73–94, doi:10.1007/978-1-0716-0826-5_3, ISBN 978-1-0716-0826-5, PMID 32804361, https://doi.org/10.1007/978-1-0716-0826-5_3