View source for Graph neural network

{{Short description|Class of artificial neural networks}}
{{Machine learning|Artificial neural network}}

A '''graph neural network''' ('''GNN''') belongs to a class of [[Artificial neural network|artificial neural network]]s for processing data that can be represented as [[Graph (abstract data type)|graphs]].<ref name="wucuipeizhao2022" /><ref name="scarselli2009" /><ref name="micheli2009" /><ref name="sanchez2021" /><ref name="daigavane2021" />

[[File:GNN building blocks.png|thumb|Basic building blocks of a graph neural network (GNN). <math>(1)</math> Permutation equivariant layer. <math>(2)</math> Local pooling layer. <math>(3)</math> Global pooling (or readout) layer. Colors indicate [[Feature (machine learning)|features]].]]

In the more general subject of "geometric [[Deep learning|deep learning]]", certain existing neural network architectures can be interpreted as GNNs operating on suitably defined graphs.<ref name=bronstein2021 /> A [[Convolutional neural network|convolutional neural network]] layer, in the context of [[Computer vision|computer vision]], can be seen as a GNN applied to graphs whose nodes are [[Pixel|pixel]]s and only adjacent pixels are connected by edges in the graph. A [[Transformer (machine learning model)|transformer]] layer, in [[Natural language processing|natural language processing]], can be seen as a GNN applied to [[Complete graph|complete graph]]s whose nodes are words or tokens in a passage of [[Philosophy:Natural language|natural language]] text.

The key design element of GNNs is the use of ''pairwise message passing'', such that graph nodes iteratively update their representations by exchanging information with their neighbors. Since their inception, several different GNN architectures have been proposed,<ref name="scarselli2009" /><ref name="micheli2009" /><ref name="kipf2016" /><ref name="hamilton2017" /><ref name="velickovic2018" /> which implement different flavors of message passing,<ref name=bronstein2021 /> started by recursive<ref name="scarselli2009" /> or convolutional constructive<ref name="micheli2009" /> approaches. {{As of|2022}}, whether it is possible to define GNN architectures "going beyond" message passing, or if every GNN can be built on message passing over suitably defined graphs, is an open research question.<ref name=velickovic2022 />

Relevant application domains for GNNs include Natural Language Processing, <ref name="wuchen2023" /> social networks,<ref name="ying2018" /> [[Citation graph|citation networks]],<ref name="stanforddata" /> [[Biology:Molecular biology|molecular biology]],<ref>{{cite journal |last1=Zhang |first1=Weihang |last2=Cui |first2=Yang |last3=Liu |first3=Bowen |last4=Loza |first4=Martin |last5=Park |first5=Sung-Joon |last6=Nakai |first6=Kenta |title=HyGAnno: Hybrid graph neural network-based cell type annotation for single-cell ATAC sequencing data |date=1 December 2023 |doi=10.1101/2023.11.29.569114}}</ref> chemistry,<ref name="gilmer2017" /><ref>{{Cite journal |last1=Coley |first1=Connor W. |last2=Jin |first2=Wengong |last3=Rogers |first3=Luke |last4=Jamison |first4=Timothy F. |last5=Jaakkola |first5=Tommi S. |last6=Green |first6=William H. |last7=Barzilay |first7=Regina |last8=Jensen |first8=Klavs F. |date=2019-01-02 |title=A graph-convolutional neural network model for the prediction of chemical reactivity |journal=Chemical Science |language=en |volume=10 |issue=2 |pages=370–377 |doi=10.1039/C8SC04228D |pmid=30746086 |pmc=6335848 |issn=2041-6539|doi-access=free }}</ref> [[Physics:Physics|physics]]<ref name=qasim2019 /> and NP-hard [[Combinatorial optimization|combinatorial optimization]] problems.<ref name="li2018" />

Several [[Open source|open source]] [[Library (computing)|libraries]] implementing graph neural networks are available, such as PyTorch Geometric<ref name=fey2019 /> ([[Software:PyTorch|PyTorch]]), TensorFlow GNN<ref name=tfgnn2022 /> ([[Software:TensorFlow|TensorFlow]]), jraph<ref name=jraph2022/> ([[Software:Google JAX|Google JAX]]), and GraphNeuralNetworks.jl<ref name=Lucibello2021GNN/> ([[Julia (programming language)|Julia]], [[Software:Flux (machine-learning framework)|Flux]]).

== Architecture ==

The architecture of a generic GNN implements the following fundamental [[Layer (deep learning)|layers]]:<ref name=bronstein2021 />
# <em>Permutation equivariant</em>: a permutation equivariant layer [[Map (mathematics)|maps]] a representation of a graph into an updated representation of the same graph. In the literature, permutation equivariant layers are implemented via pairwise message passing between graph nodes.<ref name="bronstein2021" /><ref name="velickovic2022" /> Intuitively, in a message passing layer, nodes <em>update</em> their representations by <em>aggregating</em> the <em>messages</em> received from their immediate neighbours. As such, each message passing layer increases the receptive field of the GNN by one hop.
# <em>Local pooling</em>: a local pooling layer coarsens the graph via [[Downsampling (signal processing)|downsampling]]. Local pooling is used to increase the receptive field of a GNN, in a similar fashion to pooling layers in [[Convolutional neural network|convolutional neural network]]s. Examples include [[Nearest neighbor graph|k-nearest neighbours pooling]], top-k pooling,<ref name="gao2019" /> and self-attention pooling.<ref name="lee2019" />
# <em>Global pooling</em>: a global pooling layer, also known as ''readout'' layer, provides fixed-size representation of the whole graph. The global pooling layer must be permutation invariant, such that permutations in the ordering of graph nodes and edges do not alter the final output.<ref name="lui2022" /> Examples include element-wise sum, mean or maximum.

It has been demonstrated that GNNs cannot be more expressive than the [[Weisfeiler Leman graph isomorphism test|Weisfeiler–Leman Graph Isomorphism Test]].<ref name="douglas2011" /><ref name="xu2019" /> In practice, this means that there exist different graph structures (e.g., molecules with the same atoms but different [[Chemistry:Chemical bond|bonds]]) that cannot be distinguished by GNNs. More powerful GNNs operating on higher-dimension geometries such as [[Simplicial complex|simplicial complex]]es can be designed.<ref name=bronstein2021-2 /><ref name=grady2011discrete /> {{As of|2022}}, whether or not future architectures will overcome the message passing primitive is an open research question.<ref name=velickovic2022 />

[[File:GNN representational limits.png|thumb|[[Graph isomorphism|Non-isomorphic]] graphs that cannot be distinguished by a GNN due to the limitations of the Weisfeiler-Lehman Graph Isomorphism Test. Colors indicate node [[Feature (machine learning)|features]].]]

== Message passing layers ==
[[File:Message Passing Neural Network.png|thumb|Node representation update in a Message Passing Neural Network (MPNN) layer. Node <math>\mathbf{x}_0</math> receives messages sent by all of its immediate neighbours <math>\mathbf{x}_1</math> to <math>\mathbf{x}_4</math>. Messages are computing via the message function <math>\phi</math>, which accounts for the features of both senders and receiver.]]

Message passing layers are permutation-equivariant layers mapping a graph into an updated representation of the same graph. Formally, they can be expressed as message passing neural networks (MPNNs).<ref name="bronstein2021" />

Let <math>G = (V,E)</math> be a [[Graph (discrete mathematics)|graph]], where <math>V</math> is the node set and <math>E</math> is the edge set. Let <math>N_u</math> be the [[Neighbourhood (graph theory)|neighbourhood]] of some node <math>u \in V</math>. Additionally, let <math>\mathbf{x}_u</math> be the [[Feature (machine learning)|features]] of node <math>u \in V</math>, and <math>\mathbf{e}_{uv}</math> be the features of edge <math>(u, v) \in E</math>. An MPNN [[Layer (deep learning)|layer]] can be expressed as follows:<ref name="bronstein2021" />

:<math>\mathbf{h}_u = \phi \left( \mathbf{x}_u, \bigoplus_{v \in N_u} \psi(\mathbf{x}_u, \mathbf{x}_v, \mathbf{e}_{uv}) \right)</math>

where <math>\phi</math> and <math>\psi</math> are differentiable functions (e.g., [[Artificial neural network|artificial neural network]]s), and <math>\bigoplus</math> is a [[Permutation|permutation]] [[Invariant (mathematics)|invariant]] aggregation operator that can accept an arbitrary number of inputs (e.g., element-wise sum, mean, or max). In particular, <math>\phi</math> and <math>\psi</math> are referred to as ''update'' and ''message'' functions, respectively. Intuitively, in an MPNN computational block, graph nodes ''update'' their representations by ''aggregating'' the ''messages'' received from their neighbours.

The outputs of one or more MPNN layers are node representations <math>\mathbf{h}_u</math> for each node <math>u \in V</math> in the graph. Node representations can be employed for any downstream task, such as node/graph [[Statistical classification|classification]] or edge prediction.

Graph nodes in an MPNN update their representation aggregating information from their immediate neighbours. As such, stacking <math>n</math> MPNN layers means that one node will be able to communicate with nodes that are at most <math>n</math> "hops" away. In principle, to ensure that every node receives information from every other node, one would need to stack a number of MPNN layers equal to the graph [[Distance (graph theory)|diameter]]. However, stacking many MPNN layers may cause issues such as oversmoothing<ref name=chen2021 /> and oversquashing.<ref name=alon2021 /> Oversmoothing refers to the issue of node representations becoming indistinguishable. Oversquashing refers to the bottleneck that is created by squeezing long-range dependencies into fixed-size representations. Countermeasures such as skip connections<ref name=hamilton2017 /><ref name=xu2021 /> (as in [[Residual neural network|residual neural network]]s), gated update rules<ref name=li2016 /> and jumping knowledge<ref name=xu2018 /> can mitigate oversmoothing. Modifying the final layer to be a fully-adjacent layer, i.e., by considering the graph as a [[Complete graph|complete graph]], can mitigate oversquashing in problems where long-range dependencies are required.<ref name=alon2021 />

Other "flavours" of MPNN have been developed in the literature,<ref name=bronstein2021 /> such as graph convolutional networks<ref name=kipf2016 /> and graph attention networks,<ref name=velickovic2018 /> whose definitions can be expressed in terms of the MPNN formalism.

=== Graph convolutional network ===
The graph convolutional network (GCN) was first introduced by Thomas Kipf and [[Biography:Max Welling|Max Welling]] in 2017.<ref name=kipf2016 />

A GCN layer defines a [[Order of approximation|first-order approximation]] of a localized spectral [[Filter (signal processing)|filter]] on graphs. GCNs can be understood as a generalization of [[Convolutional neural network|convolutional neural network]]s to graph-structured data.

The formal expression of a GCN layer reads as follows:

:<math>\mathbf{H} = \sigma\left(\tilde{\mathbf{D}}^{-\frac{1}{2}} \tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-\frac{1}{2}} \mathbf{X} \mathbf{\Theta}\right)</math>

where <math>\mathbf{H}</math> is the matrix of node representations <math>\mathbf{h}_u</math>, <math>\mathbf{X}</math> is the matrix of node features <math>\mathbf{x}_u</math>, <math>\sigma(\cdot)</math> is an [[Activation function|activation function]] (e.g., ReLU), <math>\tilde{\mathbf{A}}</math> is the graph [[Adjacency matrix|adjacency matrix]] with the addition of self-loops, <math>\tilde{\mathbf{D}}</math> is the graph [[Degree matrix|degree matrix]] with the addition of self-loops, and <math>\mathbf{\Theta}</math> is a matrix of trainable parameters.

In particular, let <math>\mathbf{A}</math> be the graph adjacency matrix: then, one can define <math>\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}</math> and <math>\tilde{\mathbf{D}}_{ii} = \sum_{j \in V} \tilde{A}_{ij}</math>, where <math>\mathbf{I}</math> denotes the [[Identity matrix|identity matrix]]. This normalization ensures that the eigenvalues of <math>\tilde{\mathbf{D}}^{-\frac{1}{2}} \tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-\frac{1}{2}}</math> are bounded in the range <math>[0, 1]</math>, avoiding [[Numerical stability|numerical instabilities]] and [[Vanishing gradient problem|exploding/vanishing gradients]].

A limitation of GCNs is that they do not allow multidimensional edge features <math>\mathbf{e}_{uv}</math>.<ref name=kipf2016 /> It is however possible to associate scalar weights <math>w_{uv}</math> to each edge by imposing <math>A_{uv} = w_{uv}</math>, i.e., by setting each nonzero entry in the adjacency matrix equal to the weight of the corresponding edge.

=== Graph attention network ===
The graph attention network (GAT) was introduced by Petar Veličković et al. in 2018.<ref name=velickovic2018 />

Graph attention network is a combination of a graph neural network and an attention layer.
The implementation of attention layer in graphical neural networks helps provide attention or focus to the important information from the data instead of focusing on the whole data.

A multi-head GAT layer can be expressed as follows:

:<math> \mathbf{h}_u = \overset{K}{\underset{k=1}{\Big\Vert}} \sigma \left(\sum_{v \in N_u} \alpha_{uv} \mathbf{W}^k \mathbf{x}_v\right) </math>

where <math>K</math> is the number of [[Attention (machine learning)|attention]] heads, <math>\Big\Vert</math> denotes [[Concatenation|vector concatenation]], <math>\sigma(\cdot)</math> is an [[Activation function|activation function]] (e.g., ReLU), <math>\alpha_{ij}</math> are attention coefficients, and <math>W^k</math> is a matrix of trainable parameters for the <math>k</math>-th attention head.

For the final GAT layer, the outputs from each attention head are averaged before the application of the activation function. Formally, the final GAT layer can be written as:

:<math> \mathbf{h}_u = \sigma \left(\frac{1}{K}\sum_{k=1}^K \sum_{v \in N_u} \alpha_{uv} \mathbf{W}^k \mathbf{x}_v\right) </math>

[[Attention (machine learning)|Attention]] in Machine Learning is a technique that mimics [[Philosophy:Attention|cognitive attention]]. In the context of learning on graphs, the attention coefficient <math>\alpha_{uv}</math> measures ''how important'' is node <math>u \in V</math> to node <math>v \in V</math>.

Normalized attention coefficients are computed as follows:

:<math>\alpha_{uv} = \frac{\exp(\text{LeakyReLU}\left(\mathbf{a}^T [\mathbf{W} \mathbf{h}_u \Vert \mathbf{W} \mathbf{h}_v \Vert \mathbf{e}_{uv}]\right))}{\sum_{z \in N_u}\exp(\text{LeakyReLU}\left(\mathbf{a}^T [\mathbf{W} \mathbf{h}_u \Vert \mathbf{W} \mathbf{h}_z \Vert \mathbf{e}_{uz}]\right))}</math>

where <math>\mathbf{a}</math> is a vector of learnable weights, <math>\cdot^T</math> indicates [[Transpose|transposition]], and <math>\text{LeakyReLU}</math> is a [[Rectifier (neural networks)|modified ReLU]] activation function. Attention coefficients are normalized to make them easily comparable across different nodes.<ref name=velickovic2018 />

A GCN can be seen as a special case of a GAT where attention coefficients are not learnable, but fixed and equal to the edge weights <math>w_{uv}</math>.

=== Gated graph sequence neural network ===
The gated graph sequence neural network (GGS-NN) was introduced by Yujia Li et al. in 2015.<ref name=li2016 /> The GGS-NN extends the GNN formulation by Scarselli et al.<ref name=scarselli2009 /> to output sequences. The message passing framework is implemented as an update rule to a [[Gated recurrent unit|gated recurrent unit]] (GRU) cell.

A GGS-NN can be expressed as follows:

:<math>\mathbf{h}_u^{(0)} = \mathbf{x}_u \, \Vert \, \mathbf{0}</math>
:<math>\mathbf{m}_u^{(l+1)} = \sum_{v \in N_u} \mathbf{\Theta} \mathbf{h}_v</math>
:<math>\mathbf{h}_u^{(l+1)} = \text{GRU}(\mathbf{m}_u^{(l+1)}, \mathbf{h}_u^{(l)})</math>

where <math>\Vert</math> denotes [[Concatenation|vector concatenation]], <math>\mathbf{0}</math> is a vector of zeros, <math>\mathbf{\Theta}</math> is a matrix of learnable parameters, <math>\text{GRU}</math> is a GRU cell, and <math>l</math> denotes the sequence index. In a GGS-NN, the node representations are regarded as the hidden states of a GRU cell. The initial node features <math>\mathbf{x}_u^{(0)}</math> are [[Data structure alignment|zero-padded]] up to the hidden state dimension of the GRU cell. The same GRU cell is used for updating representations for each node.

== Local pooling layers ==
Local pooling layers coarsen the graph via downsampling. We present here several learnable local pooling strategies that have been proposed.<ref name=lui2022 /> For each cases, the input is the initial graph is represented by a matrix <math>\mathbf{X}</math> of node features, and the graph adjacency matrix <math>\mathbf{A}</math>. The output is the new matrix <math>\mathbf{X}'</math>of node features, and the new graph adjacency matrix <math>\mathbf{A}'</math>.

=== Top-k pooling ===
We first set

<math>\mathbf{y} = \frac{\mathbf{X}\mathbf{p}}{\Vert\mathbf{p}\Vert}</math>

where <math>\mathbf{p}</math> is a learnable [[Projection (mathematics)|projection]] vector. The projection vector <math>\mathbf{p}</math> computes a scalar projection value for each graph node.

The top-k pooling layer <ref name="gao2019" /> can then be formalised as follows:
:<math>\mathbf{X}' = (\mathbf{X} \odot \text{sigmoid}(\mathbf{y}))_{\mathbf{i}}</math>

:<math>\mathbf{A}' = \mathbf{A}_{\mathbf{i}, \mathbf{i}}</math>

where <math>\mathbf{i} = \text{top}_k(\mathbf{y})</math> is the subset of nodes with the top-k highest projection scores, <math>\odot</math> denotes element-wise [[Matrix multiplication|matrix multiplication]], and <math>\text{sigmoid}(\cdot)</math> is the [[Sigmoid function|sigmoid function]]. In other words, the nodes with the top-k highest projection scores are retained in the new adjacency matrix <math>\mathbf{A}'</math>. The <math>\text{sigmoid}(\cdot)</math> operation makes the projection vector <math>\mathbf{p}</math> trainable by [[Backpropagation|backpropagation]], which otherwise would produce discrete outputs.<ref name="gao2019" />

=== Self-attention pooling ===
We first set

:<math>\mathbf{y} = \text{GNN}(\mathbf{X}, \mathbf{A})</math>
where <math>\text{GNN}</math> is a generic permutation equivariant GNN layer (e.g., GCN, GAT, MPNN).

The Self-attention pooling layer<ref name="lee2019" /> can then be formalised as follows:

:<math>\mathbf{X}' = (\mathbf{X} \odot \mathbf{y})_{\mathbf{i}}</math>

:<math>\mathbf{A}' = \mathbf{A}_{\mathbf{i}, \mathbf{i}}</math>

where <math>\mathbf{i} = \text{top}_k(\mathbf{y})</math> is the subset of nodes with the top-k highest projection scores, <math>\odot</math> denotes element-wise [[Matrix multiplication|matrix multiplication]].

The self-attention pooling layer can be seen as an extension of the top-k pooling layer. Differently from top-k pooling, the self-attention scores computed in self-attention pooling account both for the graph features and the graph topology.

== Applications ==

=== Protein folding ===

Graph neural networks are one of the main building blocks of [[AlphaFold]], an artificial intelligence program developed by [[Company:Google|Google]]'s [[Company:DeepMind|DeepMind]] for solving the [[Biology:Protein folding|protein folding]] problem in [[Biology:Biology|biology]]. AlphaFold achieved first place in several [[Biology:CASP|CASP]] competitions.<ref name=guardian2018 /><ref name=mit2020 /><ref name=xu2018 />

=== Social networks ===
Social networks are a major application domain for GNNs due to their natural representation as [[Social:Social graph|social graph]]s. GNNs are used to develop recommender systems based on both social relations and item relations.<ref name=fan2019 /><ref name=ying2018/>

=== Combinatorial optimization ===
GNNs are used as fundamental building blocks for several combinatorial optimization algorithms.<ref name=cappart2021 /> Examples include computing [[Shortest path problem|shortest paths]] or [[Eulerian path|Eulerian circuits]] for a given graph,<ref name=li2016/> deriving [[Engineering:Placement (electronic design automation)|chip placements]] superior or competitive to handcrafted human solutions,<ref name=mirhoseini2021 /> and improving expert-designed branching rules in [[Branch and bound|branch and bound]].<ref name=gasse2019 />

=== Cyber security ===
When viewed as a graph, a network of computers can be analyzed with GNNs for anomaly detection. Anomalies within provenance graphs often correlate to malicious activity within the network. GNNs have been used to identify these anomalies on individual nodes<ref>{{Cite journal |last1=Wang |first1=Su |last2=Wang |first2=Zhiliang |last3=Zhou |first3=Tao |last4=Sun |first4=Hongbin |last5=Yin |first5=Xia |last6=Han |first6=Dongqi |last7=Zhang |first7=Han |last8=Shi |first8=Xingang |last9=Yang |first9=Jiahai |date=2022 |title=Threatrace: Detecting and Tracing Host-Based Threats in Node Level Through Provenance Graph Learning |url=https://ieeexplore.ieee.org/document/9899459/;jsessionid=NzAXdLahhjEX-xmrFzOROk4qxoaz40aJFvKcZRgjck8-zCOucJi7!380715771 |journal=IEEE Transactions on Information Forensics and Security |volume=17 |pages=3972–3987 |doi=10.1109/TIFS.2022.3208815 |issn=1556-6021|arxiv=2111.04333 |s2cid=243847506 }}</ref> and within paths<ref>{{Cite journal |last1=Wang |first1=Qi |last2=Hassan |first2=Wajih Ul |last3=Li |first3=Ding |last4=Jee |first4=Kangkook |last5=Yu |first5=Xiao |date=2020 |title=You Are What You Do: Hunting Stealthy Malware via Data Provenance Analysis. |journal=Network and Distributed Systems Security (NDSS) Symposium|doi=10.14722/ndss.2020.24167 |isbn=978-1-891562-61-7 |s2cid=211267791 |doi-access=free }}</ref> to detect malicious processes, or on the edge level<ref>{{Cite journal |last1=King |first1=Isaiah J. |last2=Huang |first2=H. Howie |date=2022 |title=Euler: Detecting Network Lateral Movement via Scalable Temporal Link Prediction |url=https://www.ndss-symposium.org/wp-content/uploads/2022-107A-paper.pdf |journal=In Proceedings of the 29th Network and Distributed Systems Security Symposium (NDSS)|doi=10.14722/ndss.2022.24107 |s2cid=248221601 }}</ref> to detect lateral movement.

==References==
{{Reflist|refs=
<ref name="wuchen2023">{{Cite journal|last1=Wu|first1=Lingfei|last2=Chen|first2=Yu|last3=Shen |first3=Kai|last4=Guo|first4=Xiaojie|last5=Gao|first5=Hanning|last6=Li|first6=Shucheng|last7=Pei|first7=Jian|last8=Long|first8=Bo|date=2023|title=Graph Neural Networks for Natural Language Processing: A Survey
|url=https://www.nowpublishers.com/article/Details/MAL-096|journal=Foundations and Trends in Machine Learning|volume=16|issue=2|pages=119–328|doi=10.1561/2200000096 |pmid=19068426|s2cid=206756462|issn=1941-0093|arxiv=2106.06090}}</ref>
<ref name="wucuipeizhao2022">{{Cite journal|last1=Wu|first1=Lingfei|last2=Cui|first2=Peng|last3=Pei |first3=Jian|last4=Zhao|first4=Liang|date=2022|title=Graph Neural Networks: Foundations, Frontiers, and Applications|url=https://graph-neural-networks.github.io/|journal=Springer Singapore|pages=725|url-access=<!--WP:URLACCESS-->}}</ref>
<ref name="scarselli2009">{{Cite journal|last1=Scarselli|first1=Franco|last2=Gori|first2=Marco|last3=Tsoi |first3=Ah Chung|last4=Hagenbuchner|first4=Markus|last5=Monfardini|first5=Gabriele|date=2009|title=The Graph Neural Network Model|url=https://ieeexplore.ieee.org/document/4700287|journal=IEEE Transactions on Neural Networks|volume=20|issue=1|pages=61–80|doi=10.1109/TNN.2008.2005605 |pmid=19068426|s2cid=206756462|issn=1941-0093}}</ref>
<ref name="micheli2009">{{Cite journal|last1=Micheli|first1=Alessio|title=Neural Network for Graphs: A Contextual Constructive Approach|url=https://ieeexplore.ieee.org/document/4700287|journal=IEEE Transactions on Neural Networks|year=2009 |volume=20|issue=3|pages=498–511|doi=10.1109/TNN.2008.2010350 |pmid=19193509|s2cid=17486263|issn=1045-9227}}</ref>
<ref name="sanchez2021">{{Cite journal|last1=Sanchez-Lengeling|first1=Benjamin|last2=Reif|first2=Emily |last3=Pearce|first3=Adam|last4=Wiltschko|first4=Alex|date=2021-09-02|title=A Gentle Introduction to Graph Neural Networks|url=https://distill.pub/2021/gnn-intro|journal=Distill|volume=6|issue=9|pages=e33 |doi=10.23915/distill.00033|issn=2476-0757|doi-access=free}}</ref>
<ref name="daigavane2021">{{Cite journal|last1=Daigavane|first1=Ameya|last2=Ravindran|first2=Balaraman |last3=Aggarwal|first3=Gaurav|date=2021-09-02|title=Understanding Convolutions on Graphs |url=https://distill.pub/2021/understanding-gnns|journal=Distill|volume=6|issue=9|pages=e32 |doi=10.23915/distill.00032|s2cid=239678898|issn=2476-0757|doi-access=free}}</ref>
<ref name="gilmer2017">{{Cite journal|last1=Gilmer|first1=Justin|last2=Schoenholz|first2=Samuel S. |last3=Riley|first3=Patrick F.|last4=Vinyals|first4=Oriol|last5=Dahl|first5=George E.|date=2017-07-17|title=Neural Message Passing for Quantum Chemistry|url=http://proceedings.mlr.press/v70/gilmer17a.html |journal=Proceedings of Machine Learning Research|language=en|pages=1263–1272|arxiv=1704.01212}}</ref>
<ref name="kipf2016">{{Cite journal|last1=Kipf|first1=Thomas N|last2=Welling|first2=Max|date=2016 |title=Semi-supervised classification with graph convolutional networks|journal=IEEE Transactions on Neural Networks |url=https://ieeexplore.ieee.org/document/4700287 |volume=5|issue=1|pages=61–80 |doi=10.1109/TNN.2008.2005605|pmid=19068426|arxiv=1609.02907|s2cid=206756462}}</ref>
<ref name="hamilton2017">{{Cite journal|last1=Hamilton|first1=William|last2=Ying|first2=Rex |last3=Leskovec|first3=Jure|date=2017|title=Inductive Representation Learning on Large Graphs|url=https://cs.stanford.edu/people/jure/pubs/graphsage-nips17.pdf|journal=Neural Information Processing Systems|volume=31|arxiv=1706.02216|via=Stanford}}</ref>
<ref name="velickovic2018">{{Cite arXiv|last1=Veličković|first1=Petar|last2=Cucurull|first2=Guillem |last3=Casanova|first3=Arantxa|last4=Romero|first4=Adriana|last5=Liò|first5=Pietro|last6=Bengio |first6=Yoshua|date=2018-02-04 |title=Graph Attention Networks|eprint=1710.10903 |class=stat.ML}}</ref>
<ref name=stanforddata>{{Cite web|title=Stanford Large Network Dataset Collection |url=https://snap.stanford.edu/data/|access-date=2021-07-05|website=snap.stanford.edu}}</ref>
<ref name="li2018">{{cite journal |last1=Li |first1=Zhuwen |last2=Chen |first2=Qifeng |last3=Koltun |first3=Vladlen |title=Combinatorial optimization with graph convolutional networks and guided tree search |journal=Neural Information Processing Systems |date=2018 |volume=31 |pages=537–546 |arxiv=1810.10659}}</ref>
<ref name="bronstein2021">{{cite arXiv |last1=Bronstein |first1=Michael M. |last2=Bruna |first2=Joan |last3=Cohen |first3=Taco |last4=Veličković |first4=Petar |title=Geometric Deep Learning: Grids, Groups, Graphs Geodesics and Gauges |date=May 4, 2021 |class=cs.LG |eprint=2104.13478}}</ref>
<ref name=douglas2011>{{cite arXiv|last=Douglas|first=B. L.|date=2011-01-27|title=The Weisfeiler–Lehman Method and Graph Isomorphism Testing|class=math.CO|eprint=1101.5211}}</ref>
<ref name=xu2019>{{Cite arXiv|last1=Xu|first1=Keyulu|last2=Hu|first2=Weihua|last3=Leskovec|first3=Jure |last4=Jegelka|first4=Stefanie|date=2019-02-22|title=How Powerful are Graph Neural Networks? |eprint=1810.00826 |class=cs.LG}}</ref>
<ref name=velickovic2022>{{cite arXiv |last1=Veličković |first1=Petar |title=Message passing all the way up |year=2022 |class=cs.LG |eprint=2202.11097}}</ref>
<ref name=qasim2019>{{cite journal |last1=Qasim |first1=Shah Rukh |last2=Kieseler |first2=Jan |last3=Iiyama |first3=Yutaro |last4=Pierini |first4=Maurizio Pierini |title=Learning representations of irregular particle-detector geometry with distance-weighted graph networks |journal=The European Physical Journal C |date=2019 |volume=79 |issue=7 |doi=10.1140/epjc/s10052-019-7113-9|s2cid=88518244 |doi-access=free |arxiv=1902.07987 }}</ref>
<ref name=lui2022>{{cite arXiv |last1=Liu |first1=Chuang |last2=Zhan |first2=Yibing |last3=Li |first3=Chang |last4=Du |first4=Bo |last5=Wu |first5=Jia |last6=Hu |first6=Wenbin |last7=Liu |first7=Tongliang |last8=Tao |first8=Dacheng |title=Graph Pooling for Graph Neural Networks: Progress, Challenges, and Opportunities |date=2022 |class=cs.LG |eprint=2204.07321}}</ref>
<ref name=bronstein2021-2>{{cite arXiv |last1=Bodnar |first1=Christian |last2=Frasca |first2=Fabrizio |last3=Guang Wang |first3=Yu |last4=Otter |first4=Nina |last5=Montúfar |first5=Guido |last6=Liò |first6=Pietro |last7=Bronstein |first7=Michael |title=Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks |date=2021 |class=cs.LG |eprint=2103.03212}}</ref>
<ref name=gao2019>{{cite arXiv |last1=Gao |first1=Hongyang |last2=Ji |first2=Shuiwang Ji |title=Graph U-Nets |date=2019 |class=cs.LG |eprint=1905.05178}}</ref>
<ref name=lee2019>{{cite arXiv |last1=Lee |first1=Junhyun |last2=Lee |first2=Inyeop |last3=Kang |first3=Jaewoo |title=Self-Attention Graph Pooling  |date=2019 |class=cs.LG |eprint=1904.08082}}</ref>
<ref name=alon2021>{{cite arXiv |last1=Alon |first1=Uri |last2=Yahav |first2=Eran |title=On the Bottleneck of Graph Neural Networks and its Practical Implications |date=2021 |class=cs.LG |eprint=2006.05205}}</ref>
<ref name=chen2021>{{cite journal |last1=Chen |first1=Deli |last2=Lin |first2=Yankai |last3=Li |first3=Wei |last4=Li |first4=Peng |last5=Zhou |first5=Jie |last6=Sun |first6=Xu |title=Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View  |journal=Proceedings of the AAAI Conference on Artificial Intelligence |date=2020 |volume=34 |issue=4 |pages=3438–3445 |doi=10.1609/aaai.v34i04.5747 |s2cid=202539008 |arxiv=1909.03211}}</ref>
<ref name=guardian2018>{{cite news|last1=Sample|first1=Ian|date=2 December 2018|title=Google's DeepMind predicts 3D shapes of proteins|work=The Guardian|url=https://www.theguardian.com/science/2018/dec/02/google-deepminds-ai-program-alphafold-predicts-3d-shapes-of-proteins|access-date=30 November 2020}}</ref>
<ref name=mit2020>{{cite web |title=DeepMind's protein-folding AI has solved a 50-year-old grand challenge of biology |url=https://www.technologyreview.com/2020/11/30/1012712/deepmind-protein-folding-ai-solved-biology-science-drugs-disease/ |website=MIT Technology Review |access-date=30 November 2020 |language=en}}</ref>
<ref name=fan2019>{{cite book |last1=Fan |first1=Wenqi |last2=Ma |first2=Yao |last3=Li |first3=Qing |last4=He |first4=Yuan |last5=Zhao |first5=Eric |last6=Tang |first6=Jiliang |last7=Yin |first7=Dawei |title=Graph Neural Networks for Social Recommendation |date=2019 |pages=417–426 |doi=10.1145/3308558.3313488 |hdl=10397/81232 |isbn=9781450366748 |s2cid=67769538 |arxiv=1902.07243}}</ref>
<ref name=ying2018>{{cite book |last1=Ying |first1=Rex |last2=He |first2=Ruining |last3=Chen |first3=Kaifeng |last4=Eksombatchai |first4=Pong |last5=Hamilton |first5=William L. |last6=Leskovec |first6=Jure |title=Graph Convolutional Neural Networks for Web-Scale Recommender Systems  |date=2018 |pages=974–983 |doi=10.1145/3219819.3219890 |isbn=9781450355520 |s2cid=46949657 |arxiv=1806.01973}}</ref>
<ref name=gasse2019>{{cite arXiv |last1=Gasse |first1=Maxime |last2=Chételat |first2=Didier |last3=Ferroni |first3=Nicola |last4=Charlin |first4=Laurent |last5=Lodi |first5=Andrea |title=Exact Combinatorial Optimization with Graph Convolutional Neural Networks |date=2019 |class=cs.LG |eprint=1906.01629}}</ref>
<ref name=cappart2021>{{cite arXiv |last1=Cappart |first1=Quentin |last2=Chételat |first2=Didier |last3=Khalil |first3=Elias |last4=Lodi |first4=Andrea |last5=Morris |first5=Christopher |last6=Veličković |first6=Petar |title=Combinatorial optimization and reasoning with graph neural networks |year=2021 |class=cs.LG |eprint=2102.09544}}</ref>
<ref name=mirhoseini2021>{{cite journal |last1=Mirhoseini |first1=Azalia |last2=Goldie |first2=Anna |last3=Yazgan |first3=Mustafa |last4=Jiang |first4=Joe Wenjie |last5=Songhori |first5=Ebrahim |last6=Wang |first6=Shen |last7=Lee |first7=Young-Joon |last8=Johnson |first8=Eric |last9=Pathak |first9=Omkar |last10=Nazi |first10=Azade |last11=Pak |first11=Jiwoo |last12=Tong |first12=Andy |last13=Srinivasa |first13=Kavya |last14=Hang |first14=William |last15=Tuncer |first15=Emre |last16=Le |first16=Quoc V. |last17=Laudon |first17=James |last18=Ho |first18=Richard |last19=Carpenter |first19=Roger |last20=Dean |first20=Jeff |title=A graph placement methodology for fast chip design |journal=Nature |date=2021 |volume=594 |issue=7862 |pages=207–212 |doi=10.1038/s41586-021-03544-w|pmid=34108699 |s2cid=235395490 }}</ref>
<ref name=fey2019>{{cite arXiv |last1=Matthias |first1=Fey |last2=Lenssen |first2=Jan E. |title=Fast Graph Representation Learning with PyTorch Geometric |date=2019 |class=cs.LG |eprint=1903.02428}}</ref>
<ref name=tfgnn2022>{{cite web |title=Tensorflow GNN |website=[[GitHub]] |url=https://github.com/tensorflow/gnn |access-date=30 June 2022}}</ref>
<ref name=jraph2022>{{cite web |title=jraph |website=[[GitHub]] |url=https://github.com/deepmind/jraph |access-date=30 June 2022}}</ref>
<ref name=xu2021>{{cite arXiv |last1=Xu |first1=Keyulu |last2=Zhang |first2=Mozhi |last3=Jegelka |first3=Stephanie |last4=Kawaguchi |first4=Kenji |title=Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth  |date=2021 |class=cs.LG |eprint=2105.04550}}</ref>
<ref name=li2016>{{cite arXiv |last1=Li |first1=Yujia |last2=Tarlow |first2=Daniel |last3=Brockschmidt |first3=Mark |last4=Zemel |first4=Richard |title=Gated Graph Sequence Neural Networks |date=2016 |class=cs.LG |eprint=1511.05493 }}</ref>
<ref name=grady2011discrete>{{cite book |last1=Grady |first1=Leo |last2=Polimeni |first2=Jonathan |title=Discrete Calculus: Applied Analysis on Graphs for Computational Science |url=http://leogrady.net/wp-content/uploads/2017/01/grady2010discrete.pdf |date=2011 |publisher=Springer }}</ref>
<ref name=xu2018>{{cite arXiv |last1=Xu |first1=Keyulu |last2=Li |first2=Chengtao |last3=Tian |first3=Yonglong |last4=Sonobe |first4=Tomohiro |last5=Kawarabayashi |first5=Ken-ichi |last6=Jegelka |first6=Stefanie |title=Representation Learning on Graphs with Jumping Knowledge Networks  |date=2018 |class=cs.LG |eprint=1806.03536}}</ref>
<ref name=Lucibello2021GNN>{{cite web |last=Lucibello |first=Carlo |title=GraphNeuralNetworks.jl |url=https://github.com/CarloLucibello/GraphNeuralNetworks.jl |year=2021 |access-date=2023-09-21}}</ref>
}}

== External links ==
* https://distill.pub/2021/gnn-intro/

[[Category:Unsupervised learning]]
[[Category:Artificial neural networks]]
[[Category:Graph algorithms]]

{{Sourceattribution|Graph neural network}}