View source for Physics:Energy based model

An '''energy-based model''' (EBM) is a form of [[Generative model|generative model]] (GM) imported directly from [[Physics:Statistical physics|statistical physics]] to learning. GMs learn an underlying data distribution by analyzing a sample dataset. Once trained, a GM can produce other datasets that also match the data distribution.<ref name=":1">{{Cite web|url=https://openai.com/blog/energy-based-models/|title=Implicit Generation and Generalization Methods for Energy-Based Models|date=2019-03-21|website=OpenAI|language=en|access-date=2019-12-27}}</ref> EBMs provide a unified framework for many probabilistic and non-probabilistic approaches to such learning, particularly for training [[Graphical model|graphical]] and other structured models.<ref name=":0">{{Cite web|url=https://towardsdatascience.com/generating-training-datasets-using-energy-based-models-that-actually-scale-4e1f83bb9e00|title=Generating Training Datasets Using Energy Based Models that Actually Scale|last=Rodriguez|first=Jesus|date=2019-04-01|website=Medium|language=en|access-date=2019-12-27}}</ref>

An EBM learns the characteristics of a target dataset and generates a similar but larger dataset. EBMs detect the latent variables of a dataset and generate new datasets with a similar distribution.<ref name=":0" />

Target applications include [[Natural language processing|natural language processing]], robotics and [[Computer vision|computer vision]].<ref name=":0" />

== History ==
The term "energy-based models" was first coined in a JMLR paper <ref>{{Cite journal|url=https://www.jmlr.org/papers/v4/teh03a.html|title=Energy-Based Models for Sparse Overcomplete Representations|last1=Teh|first1=Yee Whye|last2=Welling|first2=Max|last3=Osindero|first3=Simon|last4=Hinton|first4=Geoffrey E.|journal=JMLR|date=December 2003}}</ref> where the authors defined a generalisation of independent components analysis to the overcomplete setting using EBMs.
Other early work on EBMs proposed models that represented energy as a composition of latent and observable variables. EBMs surfaced in 2003.<ref>{{Cite web|url=https://cs.nyu.edu/~yann/research/ebm/|title=CBLL, Research Projects, Computational and Biological Learning Lab, Courant Institute, NYU|last=LeCun|first=Yann|date=September 2003|website=cs.nyu.edu|access-date=2019-12-27}}</ref>

== Approach ==
EBMs capture dependencies by associating an unnormalized probability scalar (''energy'') to each configuration of the combination of observed and latent variables. Inference consists of finding (values of) latent variables that minimize the energy given a set of (values of) the observed variables. Similarly, the model learns a function that associates low energies to correct values of the latent variables, and higher energies to incorrect values.<ref name=":0" />

Traditional EBMs rely on [[Stochastic gradient descent|stochastic gradient-descent]] (SGD) optimization methods that are typically hard to apply to high-dimension datasets. In 2019, [[Company:OpenAI|OpenAI]] publicized a variant that instead used [[Physics:Langevin dynamics|Langevin dynamics]] (LD). LD is an iterative optimization algorithm that introduces noise to the estimator as part of learning an [[Loss function|objective function]]. It can be used for [[Bayesian inference|Bayesian learning]] scenarios by producing samples from a posterior distribution.<ref name=":0" />

EBMs do not require that energies be normalized as probabilities. In other words, energies do not need to sum to 1. Since there is no need to estimate the [[Normalization (statistics)|normalization]] constant like probabilistic models do, certain forms of inference and learning with EBMs are more tractable and flexible.<ref name=":0" />

Samples are generated implicitly via a [[Markov chain Monte Carlo]] approach.<ref name=":2">{{cite arXiv|last1=Du|first1=Yilun|last2=Mordatch|first2=Igor|date=2019-03-20|title=Implicit Generation and Generalization in Energy-Based Models|eprint=1903.08689|class=cs.LG}}</ref> A replay buffer of past images is used with LD to initialize the optimization module.<ref name=":0" />

== Characteristics ==
EBMs demonstrate useful properties:<ref name=":0" />

* Simplicity and stability–The EBM is the only object that needs to be designed and trained. Separate networks need not be trained to ensure balance.
* Adaptive computation time–An EBM can generate sharp, diverse samples or (more quickly) coarse, less diverse samples. Given infinite time, this procedure produces true samples.<ref name=":1" />
* Flexibility–In [[Autoencoder|Variational Autoencoders]] (VAE) and flow-based models, the generator learns a map from a continuous space to a (possibly) discontinuous space containing different data modes. EBMs can learn to assign low energies to disjoint regions (multiple modes).
* Adaptive generation–EBM generators are implicitly defined by the probability distribution, and automatically adapt as the distribution changes (without training), allowing EBMs to address domains where generator training is impractical, as well as minimizing mode collapse and avoiding spurious modes from out-of-distribution samples.<ref name=":2" />
* Compositionality–Individual models are unnormalized probability distributions, allowing models to be combined through [[Product of experts|product of experts]] or other hierarchical techniques.

== Experimental results ==
On image datasets such as [[CIFAR-10]] and [[ImageNet]] 32x32, an EBM model generated high-quality images relatively quickly. It supported combining features learned from one type of image for generating other types of images. It was able to generalize using out-of-distribution datasets, outperforming flow-based and [[Autoregressive model|autoregressive model]]s. EBM was relatively resistant to adversarial perturbations, behaving better than models explicitly trained against them with training for classification.<ref name=":0" />

== Alternatives ==
EBMs compete with techniques such as [[Variational autoencoder|variational autoencoder]]s (VAEs) or [[Generative adversarial network|Generative Adversarial Neural Networks]] (GANs).<ref name=":0" />

== References ==
{{Reflist}}

== External links ==

* {{Cite web|url=http://www.cs.toronto.edu/~vnair/ciar/|title=CIAR NCAP Summer School|website=www.cs.toronto.edu|access-date=2019-12-27}}
*{{Citation|last1=Dayan|first1=Peter|title=Helmholtz Machine|date=1999|work=Unsupervised Learning|publisher=The MIT Press|isbn=978-0-262-28803-3|last2=Hinton|first2=Geoffrey|last3=Neal|first3=Radford|last4=Zemel|first4=Richard S.|doi=10.7551/mitpress/7011.003.0017}}
*{{Cite journal|last=Hinton|first=Geoffrey E.|date=August 2002|title=Training Products of Experts by Minimizing Contrastive Divergence|journal=Neural Computation|volume=14|issue=8|pages=1771–1800|doi=10.1162/089976602760128018|pmid=12180402|s2cid=207596505|issn=0899-7667}}
*{{Cite journal|last1=Salakhutdinov|first1=Ruslan|last2=Hinton|first2=Geoffrey|date=2009-04-15|title=Deep Boltzmann Machines|url=http://proceedings.mlr.press/v5/salakhutdinov09a.html|journal=Artificial Intelligence and Statistics|language=en|pages=448–455}}

[[Category:Statistical models]]
[[Category:Machine learning]]
[[Category:Statistical mechanics]]
[[Category:Hamiltonian mechanics]]

{{Sourceattribution|Energy based model|1}}