State-dependent information

Short description: State-dependent measures that converge to the mutual information

In information theory, state-dependent information is the generic name given to the family of state-dependent measures that in expectation converge to the mutual information.

State-dependent informations often appear in neuroscience applications.

Let $X$ and $Y$ be random variables and $y$ be a state within $Y$. The state-dependent information between a random variable $X$ and a state $Y=y$ is written as $I(X;Y=y)$. There are currently three known varieties of state-dependent information: specific-surprise, specific-information, and state-specific-information.

Specific-Surprise

The specific-surprise, ${\mathrm{I}}_{\mathrm{s}\mathrm{s}}$, is defined by a Kullback–Leibler divergence,

${\mathrm{I}}_{\mathrm{s}\mathrm{s}}(X;Y=y)\equiv {\mathrm{D}}_{\mathrm{K}\mathrm{L}}[P_{X|y}\parallel P_X]$ .

As a special case of the chain-rule for Kullback-Liebler divergerences, specific-surprise follows the chain-rule for variables. Using $Z$ as a random variable, this is specifically,

${\mathrm{I}}_{\mathrm{s}\mathrm{s}}(XZ;Y=y)={\mathrm{I}}_{\mathrm{s}\mathrm{s}}(X;Y=y)+{\mathrm{I}}_{\mathrm{s}\mathrm{s}}(Z;Y=y|X)$.

Intuitively, specific-surprise is thought of as “how much did my beliefs about $X$ change upon learning that $Y=y$”? Which is zero when there’s no change. It is nonnegative. Specific-surprise has also been called “Bayesian Surprise”.

Specific-Information

The specific-information, ${\mathrm{I}}_{\mathrm{s}\mathrm{i}}$, is defined by a difference of entropies,

${\mathrm{I}}_{\mathrm{s}\mathrm{i}}(X;Y=y)\equiv \mathrm{H}(X)-\mathrm{H}(X|Y=y)$ .

Specific-information follows the chain-rule for states. Using a state $z\in Z$ as a state of random variable $Z$, this is specifically,

${\mathrm{I}}_{\mathrm{s}\mathrm{i}}(X;YZ=yz)={\mathrm{I}}_{\mathrm{s}\mathrm{i}}(X;Y=y)+{\mathrm{I}}_{\mathrm{s}\mathrm{i}}(X;Z=z|Y=y)$ .

Specific-information is interpreted as "how did the uncertainty about $X$ change upon learning $Y=y$?" This can be in the positive or negative. When $X$ follows a uniform distribution, the ${\mathrm{I}}_{\mathrm{s}\mathrm{s}}$ and ${\mathrm{I}}_{\mathrm{s}\mathrm{i}}$ are equivalent.

State-Specific-Information

The state-specific information, ${\mathrm{I}}_{\mathrm{s}\mathrm{s}\mathrm{i}}$, is a synonym for the Pointwise mutual information.

References

• Timme, Nicholas; Lapish, Christopher (2018). "A Tutorial for Information Theory in Neuroscience". eNeuro 5 (3). doi:10.1523/ENEURO.0052-18.2018. PMC 6131830. https://www.eneuro.org/content/5/3/ENEURO.0052-18.2018.long. 
• Deweese, Michael; Meister, Markus (1999). "How to measure the information gained from one symbol". Network: Computation in Neural Systems 10 (4): 325–40. doi:10.1088/0954-898X/10/4/303. PMID 10695762. 
• Butts, Daniel (2003). "How much information is associated with a particular stimulus?". Network: Computation in Neural Systems 14 (2): 177–87. doi:10.1088/0954-898X/14/2/301. PMID 12790180. 
• Itti, Laurent; Baldi, Pierre (2009). "Bayesian surprise attracts human attention". Vision Research 49 (10): 1295–1306. doi:10.1016/j.visres.2008.09.007. PMC 2782645. https://www.sciencedirect.com/science/article/pii/S0042698908004380.