Partial information decomposition

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.

Motivation

Information theory can quantify the amount of information a single source variable $X_1$ has about a target variable $Y$ via the mutual information $I(X_1;Y)$. If we now consider a second source variable $X_2$, classical information theory can only describe the mutual information of the joint variable $\{X_1,X_2\}$ with $Y$, given by $I(X_1,X_2;Y)$. In general however, it would be interesting to know how exactly the individual variables $X_1$ and $X_2$ and their interactions relate to $Y$.

Consider that we are given two source variables $X_1,X_2\in \{0,1\}$ and a target variable $Y=XOR(X_1,X_2)$. In this case the total mutual information $I(X_1,X_2;Y)=1$, while the individual mutual information $I(X_1;Y)=I(X_2;Y)=0$. That is, there is synergistic information arising from the interaction of $X_1,X_2$ about $Y$, which cannot be easily captured with classical information theoretic quantities.

As mutual information is able to capture non-linear, non-monotone statistical relationships between variables, the PID framework would be able to quantify mulitvariate statistical dependencies in arbitrary complex systems in a much more general way than e.g. the correlation coefficient and is especially able to distinguish between different kinds of interaction between variables.

Definition

Partial information decomposition further decomposes the mutual information between the source variables $\{X_1,X_2\}$ with the target variable $Y$ as

$I(X_1,X_2;Y)=\text{Unq}(X_1;Y\setminus X_2)+\text{Unq}(X_2;Y\setminus X_1)+\text{Syn}(X_1,X_2;Y)+\text{Red}(X_1,X_2;Y)$

Here the individual information atoms are defined as

• $\text{Unq}(X_1;Y\setminus X_2)$ is the unique information that $X_1$ has about $Y$, which is not in $X_2$
• $\text{Syn}(X_1,X_2;Y)$ is the synergistic information that is in the interaction of $X_1$ and $X_2$ about $Y$
• $\text{Red}(X_1,X_2;Y)$ is the redundant information that is in both $X_1$ or $X_2$ about $Y$

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.

However, once an appropriate definition of redundant information has been chosen, the decomposition reduces to a Möbius inversion and can be calculated using the Fast Möbius Transform.

Applications

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology, neuroscience sociology, and machine learning Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems and may be relevant to formal theories of consciousness.

See also

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information