Transfer entropy

From HandWiki

Transfer entropy is a non-parametric statistic measuring the amount of directed (time-asymmetric) transfer of information between two random processes.[1][2][3] Transfer entropy from a process X to another process Y is the amount of uncertainty reduced in future values of Y by knowing the past values of X given past values of Y. More specifically, if [math]\displaystyle{ X_t }[/math] and [math]\displaystyle{ Y_t }[/math] for [math]\displaystyle{ t\in \mathbb{N} }[/math] denote two random processes and the amount of information is measured using Shannon's entropy, the transfer entropy can be written as:

[math]\displaystyle{ T_{X\rightarrow Y} = H\left( Y_t \mid Y_{t-1:t-L}\right) - H\left( Y_t \mid Y_{t-1:t-L}, X_{t-1:t-L}\right), }[/math]

where H(X) is Shannon entropy of X. The above definition of transfer entropy has been extended by other types of entropy measures such as Rényi entropy.[3][4]

Transfer entropy is conditional mutual information,[5][6] with the history of the influenced variable [math]\displaystyle{ Y_{t-1:t-L} }[/math] in the condition:

[math]\displaystyle{ T_{X\rightarrow Y} = I(Y_t ; X_{t-1:t-L} \mid Y_{t-1:t-L}). }[/math]

Transfer entropy reduces to Granger causality for vector auto-regressive processes.[7] Hence, it is advantageous when the model assumption of Granger causality doesn't hold, for example, analysis of non-linear signals.[8][9] However, it usually requires more samples for accurate estimation.[10] The probabilities in the entropy formula can be estimated using different approaches (binning, nearest neighbors) or, in order to reduce complexity, using a non-uniform embedding.[11] While it was originally defined for bivariate analysis, transfer entropy has been extended to multivariate forms, either conditioning on other potential source variables[12] or considering transfer from a collection of sources,[13] although these forms require more samples again.

Transfer entropy has been used for estimation of functional connectivity of neurons,[13][14][15] social influence in social networks[8] and statistical causality between armed conflict events.[16] Transfer entropy is a finite version of the Directed Information which was defined in 1990 by James Massey[17] as [math]\displaystyle{ I(X^n\to Y^n) =\sum_{i=1}^n I(X^i;Y_i|Y^{i-1}) }[/math], where [math]\displaystyle{ X^n }[/math] denotes the vector [math]\displaystyle{ X_1,X_2,...,X_n }[/math] and [math]\displaystyle{ Y^n }[/math] denotes [math]\displaystyle{ Y_1,Y_2,...,Y_n }[/math]. The directed information places an important role in characterizing the fundamental limits (channel capacity) of communication channels with or without feedback [18] [19] and gambling with causal side information.[20]

See also

References

  1. ↑ Schreiber, Thomas (1 July 2000). "Measuring information transfer". Physical Review Letters 85 (2): 461–464. doi:10.1103/PhysRevLett.85.461. PMID 10991308. Bibcode: 2000PhRvL..85..461S. 
  2. ↑ Seth, Anil (2007). "Granger causality". Scholarpedia 2 (7): 1667. doi:10.4249/scholarpedia.1667. Bibcode: 2007SchpJ...2.1667S. 
  3. ↑ 3.0 3.1 Hlaváčková-Schindler, Katerina; Palus, M; Vejmelka, M; Bhattacharya, J (1 March 2007). "Causality detection based on information-theoretic approaches in time series analysis". Physics Reports 441 (1): 1–46. doi:10.1016/j.physrep.2006.12.004. Bibcode: 2007PhR...441....1H. 
  4. ↑ Jizba, Petr; Kleinert, Hagen; Shefaat, Mohammad (2012-05-15). "Rényi's information transfer between financial time series" (in en). Physica A: Statistical Mechanics and Its Applications 391 (10): 2971–2989. doi:10.1016/j.physa.2011.12.064. ISSN 0378-4371. Bibcode: 2012PhyA..391.2971J. 
  5. ↑ Wyner, A. D. (1978). "A definition of conditional mutual information for arbitrary ensembles". Information and Control 38 (1): 51–59. doi:10.1016/s0019-9958(78)90026-8. 
  6. ↑ Dobrushin, R. L. (1959). "General formulation of Shannon's main theorem in information theory". Uspekhi Mat. Nauk 14: 3–104. 
  7. ↑ Barnett, Lionel (1 December 2009). "Granger Causality and Transfer Entropy Are Equivalent for Gaussian Variables". Physical Review Letters 103 (23): 238701. doi:10.1103/PhysRevLett.103.238701. PMID 20366183. Bibcode: 2009PhRvL.103w8701B. 
  8. ↑ 8.0 8.1 Ver Steeg, Greg; Galstyan, Aram (2012). "Information transfer in social media". ACM. pp. 509–518. Bibcode: 2011arXiv1110.2724V. 
  9. ↑ Lungarella, M.; Ishiguro, K.; Kuniyoshi, Y.; Otsu, N. (1 March 2007). "Methods for quantifying the causal structure of bivariate time series". International Journal of Bifurcation and Chaos 17 (3): 903–921. doi:10.1142/S0218127407017628. Bibcode: 2007IJBC...17..903L. 
  10. ↑ Pereda, E; Quiroga, RQ; Bhattacharya, J (Sep–Oct 2005). "Nonlinear multivariate analysis of neurophysiological signals.". Progress in Neurobiology 77 (1–2): 1–37. doi:10.1016/j.pneurobio.2005.10.003. PMID 16289760. Bibcode: 2005nlin.....10077P. 
  11. ↑ Montalto, A; Faes, L; Marinazzo, D (Oct 2014). "MuTE: A MATLAB Toolbox to Compare Established and Novel Estimators of the Multivariate Transfer Entropy.". PLOS ONE 9 (10): e109462. doi:10.1371/journal.pone.0109462. PMID 25314003. Bibcode: 2014PLoSO...9j9462M. 
  12. ↑ Lizier, Joseph; Prokopenko, Mikhail; Zomaya, Albert (2008). "Local information transfer as a spatiotemporal filter for complex systems". Physical Review E 77 (2): 026110. doi:10.1103/PhysRevE.77.026110. PMID 18352093. Bibcode: 2008PhRvE..77b6110L. 
  13. ↑ 13.0 13.1 Lizier, Joseph; Heinzle, Jakob; Horstmann, Annette; Haynes, John-Dylan; Prokopenko, Mikhail (2011). "Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity". Journal of Computational Neuroscience 30 (1): 85–107. doi:10.1007/s10827-010-0271-2. PMID 20799057. 
  14. ↑ Vicente, Raul; Wibral, Michael; Lindner, Michael; Pipa, Gordon (February 2011). "Transfer entropy—a model-free measure of effective connectivity for the neurosciences". Journal of Computational Neuroscience 30 (1): 45–67. doi:10.1007/s10827-010-0262-3. PMID 20706781. 
  15. ↑ Shimono, Masanori; Beggs, John (October 2014). "Functional clusters, hubs, and communities in the cortical microconnectome". Cerebral Cortex 25 (10): 3743–57. doi:10.1093/cercor/bhu252. PMID 25336598. 
  16. ↑ Kushwaha, Niraj; Lee, Edward D (July 2023). "Discovering the mesoscale for chains of conflict". PNAS Nexus 2 (7). doi:10.1093/pnasnexus/pgad228. ISSN 2752-6542. PMID 37533894. PMC 10392960. https://doi.org/10.1093/pnasnexus/pgad228. 
  17. ↑ Massey, James (1990). Causality, Feedback And Directed Information. 
  18. ↑ Permuter, Haim Henry; Weissman, Tsachy; Goldsmith, Andrea J. (February 2009). "Finite State Channels With Time-Invariant Deterministic Feedback". IEEE Transactions on Information Theory 55 (2): 644–662. doi:10.1109/TIT.2008.2009849. 
  19. ↑ Kramer, G. (January 2003). "Capacity results for the discrete memoryless network". IEEE Transactions on Information Theory 49 (1): 4–21. doi:10.1109/TIT.2002.806135. 
  20. ↑ Permuter, Haim H.; Kim, Young-Han; Weissman, Tsachy (June 2011). "Interpretations of Directed Information in Portfolio Theory, Data Compression, and Hypothesis Testing". IEEE Transactions on Information Theory 57 (6): 3248–3259. doi:10.1109/TIT.2011.2136270. 

External links



Retrieved from "https://handwiki.org/wiki/index.php?title=Transfer_entropy&oldid=3365253"