Empirical dynamic modeling

From HandWiki

Empirical dynamic modeling (EDM) is a framework for analysis and prediction of nonlinear dynamical systems. Applications include population dynamics,[1][2][3][4][5][6] ecosystem service,[7] medicine,[8] neuroscience,[9][10][11] dynamical systems,[12][13][14] geophysics,[15][16][17] and human-computer interaction.[18] EDM was originally developed by Robert May and George Sugihara. It can be considered a methodology for data modeling, predictive analytics, dynamical system analysis, machine learning and time series analysis.

Description

Mathematical models have tremendous power to describe observations of real-world systems. They are routinely used to test hypothesis, explain mechanisms and predict future outcomes. However, real-world systems are often nonlinear and multidimensional, in some instances rendering explicit equation-based modeling problematic. Empirical models, which infer patterns and associations from the data instead of using hypothesized equations, represent a natural and flexible framework for modeling complex dynamics.

Donald DeAngelis and Simeon Yurek illustrated that canonical statistical models are ill-posed when applied to nonlinear dynamical systems.[19] A hallmark of nonlinear dynamics is state-dependence: system states are related to previous states governing transition from one state to another. EDM operates in this space, the multidimensional state-space of system dynamics rather than on one-dimensional observational time series. EDM does not presume relationships among states, for example, a functional dependence, but projects future states from localised, neighboring states. EDM is thus a state-space, nearest-neighbors paradigm where system dynamics are inferred from states derived from observational time series. This provides a model-free representation of the system naturally encompassing nonlinear dynamics.

A cornerstone of EDM is recognition that time series observed from a dynamical system can be transformed into higher-dimensional state-spaces by time-delay embedding with Takens's theorem. The state-space models are evaluated based on in-sample fidelity to observations, conventionally with Pearson correlation between predictions and observations.

Methods

EDM is continuing to evolve. As of 2022, the main algorithms are Simplex projection,[20] Sequential locally weighted global linear maps (S-Map) projection,[21] Multivariate embedding in Simplex or S-Map,[1] Convergent cross mapping (CCM),[22] and Multiview Embeding,[23] described below.

Nomenclature
Parameter Description
[math]\displaystyle{ E }[/math] embedding dimension
[math]\displaystyle{ k }[/math] number of nearest neighbors
[math]\displaystyle{ T_p }[/math] prediction interval
[math]\displaystyle{ X\in \R }[/math] observed time series
[math]\displaystyle{ y\in \R^{E} }[/math] vector of lagged observations
[math]\displaystyle{ \theta \geq 0 }[/math] S-Map localization
[math]\displaystyle{ X_t^E = (X_t, X_{t-1},\dots, X_{t-E+1} ) \in \R^E }[/math] lagged embedding vectors
[math]\displaystyle{ \| v \| }[/math] norm of v
[math]\displaystyle{ N = \{N_1,\dots,N_k\} }[/math] list of nearest neighbors

Nearest neighbors are found according to: [math]\displaystyle{ \text{NN}(y, X, k) = \| X_{N_i}^{E} - y\| \leq \| X_{N_j}^{E} - y\| \text{ if } 1 \leq i \leq j \leq k }[/math]

Simplex

Simplex projection[20][24][25][26] is a nearest neighbor projection. It locates the [math]\displaystyle{ k }[/math] nearest neighbors to the location in the state-space from which a prediction is desired. To minimize the number of free parameters [math]\displaystyle{ k }[/math] is typically set to [math]\displaystyle{ E+1 }[/math] defining an [math]\displaystyle{ E+1 }[/math] dimensional simplex in the state-space. The prediction is computed as the average of the weighted phase-space simplex projected [math]\displaystyle{ Tp }[/math] points ahead. Each neighbor is weighted proportional to their distance to the projection origin vector in the state-space.

  1. Find [math]\displaystyle{ k }[/math] nearest neighbor: [math]\displaystyle{ N_k \gets \text{NN}(y, X, k) }[/math]
  2. Define the distance scale: [math]\displaystyle{ d \gets \| X_{N_1}^{E} - y\| }[/math]
  3. Compute weights: For{[math]\displaystyle{ i=1,\dots,k }[/math]} : [math]\displaystyle{ w_i \gets \exp (-\| X_{N_i}^{E} - y\| / d ) }[/math]
  4. Average of state-space simplex: [math]\displaystyle{ \hat{y} \gets \sum_{i = 1}^{k} \left(w_iX_{N_i+T_p}\right) / \sum_{i = 1}^{k} w_i }[/math]

S-Map

S-Map[21] extends the state-space prediction in Simplex from an average of the [math]\displaystyle{ E+1 }[/math] nearest neighbors to a linear regression fit to all neighbors, but localised with an exponential decay kernel. The exponential localisation function is [math]\displaystyle{ F(\theta) = \text{exp}(-\theta d/D) }[/math], where [math]\displaystyle{ d }[/math] is the neighbor distance and [math]\displaystyle{ D }[/math] the mean distance. In this way, depending on the value of [math]\displaystyle{ \theta }[/math], neighbors close to the prediction origin point have a higher weight than those further from it, such that a local linear approximation to the nonlinear system is reasonable. This localisation ability allows one to identify an optimal local scale, in-effect quantifying the degree of state dependence, and hence nonlinearity of the system.

Another feature of S-Map is that for a properly fit model, the regression coefficients between variables have been shown to approximate the gradient (directional derivative) of variables along the manifold.[27] These Jacobians represent the time-varying interaction strengths between system variables.

  1. Find [math]\displaystyle{ k }[/math] nearest neighbor: [math]\displaystyle{ N \gets \text{NN}(y, X, k) }[/math]
  2. Sum of distances: [math]\displaystyle{ D \gets \frac{1}{k} \sum_{i=1}^k \| X_{N_i}^{E} - y\| }[/math]
  3. Compute weights: For{[math]\displaystyle{ i=1,\dots,k }[/math]} : [math]\displaystyle{ w_i \gets \exp (-\theta \| X_{N_i}^{E} - y\| / D ) }[/math]
  4. Reweighting matrix: [math]\displaystyle{ W \gets \text{diag}(w_i) }[/math]
  5. Design matrix: [math]\displaystyle{ A \gets \begin{bmatrix} 1 & X_{N_1} & X_{N_1- 1} & \dots & X_{N_1 - E + 1} \\ 1 & X_{N_2} & X_{N_2- 1} & \dots & X_{N_2 - E + 1} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & X_{N_k} & X_{N_k- 1} & \dots & X_{N_k - E + 1} \end{bmatrix} }[/math]
  6. Weighted design matrix: [math]\displaystyle{ A \gets WA }[/math]
  7. Response vector at [math]\displaystyle{ Tp }[/math]: [math]\displaystyle{ b \gets \begin{bmatrix} X_{N_1 + T_p} \\ X_{N_2 + T_p} \\ \vdots \\ X_{N_k + T_p} \end{bmatrix} }[/math]
  8. Weighted response vector: [math]\displaystyle{ b \gets Wb }[/math]
  9. Least squares solution (SVD): [math]\displaystyle{ \hat{c} \gets \text{argmin}_{c}\| Ac - b \|_2^2 }[/math]
  10. Local linear model [math]\displaystyle{ \hat{c} }[/math] is prediction: [math]\displaystyle{ \hat{y} \gets \hat{c}_0 + \sum_{i=1}^E\hat{c}_iy_i }[/math]

Multivariate Embedding

Multivariate Embedding[1][12][28] recognizes that time-delay embeddings are not the only valid state-space construction. In Simplex and S-Map one can generate a state-space from observational vectors, or time-delay embeddings of a single observational time series, or both.

Convergent Cross Mapping

Convergent cross mapping (CCM)[22] leverages a corollary to the Generalized Takens Theorem[12] that it should be possible to cross predict or cross map between variables observed from the same system. Suppose that in some dynamical system involving variables [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math], [math]\displaystyle{ X }[/math] causes [math]\displaystyle{ Y }[/math]. Since [math]\displaystyle{ X }[/math] and [math]\displaystyle{ Y }[/math] belong to the same dynamical system, their reconstructions (via embeddings) [math]\displaystyle{ M_{x} }[/math], and [math]\displaystyle{ M_{y} }[/math], also map to the same system.

The causal variable [math]\displaystyle{ X }[/math] leaves a signature on the affected variable [math]\displaystyle{ Y }[/math], and consequently, the reconstructed states based on [math]\displaystyle{ Y }[/math] can be used to cross predict values of [math]\displaystyle{ X }[/math]. CCM leverages this property to infer causality by predicting [math]\displaystyle{ X }[/math] using the [math]\displaystyle{ M_{y} }[/math] library of points (or vice versa for the other direction of causality), while assessing improvements in cross map predictability as larger and larger random samplings of [math]\displaystyle{ M_{y} }[/math] are used. If the prediction skill of [math]\displaystyle{ X }[/math] increases and saturates as the entire [math]\displaystyle{ M_{y} }[/math] is used, this provides evidence that [math]\displaystyle{ X }[/math] is casually influencing [math]\displaystyle{ Y }[/math].

Multiview Embedding

Multiview Embedding[23] is a Dimensionality reduction technique where a large number of state-space time series vectors are combitorially assessed towards maximal model predictability.

Extensions

Extensions to EDM techniques include:

  • Generalized Theorems for Nonlinear State Space Reconstruction[12]
  • Extended Convergent Cross Mapping[13]
  • Dynamic stability[4]
  • S-Map regularization[29]
  • Visual analytics with EDM[30]
  • Convergent Cross Sorting[31]
  • Expert system with EDM hybrid[32]
  • Sliding windows based on the extended convergent cross-mapping[33]
  • Empirical Mode Modeling[17]
  • Variable step sizes with bundle embedding[34]
  • Multiview distance regularised S-map[35]

See also

References

  1. ↑ 1.0 1.1 1.2 [1]Dixon, P. A., et al. 1999. Episodic fluctuations in larval supply. Science 283:1528–1530
  2. ↑ [2]Hao Ye, Richard J. Beamish, Sarah M. Glaser, et al. 2015. Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) E1569-E1576; DOI: 10.1073/pnas.1417063112
  3. ↑ [3]Ethan R. Deyle, Michael Fogarty, Chih-hao Hsieh, et al. 2013. Proceedings of the National Academy of Sciences Apr 2013, 110 (16) 6430-6435; DOI: 10.1073/pnas.1215506110
  4. ↑ 4.0 4.1 [4]Ushio, M., Hsieh, Ch., Masuda, R. et al., 2018. Fluctuating interaction network and time-varying stability of a natural fish community. Nature 554, 360–363
  5. ↑ [5]Deyle E.R., et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
  6. ↑ [6]Tanya L. Rogers, Stephan B. Munch, Simon D. Stewart, Eric P. Palkovacs, Alfredo Giron-Nava, Shin-ichiro S. Matsuzaki, Celia C. Symons. Ecology Letters, 23 (8) August 2020, 1287-1297
  7. ↑ [7]Park J., et al. 2021. Dynamics of Florida milk production and total phosphate in Lake Okeechobee. PLoS ONE 16(8): e0248910. doi:10.1371/journal.pone.0248910
  8. ↑ [8]George Sugihara, Walter Allan, Daniel Sobel, and Kenneth D. Allan, 1996. Nonlinear control of heart rate variability in human infants. Proc. Natl. Acad. Sci. USA. Vol. 93, pp. 2608-2613, March 1996. Medical Sciences
  9. ↑ [9]McBride, J. C., et al. Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease. Neuroimage-Clinical 7:258–265 (2015)
  10. ↑ [10]Tajima S, Yanagawa T, Fujii N, Toyoizumi T (2015) Untangling Brain-Wide Dynamics in Consciousness by Cross-Embedding. PLoS Comput Biol 11(11): e1004537. https://doi.org/10.1371/journal.pcbi.1004537
  11. ↑ [11]W. Watanakeesuntorn et al., "Massively Parallel Causal Inference of Whole Brain Dynamics at Single Neuron Resolution," 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), 2020, pp. 196-205, doi: 10.1109/ICPADS51040.2020.00035
  12. ↑ 12.0 12.1 12.2 12.3 [12] Deyle ER, Sugihara G (2011) Generalized Theorems for Nonlinear State Space Reconstruction. PLoS ONE 6(3): e18295. doi:10.1371/journal.pone.0018295
  13. ↑ 13.0 13.1 [13]Ye, H., Deyle, E., Gilarranz, L. et al., 2015. Distinguishing time-delayed causal interactions using convergent cross mapping. Sci Rep 5, 14750 (2015). doi:10.1038/srep14750
  14. ↑ [14]Cenci, S., Saavedra, S. Non-parametric estimation of the structural stability of non-equilibrium community dynamics. Nat Ecol Evol 3, 912–918 (2019). https://doi.org/10.1038/s41559-019-0879-1
  15. ↑ [15]Tsonis A. A., et al. Dynamical evidence for causality between galactic cosmic rays and interannual variation in global temperature. Proc Natl Acad Sci 112(11):3253–3256 (2015).
  16. ↑ [16]Nes EH Van, et al. Causal feedbacks in climate change. Nat Clim Chang 5(5):445–448 (2015)
  17. ↑ 17.0 17.1 [17]Park, J., et al. Empirical mode modeling. Nonlinear Dyn (2022). https://doi.org/10.1007/s11071-022-07311-y
  18. ↑ van Berkel, Niels; Dennis, Simon; Zyphur, Michael; Li, Jinjing; Heathcote, Andrew; Kostakos, Vassilis (2021-07-04). "Modeling interaction as a complex system". Human–Computer Interaction 36 (4): 279–305. doi:10.1080/07370024.2020.1715221. ISSN 0737-0024. https://doi.org/10.1080/07370024.2020.1715221. 
  19. ↑ [18]Donald L. DeAngelis, Simeon Yurek, 2015, Equation-free modeling unravels the behavior of complex ecological systems. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) 3856-3857; DOI: 10.1073/pnas.1503154112
  20. ↑ 20.0 20.1 [19] Sugihara G. and May R., 1990. Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344:734–741
  21. ↑ 21.0 21.1 [20] Sugihara G., 1994. Nonlinear forecasting for the classification of natural time series. Philosophical Transactions: Physical Sciences and Engineering, 348 (1688) : 477–495
  22. ↑ 22.0 22.1 [21] Sugihara G., May R., Ye H., et al. 2012. Detecting Causality in Complex Ecosystems. Science 338:496-500
  23. ↑ 23.0 23.1 [22] Ye H., and G. Sugihara, 2016. Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality. Science 353:922–925
  24. ↑ [23] Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence (pp. 366–381). Springer.
  25. ↑ [24] Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenomena, 35(3), 335–356.
  26. ↑ [25] Judd, K., & Mees, A. (1998). Embedding as a modeling problem. Physica D: Nonlinear Phenomena, 120(3), 273–286.
  27. ↑ [26]Deyle ER. et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
  28. ↑ [27] Sauer, T., Yorke, J. A., & Casdagli, M. (1991). Embedology. Journal of Statistical Physics, 65(3), 579–616
  29. ↑ [28]Cenci S, Sugihara G, Saavedra S, 2019. Regularized S-map for inference and forecasting with noisy ecological time series, METHODS IN ECOLOGY AND EVOLUTION, 10 (5), 650-660
  30. ↑ [29] Hiroaki Natsukawa, et al. 2021. A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling. IEEE Transactions on Visualization and Computer Graphics. Feb. 2021, 506-516, vol. 27 DOI: 10.1109/TVCG.2020.3028956
  31. ↑ [30] Breston, L., Leonardis, E.J., Quinn, L.K. et al. 2021. Convergent cross sorting for estimating dynamic coupling. Sci Rep 11, 20374 (2021). doi:10.1038/s41598-021-98864-2
  32. ↑ [31] Deyle E. R. et al. A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures. PNAS Vol. 119, No. 26 (2022).
  33. ↑ [32] Ge, X., Lin, A. Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping. Nonlinear Dyn 104, 1753–1765 (2021). https://doi.org/10.1007/s11071-021-06362-x
  34. ↑ [33] Bethany Johnson, Stephan B. Munch. 2022. An empirical dynamic modeling framework for missing or irregular samples. Ecological Modelling, Volume 468, June 2022, 109948.
  35. ↑ [34] Chang, C.-W., Miki, T., Ushio, M., et al. (2021) Reconstructing large interaction networks from empirical time series data. Ecology Letters, 24, 2763– 2774. https://doi.org/10.1111/ele.13897

Further reading

External links

Animations
Online books or lecture notes
Research groups
  • Sugihara Lab, Scripps Institution of Oceanography, University of California San Diego.



Retrieved from "https://handwiki.org/wiki/index.php?title=Empirical_dynamic_modeling&oldid=3384322"