Sensor fusion

From HandWiki
Revision as of 22:59, 6 February 2024 by DanMescoff (talk | contribs) (over-write)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Short description: Combining of sensor data from disparate sources
Eurofighter sensor fusion

Sensor fusion is the process of combining sensor data or data derived from disparate sources such that the resulting information has less uncertainty than would be possible when these sources were used individually. For instance, one could potentially obtain a more accurate location estimate of an indoor object by combining multiple data sources such as video cameras and WiFi localization signals. The term uncertainty reduction in this case can mean more accurate, more complete, or more dependable, or refer to the result of an emerging view, such as stereoscopic vision (calculation of depth information by combining two-dimensional images from two cameras at slightly different viewpoints).[1][2]

The data sources for a fusion process are not specified to originate from identical sensors. One can distinguish direct fusion, indirect fusion and fusion of the outputs of the former two. Direct fusion is the fusion of sensor data from a set of heterogeneous or homogeneous sensors, soft sensors, and history values of sensor data, while indirect fusion uses information sources like a priori knowledge about the environment and human input.

Sensor fusion is also known as (multi-sensor) data fusion and is a subset of information fusion.

Examples of sensors

Algorithms

Sensor fusion is a term that covers a number of methods and algorithms, including:

Example calculations

Two example sensor fusion calculations are illustrated below.

Let [math]\displaystyle{ {\textbf{x}}_1 }[/math] and [math]\displaystyle{ {\textbf{x}}_2 }[/math] denote two sensor measurements with noise variances [math]\displaystyle{ \scriptstyle\sigma_1^2 }[/math] and [math]\displaystyle{ \scriptstyle\sigma_2^2 }[/math] , respectively. One way of obtaining a combined measurement [math]\displaystyle{ {\textbf{x}}_3 }[/math] is to apply inverse-variance weighting, which is also employed within the Fraser-Potter fixed-interval smoother, namely [6]

[math]\displaystyle{ {\textbf{x}}_3 = \sigma_3^{2} (\sigma_1^{-2}{\textbf{x}}_1 + \sigma_2^{-2}{\textbf{x}}_2) }[/math] ,

where [math]\displaystyle{ \scriptstyle\sigma_3^{2} = (\scriptstyle\sigma_1^{-2} + \scriptstyle\sigma_2^{-2})^{-1} }[/math] is the variance of the combined estimate. It can be seen that the fused result is simply a linear combination of the two measurements weighted by their respective noise variances.

Another method to fuse two measurements is to use the optimal Kalman filter. Suppose that the data is generated by a first-order system and let [math]\displaystyle{ {\textbf{P}}_k }[/math] denote the solution of the filter's Riccati equation. By applying Cramer's rule within the gain calculation it can be found that the filter gain is given by:[citation needed]

[math]\displaystyle{ {\textbf{L}}_k = \begin{bmatrix} \tfrac{\scriptstyle\sigma_2^{2}{\textbf{P}}_k}{\scriptstyle\sigma_2^{2}{\textbf{P}}_k + \scriptstyle\sigma_1^{2}{\textbf{P}}_k + \scriptstyle\sigma_1^{2} \scriptstyle\sigma_2^{2}} & \tfrac{\scriptstyle\sigma_1^{2}{\textbf{P}}_k}{\scriptstyle\sigma_2^{2}{\textbf{P}}_k + \scriptstyle\sigma_1^{2}{\textbf{P}}_k + \scriptstyle\sigma_1^{2} \scriptstyle\sigma_2^{2}} \end{bmatrix}. }[/math]

By inspection, when the first measurement is noise free, the filter ignores the second measurement and vice versa. That is, the combined estimate is weighted by the quality of the measurements.

Centralized versus decentralized

In sensor fusion, centralized versus decentralized refers to where the fusion of the data occurs. In centralized fusion, the clients simply forward all of the data to a central location, and some entity at the central location is responsible for correlating and fusing the data. In decentralized, the clients take full responsibility for fusing the data. "In this case, every sensor or platform can be viewed as an intelligent asset having some degree of autonomy in decision-making."[7]

Multiple combinations of centralized and decentralized systems exist.

Another classification of sensor configuration refers to the coordination of information flow between sensors.[8][9] These mechanisms provide a way to resolve conflicts or disagreements and to allow the development of dynamic sensing strategies. Sensors are in redundant (or competitive) configuration if each node delivers independent measures of the same properties. This configuration can be used in error correction when comparing information from multiple nodes. Redundant strategies are often used with high level fusions in voting procedures.[10][11] Complementary configuration occurs when multiple information sources supply different information about the same features. This strategy is used for fusing information at raw data level within decision-making algorithms. Complementary features are typically applied in motion recognition tasks with Neural network,[12][13] Hidden Markov model,[14][15] Support-vector machine,[16] clustering methods and other techniques.[16][15] Cooperative sensor fusion uses the information extracted by multiple independent sensors to provide information that would not be available from single sensors. For example, sensors connected to body segments are used for the detection of the angle between them. Cooperative sensor strategy gives information impossible to obtain from single nodes. Cooperative information fusion can be used in motion recognition,[17] gait analysis, motion analysis,[18][19],.[20]

Levels

There are several categories or levels of sensor fusion that are commonly used. [21] [22] [23] [24] [25] [26]

  • Level 0 – Data alignment
  • Level 1 – Entity assessment (e.g. signal/feature/object).
    • Tracking and object detection/recognition/identification
  • Level 2 – Situation assessment
  • Level 3 – Impact assessment
  • Level 4 – Process refinement (i.e. sensor management)
  • Level 5 – User refinement

Sensor fusion level can also be defined basing on the kind of information used to feed the fusion algorithm.[27] More precisely, sensor fusion can be performed fusing raw data coming from different sources, extrapolated features or even decision made by single nodes.

  • Data level - data level (or early) fusion aims to fuse raw data from multiple sources and represent the fusion technique at the lowest level of abstraction. It is the most common sensor fusion technique in many fields of application. Data level fusion algorithms usually aim to combine multiple homogeneous sources of sensory data to achieve more accurate and synthetic readings.[28] When portable devices are employed data compression represent an important factor, since collecting raw information from multiple sources generates huge information spaces that could define an issue in terms of memory or communication bandwidth for portable systems. Data level information fusion tends to generate big input spaces, that slow down the decision-making procedure. Also, data level fusion often cannot handle incomplete measurements. If one sensor modality becomes useless due to malfunctions, breakdown or other reasons the whole systems could occur in ambiguous outcomes.
  • Feature level - features represent information computed on board by each sensing node. These features are then sent to a fusion node to feed the fusion algorithm.[29] This procedure generates smaller information spaces with respect to the data level fusion, and this is better in terms of computational load. Obviously, it is important to properly select features on which to define classification procedures: choosing the most efficient features set should be a main aspect in method design. Using features selection algorithms that properly detect correlated features and features subsets improves the recognition accuracy but large training sets are usually required to find the most significant feature subset.[27]
  • Decision level - decision level (or late) fusion is the procedure of selecting an hypothesis from a set of hypotheses generated by individual (usually weaker) decisions of multiple nodes.[30] It is the highest level of abstraction and uses the information that has been already elaborated through preliminary data- or feature level processing. The main goal in decision fusion is to use meta-level classifier while data from nodes are preprocessed by extracting features from them.[31] Typically decision level sensor fusion is used in classification an recognition activities and the two most common approaches are majority voting and Naive-Bayes.[citation needed] Advantages coming from decision level fusion include communication bandwidth and improved decision accuracy. It also allows the combination of heterogeneous sensors.[29]

Applications

One application of sensor fusion is GPS/INS, where Global Positioning System and inertial navigation system data is fused using various different methods, e.g. the extended Kalman filter. This is useful, for example, in determining the attitude of an aircraft using low-cost sensors.[32] Another example is using the data fusion approach to determine the traffic state (low traffic, traffic jam, medium flow) using road side collected acoustic, image and sensor data.[33] In the field of autonomous driving, sensor fusion is used to combine the redundant information from complementary sensors in order to obtain a more accurate and reliable representation of the environment.[34]

Although technically not a dedicated sensor fusion method, modern Convolutional neural network based methods can simultaneously process many channels of sensor data (such as Hyperspectral imaging with hundreds of bands [35]) and fuse relevant information to produce classification results.

See also


References

  1. ↑ Elmenreich, W. (2002). Sensor Fusion in Time-Triggered Systems, PhD Thesis. Vienna, Austria: Vienna University of Technology. p. 173. https://mobile.aau.at/~welmenre/papers/elmenreich_Dissertation_sensorFusionInTimeTriggeredSystems.pdf. 
  2. ↑ Haghighat, Mohammad Bagher Akbari; Aghagolzadeh, Ali; Seyedarabi, Hadi (2011). "Multi-focus image fusion for visual sensor networks in DCT domain". Computers & Electrical Engineering 37 (5): 789–797. doi:10.1016/j.compeleceng.2011.04.016. 
  3. ↑ Li, Wangyan; Wang, Zidong; Wei, Guoliang; Ma, Lifeng; Hu, Jun; Ding, Derui (2015). "A Survey on Multisensor Fusion and Consensus Filtering for Sensor Networks" (in en). Discrete Dynamics in Nature and Society 2015: 1–12. doi:10.1155/2015/683701. ISSN 1026-0226. 
  4. ↑ Badeli, Vahid; Ranftl, Sascha; Melito, Gian Marco; Reinbacher-Köstinger, Alice; Von Der Linden, Wolfgang; Ellermann, Katrin; Biro, Oszkar (2021-01-01). "Bayesian inference of multi-sensors impedance cardiography for detection of aortic dissection". COMPEL - the International Journal for Computation and Mathematics in Electrical and Electronic Engineering 41 (3): 824–839. doi:10.1108/COMPEL-03-2021-0072. ISSN 0332-1649. https://doi.org/10.1108/COMPEL-03-2021-0072. 
  5. ↑ Ranftl, Sascha; Melito, Gian Marco; Badeli, Vahid; Reinbacher-Köstinger, Alice; Ellermann, Katrin; von der Linden, Wolfgang (2019-12-31). "Bayesian Uncertainty Quantification with Multi-Fidelity Data and Gaussian Processes for Impedance Cardiography of Aortic Dissection". Entropy 22 (1): 58. doi:10.3390/e22010058. ISSN 1099-4300. PMID 33285833. 
  6. ↑ Maybeck, S. (1982). Stochastic Models, Estimating, and Control. River Edge, NJ: Academic Press. 
  7. ↑ N. Xiong; P. Svensson (2002). "Multi-sensor management for information fusion: issues and approaches". Information Fusion. p. 3(2):163–186. http://www.elsevier.com/locate/inffus. 
  8. ↑ Durrant-Whyte, Hugh F. (2016). "Sensor Models and Multisensor Integration". The International Journal of Robotics Research 7 (6): 97–113. doi:10.1177/027836498800700608. ISSN 0278-3649. 
  9. ↑ Galar, Diego; Kumar, Uday (2017). eMaintenance: Essential Electronic Tools for Efficiency. Academic Press. p. 26. ISBN 9780128111543. 
  10. ↑ Li, Wenfeng; Bao, Junrong; Fu, Xiuwen; Fortino, Giancarlo; Galzarano, Stefano (2012). "Human Postures Recognition Based on D-S Evidence Theory and Multi-sensor Data Fusion". 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012). pp. 912–917. doi:10.1109/CCGrid.2012.144. ISBN 978-1-4673-1395-7. 
  11. ↑ Fortino, Giancarlo; Gravina, Raffaele (2015). "Fall-MobileGuard: a Smart Real-Time Fall Detection System". Proceedings of the 10th EAI International Conference on Body Area Networks. doi:10.4108/eai.28-9-2015.2261462. ISBN 978-1-63190-084-6. 
  12. ↑ Tao, Shuai; Zhang, Xiaowei; Cai, Huaying; Lv, Zeping; Hu, Caiyou; Xie, Haiqun (2018). "Gait based biometric personal authentication by using MEMS inertial sensors". Journal of Ambient Intelligence and Humanized Computing 9 (5): 1705–1712. doi:10.1007/s12652-018-0880-6. ISSN 1868-5137. 
  13. ↑ Dehzangi, Omid; Taherisadr, Mojtaba; ChangalVala, Raghvendar (2017). "IMU-Based Gait Recognition Using Convolutional Neural Networks and Multi-Sensor Fusion". Sensors 17 (12): 2735. doi:10.3390/s17122735. ISSN 1424-8220. PMID 29186887. Bibcode: 2017Senso..17.2735D. 
  14. ↑ Guenterberg, E.; Yang, A.Y.; Ghasemzadeh, H.; Jafari, R.; Bajcsy, R.; Sastry, S.S. (2009). "A Method for Extracting Temporal Parameters Based on Hidden Markov Models in Body Sensor Networks With Inertial Sensors". IEEE Transactions on Information Technology in Biomedicine 13 (6): 1019–1030. doi:10.1109/TITB.2009.2028421. ISSN 1089-7771. PMID 19726268. http://www.eecs.berkeley.edu/~yang/paper/GuenterbergE-Biomedicine.pdf. 
  15. ↑ 15.0 15.1 Parisi, Federico; Ferrari, Gianluigi; Giuberti, Matteo; Contin, Laura; Cimolin, Veronica; Azzaro, Corrado; Albani, Giovanni; Mauro, Alessandro (2016). "Inertial BSN-Based Characterization and Automatic UPDRS Evaluation of the Gait Task of Parkinsonians". IEEE Transactions on Affective Computing 7 (3): 258–271. doi:10.1109/TAFFC.2016.2549533. ISSN 1949-3045. 
  16. ↑ 16.0 16.1 Gao, Lei; Bourke, A.K.; Nelson, John (2014). "Evaluation of accelerometer based multi-sensor versus single-sensor activity recognition systems". Medical Engineering & Physics 36 (6): 779–785. doi:10.1016/j.medengphy.2014.02.012. ISSN 1350-4533. PMID 24636448. 
  17. ↑ Xu, James Y.; Wang, Yan; Barrett, Mick; Dobkin, Bruce; Pottie, Greg J.; Kaiser, William J. (2016). "Personalized Multilayer Daily Life Profiling Through Context Enabled Activity Classification and Motion Reconstruction: An Integrated System Approach". IEEE Journal of Biomedical and Health Informatics 20 (1): 177–188. doi:10.1109/JBHI.2014.2385694. ISSN 2168-2194. PMID 25546868. 
  18. ↑ Chia Bejarano, Noelia; Ambrosini, Emilia; Pedrocchi, Alessandra; Ferrigno, Giancarlo; Monticone, Marco; Ferrante, Simona (2015). "A Novel Adaptive, Real-Time Algorithm to Detect Gait Events From Wearable Sensors". IEEE Transactions on Neural Systems and Rehabilitation Engineering 23 (3): 413–422. doi:10.1109/TNSRE.2014.2337914. ISSN 1534-4320. PMID 25069118. 
  19. ↑ Wang, Zhelong; Qiu, Sen; Cao, Zhongkai; Jiang, Ming (2013). "Quantitative assessment of dual gait analysis based on inertial sensors with body sensor network". Sensor Review 33 (1): 48–56. doi:10.1108/02602281311294342. ISSN 0260-2288. 
  20. ↑ Kong, Weisheng; Wanning, Lauren; Sessa, Salvatore; Zecca, Massimiliano; Magistro, Daniele; Takeuchi, Hikaru; Kawashima, Ryuta; Takanishi, Atsuo (2017). "Step Sequence and Direction Detection of Four Square Step Test". IEEE Robotics and Automation Letters 2 (4): 2194–2200. doi:10.1109/LRA.2017.2723929. ISSN 2377-3766. http://irep.ntu.ac.uk/id/eprint/33502/1/10999_Magistro.pdf. 
  21. ↑ Rethinking JDL Data Fusion Levels
  22. ↑ Blasch, E., Plano, S. (2003) “Level 5: User Refinement to aid the Fusion Process”, Proceedings of the SPIE, Vol. 5099.
  23. ↑ J. Llinas; C. Bowman; G. Rogova; A. Steinberg; E. Waltz; F. White (2004). "Revisiting the JDL data fusion model II". International Conference on Information Fusion. 
  24. ↑ Blasch, E. (2006) "Sensor, user, mission (SUM) resource management and their interaction with level 2/3 fusion[yes|permanent dead link|dead link}}]" International Conference on Information Fusion.
  25. ↑ "Harnessing the full power of sensor fusion -". http://defensesystems.com/articles/2009/09/02/c4isr1-sensor-fusion.aspx. 
  26. ↑ Blasch, E., Steinberg, A., Das, S., Llinas, J., Chong, C.-Y., Kessler, O., Waltz, E., White, F. (2013) "Revisiting the JDL model for information Exploitation," International Conference on Information Fusion.
  27. ↑ 27.0 27.1 Gravina, Raffaele; Alinia, Parastoo; Ghasemzadeh, Hassan; Fortino, Giancarlo (2017). "Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges". Information Fusion 35: 68–80. doi:10.1016/j.inffus.2016.09.005. ISSN 1566-2535. 
  28. ↑ Gao, Teng; Song, Jin-Yan; Zou, Ji-Yan; Ding, Jin-Hua; Wang, De-Quan; Jin, Ren-Cheng (2015). "An overview of performance trade-off mechanisms in routing protocol for green wireless sensor networks". Wireless Networks 22 (1): 135–157. doi:10.1007/s11276-015-0960-x. ISSN 1022-0038. 
  29. ↑ 29.0 29.1 Chen, Chen; Jafari, Roozbeh; Kehtarnavaz, Nasser (2015). "A survey of depth and inertial sensor fusion for human action recognition". Multimedia Tools and Applications 76 (3): 4405–4425. doi:10.1007/s11042-015-3177-1. ISSN 1380-7501. 
  30. ↑ Banovic, Nikola; Buzali, Tofi; Chevalier, Fanny; Mankoff, Jennifer; Dey, Anind K. (2016). "Modeling and Understanding Human Routine Behavior". Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. pp. 248–260. doi:10.1145/2858036.2858557. ISBN 9781450333627. 
  31. ↑ Maria, Aileni Raluca; Sever, Pasca; Carlos, Valderrama (2015). "Biomedical sensors data fusion algorithm for enhancing the efficiency of fault-tolerant systems in case of wearable electronics device". 2015 Conference Grid, Cloud & High Performance Computing in Science (ROLCG). pp. 1–4. doi:10.1109/ROLCG.2015.7367228. ISBN 978-6-0673-7040-9. 
  32. ↑ Gross, Jason; Yu Gu; Matthew Rhudy; Srikanth Gururajan; Marcello Napolitano (July 2012). "Flight Test Evaluation of Sensor Fusion Algorithms for Attitude Estimation". IEEE Transactions on Aerospace and Electronic Systems 48 (3): 2128–2139. doi:10.1109/TAES.2012.6237583. Bibcode: 2012ITAES..48.2128G. 
  33. ↑ Joshi, V., Rajamani, N., Takayuki, K., Prathapaneni, N., Subramaniam, L. V. (2013). "Information Fusion Based Learning for Frugal Traffic State Sensing". Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence. 
  34. ↑ Mircea Paul, Muresan; Ion, Giosan; Sergiu, Nedevschi (2020-02-18). "Stabilization and Validation of 3D Object Position Using Multimodal Sensor Fusion and Semantic Segmentation". Sensors 20 (4): 1110. doi:10.3390/s20041110. PMID 32085608. Bibcode: 2020Senso..20.1110M. 
  35. ↑ Ran, Lingyan; Zhang, Yanning; Wei, Wei; Zhang, Qilin (2017-10-23). "A Hyperspectral Image Classification Framework with Spatial Pixel Pair Features". Sensors 17 (10): 2421. doi:10.3390/s17102421. PMID 29065535. Bibcode: 2017Senso..17.2421R. 

External links



  1. ↑ Haghighat, Mohammad; Abdel-Mottaleb, Mohamed; Alhalabi, Wadee (2016). "Discriminant Correlation Analysis: Real-Time Feature Level Fusion for Multimodal Biometric Recognition". IEEE Transactions on Information Forensics and Security 11 (9): 1984–1996. doi:10.1109/TIFS.2016.2569061. https://zenodo.org/record/889881. 
Retrieved from "https://handwiki.org/wiki/index.php?title=Sensor_fusion&oldid=3386304"