AVT Statistical filtering algorithm

From HandWiki

AVT Statistical filtering algorithm is an approach to improving quality of raw data collected from various sources. It is most effective in cases when there is inband noise present. In those cases AVT is better at filtering data then, band-pass filter or any digital filtering based on variation of. Conventional filtering is useful when signal/data has different frequency than noise and signal/data is separated/filtered by frequency discrimination of noise. Frequency discrimination filtering is done using Low Pass, High Pass and Band Pass filtering which refers to relative frequency filtering criteria target for such configuration. Those filters are created using passive and active components and sometimes are implemented using software algorithms based on Fast Fourier transform (FFT).

AVT filtering is implemented in software and its inner working is based on statistical analysis of raw data.

When signal frequency/(useful data distribution frequency) coincides with noise frequency/(noisy data distribution frequency) we have inband noise. In this situations frequency discrimination filtering does not work since the noise and useful signal are indistinguishable and where AVT excels. To achieve filtering in such conditions there are several methods/algorithms available which are briefly described below.

Averaging algorithm

  1. Collect n samples of data
  2. Calculate average value of collected data
  3. Present/record result as actual data

Median algorithm

  1. Collect n samples of data
  2. Sort the data in ascending or descending order. Note that order does not matter
  3. Select the data that happen to be in n/2 position and present/record it as final result representing data sample

AVT algorithm

AVT inner working

AVT algorithm stands for Antonyan Vardan Transform and its implementation explained below.

  1. Collect n samples of data
  2. Calculate the standard deviation and average value
  3. Drop any data that is greater or less than average ± one standard deviation
  4. Calculate average value of remaining data
  5. Present/record result as actual value representing data sample

This algorithm is based on amplitude discrimination and can easily reject any noise that is not like actual signal, otherwise statistically different then 1 standard deviation of the signal. Note that this type of filtering can be used in situations where the actual environmental noise is not known in advance. Notice that it is preferable to use the median in above steps than average. Originally the AVT algorithm used average value to compare it with results of median on the data window.

Filtering algorithms comparison

Using a system that has signal value of 1 and has noise added at 0.1% and 1% levels will simplify quantification of algorithm performance. The R[1] script is used to create pseudo random noise added to signal and analyze the results of filtering using several algorithms. Please refer to "Reduce Inband Noise with the AVT Algorithm" [2] article for details. This graphs show that AVT algorithm provides best results compared with Median and Averaging algorithms while using data sample size of 32, 64 and 128 values. Note that this graph was created by analyzing random data array of 10000 values. Sample of this data is graphically represented below.
From this graph it is apparent that AVT outperforms other filtering algorithms by providing 5% to 10% more accurate data when analyzing same datasets. Considering random nature of noise used in this numerical experiment that borderlines worst case situation where actual signal level is below ambient noise the precision improvements of processing data with AVT algorithm are significant.
AVT algorithm comparison AVT test data sample

AVT algorithm variations

Cascaded AVT

In some situations better results can be obtained by cascading several stages of AVT filtering. This will produce singular constant value which can be used for equipment that has known stable characteristics like thermometers, thermistors and other slow acting sensors.

Reverse AVT

  1. Collect n samples of data
  2. Calculate the standard deviation and average value
  3. Drop any data that is within one standard deviation ± average band
  4. Calculate average value of remaining data
  5. Present/record result as actual data

This is useful for detecting minute signals that are close to background noise level.

Detecting Altered/modified Jupiter Picture with AVT image filter
Detecting Altered/modified Jupiter Picture with AVT image filter

Possible applications and uses

Another version of this comparison is the Reverse AVT filter applied to the same original Jupiter Image, where we only see that altered portion as Noise that was eliminated by AVT algorithm.

Jupiter Reverse AVT comparison.png
  • Use AVT as image filtering algorithm to estimate data density from images, please see Python program program. Picture of Pillars of Creation Nebula shows data density in filtered images from Hubble and Webb. Note that image on the left has big patches of missing data marked with simpler color patterns.
Pillars of creation picture data density difference detected with AVT comparing Webb to Hubble
Pillars of creation picture data density difference detected with AVT comparing Webb to Hubble

References

  1. ↑ "The R Project for Statistical Computing". r-project.org. http://www.r-project.org/. 
  2. ↑ "Reduce Inband Noise with the AVT Algorithm | Embedded content from Electronic Design". electronicdesign.com. http://electronicdesign.com/embedded/reduce-inband-noise-avt-algorithm. 
  1. Joseph, Favis; Balinadoa, C.; Paolo Dar Santos, Gerald; Escanilla, Rio; Darell C. Aguda, John; Ramona A. Alcantara, Ma.; Belen M. Roble, Mariela; F. Bueser, Jomalyn (May 5, 2020). "13Th International Engineering Research Conference (13Th Eureca 2019)". 2233. p. 050003. doi:10.1063/5.0002323. 
  2. Vinicius, Cene; Mauricio, Tosin; J., Machado; A., Balbinot (April 2019). "Open Database for Accurate Upper-Limb Intent Detection Using Electromyography and Reliable Extreme Learning Machines". Sensors 19 (8): 1864. doi:10.3390/s19081864. PMID 31003524. PMC 6515272. Bibcode: 2019Senso..19.1864C. https://www.researchgate.net/publication/332514376. 
  3. HornCene, Vinicius; Balbinot, Alexandr (August 10, 2018), "Using the sEMG signal representativity improvement towards upper-limb movement classification reliability", Biomedical Signal Processing and Control 46: 182–191, doi:10.1016/j.bspc.2018.07.014, ISSN 1746-8094, http://www.sciencedirect.com/science/article/pii/S1746809418301848 
  4. Horn Cene, Vinicius; Ruschel dos Santos, Raphael; Balbinot, Alexandre (July 18, 2018). Honolulu, HI, USA: IEEE. pp. 5224–5227. doi:10.1109/EMBC.2018.8513468. ISBN 978-1-5386-3646-6. 
  5. AVT image Filtering algorithm in python[1]



  1. ↑ Antonyan, Vardan. "AVT Image Filter". Github. https://github.com/vantonyan/AVT_Image_Filtering. 
Retrieved from "https://handwiki.org/wiki/index.php?title=AVT_Statistical_filtering_algorithm&oldid=3009022"