Extension neural network

Extension neural network is a pattern recognition method found by M. H. Wang and C. P. Hung in 2003 to classify instances of data sets. Extension neural network is composed of artificial neural network and extension theory concepts. It uses the fast and adaptive learning capability of neural network and correlation estimation property of extension theory by calculating extension distance.
ENN was used in:

• Failure detection in machinery.
• Tissue classification through MRI.
• Fault recognition in automotive engine.
• State of charge estimation in lead-acid battery.
• Classification with incomplete survey data.

Extension Theory

Extension theory was first proposed by Cai in 1983 to solve contradictory problems. While classical mathematics is familiar with quantity and forms of objects, extension theory transforms these objects to matter-element models.

[math]\displaystyle{ R=(N,C,V) }[/math]

(1)

where in matter [math]\displaystyle{ R }[/math], [math]\displaystyle{ N }[/math] is the name or type, [math]\displaystyle{ C }[/math] is its characteristics and [math]\displaystyle{ V }[/math] is the corresponding value for the characteristic. There is a corresponding example in equation 2.

[math]\displaystyle{ R=\begin{bmatrix} Yusuf & Height, & 178cm \\ & Weight & 98kg \end{bmatrix} }[/math]

(2)

where [math]\displaystyle{ Height }[/math] and [math]\displaystyle{ Weight }[/math] characteristics form extension sets. These extension sets are defined by the [math]\displaystyle{ V }[/math] values which are range values for corresponding characteristics. Extension theory concerns the extension correlation function between matter-element models like shown in equation 2, and extension sets. Extension correlation function is used to define extension space which is composed of pairs of elements and their extension correlation functions. The extension space formula is shown in equation 3.

[math]\displaystyle{ A=\left \{ (x,y)| x\in U, y=K(x) \right \} }[/math]

(3)

where, [math]\displaystyle{ A }[/math] is the extension space, [math]\displaystyle{ U }[/math] is the object space, [math]\displaystyle{ K }[/math] is the extension correlation function, [math]\displaystyle{ x }[/math] is an element from the object space and [math]\displaystyle{ y }[/math] is the corresponding extension correlation function output of element [math]\displaystyle{ x }[/math]. [math]\displaystyle{ K(x) }[/math] maps [math]\displaystyle{ x }[/math] to a membership interval [math]\displaystyle{ \left [ -\infin,\infin \right ] }[/math]. Negative region represents an element not belonging membership degree to a class and positive region vice versa. If [math]\displaystyle{ x }[/math] is mapped to [math]\displaystyle{ \left [ 0,1 \right ] }[/math], extension theory acts like fuzzy set theory. The correlation function can be shown with the equation 4.

[math]\displaystyle{ \rho(x,X_{in})=\left|x-\frac{a+b}{2}\right|-\frac{b-a}{2} }[/math] [math]\displaystyle{ \rho(x,X_{out})=\left|x-\frac{c+d}{2}\right|-\frac{d-c}{2} }[/math]

(4)

where, [math]\displaystyle{ X_{in} }[/math] and [math]\displaystyle{ X_{out} }[/math] are called concerned and neighborhood domain and their intervals are (a,b) and (c,d) respectively. The extended correlation function used for estimation of membership degree between [math]\displaystyle{ x }[/math] and [math]\displaystyle{ X_{in} }[/math], [math]\displaystyle{ X_{out} }[/math] is shown in equation 5.

[math]\displaystyle{ K(x)= \begin{cases} -\rho(x,X_{in}) &x\in{X_{in}} \\ \frac{\rho(x,X_{in})}{\rho(x,X_{out})-\rho(x,X_{in})} &x\not \in{X_{in}} \end{cases} }[/math]

(5)

Extension Neural Network

Extension neural network has a neural network like appearance. Weight vector resides between the input nodes and output nodes. Output nodes are the representation of input nodes by passing them through the weight vector.

There are total number of input and output nodes are represented by [math]\displaystyle{ n }[/math] and [math]\displaystyle{ n_c }[/math], respectively. These numbers depend on the number of characteristics and classes. Rather than using one weight value between two layer nodes as in neural network, extension neural network architecture has two weight values. In extension neural network architecture, for instance [math]\displaystyle{ i }[/math], [math]\displaystyle{ x^p_{ij} }[/math] is the input which belongs to class [math]\displaystyle{ p }[/math] and [math]\displaystyle{ o_{ik} }[/math] is the corresponding output for class [math]\displaystyle{ k }[/math]. The output [math]\displaystyle{ o_{ik} }[/math] is calculated by using extension distance as shown in equation 6.

[math]\displaystyle{ ED_{ik}=\sum\limits_{j=0}^n \left( \frac{|x_{ij}^p-z_{kj}|-\frac{w_{kj}^U-w_{kj}^L}{2}}{|\frac{w_{kj}^U-w_{kj}^L}{2}|}+1 \right) }[/math] [math]\displaystyle{ k=1,2,....,n_c }[/math]

(6)

Estimated class is found through searching for the minimum extension distance among the calculated extension distance for all classes as summarized in equation 7, where [math]\displaystyle{ k^* }[/math] is the estimated class.

[math]\displaystyle{ k^*= arg\min_k(o_{ik}) }[/math]

(7)

Learning Algorithm

Each class is composed of ranges of characteristics. These characteristics are the input types or names which come from matter-element model. Weight values in extension neural network represent these ranges. In the learning algorithm, first weights are initialized by searching for the maximum and minimum values of inputs for each class as shown in equation 8

[math]\displaystyle{ w_{kj}^U = \max_i\{x_{ij}^k\} }[/math] [math]\displaystyle{ w_{kj}^L = \min_i\{x_{ij}^k\} }[/math] [math]\displaystyle{ i=1,...,N_p }[/math] [math]\displaystyle{ k=1,...,n_c }[/math] [math]\displaystyle{ j=1,2,....,n }[/math]

(8)

where, [math]\displaystyle{ i }[/math] is the instance number and [math]\displaystyle{ j }[/math] is represents number of input. This initialization provides classes' ranges according to given training data.

After maintaining weights, center of clusters are found through the equation 9.

[math]\displaystyle{ Z_k = \{z_{k1},z_{k2},...,z_{kn}\} }[/math] [math]\displaystyle{ z_{kj} = \frac{w_{kj}^U+w_{kj}^L}{2} }[/math] [math]\displaystyle{ k=1,2,....,n_c }[/math] [math]\displaystyle{ j=1,2,....,n }[/math]

(9)

Before learning process begins, predefined learning performance rate is given as shown in equation 10

[math]\displaystyle{ E_\tau=\frac{N_m}{N_p} }[/math]

(10)

where, [math]\displaystyle{ N_m }[/math] is the misclassified instances and [math]\displaystyle{ N_p }[/math] is the total number of instances. Initialized parameters are used to classify instances with using equation 6. If the initialization is not sufficient due to the learning performance rate, training is required. In the training step weights are adjusted to classify training data more accurately, therefore reducing learning performance rate is aimed. In each iteration, [math]\displaystyle{ E_\tau }[/math] is checked to control if required learning performance is reached. In each iteration every training instance is used for training.
Instance [math]\displaystyle{ i }[/math], belongs to class [math]\displaystyle{ p }[/math] is shown by:

[math]\displaystyle{ X_{i}^p=\{x_{i1}^p,x_{i2}^p,...,x_{in}^p\} }[/math]

[math]\displaystyle{ 1\leq p\leq n_c }[/math]

Every input data point of [math]\displaystyle{ X_i^p }[/math] is used in extension distance calculation to estimate the class of [math]\displaystyle{ X_i^p }[/math]. If the estimated class [math]\displaystyle{ k^*=p }[/math] then update is not needed. Whereas, if [math]\displaystyle{ k^* \neq p }[/math] then update is done. In update case, separators which show the relationship between inputs and classes, are shifted proportional to the distance between the center of clusters and the data points.
The update formula:

[math]\displaystyle{ z_{pj}^{new} = z_{pj}^{old} + \eta (x_{ij}^p-z_{pj}^{old}) }[/math]
[math]\displaystyle{ z_{k^*j}^{new} = z_{k^*j}^{old} - \eta (x_{ij}^p-z_{k^*j}^{old}) }[/math]
[math]\displaystyle{ w_{pj}^{L(new)} = w_{pj}^{L(old)} + \eta (x_{ij}^p-z_{pj}^{old}) }[/math]
[math]\displaystyle{ w_{pj}^{U(new)} = w_{pj}^{U(old)} + \eta (x_{ij}^p-z_{pj}^{old}) }[/math]
[math]\displaystyle{ w_{k^*j}^{L(new)} = w_{k^*j}^{L(old)} - \eta (x_{ij}^p-z_{k^*j}^{old}) }[/math]
[math]\displaystyle{ w_{k^*j}^{U(new)} = w_{k^*j}^{U(old)} - \eta (x_{ij}^p-z_{k^*j}^{old}) }[/math]

To classify the instance [math]\displaystyle{ i }[/math] accurately, separator of class [math]\displaystyle{ p }[/math] for input [math]\displaystyle{ j }[/math] moves close to data-point of instance [math]\displaystyle{ i }[/math], whereas separator of class [math]\displaystyle{ k^* }[/math] for input [math]\displaystyle{ j }[/math] moves far away. In the above image, an update example is given. Assume that instance [math]\displaystyle{ i }[/math] belongs to class A, whereas it is classified to class B because extension distance calculation gives out [math]\displaystyle{ ED_A\gt ED_B }[/math]. After the update, separator of class A moves close to the data-point of instance [math]\displaystyle{ i }[/math] whereas separator of class B moves far away. Consequently, extension distance gives out [math]\displaystyle{ ED_B\gt ED_A }[/math], therefore after update instance [math]\displaystyle{ i }[/math] is classified to class A.

References

1. Wang, M. H.; Tseng, Y. F.; Chen, H. C.; Chao, K. H. (2009). "A novel clustering algorithm based on the extension theory and genetic algorithm". Expert Systems with Applications 36 (4): 8269–8276. doi:10.1016/j.eswa.2008.10.010. 
2. Kuei-Hsiang Chao, Meng-Hui Wang, and Chia-Chang Hsu. A novel residual capacity estimation method based on extension neural network for lead-acid batteries. International Symposium on Neural Networks, pages 1145–1154, 2007
3. Kuei-Hsiang Chao, Meng-Hui Wang, Wen-Tsai Sung, and Guan-Jie Huang. Using enn-1 for fault recognition of automotive engine. Expert Systems with Applications, 37(4):29432947, 2010
4. Wang, C. M.; Wu, M. J.; Chen, J. H.; Yu, C. Y. (2009). "Extension Neural Network Approach to Classification of Brain MRI". 2009 Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing. pp. 515. doi:10.1109/IIH-MSP.2009.141. ISBN 978-1-4244-4717-6. 
5. Ye, J. (2009). "Application of extension theory in misfire fault diagnosis of gasoline engines". Expert Systems with Applications 36 (2): 1217–1221. doi:10.1016/j.eswa.2007.11.012. 
6. Juncai Zhang, Xu Qian, Yu Zhou, and Ai Deng. Condition monitoring method of the equipment based on extension neural network. Chinese Control and Decision Conference, pages 1735–1740, 2010
7. Wang, M.; Hung, C. P. (2003). "Extension neural network and its applications". Neural Networks 16 (5–6): 779–784. doi:10.1016/S0893-6080(03)00104-7. PMID 12850034.