Chaos theory in Neural Networks

ABSTRACT

Neural networks based on chaos theory can simulate the functioning the brain in a better way as most of the natural processes are chaotic. These neural networks not only help in better understanding of the brain processes but they also have real world applications. Here, I present a review of the literature to put forth the importance of chaotic neural networks.


KEYWORDS
Chaotic networks, neural networks, learning, memory, brain processes.


1. INTRODUCTION.

Research results show that chaotic dynamics is present in the functioning of biological neural systems, ranging from chaotic firing of individual neurons to chaotic spatio-temporal patterns in the EEG. Chaotic behaviour of neural networks is found to have application in biological modeling. In this paper, first I describe the chaos theory and its main characteristics. Next, I show that neural networks are motivated from biological processes in the brain and at last, I present arguments for the use of chaos theory in neural networks.


2. MEANING OF CHAOS.

Chaos theory studies systems whose behavior lies between rigid regularity and randomness. The common meaning of the word “chaos” suggests complete disorder, but that is not how chaotic systems behave. Although, statistically chaotic activity is indistinguishable from randomness.

The typical features of chaos include:
a. Nonlinearity – All chaotic systems are non-linear.
b. Determinism – There are deterministic underlying rules to specify the future state of the system.
c. Sensitivity to initial conditions – Slight changes in initial conditions lead to radically different behavior in the final state of the systems.

It can be explained with a simple example of pseudo-number generator. The rule for pseudo-number generator is a deterministic formula e.g. Xn+1 = c Xn mod(m). The resulting solutions of the equation are very irregular, unpredictable, and sensitive to initial conditions.

In fact, most of the physical systems are chaotic. It seems nature uses chaotic systems more often than it uses linear systems.

3. NEURAL NETWORKS.

Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. In spite of the growing number of publications in the field, there is no consensus on the precise definition of ANNs. The reason for this is that there are too many types of them. Sometimes a more
general term called “connectionism" is used for ANNs. The term connectionism means a methodology of making a complex system by a combination of connected elements that are similar or identical.
The basic nonlinear elements of an ANN are called neurons.
ANN is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. ANNs, like people, learn by example. Learning in biological systems involves adjustments to the synaptic connections that exist between the neurons. This is true of ANNs as well.

Applications of neural networks:
In real life applications, neural networks perform particularly well on the following common tasks:
1. Function approximation.
2. Pattern recognition.
3. Time-Series Forecasting.

Functioning of ANNs is based on functioning of neurons in the human brain. Therefore, this is also a good method to study brain.

4. MOTIVATION OF USING CHAOS IN ANNS.

There is a wealth of experimental evidence indicating the significance of complex, non-periodic spatio-temporal oscillations at microscopic, mesoscopic, and macroscopic levels of neural organization using EEG, fMRI, and MEG techniques. Initially it was believed that chaotic behavior could be responsible for epilepsy, insomnia and other such neural disorders, but now the positive aspects of chaotic behavior are being considered. According to Freeman, chaotic behavior is what helps us to recognize a familiar face almost instantaneously. Further, he believes that chaotic activity in the brain is necessary for learning.
Now I will present some points which motivate the usage of chaos theory in ANNs:


1. Assuming that the brain processes exhibit chaotic behaviour. The models if ANNs which use chaos theory can help in better understanding of brain processes.

“One way to find answers to questions about neural chaos is to build models that produce similar emergent chaotic behavior in artificial neural networks” (Andras, 2004).

Another benefit of designing chaotic neural networks is that simple yet biologically plausible networks can be designed which exhibit complex activity of neurons. These networks being simple can help to understand the brain processes in a better way.

2. Chaos could explain the capacity of human memory.

(a) “A study of the various routes to chaos in dynamical systems reveals that significant computation occurs at the onset of chaos. At first blush this is not surprising since statistical mechanics views these as phase transitions with infinite temporal correlations. In computational terms, processes that are in a critical state, like those at the onset of chaos considered here, have an infinite memory” (Crutchfield, 1993).

(b) “Indeed, chaos offers many advantages over alternative memory storage mechanisms used in artificial neural networks. One is that chaotic dynamics are significantly easier to control than other linear or non-linear systems, requiring only small appropriately timed perturbations to constrain them within specific Unstable Periodic Orbits
(UPOs). Another is that chaotic attractors contain an infinite number of these UPOs. If individual UPOs can be made to represent specific internal memory states of a system, then in theory a chaotic attractor can provide an infinite memory store for the system” (Crook and Scheper, 2001).

3. Chaos in memory search

(a) “When a trajectory moves along a chaotic attractor, it moves sequentially from one part to another. If we associate various parts of the attractor with different patterns, then the trajectory will wander between them. In principle, this wandering can be used for the recognition or association purposes: if a trajectory spends most of its time near one of the patterns, then the latter can be considered as “recognized", and if in the sequence of visited patterns there are stable combinations, those patterns may be considered as “associated" with one another. Note that sequences of patterns can be stored into Hopfield-type networks. There is a possibility that chaos may help vary these combinations to learn new ones or to allow one pattern to participate in a number of associations simultaneously” (Potapov and Ali, 2001).

(b) “The network stems from the study of the neurophysiology of the olfactory system. It is shown that the network serves as an associative memory, which possesses chaotic dynamics. The problem addressed is machine recognition of industrial screws, bolts, etc. in simulated real time in accordance with tolerated deviations from manufacturing specifications… The existence of the chaotic dynamics provides the network with it’s capability to suppress noise and irrelevant information with respect to the recognition task”(Yao , Freeman, Burke And Yang, 1990).

4. Learning

(a) Freeman sees chaos as the building block that sets the stage for new learning. The brain is constantly being attracted towards a chaotic state in order not to bias new information in the environment by preexisting learned attractors.


(b) “If we consider a neural network as an element of a larger system interacting with the world, then dynamical chaos can emerge in rather simple models. A number of such models are known, for example, in artificial intelligence. Moreover, systems interacting with their surroundings need a source of `initiatives' to pursue exploration and learning from experience. Dynamical chaos can serve as a source of such initiatives” (Potapov and Ali , 2001).



Thus, chaotic neural networks could help in understanding the various mysteries of the brain. Also, such neural networks may have much more applications in real world.

5. CONCLUSION

I have presented arguments for the use of chaos theory in neural networks. It can be seen that such neural networks can help in understanding the functioning of memory, the quantity of memory, learning etc. In Munakata, Sinha and Ditto (2002), the logical operations like AND, OR, NOT, XOR, and NAND are realized using chaotic elements. They provide a theoretical foundation of computer architecture based on a totally new principle other than silicon chips. Chaos computing may also lead to dynamic architecture, where the hardware design itself evolves during the course of computation. Thus, apart from helping understanding the brain processes, chaos can also help in designing artificial brains.


6. REFERENCES
1. Crutchfield, James P. (1993), Critical Computation, Phase Transitions, and Hierarchical Learning(Santa Fe Institute Studies on the Sciences of Complexity), http://www.santafe.edu/ research / publications/ wpabstract/199310061.

2. Andras, P. (2004), ‘A Model for Emergent Chaotic Order
in Small Neural Networks’, Technical Report Series, University of Newcastle upon Tyne, CS-TR-860, pp. 1-10. Nonlinear Workbook: Chaos, Fractals, Cellular Automata, Neural Networks, Genetic Algorithms, Gene Expression

3. Crook, N. and Scheper T. (2001), ‘A Novel Chaotic Neural Network Architecture’ , European Symposium on Artificial Neural Networks ISBN 2-930307-01-3, pp. 295-300. Neural Nets and Chaotic Carriers (Wiley Interscience Series in Systems and Optimization)


4. Potapov, A.B. and Ali, M.K. (2001), ‘Nonlinear dynamics and chaos in information processing neural networks’, Differential Equations and Dynamical Systems, 9(3&4), pp. 259-319.

5. Albers, D.J. And Sprott, J.C. (1998), ‘Routes To Chaos In Neural Networks With Random Weights’, International Journal of Bifurcation and Chaos, 8(7), pp. 1463-1478.

6. Munakata T., Sinha S., and Ditto W.L. (2002), ‘Chaos Computing: Implementation of Fundamental Logical Gates by Chaotic Elements’, IEEE Transactions on Circuits And Systems, 49(11). CONTROL OF CHAOS IN NONLINEAR CIRCUITS AND SYSTEMS (World Scientific Series on Nonlinear Science, Series a)

7. Freeman, W.J., ‘Chaos In The CNS: Theory And Practice’, http://sulcus.berkeley.edu/FLM/MS/WJF_man2.html.