Are neural networks deterministic

Neural Networks

with an initial condition

. Reached the membrane potential at a time

the fire threshold,

, the neuron spikes and the membrane potential is set to zero:

. Refractory periods can be achieved by a time-dependent fire threshold,

, must be taken into account. For the currents that occur

one chooses the delta function according to the pulse character of the spikes, or one takes into account postsynaptic potentials in the form of a Alpha function:


for excitation and

for inhibition.

In addition to the ones described, there are other, especially more complex, neuron models that take into account the spatial structure of the neuron shown in Fig. 2. To simulate larger networks, however, simpler models are mostly used because of the lower computational effort.

3 The Hopfield model

John Hopfield proposed a neural network, the one, in 1982 Associative memory shows: Patterns can be impressed on the network by selecting the synapse strengths, and if a noisy or incomplete pattern is presented later, it is able to reproduce the original pattern. Hopfield networks are therefore used in image processing. The Hopfield model has also gained importance due to its exact solvability and its formal analogy to the Ising model in the physics of spin glasses.

The network consists of N McCulloch-Pitts neurons, where the output values ​​chosen are usually -1 (not active) and 1 (active) instead of 0 and 1. All neurons are connected to each other. Gets neuron i (

) the inputs


), this is how its output is currently calculated t +1 according to

is there

the Signum function (




), and

represents the strength of the synapse with the neuron j on neuron i projected. The dynamics of the network is asynchronous, i.e. a neuron is randomly selected in each time step i and calculating its activity


For networks with symmetrical couplings,

, there is an energy function

which is monotonically decreasing under the dynamics (2). The network thus runs into one of the local minima of the energy function for any initial conditions; such a state of affairs

is stable. If one chooses a graphical representation by color coding (activity 1 black, activity -1 white) and a rectangular arrangement of the neurons, each minimum corresponds to a stable pattern that is started in its catchment area when the initial conditions are selected (Fig. 7).

Through a suitable choice of the synapse strengths

it is possible to save patterns desired by the user in the network. Be that m-th pattern given by



); then exist for

minima of the energy function at the corresponding points.

You cannot save any number of patterns in a Hopfield network. Calculations show that, on average,

a phase transition takes place beyond which stability of the pattern can no longer be expected. The Hopfield network has the best storage properties for mutually independent random patterns, which is generally not the case with real images.

4 The back-propagation algorithm

While the synapse strengths are specified in Hopfield networks, the backpropagation algorithm sets it supervised learning process dar: be the network M. Presented input patterns, each to be associated with an output pattern. The mistake that the network makes is minimized by changing the synapse strengths according to a gradient method. Back-propagation networks have found numerous uses. The best known is probably the one developed by Terrence J. Sejnowski and Charles R. Rosenberg NETtalkwhose input consists of English text. The network converts the text into chains of phonemes which can be played back by a speech synthesizer.

Back-propagation networks (also Multilayer perceptrons called) consist of layers of analog neurons, whereby each layer with randomly initialized coupling strengths is completely coupled to the next, but there are no connections to previous layers or between the neurons of a layer (Fig. 8). These are so-called. forward-looking networks, in contrast to feedback networks such as the Hopfield networks. To the Input layer will be one after the other M. Input pattern in the form of N-dimensional real vectors

created. The input neurons have no dynamics, but pass their inputs on without change. The activity is carried out as described in Section 3 via the hidden layers Up to the K Neurons of the Output layer touted that the spending

produce. The desired output patterns are


). After presenting all the patterns, the error of the network is expressed in the form of the sum of all the square deviations between



is a function of all synapse strengths in the network, symbolized by the index w.

The learning rule, i.e. the change in synapse strengths after the presentation of the M. Pattern, has the shape of a gradient descent. With


in which

as Learning rate is referred to and

the strength of any synapse between neurons j and neuron i represents in the network.

In the following, let the output function be the Fermi function (1) with

, for which the functional equation

applies. Then the following arithmetic rule can be given for the right-hand side of (3): For each created pattern

and every expenditure euro k (

) is used for each connection path

from the output neuron back to the corresponding synapse a product

calculated by factors and it is

is the output of the neuron i when presenting the sample m. The products

are composed as follows: i) For each neuron n along the way there is a factor

, in which

the output of this neuron is. ii) For each synapse with the strength

along the way there is a factor


Take the network in Fig. 8 as an example. Are the coupling strengths from the hidden layer to the output layer with

and those from the input layer to the hidden layer

then applies to


There are numerous extensions to the backpropagation algorithm described, which aim in particular to improve the convergence properties of the gradient descent and the applicability to recursive networks.

5 Kohonen networks

In 1982 Teuvo Kohonen introduced a neural network whose synapse strengths are organized without an error function having to be evaluated; the Kohonen network uses one unsupervised learning process. The inputs to the network come from a metric space, for example a square. The network also has a spatial structure: in many cases the neurons form a linear chain or a two-dimensional grid. During the learning process, the synapse strengths adapt in such a way that the neurons represent the input space while preserving the neighborhood. One application of Kohonen networks is the modeling of biological self-organizing feature maps, which are found in the human cerebral cortex, for example: there are areas with a topographical representation of the visual environment or the surface of the skin.

The input signals to the network are vectors

one k-dimensional space

. The network

have one n-dimensional topology, and each neuron is through a place

marked. The synapse strengths between the input signal and the neuron

be with

designated. For the neural dynamics, the coupling between the neurons can be neglected. Thus, in the case of analog neurons, their output is calculated as

in which f represents the Fermi function (1). At the beginning of the learning process, the

randomly initialized. After each input

that is drawn from an arbitrary distribution becomes that neuron

whose activity is at its maximum. Instead of calculating the

according to (4) one uses the simple condition for this

The learning rule is that for each neuron

the weight vector according to

is changed.

is the learning rate, and

is a function based on the spacing of neurons



determines the strength of the learning step. Generally one chooses for

a Gaussian curve, the width of which decreases in the course of the learning process, in order to finally 'freeze' the network. (6) clearly states that all weight vectors


in the direction of the input signal

can be drawn, the respective distance being derived from the metric of the space

of neurons determined.

Equation (5) defines a mapping that is dependent on the synapse strengths


in the crowd

of the neurons (see Fig. 9). The pixel

is precisely the neuron with the strongest response to the stimulus v.

is called neural map designated. This neural map is changed during the learning process.

For the development of the synapse strengths and the resulting representation of the entrance area

there is a clear illustration. The

in the room

and the connection between the neurons in space

is indicated by connecting lines between the

symbolizes. Fig. 10 shows a schematic example of a square

and a chain

of neurons. The synapse strengths are randomly initialized (a). The input signals are evenly distributed in

, and under the learning dynamics they order and unfold (b-d), resulting in a neighborhood-preserving neural map

leads. Due to the best possible representation of the square by the neuron chain, a curve is formed which corresponds to an iterate of a Peano curve.

Kohonen cards have other interesting properties. For example, if the input signals are not evenly distributed in

, so is the density of synapse strengths in areas with higher signal density

larger, so that the area is represented by an above-average number of neurons.

6 Synchronization in networks of spiking neurons

Networks with continuous-time dynamics are often used to model neurobiological systems. The choice of a simple neuron model such as the Integrate-and-Fire-Neuron partially allows an analytical approach. A current area of ​​research is the study of the synchronization of the fire activities of populations of coupled neurons. The reason for the interest in this area lies in the assumption that synchronization occurs in the brain in connection with gestalt perception and behaviorally relevant object recognition.

In 1991, Renato E. Mirollo and Steven H. Strogatz studied the dynamics of networks of spiked model neurons that are coupled to one another in a completely excitatory manner and that receive a constant input current. They showed that such networks evolve under the neural dynamics for almost all initial conditions into a state in which the neurons fire synchronously with the phase position zero. Extensions of this investigation concern the consideration of signal propagation times between the neurons, which always occur in real systems, and the effects of inhibition and excitation. The main result is that when time delays are present, inhibitory couplings lead to the formation of stable clusters, the neurons of which fire synchronously. In the case of excitatory coupling, on the other hand, the clusters that form are not stable, but instead disintegrate again after a few periods of fire and spontaneously re-form with a changing neural composition.

7 Outlook

The field of neural networks, with a history spanning just over 50 years, is still incomplete. This applies both to the mathematical properties of even simple networks and to applications in science, technology and economics. The number of commercial applications and examples in which neural networks are superior to conventional methods is still quite small at the moment. One reason for the caution exercised, for example, when using neural networks for control tasks, is that the network dynamics are in many cases complex and therefore not transparent. The great advantage of neural networks, however, lies in their flexibility and their ability to adapt to the respective tasks during use without the need for technical skills on the part of the user.


M. A. Arbib (Ed.): The Handbook of Brain Theory and Neural Networks, Cambridge, MA: MIT Press, 1995;

J. Hertz, A. Krogh, R.G. Palmer: Introduction to the Theory of Neural Computation, Redwood City, CA: Addison-Wesley, 1991;

W. Maass, C.M. Bishop (Ed.): Pulsed Neural Networks, Cambridge, MA: MIT Press, 1999;

H. Ritter, T. Martinetz, K. Schulten: Neural Networks, Bonn: Addison-Wesley, 1991;

R. Rojas: Neural Network Theory, Berlin: Springer-Verlag, 1993;

A. Zell: Simulation of neural networks, Munich: Oldenbourg, 1994.

neural networks 1: Sub-disciplines.

neural networks 2: Schematic representation of a biological neuron.

neural networks 3: Spiketrain of a midbrain neuron of a salamander in response to a black moving rectangle presented to the animal. The action potentials are shown as lines. Scale: 500 ms. (Data courtesy of B. Lienstädt.)

neural networks 4: Model neuron for networks with discrete-time dynamics. f is the symbol for the output function.

neural networks 5: Output functions for model neurons. a) To the threshold

shifted Heaviside function for McCulloch-Pitts neurons. b) Fermi function for analog neurons with maximum rate of fire

; For

fires the neuron at the rate


neural networks 6: Equivalent circuit diagram for the Integrate-and-Fire-Neuron below the fire threshold.

neural networks 7: The letter ›T‹ as the activity pattern of a Hopfield network. If the intact letter (above) corresponds to a minimum of the energy function, it is reconstructed if the network is started with a noisy (middle) or incomplete (below) pattern.

neural networks 8: A three-tier network with

Input neurons,

hidden neurons and

Output neurons. Black dots symbolize synapses. The arrow indicates the direction of the signal flow.

neural networks 9: A neural map

that maps a two-dimensional input space onto a one-dimensional chain of neurons.

neural networks 10: Figure between a chain and a square. a) randomly chosen initial assignment; b) Rough allocation after 200 Markov steps, c) after 50,000 Markov steps. d) After 100,000 Markov steps, the assignment obtained is similar to a Peano curve.