Are neural networks deterministic
with an initial condition
. Reached the membrane potential at a time
the fire threshold,
, the neuron spikes and the membrane potential is set to zero:
. Refractory periods can be achieved by a time-dependent fire threshold,
, must be taken into account. For the currents that occur
one chooses the delta function according to the pulse character of the spikes, or one takes into account postsynaptic potentials in the form of a Alpha function:
for excitation and
In addition to the ones described, there are other, especially more complex, neuron models that take into account the spatial structure of the neuron shown in Fig. 2. To simulate larger networks, however, simpler models are mostly used because of the lower computational effort.
3 The Hopfield model
John Hopfield proposed a neural network, the one, in 1982 Associative memory shows: Patterns can be impressed on the network by selecting the synapse strengths, and if a noisy or incomplete pattern is presented later, it is able to reproduce the original pattern. Hopfield networks are therefore used in image processing. The Hopfield model has also gained importance due to its exact solvability and its formal analogy to the Ising model in the physics of spin glasses.
The network consists of N McCulloch-Pitts neurons, where the output values chosen are usually -1 (not active) and 1 (active) instead of 0 and 1. All neurons are connected to each other. Gets neuron i (
) the inputs
), this is how its output is currently calculated t +1 according to
the Signum function (
represents the strength of the synapse with the neuron j on neuron i projected. The dynamics of the network is asynchronous, i.e. a neuron is randomly selected in each time step i and calculating its activity
For networks with symmetrical couplings,
, there is an energy function
which is monotonically decreasing under the dynamics (2). The network thus runs into one of the local minima of the energy function for any initial conditions; such a state of affairs
is stable. If one chooses a graphical representation by color coding (activity 1 black, activity -1 white) and a rectangular arrangement of the neurons, each minimum corresponds to a stable pattern that is started in its catchment area when the initial conditions are selected (Fig. 7).
Through a suitable choice of the synapse strengths
it is possible to save patterns desired by the user in the network. Be that m-th pattern given by
); then exist for
minima of the energy function at the corresponding points.
You cannot save any number of patterns in a Hopfield network. Calculations show that, on average,
a phase transition takes place beyond which stability of the pattern can no longer be expected. The Hopfield network has the best storage properties for mutually independent random patterns, which is generally not the case with real images.
4 The back-propagation algorithm
While the synapse strengths are specified in Hopfield networks, the backpropagation algorithm sets it supervised learning process dar: be the network M. Presented input patterns, each to be associated with an output pattern. The mistake that the network makes is minimized by changing the synapse strengths according to a gradient method. Back-propagation networks have found numerous uses. The best known is probably the one developed by Terrence J. Sejnowski and Charles R. Rosenberg NETtalkwhose input consists of English text. The network converts the text into chains of phonemes which can be played back by a speech synthesizer.
Back-propagation networks (also Multilayer perceptrons called) consist of layers of analog neurons, whereby each layer with randomly initialized coupling strengths is completely coupled to the next, but there are no connections to previous layers or between the neurons of a layer (Fig. 8). These are so-called. forward-looking networks, in contrast to feedback networks such as the Hopfield networks. To the Input layer will be one after the other M. Input pattern in the form of N-dimensional real vectors
created. The input neurons have no dynamics, but pass their inputs on without change. The activity is carried out as described in Section 3 via the hidden layers Up to the K Neurons of the Output layer touted that the spending
produce. The desired output patterns are
). After presenting all the patterns, the error of the network is expressed in the form of the sum of all the square deviations between
is a function of all synapse strengths in the network, symbolized by the index w.
The learning rule, i.e. the change in synapse strengths after the presentation of the M. Pattern, has the shape of a gradient descent. With
as Learning rate is referred to and
the strength of any synapse between neurons j and neuron i represents in the network.
In the following, let the output function be the Fermi function (1) with
, for which the functional equation
applies. Then the following arithmetic rule can be given for the right-hand side of (3): For each created pattern
and every expenditure euro k (
) is used for each connection path
from the output neuron back to the corresponding synapse a product
calculated by factors and it is
is the output of the neuron i when presenting the sample m. The products
are composed as follows: i) For each neuron n along the way there is a factor
, in which
the output of this neuron is. ii) For each synapse with the strength
along the way there is a factor
Take the network in Fig. 8 as an example. Are the coupling strengths from the hidden layer to the output layer with
and those from the input layer to the hidden layer
then applies to
There are numerous extensions to the backpropagation algorithm described, which aim in particular to improve the convergence properties of the gradient descent and the applicability to recursive networks.
5 Kohonen networks
In 1982 Teuvo Kohonen introduced a neural network whose synapse strengths are organized without an error function having to be evaluated; the Kohonen network uses one unsupervised learning process. The inputs to the network come from a metric space, for example a square. The network also has a spatial structure: in many cases the neurons form a linear chain or a two-dimensional grid. During the learning process, the synapse strengths adapt in such a way that the neurons represent the input space while preserving the neighborhood. One application of Kohonen networks is the modeling of biological self-organizing feature maps, which are found in the human cerebral cortex, for example: there are areas with a topographical representation of the visual environment or the surface of the skin.
The input signals to the network are vectors
one k-dimensional space
. The network
have one n-dimensional topology, and each neuron is through a place
marked. The synapse strengths between the input signal and the neuron
designated. For the neural dynamics, the coupling between the neurons can be neglected. Thus, in the case of analog neurons, their output is calculated as
in which f represents the Fermi function (1). At the beginning of the learning process, the
randomly initialized. After each input
that is drawn from an arbitrary distribution becomes that neuron
whose activity is at its maximum. Instead of calculating the
according to (4) one uses the simple condition for this
The learning rule is that for each neuron
the weight vector according to
is the learning rate, and
is a function based on the spacing of neurons
determines the strength of the learning step. Generally one chooses for
a Gaussian curve, the width of which decreases in the course of the learning process, in order to finally 'freeze' the network. (6) clearly states that all weight vectors
in the direction of the input signal
can be drawn, the respective distance being derived from the metric of the space
of neurons determined.
Equation (5) defines a mapping that is dependent on the synapse strengths
in the crowd
of the neurons (see Fig. 9). The pixel
is precisely the neuron with the strongest response to the stimulus v.
is called neural map designated. This neural map is changed during the learning process.
For the development of the synapse strengths and the resulting representation of the entrance area
there is a clear illustration. The
in the room
and the connection between the neurons in space
is indicated by connecting lines between the
symbolizes. Fig. 10 shows a schematic example of a square
and a chain
of neurons. The synapse strengths are randomly initialized (a). The input signals are evenly distributed in
, and under the learning dynamics they order and unfold (b-d), resulting in a neighborhood-preserving neural map
leads. Due to the best possible representation of the square by the neuron chain, a curve is formed which corresponds to an iterate of a Peano curve.
Kohonen cards have other interesting properties. For example, if the input signals are not evenly distributed in
, so is the density of synapse strengths in areas with higher signal density
larger, so that the area is represented by an above-average number of neurons.
6 Synchronization in networks of spiking neurons
Networks with continuous-time dynamics are often used to model neurobiological systems. The choice of a simple neuron model such as the Integrate-and-Fire-Neuron partially allows an analytical approach. A current area of research is the study of the synchronization of the fire activities of populations of coupled neurons. The reason for the interest in this area lies in the assumption that synchronization occurs in the brain in connection with gestalt perception and behaviorally relevant object recognition.
In 1991, Renato E. Mirollo and Steven H. Strogatz studied the dynamics of networks of spiked model neurons that are coupled to one another in a completely excitatory manner and that receive a constant input current. They showed that such networks evolve under the neural dynamics for almost all initial conditions into a state in which the neurons fire synchronously with the phase position zero. Extensions of this investigation concern the consideration of signal propagation times between the neurons, which always occur in real systems, and the effects of inhibition and excitation. The main result is that when time delays are present, inhibitory couplings lead to the formation of stable clusters, the neurons of which fire synchronously. In the case of excitatory coupling, on the other hand, the clusters that form are not stable, but instead disintegrate again after a few periods of fire and spontaneously re-form with a changing neural composition.
The field of neural networks, with a history spanning just over 50 years, is still incomplete. This applies both to the mathematical properties of even simple networks and to applications in science, technology and economics. The number of commercial applications and examples in which neural networks are superior to conventional methods is still quite small at the moment. One reason for the caution exercised, for example, when using neural networks for control tasks, is that the network dynamics are in many cases complex and therefore not transparent. The great advantage of neural networks, however, lies in their flexibility and their ability to adapt to the respective tasks during use without the need for technical skills on the part of the user.
M. A. Arbib (Ed.): The Handbook of Brain Theory and Neural Networks, Cambridge, MA: MIT Press, 1995;
J. Hertz, A. Krogh, R.G. Palmer: Introduction to the Theory of Neural Computation, Redwood City, CA: Addison-Wesley, 1991;
W. Maass, C.M. Bishop (Ed.): Pulsed Neural Networks, Cambridge, MA: MIT Press, 1999;
H. Ritter, T. Martinetz, K. Schulten: Neural Networks, Bonn: Addison-Wesley, 1991;
R. Rojas: Neural Network Theory, Berlin: Springer-Verlag, 1993;
A. Zell: Simulation of neural networks, Munich: Oldenbourg, 1994.
neural networks 1: Sub-disciplines.
neural networks 2: Schematic representation of a biological neuron.
neural networks 3: Spiketrain of a midbrain neuron of a salamander in response to a black moving rectangle presented to the animal. The action potentials are shown as lines. Scale: 500 ms. (Data courtesy of B. Lienstädt.)
neural networks 4: Model neuron for networks with discrete-time dynamics. f is the symbol for the output function.
neural networks 5: Output functions for model neurons. a) To the threshold
shifted Heaviside function for McCulloch-Pitts neurons. b) Fermi function for analog neurons with maximum rate of fire
fires the neuron at the rate
neural networks 6: Equivalent circuit diagram for the Integrate-and-Fire-Neuron below the fire threshold.
neural networks 7: The letter ›T‹ as the activity pattern of a Hopfield network. If the intact letter (above) corresponds to a minimum of the energy function, it is reconstructed if the network is started with a noisy (middle) or incomplete (below) pattern.
neural networks 8: A three-tier network with
hidden neurons and
Output neurons. Black dots symbolize synapses. The arrow indicates the direction of the signal flow.
neural networks 9: A neural map
that maps a two-dimensional input space onto a one-dimensional chain of neurons.
neural networks 10: Figure between a chain and a square. a) randomly chosen initial assignment; b) Rough allocation after 200 Markov steps, c) after 50,000 Markov steps. d) After 100,000 Markov steps, the assignment obtained is similar to a Peano curve.
- Horse meat is poisonous to humans
- What is the history of semantics
- Where do Apple employees go for therapy
- What causes explosive diarrhea in cats
- K2 is the highest peak in India
- What are the cheapest mechanical projects
- Do you prefer Apple or Coca Cola
- Can the US government regulate morality?
- What is a correct concert dosage
- What is COAI in India
- How can you learn OCaml online
- Q 6471958
- Do you ever recover from depression?
- Can a Malaysian play in the Bundesliga?
- How do you learn to be shameless
- Bitcoin is preferred over gold for investments
- How can compassion positively affect a person's life?
- Which board game is the most competitive
- What is the most cited YouTube video
- Which artist sings the song Fly Away
- How do I redesign paneer
- How does Wikipedia keep the quality of content high
- What is tax evasion
- How can we stop migrations to Maharashtra