{"id":81,"date":"2018-07-13T05:22:50","date_gmt":"2018-07-13T05:22:50","guid":{"rendered":"http:\/\/itp6.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=81"},"modified":"2019-05-16T04:26:03","modified_gmt":"2019-05-16T04:26:03","slug":"learning-process-in-bpnn-and-hopfield-networks","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/chapter\/learning-process-in-bpnn-and-hopfield-networks\/","title":{"rendered":"Learning process in BPNN and Hopfield networks"},"content":{"raw":"
\"epgp<\/a>\r\n<\/span><\/div>\r\n
\r\n\r\n \r\n

Introduction<\/strong><\/p>\r\n \r\n

This module extends the discussion that we started in the previous module. We looked at how multilayer perceptrons or multilayer networks can be designed and how the activations are calculated for the input. Now we will see how learning is done and how weights are updated in BPNN. We will also see how Hopfield networks are used for content addressability in this module.<\/p>\r\n \r\n

Learning in Back Propagation network<\/strong><\/p>\r\n \r\n

Back propagation network or BPNN is a neural network for which learning we can apply back propagation algorithm. The figure 10.5 that we have seen in previous module is an example of back propagation network. The back propagation network is popularly known as BPNN.<\/p>\r\n \r\n

The forward activations (h<\/em>i and o<\/em>j) are calculated based on sigmoid activation function. Once the output is received for a given input, the weights must be adjusted for making sure better output is achieved next time. We have already seen that adjustment of weight is decided based on the error that we received. The output is a real value between 0 and 1 and the required value is either 0 or 1. For example if for a given face, the correct output is 0,0,0,0,1,1, and we might get 0.10, 0.01, 0.92,0.25,0.92,0.87 as the output (considering two digits after the decimal point and truncating the output accordingly) of each layer. Now the difference at each unit can be calculated as 0.10, 0.01, 0.92, 0.25, 0.08, 0.13 (the difference between actual values calculated and required output) Now we decide what our tolerance level is. Suppose if our tolerance level is 0.10, three units have correctly learned (their difference is less than or equal to 0.10) while the rest are still to be learned. Their weights are to be reduced if they are positively misclassified or increased otherwise. Look at output units 1 and 2, both of them are learned so do not require additional learning. So no weight update takes place for them. Look at the third unit. It should output 0 but provides 0.92 and thus the weights associated to the lines connecting to output unit 3 (o3) must be reduced. Similarly output unit 5 must increase its weights a little further.<\/p>\r\n \r\n

The only issue now is to find out the exact values of weights to be increased or decreased for each erring input. For that, let us assume that o<\/em>j is the output received and y<\/em>j is the actual output received. The weights associated w<\/em>ith the incoming connections are all needed to be changed. For jth output unit, the difference is y<\/em>j-o<\/em>j. The error is calculated by multiplying this value w<\/em>ith o<\/em>j and (1- o<\/em>j)1. The error is denoted by \u03b42 (Error at second weight layer). The subscript to that \u03b42 indicates the unit of the output layer.<\/p>\r\n \r\n

\"1\u00a0 Explanation to this multiplication is beyond the scope of this discussion. You can refer any book on BPNN for further discussion.\"<\/em><\/p>\r\n\r\n<\/div>\r\n \r\n

Thus the error at output unit j is = \u03b42j = (<\/span>y<\/em>j-<\/span>o<\/em>j) * <\/span>o<\/em>j * (1- <\/span>o<\/em>j). Now let us look at the difference made at hidden layer.<\/span><\/p>\r\n\r\n

\r\n\r\n \r\n

Let us try to calculate error at h<\/em>1. The h1 hidden unit is connected to all o<\/em>j (for all j from 1 to m). Each oj has \u03b42j error associated with it. All lines that bring the errors back to h1 is shown in the figure 11.1 as boldface.<\/p>\r\n \r\n\r\nThe summation of all errors and associated weights for a hidden unit h1 is found as follows\r\n\r\n \r\n\r\n= w2<\/em>11 * error at o<\/em>1 + w2<\/em>12 * error at o<\/em>2 +W213 * error at o<\/em>3 + w2<\/em>14 * error at o<\/em>4+ \u2026. + w2<\/em>1l * error at o<\/em>l(\r\n\r\n \r\n\r\n= \u2211\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j\r\n\r\n \r\n\r\nThe error at hidden unit h<\/em>1 is therefore\r\n\r\n \r\n\r\n= h<\/em>1 * (1 - h<\/em>1) \u2211\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j\r\n\r\n \r\n\r\nThus for ith hidden unit, the error is denoted as \u03b41i and is calculated as\r\n\r\n\u03b41i = h<\/em>i * (1 - h<\/em>i) \u2211\u00a0 \u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j<\/sub>\r\n\r\n \r\n

Once the errors are calculated, one must think of updating the weights. We do not really need to w<\/em>orry about reducing or incrementing weights as the sign of error itself determines that. For example if (y<\/em>j<\/sub>-o<\/em>j<\/sub>) is positive, w<\/em>e need to decrease the weights and increase otherwise. The other values that w<\/em>e multiply o<\/em>j<\/sub> * (1- o<\/em>j) is always negative and thus the multiplication (y<\/em>j<\/sub>-o<\/em>j<\/sub>) * o<\/em>j<\/sub> * (1- o<\/em>j<\/sub>) is positive when the weights are to be increased and negative when the weights to be decreased. That means, when the weight update is done, these values can be used as it is.<\/p>\r\n \r\n

Thus the weight update is done using the errors associated w<\/em>ith each weight. The error in the second layer (w2<\/sub>), is coming from the hidden unit and in the first layer (w1<\/sub>) from the input unit.<\/p>\r\n \r\n

For example updates at w2<\/em>11<\/sub>\u00a0 is based on the h1<\/sub>\u00a0 (the place from activation begins) and the error at o<\/em>1<\/sub> is \u03b421<\/sub> (Where the activation ends).<\/p>\r\n \r\n\r\nSo weight update at w2<\/em>11<\/sub> is based on h<\/em>1<\/sub> * \u03b421<\/sub>\r\n\r\n \r\n\r\nSimilarly weight update at w2<\/em>ij is based on h<\/em>i * \u03b42j \/\/ note that the subscripts of h <\/em>and \u03b42 are different\r\n\r\n \r\n\r\nThe weight updates are indicated as \u2206. Usually the weight update also includes the value called \u0273. So\r\n\r\nweight updates are w<\/em>ritten as\r\n\r\n \r\n\r\n\u2206 w2<\/em>ij = \u0273 * hi * \u03b42j\r\n\r\n<\/div>\r\n \r\n

The symbol \u0273 is known as the learning rate. Its value is kept in the range of 0.3 to 0.42<\/sup>. Thus\u00a0<\/span>New <\/span>w2<\/em>ij = Old <\/span>w2<\/em>ij \u00a0+ \u2206 <\/span>w2<\/em>ij<\/span><\/p>\r\n\r\n

\r\n\r\n \r\n\r\nAn update is usually also multiplied with a value called momentum factor \u03b1 which is kept at 0.9 after a few iterations. In the initial few iterations, the value is kept very law, 0.1 usually.\r\n\r\n \r\n\r\nNew w2<\/em>ij = Old w2<\/em>ij \u00a0+ \u03b1 * \u2206 w2<\/em>ij\r\n\r\n \r\n\r\nSimilarly updates at the first layer involves initiators at the first activation layer, x<\/em>i and also terminators at the hidden layer for each weight and weight updates are w<\/em>ritten as\r\n\r\n \r\n\r\n\u2206 w1<\/em>ij = \u0273 * x<\/em>i * \u03b41j Thus\r\n\r\nNew w1<\/em>ij = Old w1<\/em>ij \u00a0+ \u2206 w1<\/em>ij\r\n\r\n \r\n\r\nNew w1<\/em>ij = Old w1<\/em>ij \u00a0+ \u03b1 * \u2206 w1<\/em>ij (considering momentum factor)\r\n\r\n \r\n

Once these weight updates are applied, one epoch is said to be over. Multiple epochs (thousands of them in most cases) are required in getting the set of weights w<\/em>hich can classify all the inputs correctly. If w<\/em>e do not get one set, w<\/em>e may need to start all over again as there is no guarantee that w<\/em>e w<\/em>ill always get that weight vector.<\/p>\r\n \r\n\r\n2\u00a0 The values like learning rate and momentum rate etc are found empirically, researchers tries a few values and whichever value gives better performance with lesser overhead is used.<\/em>\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n

Figure 11.1 The error calculation back at hidden layer based on error found at output layer.<\/p>\r\n \r\n\r\nThe steps of the algorithm<\/strong>\r\n\r\n \r\n\r\nLet us briefly list the steps of the algorithm for training.\r\n\r\n \r\n

1. Decide about neurons for each layer, input and output first and then hidden as a geometrical mean of both values<\/p>\r\n

2. Initialize each matrix of weights with random weights of small values for example -0.1 and +0.1 or between 0.05 and -0.05. The idea of keeping the values small is that the larger weights are found to start dominating and bias the output. Keeping them small helps them to learn in a more unbiased way.<\/p>\r\n3. Epoch = 1;\r\n\r\n4. Pick up first input\r\n\r\n5. Provide the input to each x<\/em>i, and calculate each h<\/em>i as well as o<\/em>i based on the input.\r\n\r\n6. Get correct output for that input, call it y<\/em>j.\r\n\r\n7. Find out error \u03b42 based on equation 10.13 for each activation h<\/em>i.\r\n\r\n8. Find out error \u03b41 based on equation 10.2 for each activation x<\/em>i\r\n\r\n9. If the error is less than tolerance level mark that this weight is learned and go to 11\r\n\r\n10. Find out \u2206 w2<\/em>ij and \u2206 w1<\/em>ij and update weights accordingly.\r\n\r\n11. If there is another input, take it and go to 5.<\/span>\r\n\r\n12. This state is reached when all inputs are processed<\/span>\r\n\r\n13. Epoch value is incremented by 1<\/span>\r\n\r\n14. If Epoch value is more than reasonable (different for different case), the network is not learning, stop and restart the process from the beginning.<\/span>\r\n\r\n15. If there is any weight which is not learned, pick up first input again and go to 5.<\/span>\r\n\r\n16. Otherwise store the weight matrices and ready for testing. <\/span>\r\n\r\n \r\n\r\nNow let us write down the algorithm for testing<\/span>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n1. Populate the weight matrix with weights learned during the training process\r\n\r\n2. Take the first testing input.\r\n\r\n3. Provide that input and see what the output is. Now we use a different tolerance level, normally it is such that if a value is > 0.5 it is considered 1 and if it is less than 0.5 it is considered 0.\r\n\r\n4. Display the output. There is no updating of weights for testing inputs.\r\n\r\n \r\n

Geometrical view of the learning process<\/strong><\/p>\r\n \r\n

If one would like to view this process geometrically, it is about finding out a line in a plane w<\/em>hich separates two classes. If there are multiple classes, w<\/em>e need multiple lines to segregate the plane in segments w<\/em>hich correctly divides the plane into classes w<\/em>e w<\/em>ant. For example if assume face recognition process, w<\/em>e might have a plane w<\/em>ith some weight values as mentioned in figure 11.2. Multiple lines represent typical sequence of neurons. The combination of weights represent m <\/em>and c values in a conventional y = mx+ c equation of line. Thus some weights represent the slope of the line and some other represents the intersection on Y axis.<\/p>\r\n \r\n\r\n\"\"\r\n

Figure 11.2 Different regions separated in classes by different lines<\/p>\r\n\r\n<\/div>\r\n \r\n

The figure 11.2 represents a case <\/span>w<\/em>here the correct weights for all lines are found and thus those lines are able to divide each region correctly4.<\/span><\/p>\r\n\r\n

\r\n\r\n \r\n

Starting w<\/em>ith random weights means w<\/em>e are drawing an arbitrary line in the plane. Changing weights using back propagation algorithm means w<\/em>e are changing weights in a way that the new line separates the classes better than previous case.<\/p>\r\n \r\n

In fact, w<\/em>hen w<\/em>e are using multiple neurons in a single layer, it represents an n<\/em>-dimensional plane where another n<\/em>-1 dimensional plane or multiple planes are used to segregate different classes. In AI parlance, this plane (or a line in a previous case and in figure 11.2) is called a decision surface<\/strong>.<\/p>\r\n \r\n

The back propagation algorithm, in short, is to find one or multiple decision surfaces of dimension n<\/em>-1 for correctly segregating each class in an n <\/em>dimensional plane.<\/p>\r\n \r\n

Content addressability and Hopfield networks<\/strong><\/p>\r\n \r\n

Though most neural networks are used for classification, many other proposals are made to use them in other cases. An important use of neural network is in the field of content addressability. Let us take an example to understand. We have usually seen byte storing 8 bits and designed to take any combination to store any sequence of eight bits from 00000000 to 11111111. The example that we used for content addressability that is described here use same eight bits but little differently. Here a neuron is used to store a bit and each neuron is connected w<\/em>ith other neuron w<\/em>ith some weight. Each neuron is assigned\u00a0some value in the beginning. Once the values are assigned, each neuron calculates the \u2211i= n\u00a0<\/span>w<\/em>here\u00a0<\/span>n <\/em>is total number of other neurons connected to it, <\/span>x<\/em>i is the value of that neuron and <\/span>w<\/em>i is the weight connecting the neuron. For example the last neuron in the figure 11.3 is connected to 5th and 6th neuron. It will inhibit the 5th neuron by 5 while exhibit the 6th neuron by 2. The 5th neuron is 1 so sets the last neuron to zero. The 6th unit on the contrary receive positive input from that neuron but it has a strong inhibition (-2 from second neuron and -5 from 4th neuron) and the total comes out to be negative thus it resort to 0.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n

In fact only based on this connection information a few values are possible to be stored by network shown in figure 11.3. For example if w<\/em>e begin w<\/em>ith 11111111, the network will settle down into one of the few stable states. The state it settles into depends on which units are activated from the list. If we take a left to right route for example, it works like this.<\/p>\r\n \r\n\r\nThe first unit becomes active (the first 1) w<\/em>hich actives the fourth neuron. It also inhibits third neuron, thus it makes it\r\n\r\n \r\n\r\n1_01 _ _ _ _\r\n\r\n \r\n\r\nNow the second unit becomes active (second 1), it activates unit 7 and inhibits unit no 6. Thus after that the value is<\/span>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n1101_ 01_\r\n\r\n \r\n\r\nNow next is fifth unit, it is turned on than it is one and now the eighth unit w<\/em>ill be zero, as 5th unit inhibits it. The result is\r\n\r\n \r\n\r\n11011010\r\n\r\n \r\n

Thus even if the input is incorrect (not stable form), the output is in stable form. You can try a few other combinations and can see that only a few possible options possible for that network to settle into. One more option available w<\/em>ith the above case is, w<\/em>e just assume first two units are activated and then the last one is activated. You can get eighth unit on before the fifth and it exhibits the fifth neuron and result is<\/p>\r\n \r\n\r\n11010<\/strong>011<\/strong>\r\n\r\n \r\n\r\nYou can see two bits are different in this another stable state.\r\n\r\n \r\n\r\nOne typical stable state is surprisingly 00000000, all zeros. You can easily see that any other case also has that as a stable state.\r\n\r\n \r\n

You may be wondering what w<\/em>e are trying achieve by doing seemingly w<\/em>asteful exercise. Eight neurons, w<\/em>hich in true sense are capable of storing 28 values, are restricted to store much lesser (which one can count on finger tips usually), w<\/em>ith additional weights and so on.<\/p>\r\n \r\n

We are trying to demonstrate a type of network called Hopfield w<\/em>ith the ability to content address. Content addressability, if you remember, is the ability of network to get some part of data and get complete data. For example listening a few bars of music and get the entire song. Here w<\/em>e have provided a few activations and we received activations from all other units. Quite similar to looking at part of the face and recognize the face. You can assume each neuron indicating a feature, an arc as a relationship between features weights as strength of that relation. For face recognition from a partial image, such networks can be used. For some information (some part of that can even be w<\/em>rong! For example a big fish w<\/em>hich comes often to the surface and throws w<\/em>ater up like a fountain may be a query. The features are fish, fountain of water, being big in size and coming on surface regularly. All of this might get you an answer called w<\/em>hale. You may have noticed that w<\/em>e have an incorrect input (fish), w<\/em>hale is not a fish but a mammal but w<\/em>e still can get the right answer. In the case of figure 11.3 also, if w<\/em>e give 1110 as an input for example, the first two w<\/em>ill set the next two and w<\/em>e w<\/em>ill get 1101 and the rest based on the input. This is content addressability Hopfield networks are designed to provide. In our trivial examples w<\/em>e have looked only at 8 such neurons, in real w<\/em>orld case there may be thousands of neurons5.<\/p>\r\n\r\n<\/div>\r\n

\"\"<\/p>\r\n

Figure 11.3 Content Addressability<\/p>\r\n \r\n

Summary<\/strong><\/p>\r\n \r\n

The back propagation algorithm is usually applied to neural networks which are multilayer feed forward and complete. The usual neural network contains three layers, input, hidden and output. The input layer distributes the inputs to all hidden units, hidden units process them and distribute them to output units. The output units process them to generate outputs. Errors are calculated and propagated back. The weights between input and hidden and hidden and output units are changed accordingly. This process continues till the weights are set for every input correctly. The learning process basically divides inputs into output clusters so much so that every input is correct classified to a geometric region identified as a typical output. Another type of neural network is known as Hopfield networks which are able to help content addressability that means setting a small fraction of the pattern generates complete pattern.<\/p>\r\n\r\n\r\n\r\n
you can view video on Learning process in BPNN and Hopfield networks<\/strong><\/td>\r\n\"\"<\/a><\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>","rendered":"
\"epgp<\/a>
\n<\/span><\/div>\n
\n

 <\/p>\n

Introduction<\/strong><\/p>\n

 <\/p>\n

This module extends the discussion that we started in the previous module. We looked at how multilayer perceptrons or multilayer networks can be designed and how the activations are calculated for the input. Now we will see how learning is done and how weights are updated in BPNN. We will also see how Hopfield networks are used for content addressability in this module.<\/p>\n

 <\/p>\n

Learning in Back Propagation network<\/strong><\/p>\n

 <\/p>\n

Back propagation network or BPNN is a neural network for which learning we can apply back propagation algorithm. The figure 10.5 that we have seen in previous module is an example of back propagation network. The back propagation network is popularly known as BPNN.<\/p>\n

 <\/p>\n

The forward activations (h<\/em>i and o<\/em>j) are calculated based on sigmoid activation function. Once the output is received for a given input, the weights must be adjusted for making sure better output is achieved next time. We have already seen that adjustment of weight is decided based on the error that we received. The output is a real value between 0 and 1 and the required value is either 0 or 1. For example if for a given face, the correct output is 0,0,0,0,1,1, and we might get 0.10, 0.01, 0.92,0.25,0.92,0.87 as the output (considering two digits after the decimal point and truncating the output accordingly) of each layer. Now the difference at each unit can be calculated as 0.10, 0.01, 0.92, 0.25, 0.08, 0.13 (the difference between actual values calculated and required output) Now we decide what our tolerance level is. Suppose if our tolerance level is 0.10, three units have correctly learned (their difference is less than or equal to 0.10) while the rest are still to be learned. Their weights are to be reduced if they are positively misclassified or increased otherwise. Look at output units 1 and 2, both of them are learned so do not require additional learning. So no weight update takes place for them. Look at the third unit. It should output 0 but provides 0.92 and thus the weights associated to the lines connecting to output unit 3 (o3) must be reduced. Similarly output unit 5 must increase its weights a little further.<\/p>\n

 <\/p>\n

The only issue now is to find out the exact values of weights to be increased or decreased for each erring input. For that, let us assume that o<\/em>j is the output received and y<\/em>j is the actual output received. The weights associated w<\/em>ith the incoming connections are all needed to be changed. For jth output unit, the difference is y<\/em>j-o<\/em>j. The error is calculated by multiplying this value w<\/em>ith o<\/em>j and (1- o<\/em>j)1. The error is denoted by \u03b42 (Error at second weight layer). The subscript to that \u03b42 indicates the unit of the output layer.<\/p>\n

 <\/p>\n

“1\u00a0 Explanation to this multiplication is beyond the scope of this discussion. You can refer any book on BPNN for further discussion.”<\/em><\/p>\n<\/div>\n

 <\/p>\n

Thus the error at output unit j is = \u03b42j = (<\/span>y<\/em>j-<\/span>o<\/em>j) * <\/span>o<\/em>j * (1- <\/span>o<\/em>j). Now let us look at the difference made at hidden layer.<\/span><\/p>\n

\n

 <\/p>\n

Let us try to calculate error at h<\/em>1. The h1 hidden unit is connected to all o<\/em>j (for all j from 1 to m). Each oj has \u03b42j error associated with it. All lines that bring the errors back to h1 is shown in the figure 11.1 as boldface.<\/p>\n

 <\/p>\n

The summation of all errors and associated weights for a hidden unit h1 is found as follows<\/p>\n

 <\/p>\n

= w2<\/em>11 * error at o<\/em>1 + w2<\/em>12 * error at o<\/em>2 +W213 * error at o<\/em>3 + w2<\/em>14 * error at o<\/em>4+ \u2026. + w2<\/em>1l * error at o<\/em>l(<\/p>\n

 <\/p>\n

= \u2211\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j<\/p>\n

 <\/p>\n

The error at hidden unit h<\/em>1 is therefore<\/p>\n

 <\/p>\n

= h<\/em>1 * (1 – h<\/em>1) \u2211\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j<\/p>\n

 <\/p>\n

Thus for ith hidden unit, the error is denoted as \u03b41i and is calculated as<\/p>\n

\u03b41i = h<\/em>i * (1 – h<\/em>i) \u2211\u00a0 \u00a0 \u00a02\u00a0\u00a0 \u00a0* \u03b42j<\/sub><\/p>\n

 <\/p>\n

Once the errors are calculated, one must think of updating the weights. We do not really need to w<\/em>orry about reducing or incrementing weights as the sign of error itself determines that. For example if (y<\/em>j<\/sub>–o<\/em>j<\/sub>) is positive, w<\/em>e need to decrease the weights and increase otherwise. The other values that w<\/em>e multiply o<\/em>j<\/sub> * (1- o<\/em>j) is always negative and thus the multiplication (y<\/em>j<\/sub>–o<\/em>j<\/sub>) * o<\/em>j<\/sub> * (1- o<\/em>j<\/sub>) is positive when the weights are to be increased and negative when the weights to be decreased. That means, when the weight update is done, these values can be used as it is.<\/p>\n

 <\/p>\n

Thus the weight update is done using the errors associated w<\/em>ith each weight. The error in the second layer (w2<\/sub>), is coming from the hidden unit and in the first layer (w1<\/sub>) from the input unit.<\/p>\n

 <\/p>\n

For example updates at w2<\/em>11<\/sub>\u00a0 is based on the h1<\/sub>\u00a0 (the place from activation begins) and the error at o<\/em>1<\/sub> is \u03b421<\/sub> (Where the activation ends).<\/p>\n

 <\/p>\n

So weight update at w2<\/em>11<\/sub> is based on h<\/em>1<\/sub> * \u03b421<\/sub><\/p>\n

 <\/p>\n

Similarly weight update at w2<\/em>ij is based on h<\/em>i * \u03b42j \/\/ note that the subscripts of h <\/em>and \u03b42 are different<\/p>\n

 <\/p>\n

The weight updates are indicated as \u2206. Usually the weight update also includes the value called \u0273. So<\/p>\n

weight updates are w<\/em>ritten as<\/p>\n

 <\/p>\n

\u2206 w2<\/em>ij = \u0273 * hi * \u03b42j<\/p>\n<\/div>\n

 <\/p>\n

The symbol \u0273 is known as the learning rate. Its value is kept in the range of 0.3 to 0.42<\/sup>. Thus\u00a0<\/span>New <\/span>w2<\/em>ij = Old <\/span>w2<\/em>ij \u00a0+ \u2206 <\/span>w2<\/em>ij<\/span><\/p>\n

\n

 <\/p>\n

An update is usually also multiplied with a value called momentum factor \u03b1 which is kept at 0.9 after a few iterations. In the initial few iterations, the value is kept very law, 0.1 usually.<\/p>\n

 <\/p>\n

New w2<\/em>ij = Old w2<\/em>ij \u00a0+ \u03b1 * \u2206 w2<\/em>ij<\/p>\n

 <\/p>\n

Similarly updates at the first layer involves initiators at the first activation layer, x<\/em>i and also terminators at the hidden layer for each weight and weight updates are w<\/em>ritten as<\/p>\n

 <\/p>\n

\u2206 w1<\/em>ij = \u0273 * x<\/em>i * \u03b41j Thus<\/p>\n

New w1<\/em>ij = Old w1<\/em>ij \u00a0+ \u2206 w1<\/em>ij<\/p>\n

 <\/p>\n

New w1<\/em>ij = Old w1<\/em>ij \u00a0+ \u03b1 * \u2206 w1<\/em>ij (considering momentum factor)<\/p>\n

 <\/p>\n

Once these weight updates are applied, one epoch is said to be over. Multiple epochs (thousands of them in most cases) are required in getting the set of weights w<\/em>hich can classify all the inputs correctly. If w<\/em>e do not get one set, w<\/em>e may need to start all over again as there is no guarantee that w<\/em>e w<\/em>ill always get that weight vector.<\/p>\n

 <\/p>\n

2\u00a0 The values like learning rate and momentum rate etc are found empirically, researchers tries a few values and whichever value gives better performance with lesser overhead is used.<\/em><\/p>\n<\/div>\n

\n

\"\"<\/p>\n

Figure 11.1 The error calculation back at hidden layer based on error found at output layer.<\/p>\n

 <\/p>\n

The steps of the algorithm<\/strong><\/p>\n

 <\/p>\n

Let us briefly list the steps of the algorithm for training.<\/p>\n

 <\/p>\n

1. Decide about neurons for each layer, input and output first and then hidden as a geometrical mean of both values<\/p>\n

2. Initialize each matrix of weights with random weights of small values for example -0.1 and +0.1 or between 0.05 and -0.05. The idea of keeping the values small is that the larger weights are found to start dominating and bias the output. Keeping them small helps them to learn in a more unbiased way.<\/p>\n

3. Epoch = 1;<\/p>\n

4. Pick up first input<\/p>\n

5. Provide the input to each x<\/em>i, and calculate each h<\/em>i as well as o<\/em>i based on the input.<\/p>\n

6. Get correct output for that input, call it y<\/em>j.<\/p>\n

7. Find out error \u03b42 based on equation 10.13 for each activation h<\/em>i.<\/p>\n

8. Find out error \u03b41 based on equation 10.2 for each activation x<\/em>i<\/p>\n

9. If the error is less than tolerance level mark that this weight is learned and go to 11<\/p>\n

10. Find out \u2206 w2<\/em>ij and \u2206 w1<\/em>ij and update weights accordingly.<\/p>\n

11. If there is another input, take it and go to 5.<\/span><\/p>\n

12. This state is reached when all inputs are processed<\/span><\/p>\n

13. Epoch value is incremented by 1<\/span><\/p>\n

14. If Epoch value is more than reasonable (different for different case), the network is not learning, stop and restart the process from the beginning.<\/span><\/p>\n

15. If there is any weight which is not learned, pick up first input again and go to 5.<\/span><\/p>\n

16. Otherwise store the weight matrices and ready for testing. <\/span><\/p>\n

 <\/p>\n

Now let us write down the algorithm for testing<\/span><\/p>\n<\/div>\n

\n

 <\/p>\n

1. Populate the weight matrix with weights learned during the training process<\/p>\n

2. Take the first testing input.<\/p>\n

3. Provide that input and see what the output is. Now we use a different tolerance level, normally it is such that if a value is > 0.5 it is considered 1 and if it is less than 0.5 it is considered 0.<\/p>\n

4. Display the output. There is no updating of weights for testing inputs.<\/p>\n

 <\/p>\n

Geometrical view of the learning process<\/strong><\/p>\n

 <\/p>\n

If one would like to view this process geometrically, it is about finding out a line in a plane w<\/em>hich separates two classes. If there are multiple classes, w<\/em>e need multiple lines to segregate the plane in segments w<\/em>hich correctly divides the plane into classes w<\/em>e w<\/em>ant. For example if assume face recognition process, w<\/em>e might have a plane w<\/em>ith some weight values as mentioned in figure 11.2. Multiple lines represent typical sequence of neurons. The combination of weights represent m <\/em>and c values in a conventional y = mx+ c equation of line. Thus some weights represent the slope of the line and some other represents the intersection on Y axis.<\/p>\n

 <\/p>\n

\"\"<\/p>\n

Figure 11.2 Different regions separated in classes by different lines<\/p>\n<\/div>\n

 <\/p>\n

The figure 11.2 represents a case <\/span>w<\/em>here the correct weights for all lines are found and thus those lines are able to divide each region correctly4.<\/span><\/p>\n

\n

 <\/p>\n

Starting w<\/em>ith random weights means w<\/em>e are drawing an arbitrary line in the plane. Changing weights using back propagation algorithm means w<\/em>e are changing weights in a way that the new line separates the classes better than previous case.<\/p>\n

 <\/p>\n

In fact, w<\/em>hen w<\/em>e are using multiple neurons in a single layer, it represents an n<\/em>-dimensional plane where another n<\/em>-1 dimensional plane or multiple planes are used to segregate different classes. In AI parlance, this plane (or a line in a previous case and in figure 11.2) is called a decision surface<\/strong>.<\/p>\n

 <\/p>\n

The back propagation algorithm, in short, is to find one or multiple decision surfaces of dimension n<\/em>-1 for correctly segregating each class in an n <\/em>dimensional plane.<\/p>\n

 <\/p>\n

Content addressability and Hopfield networks<\/strong><\/p>\n

 <\/p>\n

Though most neural networks are used for classification, many other proposals are made to use them in other cases. An important use of neural network is in the field of content addressability. Let us take an example to understand. We have usually seen byte storing 8 bits and designed to take any combination to store any sequence of eight bits from 00000000 to 11111111. The example that we used for content addressability that is described here use same eight bits but little differently. Here a neuron is used to store a bit and each neuron is connected w<\/em>ith other neuron w<\/em>ith some weight. Each neuron is assigned\u00a0some value in the beginning. Once the values are assigned, each neuron calculates the \u2211i= n\u00a0<\/span>w<\/em>here\u00a0<\/span>n <\/em>is total number of other neurons connected to it, <\/span>x<\/em>i is the value of that neuron and <\/span>w<\/em>i is the weight connecting the neuron. For example the last neuron in the figure 11.3 is connected to 5th and 6th neuron. It will inhibit the 5th neuron by 5 while exhibit the 6th neuron by 2. The 5th neuron is 1 so sets the last neuron to zero. The 6th unit on the contrary receive positive input from that neuron but it has a strong inhibition (-2 from second neuron and -5 from 4th neuron) and the total comes out to be negative thus it resort to 0.<\/span><\/p>\n<\/div>\n

\n

 <\/p>\n

In fact only based on this connection information a few values are possible to be stored by network shown in figure 11.3. For example if w<\/em>e begin w<\/em>ith 11111111, the network will settle down into one of the few stable states. The state it settles into depends on which units are activated from the list. If we take a left to right route for example, it works like this.<\/p>\n

 <\/p>\n

The first unit becomes active (the first 1) w<\/em>hich actives the fourth neuron. It also inhibits third neuron, thus it makes it<\/p>\n

 <\/p>\n

1_01 _ _ _ _<\/p>\n

 <\/p>\n

Now the second unit becomes active (second 1), it activates unit 7 and inhibits unit no 6. Thus after that the value is<\/span><\/p>\n<\/div>\n

\n

 <\/p>\n

1101_ 01_<\/p>\n

 <\/p>\n

Now next is fifth unit, it is turned on than it is one and now the eighth unit w<\/em>ill be zero, as 5th unit inhibits it. The result is<\/p>\n

 <\/p>\n

11011010<\/p>\n

 <\/p>\n

Thus even if the input is incorrect (not stable form), the output is in stable form. You can try a few other combinations and can see that only a few possible options possible for that network to settle into. One more option available w<\/em>ith the above case is, w<\/em>e just assume first two units are activated and then the last one is activated. You can get eighth unit on before the fifth and it exhibits the fifth neuron and result is<\/p>\n

 <\/p>\n

11010<\/strong>011<\/strong><\/p>\n

 <\/p>\n

You can see two bits are different in this another stable state.<\/p>\n

 <\/p>\n

One typical stable state is surprisingly 00000000, all zeros. You can easily see that any other case also has that as a stable state.<\/p>\n

 <\/p>\n

You may be wondering what w<\/em>e are trying achieve by doing seemingly w<\/em>asteful exercise. Eight neurons, w<\/em>hich in true sense are capable of storing 28 values, are restricted to store much lesser (which one can count on finger tips usually), w<\/em>ith additional weights and so on.<\/p>\n

 <\/p>\n

We are trying to demonstrate a type of network called Hopfield w<\/em>ith the ability to content address. Content addressability, if you remember, is the ability of network to get some part of data and get complete data. For example listening a few bars of music and get the entire song. Here w<\/em>e have provided a few activations and we received activations from all other units. Quite similar to looking at part of the face and recognize the face. You can assume each neuron indicating a feature, an arc as a relationship between features weights as strength of that relation. For face recognition from a partial image, such networks can be used. For some information (some part of that can even be w<\/em>rong! For example a big fish w<\/em>hich comes often to the surface and throws w<\/em>ater up like a fountain may be a query. The features are fish, fountain of water, being big in size and coming on surface regularly. All of this might get you an answer called w<\/em>hale. You may have noticed that w<\/em>e have an incorrect input (fish), w<\/em>hale is not a fish but a mammal but w<\/em>e still can get the right answer. In the case of figure 11.3 also, if w<\/em>e give 1110 as an input for example, the first two w<\/em>ill set the next two and w<\/em>e w<\/em>ill get 1101 and the rest based on the input. This is content addressability Hopfield networks are designed to provide. In our trivial examples w<\/em>e have looked only at 8 such neurons, in real w<\/em>orld case there may be thousands of neurons5.<\/p>\n<\/div>\n

\"\"<\/p>\n

Figure 11.3 Content Addressability<\/p>\n

 <\/p>\n

Summary<\/strong><\/p>\n

 <\/p>\n

The back propagation algorithm is usually applied to neural networks which are multilayer feed forward and complete. The usual neural network contains three layers, input, hidden and output. The input layer distributes the inputs to all hidden units, hidden units process them and distribute them to output units. The output units process them to generate outputs. Errors are calculated and propagated back. The weights between input and hidden and hidden and output units are changed accordingly. This process continues till the weights are set for every input correctly. The learning process basically divides inputs into output clusters so much so that every input is correct classified to a geometric region identified as a typical output. Another type of neural network is known as Hopfield networks which are able to help content addressability that means setting a small fraction of the pattern generates complete pattern.<\/p>\n\n\n\n
you can view video on Learning process in BPNN and Hopfield networks<\/strong><\/td>\n\"\"<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n","protected":false},"author":4,"menu_order":11,"template":"","meta":{"_acf_changed":false,"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":["prof-bhushan-trivedi"],"pb_section_license":""},"chapter-type":[],"contributor":[58],"license":[],"class_list":["post-81","chapter","type-chapter","status-publish","hentry","contributor-prof-bhushan-trivedi"],"part":3,"_links":{"self":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapters\/81","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/wp\/v2\/users\/4"}],"version-history":[{"count":6,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapters\/81\/revisions"}],"predecessor-version":[{"id":458,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapters\/81\/revisions\/458"}],"part":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapters\/81\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/wp\/v2\/media?parent=81"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/pressbooks\/v2\/chapter-type?post=81"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/wp\/v2\/contributor?post=81"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/itp6\/wp-json\/wp\/v2\/license?post=81"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}