In this exercise, we are going to explore just a few aspects of what type of representations the backpropagation algorithm can develop. Sometimes its power of representation is quite surprising, though the number of learning cycles may sometimes reach very high values, even with tiny networks and small data sets.
In the simulators below the takes the currently active pattern and applies one iteration of the backpropagation learning algorithm (including upsweep activation and downsweep error-based adjustment of the weights). The does this for all patterns in the currently active pattern set. The n can be set with the Learn All Time parameter, which for backpropagation may be set high for small but difficult problems, e.g., 100 or even 1000. (Start low and then increase once you have an idea how long this will take on your computer.)
Once the network has been trained, you can inspect the hidden layer activations by loading a pattern and clicking . This will carry out just the upsweep activation, with changing any weights.
Activate the pattern set. Set Learn All Time to 100 or even 1000 and start training the network on the XOR patterns. How many learning cycles does it take before the PatternSetError is below 0.01? Inspect the resulting 'solution' by the network by switching to Connection View and looking at the receptive field of each node. Draw the network and try to understand the logic. Describe the essence of the solution the network found.
All this may already cause a little bit of reasoning. Now, imagine a pattern set with 1000 patterns and a network with 100,000 weights. The network may find a solution, but it may be extremely difficult to understand its underlying logic.
In this exercise we have a simple mapping of vectors of the type 1000 to themselves. But there is a catch: the hidden layer has only two nodes. Find out what the hidden layer activiation 'code' is for each input pattern. This network does not always converge and will need in the order of learning (Learn All) 10,000 cycles or more to find a solution (i.e., the pattern set error is then in the order of 0.02 or lower).
Note that we have a an extra 'layer' below the input layer with a single node that is always 1. This is a bias node; it is used to implement a threshold or bias parameter for each of the other nodes in the network. Without a bias node, the network will not converge to a good solution. You can view it as a tunable threshold for each node. In the original backpropagation publication by Rumelhart, Hinton, and Williams (1986) a bias term was incorporated in the activation rule. The approach here is mathematically equivalent and makes for cleaner programming code. Now, inspect the weights in the hidden and output layer. What solution 'strategy' has the network found and in which layer is the bias value important and why?
Here we have an even more extreme example of recoding. In this exercise, which is derived from Rumelhart, Hinton, and Williams (1986, p.337-339), we train the network with the following patterns: 00 → 1000, 01 → 0100, 10 → 0010, and 11 → 0001. Note that the binary input values must first be recoded to a single value by the network after which they must be decoded and mapped to the output nodes. The system can do this but it is likely to need a long time to learn this. Though it sometimes converges in about 5000 learning (Learn All) cycles, it may take as many as 50,000 to 200,000 learning cycles. In the process we can observe very long periods of time in which the pattern error hardly goes down. In rare cases, the network simply will not converge. In that case, reload the page, which will reset the weights, and try again. Like the network in Exercise 9b, we need a bias term to enable the network to find a solution.
Reference
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation.
In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing. Volume 1: Foundations (pp. 318-362).
Cambridge, MA: MIT Press.
After the network has converged (e.g., with a pattern set error of 0.02 or lower), find out how the network has solved this very difficult recoding problem. How are the activations remapped in the various layers, what are the weights? Don't just report the values but try to figure out the network's 'strategy'.