For this exercise you are supposed to use the Hopfield (1982) simulator below.
12.a Learn a single pattern (e.g., a Face). Lesion it but not so much that it cannot be completed easily and perfectly, e.g., lesion 30%. Now, start from a random activation pattern and cycle until you have retrieved the pattern (or its inverse). Learn it again. Now, you could say the lesion has partially been repaired. Repeat the lesion-learn (repair) cycle a couple of times. How many times can you do this before you start seeing mistakes?
12.b Reset the weights (reload page) and repeat this entire experiment with two (or more patterns). What is one of the great problems for the repair when you do this?
12.c The mechanism tried out here, though highly simplified, resembles a type of consolidation found in the neural network literature. What are some of the problems that you encounter when informally trying this that may in fact cause serious problems for the existing consolidation theories? How could they be addressed?
Take one easily identifiable pattern, for example, the face from the Face-House pattern set. (1) Copy it to the activation layer but do not learn it! (2) First, distort it with 30%. (3) Then, learn the distortion. (4) Repeat Step (1-3) five times.
You have now learned five strongly distorted versions of the face pattern, but never a perfect version. One might see this as having experienced (and stored) five events that are variations on a common theme, e.g., five dinner parties or indeed five views of a face but with different hair, lighting, distance, angle, shadows, etc.
Now, check what the network has learned. When cycling activations (e.g., 100 or more cycles), starting with a new distorted face, will it retrieve a strong or mild distortion or will it tend to retrieve the (unseen) original? What happens when you cycle a few hundred times with the unseen original face pattern? Try to explain what has happened here. Assuming that the brain works similarly, what would you predict for ordinary learning by humans or animals? Are all traces of the original distorted patterns always erased or are they still (partially?) available?
For this exercise, you may use the backpropagation simulator below. The architectures is a simple feedforward network with a hidden layer of size 6 and a bias node. The patterns are exceedingly simple, namely learning of a single activated node from input to output. In pattern set Small 1 we have patterns [1,0,0,0], [0,1,0,0], and [0,0,1,0]. In pattern set Small 2, there is only a single pattern: [0,0,0,1]. The same pattern is presented to both the Input and the Output layer (auto-associative learning).
First, activate pattern set Small 1. Set the Learn All cycle parameter to 100 and keep pressing until the pattern set error is below 0.10. Learning should go fairly quickly, e.g., 1000 cycles in total. Note the number of Learn All cycles, the pattern error, and the hidden layer representation for each of the three patterns.
Do the same with pattern set Small 2, including noting the Learn All cycle time, pattern error, and hidden layer representation.
Now, activate pattern set Small 1 again. What is the pattern set error (calculated automatically upon activation of a pattern set)? Present each pattern in turn and press (not Learn All!) in order to assess the hidden layer representation and the output, including the Winning Node. What are your findings? Think of ways in which this catastrophic interference could be eliminated.