The the small WalnutJS simulators below contain 1/1000th of the MNIST data set, which consists of 60,000 handwritten training digits and 10,000 test digits. They were written by 250 people, government employees and high school students. The digits were originally sampled on a 20 x 20 grid with 256 grey values, but they were centered such that their centroid is in the middle of a 28 x 28 grid (the current size). The grey values have been converted to a value from 0 to 1.
The idea is to train on the training set and test on the testing set. In this way, you can assess generalization.
Though the hidden layer is tiny, you can get the pattern set error below 10 after training for a long time (e.g., 50,000 learning [Learn All] cycles). When you take the time to do this (e.g., let run overnight). When you do long runs, you may ignore the browser warning, which in Chrome will not actually halt your simulation and which will disappear once the simulation is complete.
Even though the pattern set error will not drop to close to zero, the performance of the network is not that bad if you measure it by just looking at the winning node in the output layer; this is the node with the highest activation value, which is reported separately. For most digits, it will be the correct classification. Also, you may notice in the activation pattern in the hidden layer that each type of digit has a clear corresponding hidden layer representation. This can be verified if you inspect the weights from the hidden layer to the output layer. You may even be able to interpret the receptive field in terms of weight patterns to the four hidden layer nodes. In many cases, the backpropagation learning process finds surprisingly 'smart' solutions.
Generalization is much worse than the networks below. Both the pattern set error is high (e.g., over 20) and the winning nodes often give a wrong classification. So, the rule-of-thumb that smaller networks give better generalization clearly does not always apply.
This is the same network as in Exercise 10c, but now with a much larger hidden layer. How much better does it do with this much additional power of internal expression? Be sure to assess generalization. It suffices to merely activate the Test Digits to read out the pattern set error; do don't actually need to run anything else, though it is instructive to test some of the individual digits by activating it and pressing . Some of the digits are written in a quite different hand from the training set. These means that backprogation can probably never recognize those when just trained on the training set. This underscores the importantance of uses a very large and diverse training set, or a very cleverly chosen network architecture.
This network uses two identical small maps, in which the neurons project to a small portion of the input layer. Though the maps have identical projections, neurons in the same position often still develop diverging representations. What kind of features do they pick up? A larger version might have more layers with maps, and perhaps connections similar to Fukushima's neocognitron.
This network architecture is inspired by that of the visual system in the brain, where we find two routes: a slowly operating, fine-grained (parvocellular) pathway where the visual features are analyzed in great detail, and a fast, coarse-grained (magnocellular) path, which only does a rough classification. In the brain, these two pathways can be observed from very early in the visual system (e.g., Meissirel, Wikler, Chalupa, & Rakic (1997)), starting in the retina and the lateral geniculate nucleus. The parvocellular pathway connects mainly to the temporal cortex, also known as the 'what' pathway, following the ventral route through the brain. The magnocellular pathway connects mainly to the parietal cortex, also known as the 'where' pathway, following the dorsal route. We have chosen to have the coarse-grained pathway to also feed into the fine-grained one, because it is likely that in the brain the magnocellular pathway is faster and can exert a top-down influence on the parvocellular pathway. Standard backpropagation networks do not allow recurrent processing, so we have implemented this influence in a feedforward manner instead.
Reference
Meissirel, C., Wikler, K. C., Chalupa, L. M., & Rakic, P. (1997). Early divergence of magnocellular and parvocellular functional
subsystems in the embryonic primate visual system. Proceedings of the National Academy of Sciences, 94(11), 5900-5905.
Of course, this is only a very crude implementation of the real pathways! Still it seems very effective. It learns fast and generalizes well.
Compared to Network 1, above, which had only the tiny hidden layer, this network converges much faster: in about 20 learning cycles, the pattern set error may be below 25 and most digits are already recognized correctly as measured by the winning node. In 40 cycles, the pattern set error may approach 7. After 100 cycles, the pattern set error on the training set will typically be below 1.5. At this point, the pattern set error of the test set may already be quite low (e.g., below 10). After a total of 200 Learn All cycles, the pattern set error approaches 0.5 in a typical simlation, though the performance on the test digits will usually not improve much further.
It is instructive to inspect the representation formed in this network. Click for example on a 9 and do Cycle 1, then repeat this for other 9s. In this way you can see a stable representation in the coarse-grained layer with occasional changes in the fine-grained one, for example, to accomodate 'odd' characters in the training set, which may occur because people sometimes have quite different styles of handwriting.