On the Generalization Problem

The generalization problem considered in this paper assumes that a limited amount of input and output data from a system is available, and that from this information an estimate of the output produced by another input is required. The ideas arose in the study of neural networks, but apply equally to any approximation approach. The main result is that the type of neural network to be used for generalization should be determined by the prior knowledge about the nature of the output from the system. Without such information, either of two networks matching the training data is equally likely to be the better at estimating the output generated by the same system at a new input. Therefore, the search for an optimum generalization network for use on all problems is inappropriate. For both (0, 1) and accurate real outputs, it is shown that simple approximations exist that fit the data, so these will be equally likely to generalize better than more sophisticated networks, unless prior knowledge is available that excludes them. For noisy real outputs, it is shown that the standard least squares approach forces the neural network to approximate an incorrect process; an alternative approach is outlined, which again is much easier to learn and use.