Supervised pre-clustering for sparse regression

Kernel algorithms for large data-sets are now an active research area motivated by the many real world problems producing very large numbers of data points. However, standard kernel methods scale poorly with the size of the data-set. We propose a mathematically motivated approach to sparse function estimation that utilises the uniform continuity properties of functions in continuous reproducing kernel Hilbert spaces (RKHS) defined on compact domains. Using the uniform continuity properties of the function a similarity measure on data points is defined that allows data to be pre-clustered. Unlike previous methods for sparse function estimation using clustering the proposed approach is supervised and relies on well-defined mathematical concepts. The cluster centres are used to form a sparse RKHS subspace within which the function is estimated. The greedy pre-clustering algorithms and sparse model based on pre-clustering and committee machine techniques are presented. Results are presented to demonstrate the application of the proposed algorithms on function approximation problems.