Frequency Bias in Deep Learning

Prof. David Jacobs, Department of Computer Science and UMIACS, University of Maryland

Recent results have shown that highly overparameterized deep neural networks act as linear systems.  Fully connected networks are equivalent to kernel methods, with a neural tangent kernel (NTK).  This talk will describe our work on better understanding the properties of this kernel.  We study the eigenvalues and eigenvectors of NTK, and quantify a frequency bias in neural networks, that causes them to learn low frequency functions more quickly than high frequency ones.  In fact, these eigenvectors and eigenvalues are the same as those of the well-known Laplace kernel, implying that these two kernels interpolate functions with the same smoothness properties.  On a large number of datasets, we show that Kernel-based classification with NTK and the Laplace kernel perform quite similarly.