Analyzing Generalization of Neural Networks through Loss Path Kernels

Abstract

Deep neural networks have been increasingly used in real-world applications, making it critical to ensure their ability to adapt to new, unseen data. In this paper, we study the generalization capability of neural networks trained with (stochastic) gradient descent. We establish a new connection between the loss dynamics of gradient flow and general kernel machines using a unique kernel, the loss path kernel. This kernel measures the similarity between two data points by evaluating the agreement between loss gradients along the path determined by the gradient flow. Based on this, we derive a new generalization upper bound that applies to general neural network architectures. This new bound is tight and strongly correlated with the true generalization error. We apply our results to guide the design of neural architecture search (NAS) and demonstrate the favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.