Connections between Stochasticity of SGD and Generalizability

This is an attempt to understand how stochasticity in an optimization algorithm affects the generalization properties of a neural network. Current work includes understanding the distribution of Stochastic Gradient Noise as it directly correlates to generalization under most empirical settings. There has been a recent interest in understanding the effects of distributional properties of SGD as it affects the optimization dynamics in finding good local minima (which have superior generalization properties). With the use of hypothesis testing, we observe that for batch sizes \(256\) and above, the distribution is best described as Gaussian, at least in the early phases of training. This holds across datasets, architectures, and other choices. An interesting direction is to understand which distributional characteristics in an optimization algorithm are needed for generalization on loss landscapes such as those of neural networks.

The current results have been presented at the workshop Science meets Engineering of Deep Learning at Neural Information Processing Systems 2019.

The paper can be found here.