Non-Gaussianity of Stochastic Gradient Noise
Published at Science meets Engineering of Deep Learning (SEDL) workshop, Neural Information Processing Systems (NeurIPS), 2019
What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes
The paper has been accepted at the SEDL workshop at NeurIPS 2019.
Please find the below resources:
