Use this URL to cite or link to this record in EThOS:
Title: Revisiting generalization for deep learning : PAC-Bayes, flat minima, and generative models
Author: Dziugaite, Gintare Karolina
ISNI:       0000 0004 7968 6278
Awarding Body: University of Cambridge
Current Institution: University of Cambridge
Date of Award: 2019
Availability of Full Text:
Access from EThOS:
Full text unavailable from EThOS. Please try the link below.
Access from Institution:
In this work, we construct generalization bounds to understand existing learning algorithms and propose new ones. Generalization bounds relate empirical performance to future expected performance. The tightness of these bounds vary widely, and depends on the complexity of the learning task and the amount of data available, but also on how much information the bounds take into consideration. We are particularly concerned with data and algorithm- dependent bounds that are quantitatively nonvacuous. We begin with an analysis of stochastic gradient descent (SGD) in supervised learning. By formalizing the notion of flat minima using PAC-Bayes generalization bounds, we obtain nonvacuous generalization bounds for stochastic classifiers based on SGD solutions. Despite strong empirical performance in many settings, SGD rapidly overfits in others. By combining nonvacuous generalization bounds and structural risk minimization, we arrive at an algorithm that trades-off accuracy and generalization guarantees. We also study generalization in the context of unsupervised learning. We propose to use a two sample test statistic for training neural network generator models and bound the gap between the population and the empirical estimate of the statistic.
Supervisor: Ghahramani, Zoubin Sponsor: EPSRC
Qualification Name: Thesis (Ph.D.) Qualification Level: Doctoral
Keywords: Deep learning ; statistical learning theory ; Generalization in neural networks ; adversarial learning ; PAC-Bayesian bounds ; generative models