Use this URL to cite or link to this record in EThOS:
Title: Sample-efficient deep reinforcement learning for continuous control
Author: Gu, Shixiang
ISNI:       0000 0004 7972 9741
Awarding Body: University of Cambridge
Current Institution: University of Cambridge
Date of Award: 2019
Availability of Full Text:
Access from EThOS:
Full text unavailable from EThOS. Please try the link below.
Access from Institution:
Reinforcement learning (RL) is a powerful, generic approach to discovering optimal policies in complex sequential decision-making problems. Recently, with flexible function approximators such as neural networks, RL has greatly expanded its realm of applications, from playing computer games with pixel inputs, to mastering the game of Go, to learning parkour movements by simulated humanoids. However, the common RL approaches are known to be sample intensive, making them difficult to be applied to real-world problems such as robotics. This thesis makes several contributions toward developing RL algorithms for learning in the wild, where sample-efficiency and stability are critical. The key contributions include Normalized Advantage Functions (NAF), extending Q-learning for continuous action problems; Interpolated Policy Gradient (IPG), unifying prior policy gradient algorithm variants through theoretical analyses on bias and variance; and Temporal Difference Models (TDM), interpreting a parameterized Q-function as a generalized dynamics model for novel temporally abstracted model-based planning. Importantly, this thesis highlights that these algorithms can be seen as bridging gaps between branches of RL - model-based with modelfree, and on-policy with off-policy. The proposed algorithms not only achieve substantial improvements over the prior approaches, but also provide novel perspectives on how to mix different branches of RL effectively to gain the best of both worlds. NAF has subsequently been shown to be able to train two 7-DoF robot arms to open doors using only 2.5 hours of real-world experience, making it one of the first demonstrations of deep RL approaches on real robots.
Supervisor: Turner, Richard E. ; Ghahramani, Zoubin ; Schoelkopf, Bernhard Sponsor: Cambridge-Tuebingen PhD Fellowship in Machine Learning ; Google Focused Research Award ; NSERC
Qualification Name: Thesis (Ph.D.) Qualification Level: Doctoral
Keywords: Reinforcement Learning ; Continuous Control ; Robotics ; Deep Learning ; Machine Learning ; Model-based Planning ; Model-free Reinforcement Learning