Zoom:
Reinforcement learning (RL) involves an agent interacting with an environment over time attempting to maximize its total return. Initially the agent does not know about the environment and must learn about it from experience. As the agent navigates the environment it receives noisy observations which it can use to update its (posterior) beliefs about the environment. Therefore, the RL problem is a statistical inference problem wrapped in a control problem, and the two problems must be tackled simultaneously for good data efficiency. This is because the policy of the agent affects the data it will collect, which in turn affects the policy, and so on. This is in contrast to supervised learning, where the performance of a classifier (for instance) does not influence the data it will later observe. Failure to properly consider the statistical aspect of the RL problem will result in agents that require exponential amounts of experience for good performance. On the other hand, correctly considering the statistical inference problem and the control problem together has the potential to dramatically reduce the compute requirements to solve problems and potentially unlock new domains and capabilities far outside of the range of current agents. In this talk I will introduce these concepts and discuss how Bayesian techniques can provide principled solutions to the problem.
Brendan O'Donoghue earned his PhD in 2013 from Stanford working with Stephen Boyd on optimization and control theory. Since then he has worked at DeepMind as a research scientist working on deep reinforcement learning, optimization, and (more recently) large language models.