Bayesian Statistical Probabilistic Programming:  2020-2021

Overview

The lectures for this course will be pre-recorded. 

They will be released at the start of each week, on the Course Materials page.

They will be supported by discussion on Piazza.

Statistical probabilistic programming (SPP) is a general framework for expressing probabilistic models as computer programs. SPP systems are equipped with implementations of genericinference algorithms for answering queries about these models, such as posterior inference and marginalisation. By providing these algorithms, SPP systems enable data scientists to focuson the design of good models, utilising their domain knowledge. The task of constructing efficient generic inference engines can be left to researchers with expertise in statistical machine learning and programming languages.

The full potential of SPP will come from automating the inference of latent (unobserved) variables in the model, conditioned on the observed data. Because of their generality, SPP poses interesting and challenging research problems for programming languages and the statistical foundations of machine learning.

Learning outcomes

1. To construct probabilistic models via an expressive SPP language. We use the SPP language Anglican, but emphasise general principles that are applicable to a wide range of existing SPP languages.
2. To understand standard inference algorithms and their implementations in SPP languages; and to apply them to perform inference in a SPP system.
3. To understand the semantics of SPP programs, with a view to reasoning about their correctness.

Prerequisites

- Prelims Functional Programming, Continuous Maths, Linear Algebra, and Probability
- Part B Machine Learning

Though not strict prerequisites, the following courses are highly desirable:
- Principle of Programming Languages
- Part A Probability (MCS). The following topics covered in the Part A course will be assumed: probability spaces and random variables, conditional probability, probability distributions, expectation and variance, convergence and limit theorems, joint distribution and conditional density, basics of Markov chains.

Synopsis

• Basics of Anglican; examples of Bayesian inference problems.
• Review of (discrete) probability theory.
• Posterior inference; likelihood-weighted importance sampling and importance sampling. Linear regression.
• Generative modelling with Anglican: 2-D physics, and program induction.
• Markov Chain Monte Carlo algorithms.
• Implementing inference algorithms for SPP programs – an operational semantic account.
• Variational inference.
• Sequential Monte Carlo and amortised inference.
• Basic measure theory and programs as measures.
• Denotational semantics of SPP programs.

Reading list

Christopher M Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN9788578110796. doi: 10.1017/CBO9781107415324.004.

Cameron Davidson-Pilon. Bayesian methods for Hackers. Addison-Wesley, 2015.

Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553):452–459, 2015. ISSN 0028-0836. doi: 10.1038/nature14541. URL http://www.nature.com/doifinder/10.1038/nature14541.

Noah D Goodman and Andreas Stuhlmüller. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org, 2014.

Kevin P Murphy. Machine Learning: A Probabilistic Perspective. Adaptive Computation andMachine Learning Series. The MIT Press, Cambridge, Mass, 2012. ISBN 9780262018029.

Tom Rainforth. Automating Inference, Learning, and Design using Probabilistic Programming. PhD thesis, University of Oxford, 2017.

Stuart Russell. Unifying logic and probability. Communications of the ACM, 58(7), 2015. doi: 10.1145/2699411.

Jan-Willem van de Meent, Brooks Paige, Hongseok Yang, and Frank Wood. An introduction to probabilistic programming. CoRR, abs/1809.10756, 2018. URL http://arxiv.org/abs/1809.10756.

Frank Wood et al. Anglican – A Probabilistic Programming System. Reference webpage, 2014–2018. URL https://probprog.github.io/anglican/index.html.

Taking our courses

This form is not to be used by students studying for a degree in the Department of Computer Science, or for Visiting Students who are registered for Computer Science courses

Other matriculated University of Oxford students who are interested in taking this, or other, courses in the Department of Computer Science, must complete this online form by 17.00 on Friday of 0th week of term in which the course is taught. Late requests, and requests sent by email, will not be considered. All requests must be approved by the relevant Computer Science departmental committee and can only be submitted using this form.