Employing Categorical Probability Towards Safe AI

The last 5–10 years have seen the flourishing of categorical probability: category-theoretic techniques applied to probability theory. The power of category theory in this approach stems from its ability to organize and analyze the structure of statistical models and their composition. Categorical probability can be seen as a theoretical counterpart to the eminently successful method of probabilistic programming, which aims to structure and compose statistical models using methods from software engineering: this has already achieved practical success, through languages such as Stan and Pyro, across physical, biological and social sciences.

In this first year of the project, we focus on bringing categorical probability to bear on three aspects that have been identified as crucial for world modelling in safe AI in the ARIA programme: • imprecise probability, giving bounds on probabilities of unsafe behaviour; • stochastic dynamical systems for world-modelling with random variables; • both with a solid underpinning of semantic version control.