Neural density estimation and likelihood-free inference
View/ Open
Date
23/11/2019Author
Papamakarios, Georgios
Metadata
Abstract
I consider two problems in machine learning and statistics: the problem of estimating the joint
probability density of a collection of random variables, known as density estimation, and the
problem of inferring model parameters when their likelihood is intractable, known as likelihood-free
inference. The contribution of the thesis is a set of new methods for addressing these problems
that are based on recent advances in neural networks and deep learning.
The first part of the thesis is about density estimation. The joint probability density of a collection
of random variables is a useful mathematical description of their statistical properties, but can be
hard to estimate from data, especially when the number of random variables is large. Traditional
density-estimation methods such as histograms or kernel density estimators are effective for
a small number of random variables, but scale badly as the number increases. In contrast,
models for density estimation based on neural networks scale better with the number of random
variables, and can incorporate domain knowledge in their design. My main contribution is Masked
Autoregressive Flow, a new model for density estimation based on a bijective neural network
that transforms random noise to data. At the time of its introduction, Masked Autoregressive
Flow achieved state-of-the-art results in general-purpose density estimation. Since its publication,
Masked Autoregressive Flow has contributed to the broader understanding of neural density
estimation, and has influenced subsequent developments in the field.
The second part of the thesis is about likelihood-free inference. Typically, a statistical model can
be specified either as a likelihood function that describes the statistical relationship between model
parameters and data, or as a simulator that can be run forward to generate data. Specifying
a statistical model as a simulator can offer greater modelling flexibility and can produce more
interpretable models, but can also make inference of model parameters harder, as the likelihood
of the parameters may no longer be tractable. Traditional techniques for likelihood-free inference
such as approximate Bayesian computation rely on simulating data from the model, but often
require a large number of simulations to produce accurate results. In this thesis, I cast the problem
of likelihood-free inference as a density-estimation problem, and address it with neural density
models. My main contribution is the introduction of two new methods for likelihood-free inference:
Sequential Neural Posterior Estimation (Type A), which estimates the posterior, and Sequential
Neural Likelihood, which estimates the likelihood. Both methods use a neural density model to
estimate the posterior/likelihood, and a sequential training procedure to guide simulations. My
experiments show that the proposed methods produce accurate results, and are often orders of
magnitude faster than alternative methods based on approximate Bayesian computation.