| Speaker: Pasquale Minervini, Researcher at the University of Edinburgh and University College London | |
| Where: Zoom (Zoom login required) | |
| When: December 16th, 12:00 - 13:00 |
Combining discrete probability distributions and combinatorial optimization problems with neural network components has numerous applications but poses several challenges.
We propose Implicit Maximum Likelihood Estimation (IMLE),1 a framework for end-to-end learning of models combining discrete exponential family distributions and differentiable neural components. IMLE is widely applicable as it only requires the ability to compute the most probable states and does not rely on smooth relaxations. The framework encompasses several approaches such as perturbation-based implicit differentiation and recent methods to differentiate through black-box combinatorial solvers.
Moreover, we show that IMLE simplifies to maximum likelihood estimation when used in some recently studied learning settings that involve combinatorial solvers. One limitation of IMLE is that, due to the finite difference approximation of the gradients, it can be especially sensitive to the choice of the finite difference step size which needs to be specified by the user. In this presentation, we also introduce Adaptive IMLE (AIMLE), the first adaptive gradient estimator for complex discrete distributions: it adaptively identifies the target distribution for IMLE by trading off the density of gradient information with the degree of bias in the gradient estimates. We empirically evaluate our estimator on synthetic examples, as well as on Learning to Explain, Discrete Variational Auto-Encoders, and Neural Relational Inference tasks. In our experiments, we show that our adaptive gradient estimator can produce faithful estimates while requiring orders of magnitude fewer samples than other gradient estimators.
Speaker
Pasquale is a Lecturer in Natural Language Processing at the School of Informatics, University of Edinburgh, and an Honorary Lecturer at University College London (UCL). Previously, he was a Senior Research Fellow at UCL (2017-2022); a postdoc at the INSIGHT Centre for Data Analytics, Ireland (2016); and a postdoc at the University of Bari, Italy (2015). Pasquale’s research interests are in NLP and ML, with a focus on relational learning and learning from graph-structured data, solving knowledge-intensive tasks, hybrid neuro-symbolic models, compositional generalisation, and designing data-efficient and robust deep learning models.
Pasquale published over 60 peer-reviewed papers in top-tier AI conferences, receiving multiple awards (including one Outstanding Paper Award at ICLR 2021), and delivered several tutorials on Explainable AI and relational learning (including four AAAI tutorials). He is the main inventor of a patent assigned to Fujitsu Ltd. In 2019 he was awarded a seven-figure EU Horizon 2020 research grant on applications of relational learning to cancer research and, in 2020, his team won two tracks out of three of the Efficient Open-Domain Question Answering Challenge at NeurIPS 2020. He routinely collaborates with researchers across both academia and industry. His website is www.neuralnoise.com.
References
-
Niepert, M., Minervini, P. and Franceschi, L., 2021. Implicit MLE: backpropagating through discrete exponential family distributions. Advances in Neural Information Processing Systems, 34, pp.14567-14579. ↩