Variational Method

This page contains resources about Variational Methods, Variational Bayesian Inference, Variational Bayesian Learning and Deterministic Approximate Inference.

Subfields and Concepts

 * Variational Calculus / Calculus of Variations
 * Variational Analysis‎
 * Variational free energy
 * Free energy principle
 * Conjugate Duality
 * Exponential family
 * Conjugate prior family
 * Variance reduction techniques (VRT) in Monte Carlo Gradients
 * Control variates
 * Rao–Blackwellization
 * By linear regression
 * Reparameterization trick / Reparameterization Gradient / Coordinate Tranformation / Invertible Tranformation / Elliptical Standarization
 * Local Expectation Gradient
 * Importance Sampling
 * Generalized Reparameterization (G-REP) Gradient
 * Gradient Estimators
 * Score Function (SF) Estimator
 * Pathwise Derivative (PD) Estimator
 * Reparameterization Gradient
 * Generalized Reparameterization (G-REP) Gradient
 * Evidence Lower Bound (ELBO) / Variational Lower Bound
 * Structured Variational Inference
 * Kullback–Leibler (KL) Divergence
 * Variational Bayes
 * Variational Bayesian EM (VBEM)
 * Stochastic Variational Inference
 * Stochastic Gradient-based Variational Inference
 * Stochastic Gradient Variational Bayes (SGVB) Estimator
 * Deep Variational Bayes Filter (DVBF)
 * Wake-Sleep Algorithm
 * Auto-Encoding Variational Bayes (AEVB) Algorithm
 * Variational Autoencoder (VAE)
 * Hierarchical Variational Models
 * Expectation Propagation
 * Loopy Belief Propagation / Loopy Sum-Product Message Passing
 * Assumed Density Filtering (ADF) / Moment Matching
 * Kullback-Leibler (KL) Variational Inference / Mean field Variational Bayes
 * Structured Mean field / Structured Variational Approximation
 * Weighted Mean Field
 * Tree-based reparameterizations
 * Tree-reweighted belief propagation
 * Bethe and Kikuchi free energy
 * Generalized Belief Propagation
 * Forwared KL divergence / Moment Projection (M-Projection)
 * Reverse KL divergence / Information Projection (I-Projection)
 * Online Bayesian Variational (OBV) Inference Algorithms
 * Neural Variational Inference and Learning (NVIL)
 * Non-conjugate Variational Inference
 * Rejection Sampling Variational Inference (RSVI)
 * Reinforced Variational Inference
 * Generic and Automated Variation Inference
 * Black-Box Variational Inference (BBVI)
 * Automatic Variational Inference (AVI)
 * Automatic Differentiation Variational Inference (ADVI)
 * Generalized Reparameterization (G-REP) Gradient
 * SGVB with local expectation gradients (LeGrad)
 * SGVB with reparametrization-based gradient (ReGrad) / Reparameterization trick
 * SGVB with the log derivative trick (LdGrad) / Score Function Method
 * Overdispersed BBVI (O-BBVI)
 * Stochastic Optimization
 * Gradient Ascend on ELBO
 * Stochastic Approximation
 * Robbins-Monro Algorithm (using noisy estimates of the gradient)
 * Energy-Based Model (EBM)
 * Free energy (i.e. the contrastive term)
 * Regularization term
 * Loss functionals or Loss functions or Energy functionals
 * Energy Loss
 * Generalized Perceptron Loss
 * Generalized Margin Losses
 * Negative Log-Likelihood Loss

Video Lectures

 * Graphical Models and Variational Methods by Christopher Bishop - VideoLectures.NET
 * Approximate Inference by Tom Minka - VideoLectures.NET
 * Machine Learning: Variational Inference by Jordan Boyd-Graber
 * Variational Inference by Chieh Wu
 * Autoencoding Variational Bayes by Durk Kinga - ICLR 2014
 * Variational Autoencoders by Karol Gregor

Lecture Notes

 * COS597C: Advanced Methods in Probabilistic Modeling BY David M. Blei
 * Lecture: Variational Inference by Russ Salakhutdinov

Books and Book Chapters

 * Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). "Chapter 19: Approximate Inference". Deep Learning. MIT Press.
 * Theodoridis, S. (2015). "Chapter 13: Bayesian Learning: Approximate Inference and Nonparametric Models". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
 * Murphy, K. P. (2012). "Chapter 21: Variational inference". Machine Learning: A Probabilistic Perspective. MIT Press.
 * Barber, D. (2012). "Section 7.7: Variational Inference and Planning". Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Barber, D. (2012). "Chapter 11: Learning with Hidden Variables". Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Barber, D. (2012). "Chapter 28: Deterministic Approximate Inference". Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Koller, D., & Friedman, N. (2009). "Chapter 11: Inference as Optimization". Probabilistic Graphical Models. MIT Press.
 * Bishop, C. M. (2006). "Chapter 10: Approximate Inference". Pattern Recognition and Machine Learning. Springer.
 * Smidl, V., & Quinn, A. (2006). The Variational Bayes Method in Signal Processing. Springer Science & Business Media.
 * MacKay, D. J. (2003). "Chapter 33: Variational Methods" Information Theory, Inference and Learning Algorithms. Cambridge University Press.
 * Opper, M., & Saad, D. (2001). Advanced mean field methods: Theory and practice. MIT press.

Scholarly Articles

 * Laumann, F., & Shridhar, K. (2018). Bayesian Convolutional Neural Networks. arXiv preprint arXiv:1806.05978.
 * Louizos, C., & Welling, M. (2017). Multiplicative Normalizing Flows for Variational Bayesian Neural Networks. In International Conference on Machine Learning (pp. 2218-2227).
 * Kingma, D. P. (2017). Variational Inference & Deep Learning: A New Synthesis. PhD Diss. University of Amsterdam.
 * Fortunato, M., Blundell, C., & Vinyals, O. (2017). Bayesian recurrent neural networks. arXiv preprint arXiv:1704.02798.
 * Ruiz, F. J., Titsias, M. K., & Blei, D. M. (2016). The Generalized Reparameterization Gradient. arXiv preprint arXiv:1610.02287.
 * Ruiz, F. J., Titsias, M. K., & Blei, D. M. (2016). Overdispersed Black-Box Variational Inference. arXiv preprint arXiv:1603.01140.
 * Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2016). Variational inference: A review for statisticians. arXiv preprint arXiv:1601.00670.
 * Mandt, S., Hoffman, M. D., & Blei, D. M. (2016). A Variational Analysis of Stochastic Gradient Algorithms. arXiv preprint arXiv:1602.02666.
 * Naesseth, C. A., Ruiz, F. J., Linderman, S. W., & Blei, D. M. (2016). Rejection Sampling Variational Inference. arXiv preprint arXiv:1610.05683.
 * Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2016). Automatic Differentiation Variational Inference. arXiv preprint arXiv:1603.00788.
 * Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (pp. 1050-1059).
 * Kucukelbir, A., Ranganath, R., Gelman, A., & Blei, D. (2015). Automatic variational inference in Stan. In Advances in Neural Information Processing Systems (pp. 568-576).
 * Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight Uncertainty in Neural Network. In International Conference on Machine Learning (pp. 1613-1622).
 * Schulman, J., Heess, N., Weber, T., & Abbeel, P. (2015). Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems (pp. 3528-3536).
 * Titsias, M., & Lazaro-Gredilla, M. (2015). Local expectation gradients for black box variational inference. In Advances in Neural Information Processing Systems (pp. 2638-2646).
 * Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems (pp. 2575-2583).
 * Archer, E., Park, I. M., Buesing, L., Cunningham, J., & Paninski, L. (2015). Black box variational inference for state space models. arXiv preprint arXiv:1511.07367.
 * Hoffman, M. D., & Blei, D. M. (2015). Structured stochastic variational inference. In Artificial Intelligence and Statistics.
 * Kucukelbir, A., Ranganath, R., Gelman, A., & Blei, D. (2014). Fully automatic variational inference of differentiable probability models. In NIPS Workshop on Probabilistic Programming.
 * Salimans, T., & Knowles, D. A. (2014). On using control variates with stochastic approximation for variational Bayes and its connection to stochastic linear regression. arXiv preprint arXiv:1401.1022.
 * Ranganath, R., Gerrish, S., & Blei, D. M. (2014). Black Box Variational Inference. In AISTATS (pp. 814-822).
 * Lazaro-Gredilla, M. (2014). Doubly stochastic variational Bayes for non-conjugate inference. In  Proceedings of the 31st International Conference on Machine Learning (pp. 1971-1979).
 * Mnih, A., & Gregor, K. (2014). Neural variational inference and learning in belief networks. arXiv preprint arXiv:1402.0030.
 * Salimans, T., & Knowles, D. A. (2013). Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8(4), 837-882.
 * Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013). Stochastic variational inference.Journal of Machine Learning Research, 14(1), 1303-1347.
 * Wingate, D., & Weber, T. (2013). Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299.
 * Wang, C., & Blei, D. M. (2013). Variational inference in nonconjugate models. Journal of Machine Learning Research, 14(Apr), 1005-1031.
 * Fox, C. W., & Roberts, S. J. (2012). A tutorial on variational Bayesian inference. Artificial intelligence review, 38(2), 85-95.
 * Paisley, J., Blei, D., & Jordan, M. (2012). Variational Bayesian inference with stochastic search. arXiv preprint arXiv:1206.6430.
 * Knowles, D. A., & Minka, T. (2011). Non-conjugate variational message passing for multinomial and binary regression. In Advances in Neural Information Processing Systems (pp. 1701-1709).
 * Wainwright, M. J., & Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1(1-2), 1-305.
 * Tzikas, D. G., Likas, A. C., & Galatsanos, N. P. (2008). The variational approximation for Bayesian inference. IEEE Signal Processing Magazine,25(6), 131-146.
 * Wainwright, M., & Jordan, M. (2005). A variational principle for graphical models. New Directions in Statistical Signal Processing, 155.
 * Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory, 51(7), 2282-2312.
 * Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference. Ph.D. Dissertation, University College London.
 * Xing, E. P., Jordan, M. I., & Russell, S. (2003). A generalized mean field algorithm for variational inference in exponential families. In Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence (pp. 583-591). Morgan Kaufmann Publishers Inc.
 * Wainwright, M. J., & Jordan, M. I. (2003). Variational inference in graphical models: The view from the marginal polytope. In Proceeding of Annual Allerton Conference of Communication Control and Computing (Vol. 41, No. 2, pp. 961-971).
 * Lawrence, N. D. (2001). Variational inference in probabilistic models. Ph.D. Dissertation, University of Cambridge.
 * Minka, T. P. (2001). A family of algorithms for approximate Bayesian inference. Ph.D. Dissertation, Massachusetts Institute of Technology.
 * Ghahramani, Z., & Beal, M. J. (2001). Propagation algorithms for variational Bayesian learning. In Advances in Neural Information Processing Systems, 507-513.
 * Attias, H. (2000). A variational Bayesian framework for graphical models. In Advances in Neural Information Processing Systems, 209-215.
 * Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine learning,37(2), 183-233.

Tutorials

 * Challenges in Variational Inference: Optimization, Automation, and Accuracy by Rajesh Ranganath - NIPS 2015
 * Variational Auto-Encoders and Extensions by Durk Kingma - NIPS 2015
 * Stochastic Backpropagation, Variational Inference, and Semi-Supervised Learning by Durk Kingma - NIPS 2014
 * Auto-Encoding Variational Bayes by Durk Kingma - 2014
 * Auto-Encoding Variational Bayes by Durk Kingma (Video) - ICLR 2014
 * Stochastic Gradient VB. Intractable posterior distributions? Gradients to the rescue! by Durk Kingma - 2014
 * Speeding up Gradient-Based Inference and Learning in deep/recurrent Bayes Nets with Continuous Latent Variables by Durk Kingma - 2014
 * Variational Bayesian inference by Kay H. Brodersen - 2013
 * High-Level Explanation of Variational Inference by Jason Eisner - 2011
 * Graphical models and variational methods by Martin Wainwright - ICML 2008
 * Variational Methods by Zubin Ghahramani - 2003
 * Variational Mean Field for Graphical Models by Baback Moghaddam

Software

 * Vilds - Black box variational inference for state space models in Python
 * Edward: A library for probabilistic modeling, inference, and criticism - Python with TensorFlow
 * Edward2 - Python with TensorFlow
 * InferPy - Python with Edward
 * Pyro - Python with PyTorch
 * VIBES
 * VBA toolbox - MATLAB

Other Resources

 * Variational-Bayes - A repository of research papers, software, and links related to the use of variational methods for approximate Bayesian learning up to 2003
 * The lure of free energy - Blog post
 * High Level Explanation of Variational Inference
 * Approximate Inference - Zoubin Ghahramani
 * BayesByHypernet (GitHub) - code
 * MNF_VBNN (GitHub) - code
 * Variational-DQN (GitHub) - code
 * Variational Inference using Implicit Models (Part I Part II Part III Part IV)
 * boosting-bbvi (GitHub) - code
 * var-attn (GitHub) - code