This wiki has had no edits or log actions made within the last 45 days and has been automatically marked as inactive. If you would like to prevent this wiki from being closed, please start showing signs of activity here. If there are no signs of this wiki being used within the next 15 days, this wiki will be closed in accordance to the Dormancy Policy (which all wiki founders accept when requesting a wiki). If this wiki is closed and no one reopens it 135 days from now, this wiki will become eligible for deletion. Note: If you are a bureaucrat, you can go to Special:ManageWiki and uncheck "inactive" yourself.

Variational Method

From Ioannis Kourouklides

This page contains resources about Variational Methods, Variational Bayesian Inference, Variational Bayesian Learning and Deterministic Approximate Inference.

Subfields and Concepts[edit]

  • Variational Calculus / Calculus of Variations
  • Variational Analysis‎
  • Variational free energy
  • Free energy principle
  • Conjugate Duality
  • Exponential family
  • Conjugate prior family
  • Variance reduction techniques (VRT) in Monte Carlo Gradients
    • Control variates
    • Rao–Blackwellization
    • By linear regression
    • Reparameterization trick / Reparameterization Gradient / Coordinate Tranformation / Invertible Tranformation / Elliptical Standarization
    • Local Expectation Gradient
    • Importance Sampling
    • Generalized Reparameterization (G-REP) Gradient
  • Gradient Estimators
    • Score Function (SF) Estimator
    • Pathwise Derivative (PD) Estimator
    • Reparameterization Gradient
    • Generalized Reparameterization (G-REP) Gradient
  • Evidence Lower Bound (ELBO) / Variational Lower Bound
  • Structured Variational Inference
  • Kullback–Leibler (KL) Divergence
  • Variational Bayes
  • Variational Bayesian EM (VBEM)
  • Stochastic Variational Inference
  • Stochastic Gradient-based Variational Inference
  • Stochastic Gradient Variational Bayes (SGVB) Estimator
  • Deep Variational Bayes Filter (DVBF)
  • Wake-Sleep Algorithm
  • Auto-Encoding Variational Bayes (AEVB) Algorithm
  • Variational Autoencoder (VAE)
  • Hierarchical Variational Models
  • Expectation Propagation
    • Loopy Belief Propagation / Loopy Sum-Product Message Passing
    • Assumed Density Filtering (ADF) / Moment Matching
  • Kullback-Leibler (KL) Variational Inference / Mean field Variational Bayes
    • Structured Mean field / Structured Variational Approximation
    • Weighted Mean Field
  • Tree-based reparameterizations
  • Tree-reweighted belief propagation
  • Bethe and Kikuchi free energy
  • Generalized Belief Propagation
  • Forwared KL divergence / Moment Projection (M-Projection)
  • Reverse KL divergence / Information Projection (I-Projection)
  • Online Bayesian Variational (OBV) Inference Algorithms
  • Neural Variational Inference and Learning (NVIL)
  • Non-conjugate Variational Inference
  • Rejection Sampling Variational Inference (RSVI)
  • Reinforced Variational Inference
  • Generic and Automated Variation Inference
    • Black-Box Variational Inference (BBVI)
    • Automatic Variational Inference (AVI)
    • Automatic Differentiation Variational Inference (ADVI)
    • Generalized Reparameterization (G-REP) Gradient
    • SGVB with local expectation gradients (LeGrad)
    • SGVB with reparametrization-based gradient (ReGrad) / Reparameterization trick
    • SGVB with the log derivative trick (LdGrad) / Score Function Method 
  • Overdispersed BBVI (O-BBVI)
  • Stochastic Optimization
    • Gradient Ascend on ELBO
  • Stochastic Approximation
    • Robbins-Monro Algorithm (using noisy estimates of the gradient)
  • Energy-Based Model (EBM)
    • Free energy (i.e. the contrastive term)
    • Regularization term
    • Loss functionals or Loss functions or Energy functionals
      • Energy Loss
      • Generalized Perceptron Loss
      • Generalized Margin Losses
      • Negative Log-Likelihood Loss

Online Courses[edit]

Video Lectures[edit]

Lecture Notes[edit]

Books and Book Chapters[edit]

  • Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). "Chapter 19: Approximate Inference". Deep Learning. MIT Press.
  • Theodoridis, S. (2015). "Chapter 13: Bayesian Learning: Approximate Inference and Nonparametric Models". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
  • Murphy, K. P. (2012). "Chapter 21: Variational inference". Machine Learning: A Probabilistic Perspective. MIT Press.
  • Barber, D. (2012). "Section 7.7: Variational Inference and Planning". Bayesian Reasoning and Machine Learning. Cambridge University Press.
  • Barber, D. (2012). "Chapter 11: Learning with Hidden Variables". Bayesian Reasoning and Machine Learning. Cambridge University Press.
  • Barber, D. (2012). "Chapter 28: Deterministic Approximate Inference". Bayesian Reasoning and Machine Learning. Cambridge University Press.
  • Koller, D., & Friedman, N. (2009). "Chapter 11: Inference as Optimization". Probabilistic Graphical Models. MIT Press.
  • Bishop, C. M. (2006). "Chapter 10: Approximate Inference". Pattern Recognition and Machine Learning. Springer.
  • Smidl, V., & Quinn, A. (2006). The Variational Bayes Method in Signal Processing. Springer Science & Business Media.
  • MacKay, D. J. (2003). "Chapter 33: Variational Methods" Information Theory, Inference and Learning Algorithms. Cambridge University Press.
  • Opper, M., & Saad, D. (2001). Advanced mean field methods: Theory and practice. MIT press.

Scholarly Articles[edit]

  • Laumann, F., & Shridhar, K. (2018). Bayesian Convolutional Neural Networks. arXiv preprint arXiv:1806.05978.
  • Louizos, C., & Welling, M. (2017). Multiplicative Normalizing Flows for Variational Bayesian Neural Networks. In International Conference on Machine Learning (pp. 2218-2227).
  • Kingma, D. P. (2017). Variational Inference & Deep Learning: A New Synthesis. PhD Diss. University of Amsterdam.
  • Fortunato, M., Blundell, C., & Vinyals, O. (2017). Bayesian recurrent neural networks. arXiv preprint arXiv:1704.02798.
  • Ruiz, F. J., Titsias, M. K., & Blei, D. M. (2016). The Generalized Reparameterization Gradient. arXiv preprint arXiv:1610.02287.
  • Ruiz, F. J., Titsias, M. K., & Blei, D. M. (2016). Overdispersed Black-Box Variational Inference. arXiv preprint arXiv:1603.01140.
  • Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2016). Variational inference: A review for statisticians. arXiv preprint arXiv:1601.00670.
  • Mandt, S., Hoffman, M. D., & Blei, D. M. (2016). A Variational Analysis of Stochastic Gradient Algorithms. arXiv preprint arXiv:1602.02666.
  • Naesseth, C. A., Ruiz, F. J., Linderman, S. W., & Blei, D. M. (2016). Rejection Sampling Variational Inference. arXiv preprint arXiv:1610.05683.
  • Kucukelbir, A., Tran, D., Ranganath, R., Gelman, A., & Blei, D. M. (2016). Automatic Differentiation Variational Inference. arXiv preprint arXiv:1603.00788.
  • Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (pp. 1050-1059).
  • Kucukelbir, A., Ranganath, R., Gelman, A., & Blei, D. (2015). Automatic variational inference in Stan. In Advances in Neural Information Processing Systems (pp. 568-576).
  • Blundell, C., Cornebise, J., Kavukcuoglu, K., & Wierstra, D. (2015). Weight Uncertainty in Neural Network. In International Conference on Machine Learning (pp. 1613-1622).
  • Schulman, J., Heess, N., Weber, T., & Abbeel, P. (2015). Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems (pp. 3528-3536).
  • Titsias, M., & Lazaro-Gredilla, M. (2015). Local expectation gradients for black box variational inference. In Advances in Neural Information Processing Systems (pp. 2638-2646).
  • Kingma, D. P., Salimans, T., & Welling, M. (2015). Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems (pp. 2575-2583).
  • Archer, E., Park, I. M., Buesing, L., Cunningham, J., & Paninski, L. (2015). Black box variational inference for state space models. arXiv preprint arXiv:1511.07367.
  • Hoffman, M. D., & Blei, D. M. (2015). Structured stochastic variational inference. In Artificial Intelligence and Statistics.
  • Kucukelbir, A., Ranganath, R., Gelman, A., & Blei, D. (2014). Fully automatic variational inference of differentiable probability models. In NIPS Workshop on Probabilistic Programming.
  • Salimans, T., & Knowles, D. A. (2014). On using control variates with stochastic approximation for variational Bayes and its connection to stochastic linear regression. arXiv preprint arXiv:1401.1022.
  • Ranganath, R., Gerrish, S., & Blei, D. M. (2014). Black Box Variational Inference. In AISTATS (pp. 814-822).
  • Lazaro-Gredilla, M. (2014). Doubly stochastic variational Bayes for non-conjugate inference. In Proceedings of the 31st International Conference on Machine Learning (pp. 1971-1979).
  • Mnih, A., & Gregor, K. (2014). Neural variational inference and learning in belief networks. arXiv preprint arXiv:1402.0030.
  • Salimans, T., & Knowles, D. A. (2013). Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis8(4), 837-882.
  • Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013). Stochastic variational inference.Journal of Machine Learning Research14(1), 1303-1347.
  • Wingate, D., & Weber, T. (2013). Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299.
  • Wang, C., & Blei, D. M. (2013). Variational inference in nonconjugate models. Journal of Machine Learning Research14(Apr), 1005-1031.
  • Fox, C. W., & Roberts, S. J. (2012). A tutorial on variational Bayesian inference. Artificial intelligence review38(2), 85-95.
  • Paisley, J., Blei, D., & Jordan, M. (2012). Variational Bayesian inference with stochastic search. arXiv preprint arXiv:1206.6430.
  • Knowles, D. A., & Minka, T. (2011). Non-conjugate variational message passing for multinomial and binary regression. In Advances in Neural Information Processing Systems (pp. 1701-1709).
  • Wainwright, M. J., & Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning1(1-2), 1-305.
  • Tzikas, D. G., Likas, A. C., & Galatsanos, N. P. (2008). The variational approximation for Bayesian inference. IEEE Signal Processing Magazine,25(6), 131-146.
  • Wainwright, M., & Jordan, M. (2005). A variational principle for graphical models. New Directions in Statistical Signal Processing155.
  • Yedidia, J. S., Freeman, W. T., & Weiss, Y. (2005). Constructing free-energy approximations and generalized belief propagation algorithms. IEEE Transactions on Information Theory51(7), 2282-2312.
  • Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference. Ph.D. Dissertation, University College London.
  • Xing, E. P., Jordan, M. I., & Russell, S. (2003). A generalized mean field algorithm for variational inference in exponential families. In Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence (pp. 583-591). Morgan Kaufmann Publishers Inc.
  • Wainwright, M. J., & Jordan, M. I. (2003). Variational inference in graphical models: The view from the marginal polytope. In Proceeding of Annual Allerton Conference of Communication Control and Computing (Vol. 41, No. 2, pp. 961-971).
  • Lawrence, N. D. (2001). Variational inference in probabilistic models. Ph.D. Dissertation, University of Cambridge.
  • Minka, T. P. (2001). A family of algorithms for approximate Bayesian inference. Ph.D. Dissertation, Massachusetts Institute of Technology.
  • Ghahramani, Z., & Beal, M. J. (2001). Propagation algorithms for variational Bayesian learning. In Advances in Neural Information Processing Systems, 507-513.
  • Attias, H. (2000). A variational Bayesian framework for graphical models. In Advances in Neural Information Processing Systems, 209-215.
  • Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine learning,37(2), 183-233.



See also[edit]

Other Resources[edit]