Optimization
This page contains resources about Mathematical Optimization, Computational Optimization and Operations Research.
More specific information is included in each subfield.
Subfields and ConceptsEdit
See Category:Optimization for some of its subfields.
- Cost Function / Loss Function / Objective Function
- Convex Optimization
- Linear Programming
- Big M method
- Simplex Algorithm
- Linear Programming
- Quadratic Programming
- Nonlinear Programming
- KarushâKuhnâTucker (KKT) conditions
- Combinatorial / Discrete Optimization
- Integer Programming
- Dynamic Programming / Optimal Control
- Deterministic Optimal Control
- Stochastic Optimal Control
- Lyapunov Optimization
- Greedy Algorithm
- Travelling Salesman Problem (TSP)
- Approximation Algorithms
- Online Convex Optimization
- Mini-Batch Learning
- Variational Analysisâ
- Calculus of variations
- Robust Optimization
- Lagrange Multipliers
- Online Optimization
- Iterative Methods
- Powell's Method / Powell's Conjugate Direction Method
- NelderâMead Method / Downhill Simplex Method / Amoeba Method
- Constrained Optimization by Linear Approximation (COBYLA)
- Expectation-Maximization (EM) Algorithm
- LevenbergâMarquardt Algorithm
- Iteratively Reweighted Least Squares
- Nonlinear Least Squares
- Ordinary Least Squares / Linear Least Squares
- Weighted Least Squares
- Sequential Quadratic Programming (SQP) / Sequential Least Squares Programming (SLSQP)
- GaussâNewton Algorithm
- Subgradient Methods (used for Convex Minimization)
- Gradient Descent / Steepest Descent
- Krylov Subspace Methods
- Conjugate Gradient Method
- Biconjugate Gradient stabilized (BiCGSTAB) Method
- Arnoldi Method
- Lanczos Method
- Generalized Minimal Residual (GMRES) Method
- BroydenâFletcherâGoldfarbâShanno (BFGS) Algorithm
- Limited-memory BFGS (L-BFGS)
- Truncated Newton Methods / Hessian-free Optimization
- Resilient Backpropagation (Rprop)
- Bayesian Optimization
- Stochastic Optimization
- Gradient Descent Methods (either full-batch or mini-batch or both)
- Stochastic Gradient Descent (SGD)
- Stochastic Gradient Descent with Cyclical Learning Rates (using Triangular Policy)
- Stochastic Gradient Descent with Restarts (SGDR) / Cyclic Cosine Annealing
- Stochastic Weight Averaging (SWA)
- SGD with Momentum
- Averaged SGD
- AdaDelta
- AdaGrad
- Adam
- RMSprop
- Nesterovâs Accelerated Gradient (NAG) Descent
- NAdam (NAG/Nesterov Adam)
- Projected Gradient Descent
- Particle Mirror Descent (PMD)
- Regularized Dual Averaging (RDA)
- Follow the regularised leader (FTRL)
- Online Gradient Descent
- Adaptive Online Gradient Descent
- Natural Gradient Descent
- Stochastic Gradient Fisher Scoring
- Stochastic Gradient Langevin Dynamics (SGLD)
- Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
- Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC)
- Stochastic Gradient Markov Chain Monte Carlo (SGMCMC)
- Stochastic Gradient Nose-Hoover Thermostat (SGNHT)
- Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo
- Stochastic Approximation
- Robbins-Monro Algorithm (using noisy estimates of the gradient)
- Metaheuristics
- Population-based search
- Evolutionary Algorithms
- Genetic Algorithms
- Swarm Intelligence
- Single point search / Trajectory methods / Single-state methods
- Hill-Climbing
- Simulated Annealing
- Tabu search
- Explorative search methods
- Greedy Randomized Adaptive Search Procedure (GRASP)
- Variable Neighborhood Search (VNS)
- Guided Local Search (GLS)
- Iterated Local Search (ILS)
- Population-based search
- Gradient Descent Methods (either full-batch or mini-batch or both)
- Inverse Problems
- Regularization
- L1-regularization / Least absolute shrinkage and selection operator (LASSO)
- L2-regularization / Ridge Regression / Tikhonov Regularization
- Early Stopping
- Total Variation (TV) Regularization
- Dropout
- Stochastic Simulation / Monte Carlo Methods
- Regularization
- Multi-Objective Optimization / Multicriteria Optimization / Pareto Optimization
- Multi-Objective Linear Programming
Online CoursesEdit
Video LecturesEdit
- Optimization Course by Michael Zibulevsky
- Convex Optimization I by Stephen P. Boyd
- Convex Optimization II by Stephen P. Boyd
- Discrete Optimization by Professor Pascal Van Hentenryck - Coursera
- Linear and Discrete Optimization by Friedrich Eisenbrand - Coursera
- Advanced Optimization and Randomized Methods by A. Smola and S. Sra
- Optimization, Learning and Systems by Martin Jaggi
- Approximate Dynamic Programming Lectures by Dimitri P. Bertsekas
- Optimization
Lecture NotesEdit
IntroductoryEdit
- Practical Optimization: A Gentle Introduction by John W. Chinneck - a very introductory course
- Lecture Notes on Optimization by Pravin Varaiya
- Lecture Notes on Engineering Optimization by Fraser J. Forbes and Ilyasse Aksikas
- Optimization by Geoff Gordon and Ryan Tibshirani
- Optimization: An introduction by A. AstolïŹ
- Optimization for Differential Equations by Martin Berggren and Krister Svanberg
- Quantitative Methods by Michael Trick
- Operations Research. Linear Programming by Ana Zelaia Jauregi
SpecializedEdit
- Convex Analysis and Optimization by Dimitri Bertsekas
- Combinatorial Optimization by Santosh Vempala
- Discrete Optimization for Vision and Learning by Nikos Komodakis and Pawan Kumar
- Lectures on Modern Convex Optimization by Aharon Ben-Tal and Arkadi Nemirovski
- Numerical Optimization by Christopher Griffin
- Lecture Notes on Continuous Optimization by Kok Lay Teo and Song Wang
- Introduction to Online Optimization by Sébastien Bubeck
- Lecture Notes on Online Learning by Alexander Rakhlin
- EECS260 Optimization by Miguel Ă. Carreira-Perpiñån
- Optimization Techniques in Engineering by Alan R. Parkinson and John D. Hedengren
- Multidisciplinary System Design Optimization by Olivier de Weck and Karen Willcox
- ORF523: Advanced Optimization by Sébastien Bubeck
- Dynamic Programming and Optimal Control by Raffaello D'Andrea
- An Introduction to Inverse Problems by Colin Fox, Geoff K. Nicholls and Sze M. Tan
- Inverse Problem in Imaging by Simon Arridge
BooksEdit
IntroductoryEdit
- Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons.
- Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer.
SpecializedEdit
- Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control. 4th Ed. Athena Scientific.
- Bertsekas, D. P. (2016). Nonlinear Programming. 3rd Ed. Athena scientific.
- Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and TrendsŸ in Optimization, 2(3-4), 157-325.
- Bertsekas, D. P. (2015). Convex Optimization Algorithms. Athena Scientific.
- Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
- Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons.
- Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and TrendsŸ in Machine Learning, 4(2), 107-194.
- Sra, S., Nowozin, S., & Wright, S. J. (2012). Optimization for machine learning. MIT Press.
- Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu.
- Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press.
- Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer.
- RuszczyĆski, A. (2006). Nonlinear Optimization. Princeton University Press.
- Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press.
- Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Springer.
- Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam.
- Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam.
- Kelley, C. T. (1999). Iterative Methods for Optimization. Siam.
- Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam.
- Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier.
- Scales, L. E. (1985). Introduction to non-linear optimization. Springer-Verlag.
SoftwareEdit
See List of Optimization Software for the complete list.
- Optimization Toolbox - MATLAB
- Optimization (scipy.optimize) - Python
- climin - Python
- PyOpt - Python
- dlib - C++ (with Python API)
- CVXPY - Python
- CVX - MATLAB
- libLBFGS - C
- PyLBFGS - Python
- NLopt - C, C++, Fortran, Matlab or GNU Octave, Python, GNU Guile, Julia, GNU R, Lua, OCaml
- hessianfree - Hessian-free Optimization for Deep Networks
- MOSEK Fusion API - Python framework for conic optimization
- MOSEK - C, Java, MATLAB, .NET, Python, R
- Gurobi - various languages
See alsoEdit
Other ResourcesEdit
- Mathematical Optimization - Google Scholar Metrics (Top Publications)
- NEOS Optimization Guide
- COIN-OR - Computational Infrastructure for Operations Research
- Decision Tree for Optimization Software Links to optimization source codes
- Mathematical Programming Glossary
- Optimization Related Links
- Optimization - Notebook
- Stochastic Approximation - Notebook
- Convex Optimization - Metacademy
- OPTML - NIPS Workshop on Optimization for Machine Learning
- Metaheuristics and Local Search
- A good book for Inverse Problems - Math StackExchange
- A Global Optimization Algorithm Worth Using (for hyperparameters) - blog post in dlib
- opt (GitHub) - code
- Linear Regression â Theory - blog post
- Gradient Descent - blog post
- Implement Gradient Descent in Python (Medium) - blog post
- Visualizing the gradient descent method - blog post