Open main menu
Ioannis Kourouklides

Optimization

Optimization.jpg

This page contains resources about Mathematical Optimization, Computational Optimization and Operations Research.

More specific information is included in each subfield.

Contents

Subfields and ConceptsEdit

See Category:Optimization for some of  its subfields.

  • Cost Function / Loss Function / Objective Function
  • Convex Optimization
    • Linear Programming
  • Quadratic Programming
  • Nonlinear Programming
    • Karush–Kuhn–Tucker (KKT) conditions
  • Combinatorial / Discrete Optimization
    • Integer Programming
    • Dynamic Programming / Optimal Control
      • Deterministic Optimal Control
      • Stochastic Optimal Control
      • Lyapunov Optimization
    • Greedy Algorithm
    • Travelling Salesman Problem (TSP)
    • Approximation Algorithms
  • Online Convex Optimization
    • Mini-Batch Learning
  • Variational Analysis‎
    • Calculus of variations
  • Robust Optimization
  • Lagrange Multipliers
  • Online Optimization
  • Iterative Methods
    • Powell's Method / Powell's Conjugate Direction Method
    • Nelder–Mead Method / Downhill Simplex Method / Amoeba Method
    • Constrained Optimization by Linear Approximation (COBYLA)
    • Expectation-Maximization (EM) Algorithm
    • Levenberg–Marquardt Algorithm
    • Iteratively Reweighted Least Squares
    • Nonlinear Least Squares
    • Ordinary Least Squares / Linear Least Squares
    • Weighted Least Squares
    • Sequential Quadratic Programming (SQP) / Sequential Least Squares Programming (SLSQP)
    • Gauss–Newton Algorithm
    • Subgradient Methods (used for Convex Minimization)
    • Gradient Descent / Steepest Descent
    • Krylov Subspace Methods
      • Conjugate Gradient Method
      • Biconjugate Gradient stabilized (BiCGSTAB) Method
      • Arnoldi Method
      • Lanczos Method
      • Generalized Minimal Residual (GMRES) Method
    • Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm
    • Limited-memory BFGS (L-BFGS)
    • Truncated Newton Methods / Hessian-free Optimization
    • Resilient Backpropagation (Rprop)
  • Bayesian Optimization
  • Stochastic Optimization
    • Gradient Descent Methods (either full-batch or mini-batch or both)
      • Stochastic Gradient Descent (SGD)
      • Stochastic Gradient Descent with Cyclical Learning Rates (using Triangular Policy)
      • Stochastic Gradient Descent with Restarts (SGDR) / Cyclic Cosine Annealing
      • Stochastic Weight Averaging (SWA)
      • SGD with Momentum
      • Averaged SGD
      • AdaDelta
      • AdaGrad
      • Adam
      • RMSprop
      • Nesterov’s Accelerated Gradient (NAG) Descent
      • NAdam (NAG/Nesterov Adam)
      • Projected Gradient Descent
      • Particle Mirror Descent (PMD)
      • Regularized Dual Averaging (RDA)
      • Follow the regularised leader (FTRL)
      • Online Gradient Descent
      • Adaptive Online Gradient Descent
      • Natural Gradient Descent
    • Stochastic Gradient Fisher Scoring
    • Stochastic Gradient Langevin Dynamics (SGLD)
    • Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
    • Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC)
    • Stochastic Gradient Markov Chain Monte Carlo (SGMCMC)
    • Stochastic Gradient Nose-Hoover Thermostat (SGNHT)
    • Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo
    • Stochastic Approximation
      • Robbins-Monro Algorithm (using noisy estimates of the gradient)
    • Metaheuristics
      • Population-based search
        • Evolutionary Algorithms
        • Genetic Algorithms
        • Swarm Intelligence
      • Single point search / Trajectory methods / Single-state methods
        • Hill-Climbing
        • Simulated Annealing
        • Tabu search
        • Explorative search methods
          • Greedy Randomized Adaptive Search Procedure (GRASP)
          • Variable Neighborhood Search (VNS)
          • Guided Local Search (GLS)
          • Iterated Local Search (ILS)
  • Inverse Problems
  • Multi-Objective Optimization / Multicriteria Optimization / Pareto Optimization
    • Multi-Objective Linear Programming

Online CoursesEdit

Video LecturesEdit


Lecture NotesEdit

IntroductoryEdit

SpecializedEdit

BooksEdit

IntroductoryEdit

  • Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons.
  • Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer.

SpecializedEdit

  • Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control. 4th Ed. Athena Scientific.
  • Bertsekas, D. P. (2016). Nonlinear Programming. 3rd Ed. Athena scientific.
  • Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2(3-4), 157-325.
  • Bertsekas, D. P. (2015). Convex Optimization Algorithms. Athena Scientific.
  • Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
  • Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons.
  • Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and Trends® in Machine Learning, 4(2), 107-194.
  • Sra, S., Nowozin, S., & Wright, S. J. (2012). Optimization for machine learning. MIT Press.
  • Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu.
  • Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press.
  • Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer.
  • Ruszczyński, A. (2006). Nonlinear Optimization. Princeton University Press.
  • Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press.
  • Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Springer.
  • Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam.
  • Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam.
  • Kelley, C. T. (1999). Iterative Methods for Optimization. Siam.
  • Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam.
  • Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier.
  • Scales, L. E. (1985). Introduction to non-linear optimization. Springer-Verlag.

SoftwareEdit

See List of Optimization Software  for the complete list.

See alsoEdit

Other ResourcesEdit

Retrieved from "https://wiki.kourouklides.com/w/index.php?title=Optimization&oldid=1806"