Open main menu
Ioannis Kourouklides

Optimization

Subfields and ConceptsEdit

See Category:Optimization for some of  its subfields.

  • Cost Function / Loss Function / Objective Function
  • Convex Optimization
    • Linear Programming
  • Quadratic Programming
  • Nonlinear Programming
    • Karush–Kuhn–Tucker (KKT) conditions
  • Combinatorial / Discrete Optimization
    • Integer Programming
    • Dynamic Programming / Optimal Control
      • Deterministic Optimal Control
      • Stochastic Optimal Control
      • Lyapunov Optimization
    • Greedy Algorithm
    • Travelling Salesman Problem (TSP)
    • Approximation Algorithms
  • Online Convex Optimization
    • Mini-Batch Learning
  • Variational Analysis‎
    • Calculus of variations
  • Robust Optimization
  • Lagrange Multipliers
  • Online Optimization
  • Iterative Methods
    • Powell's Method / Powell's Conjugate Direction Method
    • Nelder–Mead Method / Downhill Simplex Method / Amoeba Method
    • Constrained Optimization by Linear Approximation (COBYLA)
    • Expectation-Maximization (EM) Algorithm
    • Levenberg–Marquardt Algorithm
    • Iteratively Reweighted Least Squares
    • Nonlinear Least Squares
    • Ordinary Least Squares / Linear Least Squares
    • Weighted Least Squares
    • Sequential Quadratic Programming (SQP) / Sequential Least Squares Programming (SLSQP)
    • Gauss–Newton Algorithm
    • Subgradient Methods (used for Convex Minimization)
    • Gradient Descent / Steepest Descent
    • Krylov Subspace Methods
      • Conjugate Gradient Method
      • Biconjugate Gradient stabilized (BiCGSTAB) Method
      • Arnoldi Method
      • Lanczos Method
      • Generalized Minimal Residual (GMRES) Method
    • Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm
    • Limited-memory BFGS (L-BFGS)
    • Truncated Newton Methods / Hessian-free Optimization
    • Resilient Backpropagation (Rprop)
  • Bayesian Optimization
  • Stochastic Optimization
    • Gradient Descent Methods (either full-batch or mini-batch or both)
      • Stochastic Gradient Descent (SGD)
      • Stochastic Gradient Descent with Cyclical Learning Rates (using Triangular Policy)
      • Stochastic Gradient Descent with Restarts (SGDR) / Cyclic Cosine Annealing
      • Stochastic Weight Averaging (SWA)
      • SGD with Momentum
      • Averaged SGD
      • AdaDelta
      • AdaGrad
      • Adam
      • RMSprop
      • Nesterov’s Accelerated Gradient (NAG) Descent
      • NAdam (NAG/Nesterov Adam)
      • Projected Gradient Descent
      • Particle Mirror Descent (PMD)
      • Regularized Dual Averaging (RDA)
      • Follow the regularised leader (FTRL)
      • Online Gradient Descent
      • Adaptive Online Gradient Descent
      • Natural Gradient Descent
    • Stochastic Gradient Fisher Scoring
    • Stochastic Gradient Langevin Dynamics (SGLD)
    • Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
    • Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC)
    • Stochastic Gradient Markov Chain Monte Carlo (SGMCMC)
    • Stochastic Gradient Nose-Hoover Thermostat (SGNHT)
    • Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo
    • Stochastic Approximation
      • Robbins-Monro Algorithm (using noisy estimates of the gradient)
    • Metaheuristics
      • Population-based search
        • Evolutionary Algorithms
        • Genetic Algorithms
        • Swarm Intelligence
      • Single point search / Trajectory methods / Single-state methods
        • Hill-Climbing
        • Simulated Annealing
        • Tabu search
        • Explorative search methods
          • Greedy Randomized Adaptive Search Procedure (GRASP)
          • Variable Neighborhood Search (VNS)
          • Guided Local Search (GLS)
          • Iterated Local Search (ILS)
  • Inverse Problems
  • Multi-Objective Optimization / Multicriteria Optimization / Pareto Optimization
    • Multi-Objective Linear Programming

Online CoursesEdit

Video LecturesEdit


Lecture NotesEdit

IntroductoryEdit

SpecializedEdit

BooksEdit

IntroductoryEdit

  • Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons.
  • Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer.

SpecializedEdit

  • Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control. 4th Ed. Athena Scientific.
  • Bertsekas, D. P. (2016). Nonlinear Programming. 3rd Ed. Athena scientific.
  • Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and Trends® in Optimization, 2(3-4), 157-325.
  • Bertsekas, D. P. (2015). Convex Optimization Algorithms. Athena Scientific.
  • Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
  • Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons.
  • Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and Trends® in Machine Learning, 4(2), 107-194.
  • Sra, S., Nowozin, S., & Wright, S. J. (2012). Optimization for machine learning. MIT Press.
  • Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu.
  • Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press.
  • Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer.
  • Ruszczyński, A. (2006). Nonlinear Optimization. Princeton University Press.
  • Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press.
  • Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course. Springer.
  • Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam.
  • Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam.
  • Kelley, C. T. (1999). Iterative Methods for Optimization. Siam.
  • Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam.
  • Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier.
  • Scales, L. E. (1985). Introduction to non-linear optimization. Springer-Verlag.

SoftwareEdit

See List of Optimization Software  for the complete list.

See alsoEdit

Other ResourcesEdit

Retrieved from "https://wiki.kourouklides.com/w/index.php?title=Optimization&oldid=1806"