Machine Learning



This page contains resources about Pattern Recognition and Machine Learning in general, including Computational Statistics (which it usually refers to Monte Carlo Methods instead of Machine Learning)

More specific information is included in each subfield.

Subfields and Concepts
See Category:Machine Learning for some of its subfields.
 * Supervised Learning
 * Classification
 * Discriminative Model Vs Generative Model
 * Regression
 * Parametric Model Vs Nonparametric Model
 * Structured Learning
 * Generalized Linear Model (GLM or GLIM)
 * Support Vector Machine
 * Supervised Dimensionality Reduction
 * Adaptive Basis Function Model
 * Decision Tree Learning
 * Classification and Regression Tree (CART)
 * ID3 Algorithm
 * Artificial Neural Network
 * Feedforward Neural Network
 * Recurrent Neural Network
 * Radial Basis Function (RBF) Network
 * Kohonen Network
 * Supervised Ensemble Learning
 * Bayesian Averaging
 * Bagging
 * Boosting
 * Bayes Optimal Classifier
 * Decision Forest / Random Forest
 * Supervised Dictionary Learning
 * Supervised Deep Learning
 * Deep Belief Network
 * Unsupervised Learning
 * Clustering / Discrete Latent Variable Models
 * Unsupervised Dimensionality Reduction / Continuous Latent Variable Models
 * Manifold Learning (although supervised variants exist)
 * Autoencoder
 * Unsupervised Ensemble Learning
 * Unsupervised Dictionary Learning
 * Unsupervised Deep Learning
 * Deep Autoencoder
 * Deep Belief Network
 * Self-Supervised Learning
 * Generative models
 * Semi-supervised Learning
 * Active Learning
 * Inductive Learning
 * Supervised Learning
 * Semi-supervised Learning
 * Analytical Learning
 * Transfer Learning / Inductive Transfer
 * Learning to Learn / Meta Learning
 * Multi-task Learning
 * Reinforcement Learning
 * Multi-Armed Bandit
 * Finite Markov Decision Process
 * Temporal-Difference Learning
 * Q-Learning
 * Adaptive Dynamic Programming
 * Deep Reinforcement Learning
 * Probabilistic Machine Learning
 * Bayesian Network (directed graphical models)
 * Markov Random Field (undirected graphical models)
 * Mixture Model
 * Stochastic Model
 * Bayesian Machine Learning
 * Variational Bayesian Learning
 * Statistical Learning Theory and Computational Learning Theory
 * Online Learning and Sequential Prediction
 * Evolutionary Computation
 * Genetic Algorithms
 * Lazy Learning / Instance-based Learning
 * k-Nearest Neighbors (k-NN) Algorithm
 * Case-based Reasoning (CBR)
 * Eager Learning
 * RBF Network
 * Kernel Machine
 * Decision Tree Learning
 * Backpropagation
 * Naive Bayes
 * Applications
 * Bioinformatics
 * Compressed Sensing
 * Computational Finance
 * Computer Vision
 * Control Theory
 * Econometrics
 * Energy
 * Environmetrics
 * Geospatial Data
 * Medical Imaging
 * Natural Language Processing
 * Robotics
 * Recommender Systems
 * Sequential Data
 * Speech Processing

Video Lectures

 * Foundations of Machine Learning by David S. Rosenberg
 * Advanced Machine Learning Specialization - Coursera
 * Machine Learning by Andrew Ng - Coursera
 * Machine Learning by Pedro Domingos - Coursera
 * Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
 * Practical Machine Learning by Jeff Leek - Coursera
 * NYU Course on Big Data, Large Scale Machine Learning by John Langford and Yann LeCun
 * Learning from Data by Yaser Abu-Mostafa
 * Introduction to Machine Learning by Barnabas Poczos and Alex Smola
 * Machine Learning by Nando de Freitas
 * Pattern Recognition by Fred A. Hamprecht (2011 2012 )
 * Machine Learning and Pattern Recognition by Charles Sutton (Lecture notes )
 * Machine Learning by Joachim M. Buhmann
 * Pattern Recognition by P.S.Sastry - NPTEL
 * CS188 Intro to AI by Dan Klein and Pieter Abbeel
 * Introduction to Computational Thinking and Data Science
 * PASCAL Lecture Series - VideoLectures.Net

Introductory

 * CO395: Machine Learning by Maja Pantic - very introductory course
 * Prediction: Machine Learning and Statistics by Cynthia Rudin- very introductory course
 * Machine Learning by Michael Littman
 * SGN-2506: Introduction to Pattern Recognition by Jussi Tohka
 * CSCI1950-F: Introduction to Machine Learning by Erik Sudderth
 * CS 229: Machine Learning by Andrew Ng
 * CSC 411: Machine Learning and Data Mining by Aaron Hertzmann
 * CS 760: Machine Learning by David Page
 * CSE446: Machine Learning
 * Introduction To Machine Learning by David Sontag
 * CSC321: Introduction to Neural Networks and Machine Learning by Tijmen Tieleman - this might be a bit advanced for beginners
 * CSC2515: Introduction to Machine Learning by Geoffrey Hinton - this is very similar to the above
 * Introduction to Pattern Recognition by Sargur Srihari
 * Introduction to Machine Learning Course by Sargur Srihari - this might be a bit advanced for beginners
 * Introduction to Machine Learning by Shai Shalev-Shwartz - this might be a bit advanced for beginners
 * COS 511: Foundations of Machine Learning by Rob Schapire
 * CS 2750 Machine Learning by Milos Hauskrecht
 * Introductory Applied Machine Learning by Victor Lavrenko and Nigel Goddard
 * Machine Learning and Pattern Recognition by Yann LeCun
 * Machine Learning by Tom Mitchell
 * Machine Learning by Tommi Jaakkola
 * Machine Learning by Andrew Zisserman
 * Pattern Recognition and Analysis by Rosalind W. Picard
 * CSCE 666: Pattern Analysis Fall by Ricardo Gutierrez-Osuna
 * Pattern Recognition by Richard Zanibbi
 * Neural Networks and Pattern Recognition by Ömer Cengiz ÇELEBİ
 * Machine Learning by Carl Edward Rasmussen and Zoubin Ghahramani
 * Learning from Data by Amos Storkey
 * Machine Learning: Pattern Recognition by Gwenn Englebienne
 * Machine Learning by Tony Jebara
 * Machine Learning I by Le Song
 * Advanced data mining: theory and applications by Dmitry Efimov
 * Artificial Intelligence by Emma Brunskill, Ariel Procaccia and Gianni Di Caro

Advanced

 * CS281: Advanced Machine Learning by Ryan Adams
 * CSC2535: Advanced Machine Learning by Geoffrey Hinton
 * Advanced Topics in Machine Learning by Andreas Krause
 * Advanced Topics in Machine Learning (Kernel Methods) by Arthur Gretton - Gatsby
 * Pattern Recognition by Ricardo Gutierrez-Osuna
 * Introduction to Pattern Recognition by Jason Corso
 * Pattern Recognition by Olga Veksler
 * Pattern Recognition by Charles Robertson
 * Pattern Recognition by Esa Alhoniemi
 * Advanced Machine Learning by Mehryar Mohri
 * Advanced Machine Learning by Tony Jebara
 * Machine Learning II by Le Song
 * Machine Learning by Byron Boots

Specialized

 * STA 663 Statistical Computing and Computation by Cliburn Chan and Janice McCarthy
 * Statistical Machine Learning from Data by Samy Bengio
 * STA 4273H: Statistical Machine Learning by Ruslan Salakhutdinov
 * CS59000: Statistical Machine Learning by Alan Qi
 * Statistical Machine Learning and Data Mining by Yee Whye Teh
 * Identification, Estimation, and Learning by Harry Asada
 * Unsupervised Learning by Zoubin Ghahramani
 * Probabilistic and Unsupervised Learning by Maneesh Sahani - Gatsby
 * Approximate Inference and Learning in Probabilistic Models by Maneesh Sahani - Gatsby
 * Reinforcement Learning by David Silver
 * Reinforcement Learning by Michael Herrmann
 * Pattern Recognition by Xia Hong
 * Pattern Recognition for Machine Vision by Bernd Heisele and Yuri Ivanov
 * CS 5785: Modern Analytics by Serge J. Belongie
 * Advanced Statistical Machine Learning by Stefanos Zafeiriou
 * Machine Learning Practical by Steve Renals
 * Statistical Machine Learning by Pier Francesco Palamara
 * Computational Statistics by Martin Maechler and Peter Bühlmann

Practical

 * Witten, I. H., & Frank, E. (2005). Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann.
 * Martinez, W. L., & Martinez, A. R. (2007). Computational statistics handbook with MATLAB. 2nd Ed. CRC press.
 * Martinez, W. L., Martinez, A. R., Martinez, A., & Solka, J. (2010). Exploratory data analysis with MATLAB. 2nd Ed. CRC Press.
 * Sammut, C., & Webb, G. I. (Eds.). (2011). Encyclopedia of Machine Learning. Springer Science & Business Media.
 * Han, J., Pei, J., & Kamber, M. (2011). Data mining: concepts and techniques. 3rd Ed. Morgan Kaufmann.
 * Conway, D., & White, J. (2012). Machine Learning for Hackers. O'Reilly Media.
 * McKinney, W. (2012). Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O'Reilly Media.
 * Rajaraman, A., & Ullman, J. D. (2012). Mining of Massive Datasets. Cambridge University Press.
 * Brownlee, J. (2013). Clever Algorithms: Statistical Machine Learning Recipes. Jason Brownlee.
 * Kutz, J. N. (2013). Data-driven modeling & scientific computation: methods for complex systems & big data. Oxford University Press.
 * Schutt, R., & O'Neil, C. (2013). Doing data science: Straight talk from the frontline. O'Reilly Media, Inc.
 * Battiti, R., & Brunato, M. (2014). The LION Way. Machine Learning Plus Intelligent Optimization. CreateSpace.
 * Zumel, N., Mount, J., & Porzak, J. (2014). Practical data science with R. Manning.
 * Nolan, D., & Lang, D. T. (2015). Data Science in R: A Case Studies Approach to Computational Reasoning and Problem Solving. CRC Press.
 * Davidson-Pilon, C. (2015). Bayesian Methods for Hackers: Probabilistic Programming and Bayesian Inference. Addison-Wesley Professional.
 * Elston, S. F. (2015). Data Science in the Cloud with Microsoft Azure Machine Learning and R. O'Reilly Media, Inc.
 * Kelleher, J. D., Mac Namee, B., & D'Arcy, A. (2015). Fundamentals of machine learning for predictive data analytics: algorithms, worked examples, and case studies. MIT Press.
 * Lantz, B. (2015). Machine Learning with R. 2nd Ed. Packt Publishing Ltd.
 * Yu-Wei, C. D. C. (2015). Machine Learning with R cookbook. Packt Publishing Ltd.
 * Raschka, S. (2015). Python Machine Learning. Packt Publishing Ltd.
 * Ankan, A., & Panda, A. (2015). Mastering Probabilistic Graphical Models Using Python. Packt Publishing Ltd.
 * Grus, J. (2015). Data Science from Scratch: First Principles with Python. O'Reilly Media.
 * Madhavan, S. (2015). Mastering Python for Data Science. Packt Publishing Ltd.
 * Zaccone, G. (2016). Getting started with TensorFlow. Packt Publishing Ltd.
 * VanderPlas, J. (2016). Python Data Science Handbook: Essential Tools for Working with Data. O'Reilly Media.
 * Muller, A. C., & Guido, S. (2016). Introduction to Machine Learning with Python: A Guide for Data Scientists. O'Reilly Media, Inc.
 * Guttag, J. V. (2016). Introduction to Computation and Programming Using Python: With Application to Understanding Data. MIT Press.
 * Dixon, J. (2016). Mastering. NET Machine Learning. Packt Publishing Ltd.
 * Wickham, H., & Grolemund, G. (2017). R for Data Science. O'Reilly Media.
 * Geron, A. (2017). Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems.O'Reilly Media.

Introductory

 * Mitchell, T. M. (1997). Machine Learning. McGraw Hill.
 * Smola, A., & Vishwanathan, S. V. N. (2008). Introduction to Machine Learning. Cambridge University Press.
 * Alpaydin, E. (2010). Introduction to Machine Learning. MIT Press.
 * Theodoridis, S., Pikrakis, A., Koutroumbas, K., & Cavouras, D. (2010). Introduction to Pattern Recognition: A Matlab Approach. Academic Press.
 * Rogers, S., & Girolami, M. (2011). A First Course in Machine Learning. CRC Press.
 * Abu-Mostafa, Y. S., Magdon-Ismail, M., & Lin, H. T. (2012). Learning From Data. AMLBook.
 * Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling. Springer.
 * Kruse, R., Borgelt, C., Klawonn, F., Moewes, C., Steinbrecher, M., & Held, P. (2013). Computational Intelligence: A Methodological Introduction. Springer Science & Business Media.
 * Aggarwal, C. C. (2015). Data Mining: The Textbook. Springer.
 * Nilsson, N. J. (2015). Introduction to machine learning. An early draft of a proposed textbook.
 * James, G., Witten, D., & Hastie, T. (2017). An Introduction to Statistical Learning: With Applications in R.

Advanced

 * Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
 * Theodoridis, S., Koutroumbas, K., (2009). Pattern Recognition, 4th Ed., Academic Press
 * Duda, R. O., Hart, P. E., & Stork, D. G. (2012). Pattern Classification. John Wiley & Sons.
 * Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2012). Foundations of Machine Learning. MIT press.
 * Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.
 * Theodoridis, S. (2015). Machine Learning: A Bayesian and Optimization Perspective. Academic Press.

Specialized

 * Rojas, R. (1996). Neural Networks: A Systematic Introduction. Springer Science & Business Media.
 * Ripley, B. D. (1996). Pattern recognition and neural networks. Cambridge University Press.
 * Moon, T. K., &  Stirling, W. C. (2000). Mathematical methods and algorithms for signal processing. Pearson.
 * Webb, A. R. (2002). Statistical Pattern Recognition. 2nd Ed. John Wiley & Sons.
 * MacKay, D. J. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press.
 * Kushner, H., & Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications (Vol. 35). 2nd Ed. Springer Science & Business Media.
 * Taylor,J. S. & Cristianini, N. (2004). Kernel Methods for Pattern Analysis. Cambridge University Press.
 * Williams, C. K., & Rasmussen, C. E. (2006). Gaussian Processes for Machine Learning. MIT Press.
 * Vapnik, V. (2006). Estimation of dependences based on empirical data. Springer Science & Business Media.
 * Hastie, T., Tibshirani, R., Friedman, J., Hastie, T., Friedman, J., & Tibshirani, R. (2009). The Elements of Statistical Learning. 2nd Ed. New York: Springer.
 * Koller, D., & Friedman, N. (2009). Probabilistic Graphical Models. MIT Press.
 * Haykin, S. O. (2009). Neural Networks and Learning Machines. 3rd Ed. Pearson.
 * Gentle, J. E. (2009). Computational statistics. Springer.
 * Russell, S. J., & Norvig, P. (2010). "Part IV: Uncertain knowledge and reasoning". Artificial Intelligence: A Modern Approach. Prentice Hall.
 * Bühlmann, P., & Van De Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer Science & Business Media.
 * Givens, G. H., & Hoeting, J. A. (2012). Computational statistics. 2nd Ed. John Wiley & Sons.
 * Murphy, K. P. (2012). Machine Learning: A Probabilistic Perspective. MIT Press.
 * Barber, D. (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Bubeck, S. & Cesa-Bianchi, N. (2012). Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Foundations and Trends® in Machine Learning, 5(1), 1-122. Now Publishers.
 * Jebara, T. (2012). Machine Learning: Discriminative and Generative. Springer Science & Business Media.
 * Flach, P. (2012). Machine learning: the art and science of algorithms that make sense of data. Cambridge University Press.
 * Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.
 * Blum, A., Hopcroft, J., & Kannan, R. (2015). Foundations of Data Science. (link)
 * Goodman, N. D., & Tenenbaum, J. B. (2016). Probabilistic Models of Cognition. 2nd Ed. (link)
 * Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). Deep Learning. MIT Press.

Software

 * PRMLT - MATLAB Toolbox for the book of PRML by C. Bishop
 * pmtk3 - Probabilistic Modeling Toolkit for MLPP book by Murphy in Matlab/Octave (3rd edition)
 * pyprobml - Python code for MLPP book by K. Murphy
 * BRMLtoolbox - MATLAB and Julia code for the BRML book by D. Barber
 * PyBRML - Python code for the BRML book by D. Barber
 * PythonDataScienceHandbook - Python code for the PDSH book by J. VanderPlas
 * handson-ml - Python code for the HOML book by A. Geron
 * OpenML
 * Microsoft R
 * BigML - Prediction and Analytics tasks under 16MB are free
 * Neural Network Toolbox - MATLAB
 * Torch7 - a scientific computing framework for Machine Learning algorithms (based on Lua)
 * Lush - an OOP language for large-scale numerical and graphic applications (based on Lisp)
 * TensorFlow - Google
 * CNTK - Microsoft
 * Pylearn2 - A Machine Learning research library
 * scikit-learn - Python
 * mlpy - Python
 * Orange - Data Visualization and Analysis
 * Matlab Machine Learning Toolboxes
 * mloss.org. Machine Learning open source software
 * April-ANN - A Pattern Recognizer In Lua with ANNs
 * Weka- Data mining software in Java
 * MLTK - Machine Learning Toolkit in Java
 * OpenNN
 * Bayesian Modeling and Monte Carlo Methods
 * The Lightspeed Matlab Toolbox
 * MCML - broad range support for Monte Carlo methods to implement Machine Learning applications
 * Orange - Visual programming language
 * MLPACK - C++
 * Shogun - C++ toolbox that offers interfaces for MATLAB,Octave, Python, R, Java, Lua, Ruby and C# (mainly for Kernel Machines)
 * autograd - Efficiently computes derivatives of numpy code (Python)
 * pySPACE - Signal Processing And Classification Environment (SPACE) in Python
 * dlib - C++ (with Python API)
 * Computational Statistics Toolbox - MATLAB
 * Exploratory Data Analysis (EDA) Toolbox - MATLAB
 * aimacode - Code for the AIMA book by Russell and Norvig
 * Core ML - Apple
 * pycm - Python

Datasets

 * UCI Machine Learning Repository - a large collection of standard datasets for testing learning algorithms
 * Awesome Public Datasets (GitHub)
 * Datasets on Kaggle
 * DeepLearning.Net - a list of datasets that can be used for benchmarking Deep Learning algorithms
 * MLdata
 * List of datasets for machine learning research

Other Resources

 * Artificial Intelligence - Google Scholar Metrics (Top Publications)
 * Computer Vision and Pattern Recognition - Google Scholar Metrics (Top Publications)
 * Data Mining and Analysis - Google Scholar Metrics (Top Publications)
 * Machine Learning - Nature
 * NIPS - A top-tier Conference in Machine Learning (and other topics)
 * ICML - A top-tier Conference in Machine Learning (and other topics)
 * Machine Learning Types - Medium
 * Video Tutorials - Youtube channel of 'Mathematical Monk'
 * Basic Concepts in Machine Learning
 * Awesome-Machine-Learning (Github) - A curated list of Machine Learning frameworks, libraries and software (by language)
 * Computational Statistics in Python (2016 version, Github)
 * Comparison of software toolkits
 * Software for Data Mining, Analytics, Data Science, and Knowledge Discovery - KDnuggets
 * Machine Learning and Statistical Learning in R
 * Metacademy - List of concepts in Machine Learning
 * Machine Learning, Statistical Inference and Induction - Notebook
 * A Course in Machine Learning by Hal Daumé III - textbook
 * Courses on Statistical Pattern Recognition - summary of 33 courses
 * FastML - List of Machine Learning courses online
 * Results and Errors Percentage on Standard Datasets - MNIST, CIFAR, Pascal VOC, etc.
 * Machine Learning Surveys - List of literature surveys, reviews, and tutorials on Machine Learning and related topics
 * Machine Learning on Google+ - online community
 * The Shape of Data - Data Mining and Machine Learning blog
 * Data Mining & Machine Learning - a mindmap diagram comparing the two areas
 * Basic Glossary of Machine Learning
 * Machine Learning with MATLAB - free ebook
 * References for Machine Learning - Wikia
 * Which machine learning algorithm should I use? - blog post
 * Tutorial: Getting Started with Machine Learning in Python - blog post
 * How to know that your machine learning problem is hopeless? - Stack Exchange
 * Yandex School of Data Analysis - Github
 * practical-ml (GitHub) - code
 * Hyperparameter Optimization in Machine Learning Models - blog post
 * Hyperparameter tuning for machine learning models - blog post