Deep Learning

This page contains resources about Deep Learning and Representation Learning.

Subfields and Concepts

 * Deep Generative Models
 * Deep Directed Networks (directed graphical models)
 * Sigmoid Belief Net
 * Differentiable Generator Net
 * Variational Autoencoder (VAE)
 * Generative Adversarial Network (GAN)
 * Generative Moment Matching Network
 * Convolutional Generative Network
 * Auto-Regressive Network / Fully-visible Bayes Network (FVBN)
 * Deep Latent Gaussian Model (DLGM)
 * Deep AutoRegressive Network (DARN)
 * Deep Boltzmann Machines (undirected graphical models)
 * Deep Belief Networks (mixed graphs)
 * Deep Neural Networks (i.e. more than two hidden layers)
 * Deep Multi-Layer Perceptron (i.e. Stacked RBMs)
 * Deep Autoencoders (i.e. two symmetrical DBN)
 * DARN
 * Deep Neural Decision Forests
 * Convolutional Deep Belief Network (i.e. Stacked CRBMs)
 * Sparse Coding / Dictionary Learning
 * Sparse Autoencoders
 * Stacked Denoising Autoencoders
 * Bayesian Deep Learning
 * Bayesian Neural Networks

Video Lectures

 * Introduction to Deep Learning - Coursera
 * Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
 * Deep Learning Specialization by Andrew Ng - Coursera
 * Deep Learning by Google - Udacity
 * Neural networks class by Hugo Larochelle (Youtube )
 * Deep Learning and Neural Networks by Kevin Duh
 * Computer Perception with Deep Learning by Yann LeCun  (Part 1, Part 2 )
 * Computational Neuroscience and Learning by Eugenio Culurciello (Youtube)
 * A tutorial on Deep Learning by Geoffrey Hinton - VideoLectures.Net
 * Deep Learning with TensorFlow: Applications of Deep Neural Networks to Machine Learning Tasks by Jon Krohn

Lecture Notes

 * Practical Deep Learning for coders by Jeremy Howard
 * Deep Learning by Yann LeCun
 * Unsupervised Feature Learning and Deep Learning (UFLDL) by Andrew Ng
 * Deep Learning by Bhiksha Raj
 * Representation Learning by Yoshua Bengio
 * Convolutional Neural Networks for Visual Recognition by Fei-Fei Li & Andrej Karpathy
 * Deep Learning by Sargur Srihari
 * DataLab Cup 5: Deep Reinforcement Learning
 * Deep Learning by Shan-Hung Wu
 * Cutting-Edge Trends in Deep Learning and Recognition by Svetlana Lazebnik

Books and Book Chapters

 * Santana, E. (2018). Eder Santana's Deep Learning with Python. Packt Publishing.
 * Shukla, N. (2018). Machine learning with TensorFlow. Manning.
 * Zaccone, G., Karim, Md. R., & Menshawy, A. (2017). Deep Learning with TensorFlow. Packt Publishing.
 * McClure, N. (2017). TensorFlow Machine Learning Cookbook. Packt Publishing.
 * Gulli, A., & Pal, S. (2017). Deep Learning with Keras. Packt Publishing.
 * Chollet, F. (2017). Deep Learning with Python. Manning Publications.
 * Gulli, A., & Kapoor, A. (2017). TensorFlow 1.x Deep Learning Cookbook. Packt Publishing.
 * Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). Deep Learning. MIT Press. (link)
 * Gibson, A., & Patterson J. (2016). Deep Learning: A Practitioner's Approach. O'Reilly Media.
 * Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.
 * Theodoridis, S. (2015). "Chapter 18: Neural Networks and Deep Learning". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
 * Odense, S. (2015). Universal approximation theory of neural networks. MSc Diss. University of Victoria.
 * Du, K. L., & Swamy, M. N. (2014). Neural networks and statistical learning. Springer Science & Business Media.
 * Deng, L., & Yu, D. (2014). Deep Learning. Foundations and Trends in Signal Processing, 7, 3-4.
 * Bengio, Y.,  & Courville, A. (2013). Deep Learning of Representations. Springer.
 * Barber, D. (2012). "Chapter 26: Distributed Computation". Bayesian Reasoning and Machine Learning. Cambridge University Press.
 * Neal, R. M. (2012). Bayesian learning for neural networks. Springer Science & Business Media.
 * Orr, G. B., & Muller, K. R. (2012). Neural Networks: Tricks of the Trade. Springer.
 * Murphy, K. P. (2012). "Chapter 28: Deep Learning". Machine Learning: A Probabilistic Perspective. MIT Press.
 * Alpaydin, E. (2010). "Chapter 11: Multilayer Perceptrons". Introduction to Machine Learning. MIT Press.
 * Haykin, S. S., Haykin, S. S., Haykin, S. S., & Haykin, S. S. (2009). Neural Networks and Learning Machines. 3rd Ed. Pearson.
 * Bengio, Y. (2009). Learning Deep Architectures for AI. Foundations and Trends® in Machine Learning, 2(1), 1-127. Now Publishers.
 * LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). "A Tutorial on Energy-Based Learning". Predicting Structured Data. MIT Press.
 * Bishop, C. M. (2006). "Chapter 5: Neural Networks". Pattern Recognition and Machine Learning. Springer.
 * MacKay, D. J. (2003). "Chapter 38: Introduction to Neural Networks" Information Theory, Inference and Learning Algorithms. Cambridge University Press.
 * Mandic, D. P., & Chambers, J. (2001). Recurrent neural networks for prediction: learning algorithms, architectures and stability. John Wiley & Sons.
 * Rojas, R. (1996). Neural networks: a systematic introduction. Springer Science & Business Media. (link)
 * Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford University Press.
 * Hassoun, M. H. (1995). Fundamentals of artificial neural networks. MIT press.

Scholarly Articles
See Reading List and Recommended Readings for the complete list.
 * Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021). Self-supervised Learning: Generative or Contrastive. IEEE Transactions on Knowledge and Data Engineering.
 * Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., ... & Sun, M. (2020). Graph Neural Networks: A Review of Methods and Applications. AI Open, 1, 57-81.
 * Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Philip, S. Y. (2020). A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1), 4-24.
 * Shwartz-Ziv, R., & Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv preprint arXiv:1703.00810.
 * Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., ... & Kelley, T. D. (2017). Interpretability of Deep Learning Models: A Survey of Results.
 * Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural networks, Volume 61, 85-117.
 * Paul, A., & Venkatasubramanian, S. (2014). Why does Deep Learning work?-A perspective from Group Theory. arXiv preprint arXiv:1412.6621.
 * Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828.

Tutorials

 * Information Theory of Deep Learning by Prof Naftali Tishby
 * UFLDL Tutorial
 * A Deep Learning Tutorial: From Perceptrons to Deep Networks - with Java examples
 * Neural Networks, Manifolds, and Topology - advanced
 * Deep Learning Tutorials by Olver Dürr - Lasagne and TensorFlow
 * Efficient Deep  Learning  with Humans  in  the  Loop by Zachary  Chase  Lipton
 * Deep Generative Models by Durk Kingma (2014)
 * Learning Invariant Feature Hierarchies (2013) and its Panel Discussion
 * Deep Learning Tutorial (ICML 2013)
 * Deep Learning for Computer Vision (NIPS 2013)
 * Deep Learning for NLP (NAACL 2013)
 * Deep Support Vector Machines (ROKS 2013)
 * Deep Learning of Representaions (SSTiC 2013)
 * Deep Learning for Machine Vision (BMVC 2013)
 * Deep Learning for Computer Vision (NIPS 2013) (Video )
 * Deep Learning Methods for Vision (CVPR 2012)
 * Deep Learning for NLP (ACL 2012)
 * Representation Learning (ICML 2012)
 * Classification with Deep Invariant Scattering Networks (NIPS 2012)
 * A tutorial on deep and unsupervised feature learning for activity recognition (2011)
 * Tutorial on Deep Learning and Applications (NIPS 2010)
 * A tutorial on Deep Learning (2009)
 * Tutorial on Learning Deep Architectures (ICML 2009)
 * Learning Deep Hierarchies of Representations (2009)
 * Deep Learning with Multiplicative Interactions (NIPS 2009)
 * Learning Feature Hierarchies (MLSS 2009)
 * Deep Belief Networks (MLSS 2009)

Software
See Software Links for the complete list.
 * TensorFlow - Python
 * Theano - Python (discontinued; no longer supported or developed)
 * Keras - Python library for TensorFlow and Theano
 * Lasagne - Lightweight library to build and train neural networks in Theano
 * Caffe - C/C++, Python, MATLAB, command line
 * Torch - Lua
 * PyTorch - Python and C++
 * fastai - Python
 * pydlt - PyTorch based Deep Learning Toolbox
 * CNTK - The Microsoft Cognitive Toolkit
 * ONNX
 * cuDNN - CUDA GPU library supporting TensorFlow, Theano, Torch, Caffe, Keras, CNTK and others
 * TFLean - Deep Learning Python library featuring a higher-level API for TensorFlow
 * Blocks - A Theano framework for training neural networks
 * Pylearn2 - Python
 * deeplearning4j - Java
 * MXNet - MATLAB, Python, C++, R, Julia, Scala, Go, Javascript and more
 * Chainer - Python
 * CudaCnn - MATLAB
 * hebel - Python
 * ConvNetJS - Deep Learning models (mainly Neural Networks) entirely in your browser
 * OpenNN - C++
 * visual-rbm
 * MatCovNet - MATLAB
 * Learning Deep Boltzmann Machines - MATLAB
 * Estimating Partition Functions of RBM's - MATLAB
 * Deep Belief Networks - MATLAB
 * DeepLearnToolbox - MATLAB/Octave
 * Netlab neural network software - MATLAB
 * Neural Network Toolbox - MATLAB
 * PyBrain - Python
 * DyNet - Python and C++
 * handson-ml - Python
 * DLL - C++
 * DeepRosetta - Python
 * torchgeometry - Python
 * distiller - Python

Other Resources

 * Deep Learning with Python by Jason Brownlee - practical book
 * Deep Learning Reading List
 * DeepLearning.Net - Tutorials and a general point of reference
 * Toronto Deep Learning Demos - source code
 * Deep learning from the bottom up - Metacademy
 * Neural Networks and Deep Learning - free online book
 * Awesome-Deep-Vision (Github) - A curated list of Deep Learning resources for Computer Vision
 * Awesome-Deep-Learning (Github) - A curated list of resources
 * Awesome-Deep-Learning-papers (Github)
 * Awesome-TensoFflow (Github)
 * Deep Learning Libraries by Language
 * Deep Learning (Building Intelligent Probabilistic Systems)  - Blog by Harvard University
 * Benchmark of Deep Learning Representations for Visual Recognition
 * Deep Learning Playlist - Youtube collection of video lectures and tutorials
 * Deep Learning on Google+ - online community
 * A Short History of and Introduction to Deep Learning - Presentation by John Kaufhold
 * An Introduction to Deep Learning: From Perceptrons to Deep Networks - tutorial with Java examples
 * Graduate Summer School: Deep Learning, Feature Learning by IPAM, UCLA
 * What Does a Neural Network Actually Do? - Neural Networks and Deep Learning
 * Ersatz - Deep Neural Networks in the cloud
 * Bibliography in Deep Learning - collection of papers categorized according to type of application
 * Deep Learning Papers Reading Roadmap
 * What My Deep Model Doesn't Know - Blog post on Bayesian Neural Networks
 * New Theory Cracks Open the Black Box of Deep Learning
 * Why does Deep Learning work? by Charles H Martin - blog post
 * Why Deep Learning Works II: The Renormalization Group by Charles H Martin - blog post
 * Practical_DL - Github
 * deep-learning-book - Github
 * Deep Learning: Theory & Practice - blog post
 * From Topological Data Analysis to Deep Learning: No Pain No Gain - blog post
 * Deep learning: the final frontier for signal processing and time series analysis? - blog post