Deep Learning

This page contains resources about Deep Learning and Representation Learning .

Subfields and Concepts

• Deep Generative Models
  • Deep Directed Networks (directed graphical models)
    • Sigmoid Belief Net
    • Differentiable Generator Net
    • Variational Autoencoder (VAE)
    • Generative Adversarial Network (GAN)
    • Generative Moment Matching Network
    • Convolutional Generative Network
    • Auto-Regressive Network / Fully-visible Bayes Network (FVBN)
    • Deep Latent Gaussian Model (DLGM)
    • Deep AutoRegressive Network (DARN)
  • Deep Boltzmann Machines (undirected graphical models)
  • Deep Belief Networks (mixed graphs)
• Deep Neural Networks (i.e. more than two hidden layers)
  • Deep Multi-Layer Perceptron (i.e. Stacked RBMs) 
  • Deep Autoencoders (i.e. two symmetrical DBN)
    • DARN 
  • Deep Neural Decision Forests 
  • Convolutional Deep Belief Network (i.e. Stacked CRBMs)  
• Sparse Coding / Dictionary Learning
  • Sparse Autoencoders
  • Stacked Denoising Autoencoders
• Bayesian Deep Learning
  • Bayesian Neural Networks

Online Courses

Video Lectures

• Introduction to Deep Learning - Coursera
• Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
• Deep Learning Specialization by Andrew Ng - Coursera
• Deep Learning by Google - Udacity
• Neural networks class by Hugo Larochelle (Youtube )
• Deep Learning and Neural Networks by Kevin Duh
• Computer Perception with Deep Learning by Yann LeCun  (Part 1 , Part 2 )
• Computational Neuroscience and Learning by Eugenio Culurciello (Youtube)
• A tutorial on Deep Learning by Geoffrey Hinton - VideoLectures.Net
• Deep Learning with TensorFlow: Applications of Deep Neural Networks to Machine Learning Tasks by Jon Krohn

Lecture Notes

• Practical Deep Learning for coders by Jeremy Howard
• Deep Learning by Yann LeCun
• Unsupervised Feature Learning and Deep Learning (UFLDL) by Andrew Ng
• Deep Learning by Bhiksha Raj
• Representation Learning by Yoshua Bengio
• Convolutional Neural Networks for Visual Recognition by Fei-Fei Li & Andrej Karpathy
• Deep Learning by Sargur Srihari
• DataLab Cup 5: Deep Reinforcement Learning
• Deep Learning by Shan-Hung Wu
• Cutting-Edge Trends in Deep Learning and Recognition by Svetlana Lazebnik

Books and Book Chapters

• Santana, E. (2018). Eder Santana's Deep Learning with Python. Packt Publishing.
• Shukla, N. (2018). Machine learning with TensorFlow. Manning.
• Zaccone, G., Karim, Md. R., & Menshawy, A. (2017). Deep Learning with TensorFlow. Packt Publishing.
• McClure, N. (2017). TensorFlow Machine Learning Cookbook. Packt Publishing.
• Gulli, A., & Pal, S. (2017). Deep Learning with Keras. Packt Publishing.
• Chollet, F. (2017). Deep Learning with Python. Manning Publications.
• Gulli, A., & Kapoor, A. (2017). TensorFlow 1.x Deep Learning Cookbook. Packt Publishing.
• Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). Deep Learning. MIT Press. (link)
• Gibson, A., & Patterson J. (2016). Deep Learning: A Practitioner's Approach. O'Reilly Media.
• Nielsen, M. A. (2015). Neural Networks and Deep Learning. Determination Press.
• Theodoridis, S. (2015). "Chapter 18: Neural Networks and Deep Learning". Machine Learning: A Bayesian and Optimization Perspective. Academic Press.
• Odense, S. (2015). Universal approximation theory of neural networks. MSc Diss. University of Victoria.
• Du, K. L., & Swamy, M. N. (2014). Neural networks and statistical learning. Springer Science & Business Media.
• Deng, L., & Yu, D. (2014). Deep Learning. Foundations and Trends in Signal Processing, 7, 3-4.
• Bengio, Y.,  & Courville, A. (2013). Deep Learning of Representations. Springer.
• Barber, D. (2012). "Chapter 26: Distributed Computation". Bayesian Reasoning and Machine Learning. Cambridge University Press.
• Neal, R. M. (2012). Bayesian learning for neural networks. Springer Science & Business Media.
• Orr, G. B., & Muller, K. R. (2012). Neural Networks: Tricks of the Trade. Springer.
• Murphy, K. P. (2012). "Chapter 28: Deep Learning". Machine Learning: A Probabilistic Perspective. MIT Press.
• Alpaydin, E. (2010). "Chapter 11: Multilayer Perceptrons". Introduction to Machine Learning. MIT Press.
• Haykin, S. S., Haykin, S. S., Haykin, S. S., & Haykin, S. S. (2009). Neural Networks and Learning Machines. 3rd Ed. Pearson.
• Bengio, Y. (2009). Learning Deep Architectures for AI. Foundations and Trends® in Machine Learning, 2(1), 1-127. Now Publishers.
• LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., & Huang, F. (2006). "A Tutorial on Energy-Based Learning". Predicting Structured Data. MIT Press.
• Bishop, C. M. (2006). "Chapter 5: Neural Networks". Pattern Recognition and Machine Learning. Springer.
• MacKay, D. J. (2003). "Chapter 38: Introduction to Neural Networks" Information Theory, Inference and Learning Algorithms. Cambridge University Press.
• Mandic, D. P., & Chambers, J. (2001). Recurrent neural networks for prediction: learning algorithms, architectures and stability. John Wiley & Sons.
• Rojas, R. (1996). Neural networks: a systematic introduction. Springer Science & Business Media. (link)
• Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford University Press.
• Hassoun, M. H. (1995). Fundamentals of artificial neural networks. MIT press.

Scholarly Articles

See Reading List and Recommended Readings for the complete list.

• Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021). Self-supervised Learning: Generative or Contrastive. IEEE Transactions on Knowledge and Data Engineering.
• Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., ... & Sun, M. (2020). Graph Neural Networks: A Review of Methods and Applications. AI Open, 1, 57-81.
• Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Philip, S. Y. (2020). A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1), 4-24.
• Shwartz-Ziv, R., & Tishby, N. (2017). Opening the Black Box of Deep Neural Networks via Information. arXiv preprint arXiv:1703.00810.
• Chakraborty, S., Tomsett, R., Raghavendra, R., Harborne, D., Alzantot, M., Cerutti, F., ... & Kelley, T. D. (2017). Interpretability of Deep Learning Models: A Survey of Results.
• Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural networks, Volume 61, 85-117.
• Paul, A., & Venkatasubramanian, S. (2014). Why does Deep Learning work?-A perspective from Group Theory. arXiv preprint arXiv:1412.6621.
• Bengio, Y., Courville, A., & Vincent, P. (2013). Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828.

Tutorials

• Information Theory of Deep Learning by Prof Naftali Tishby
• UFLDL Tutorial
• A Deep Learning Tutorial: From Perceptrons to Deep Networks - with Java examples
• Neural Networks, Manifolds, and Topology - advanced
• Deep Learning Tutorials by Olver Dürr - Lasagne and TensorFlow
• Efficient Deep Learning with Humans in the Loop by Zachary Chase Lipton
• Deep Generative Models by Durk Kingma (2014)
• Learning Invariant Feature Hierarchies (2013) and its Panel Discussion
• Deep Learning Tutorial (ICML 2013)
• Deep Learning for Computer Vision (NIPS 2013)
• Deep Learning for NLP (NAACL 2013)
• Deep Support Vector Machines (ROKS 2013)
• Deep Learning of Representaions (SSTiC 2013)
• Deep Learning for Machine Vision (BMVC 2013)
• Deep Learning for Computer Vision (NIPS 2013) (Video )
• Deep Learning Methods for Vision (CVPR 2012)
• Deep Learning for NLP (ACL 2012)
• Representation Learning (ICML 2012)
• Classification with Deep Invariant Scattering Networks (NIPS 2012)
• A tutorial on deep and unsupervised feature learning for activity recognition (2011)
• Tutorial on Deep Learning and Applications (NIPS 2010)
• A tutorial on Deep Learning (2009)
• Tutorial on Learning Deep Architectures (ICML 2009)
• Learning Deep Hierarchies of Representations (2009)
• Deep Learning with Multiplicative Interactions (NIPS 2009)
• Learning Feature Hierarchies (MLSS 2009)
• Deep Belief Networks (MLSS 2009)

Software

See Software Links for the complete list.

• TensorFlow - Python
• Theano - Python (discontinued; no longer supported or developed)
• Keras - Python library for TensorFlow and Theano
• Lasagne - Lightweight library to build and train neural networks in Theano
• Caffe - C/C++, Python, MATLAB, command line
• Torch - Lua
• PyTorch - Python and C++
• fastai - Python
• pydlt - PyTorch based Deep Learning Toolbox
• CNTK - The Microsoft Cognitive Toolkit
• ONNX
• cuDNN - CUDA GPU library supporting TensorFlow, Theano, Torch, Caffe, Keras, CNTK and others
• TFLean - Deep Learning Python library featuring a higher-level API for TensorFlow
• Blocks - A Theano framework for training neural networks
• Pylearn2 - Python
• deeplearning4j - Java
• MXNet - MATLAB, Python, C++, R, Julia, Scala, Go, Javascript and more
• Chainer - Python
• CudaCnn - MATLAB
• hebel - Python
• ConvNetJS - Deep Learning models (mainly Neural Networks) entirely in your browser
• OpenNN - C++
• visual-rbm
• MatCovNet - MATLAB
• Learning Deep Boltzmann Machines - MATLAB
• Estimating Partition Functions of RBM's - MATLAB
• Deep Belief Networks - MATLAB
• DeepLearnToolbox - MATLAB/Octave
• Netlab neural network software - MATLAB
• Neural Network Toolbox - MATLAB
• PyBrain - Python
• DyNet - Python and C++
• handson-ml - Python
• DLL - C++
• DeepRosetta - Python
• torchgeometry - Python
• distiller - Python

See also

• Deep Learning in the news (blog)
• Probabilistic Graphical Models
• Sparse Coding

Other Resources

• Deep Learning with Python by Jason Brownlee - practical book
• Deep Learning Reading List
• DeepLearning.Net - Tutorials and a general point of reference
• Toronto Deep Learning Demos - source code
• Deep learning from the bottom up - Metacademy
• Neural Networks and Deep Learning - free online book
• Awesome-Deep-Vision (Github) - A curated list of Deep Learning resources for Computer Vision
• Awesome-Deep-Learning (Github) - A curated list of resources
• Awesome-Deep-Learning-papers (Github)
• Awesome-TensoFflow (Github)
• Deep Learning Libraries by Language
• Deep Learning (Building Intelligent Probabilistic Systems)  - Blog by Harvard University
• Benchmark of Deep Learning Representations for Visual Recognition
• Deep Learning Playlist - Youtube collection of video lectures and tutorials
• Deep Learning on Google+ - online community
• A Short History of and Introduction to Deep Learning - Presentation by John Kaufhold
• An Introduction to Deep Learning: From Perceptrons to Deep Networks - tutorial with Java examples
• Graduate Summer School: Deep Learning, Feature Learning by IPAM, UCLA
• What Does a Neural Network Actually Do? - Neural Networks and Deep Learning
• Ersatz - Deep Neural Networks in the cloud
• Bibliography in Deep Learning - collection of papers categorized according to type of application
• Deep Learning Papers Reading Roadmap
• What My Deep Model Doesn't Know - Blog post on Bayesian Neural Networks
• New Theory Cracks Open the Black Box of Deep Learning
• Why does Deep Learning work? by Charles H Martin - blog post
• Why Deep Learning Works II: The Renormalization Group by Charles H Martin - blog post
• Practical_DL - Github
• deep-learning-book - Github
• Deep Learning: Theory & Practice - blog post
• From Topological Data Analysis to Deep Learning: No Pain No Gain - blog post
• Deep learning: the final frontier for signal processing and time series analysis? - blog post