Artificial Neural Network

This page contains resources about Artificial Neural Networks.

For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems.

Subfields and Concepts

 * Feedforward Neural Network (FNN)
 * Single-Layer Perceptron (i.e. with no hidden layers)
 * Multi-Layer Perceptron (MLP) / Standard Neural Network (SNN)
 * Radial Basis Function (RBF) Network
 * Extreme Learning Machine (ELM)
 * Convolutional Neural Network (CNN or ConvNet)
 * Capsule Network (CapsNet)
 * Recurrent Neural Network (RNN)
 * Hopfield Network
 * Boltzmann Machine
 * Bidirectional RNN
 * Bidirectional associative memory (BAM)
 * Long short-term memory (LSTM)
 * Memory Network
 * Gated Rectified Unit RNN (GRU-RNN)
 * Vanilla Recurrent Network
 * Continuous Time RNN (CTRNN)
 * RNN-RBM
 * Echo State Network (ESN)
 * Unitary RNN (uRNN)
 * Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
 * Helmholtz Machine
 * Boltzmann Machine
 * Restricted Boltzmann Machine (RBM)
 * Conditional RBM (CRBM)
 * Autoassociative memory
 * Generative Stochastic Network
 * Generative Adversarial Network
 * Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
 * Stochastic Computation Graph
 * Variational Autoencoder (VAE)
 * Natural-Parameter Network
 * Variance Network
 * Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
 * Probabilistic Nerual Network
 * Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
 * Probabilistic Backpropagation
 * Bayes by Backprop
 * Bayesian Dark Knowledge (BDK)
 * Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
 * Gamma NPN
 * Gaussian NPN
 * Poisson NPN
 * Random Neural Network
 * Autoencoder (used for Dimensionality Reduction)
 * Linear Autoencoder (equivalent to PCA)
 * Stacked Denoising Autoencoder
 * Generalized Denoising Autoencoder
 * Sparse Autoencoder
 * Contractive Autoencoder (CAE)
 * Variational Autoencoder (VAE)
 * Deep Neural Network (i.e. more than two hidden layers)
 * Deep Multi-Layer Perceptron
 * Deep Belief Network (DBN)
 * Convolutional Deep Neural Network
 * Long short-term memory (LSTM)
 * Deep Autoencoder (i.e. two symmetrical DBN)
 * Neural Module Network (NMN)
 * HyperNetwork
 * HyperLSTM
 * Training
 * Automatic Differentiation
 * Backpropagation Algorithm
 * Backpropagation Through Time (for training RNNs)
 * Stochastic Backpropagation
 * Optimization
 * Stochastic Gradient Methods
 * Stochastic Gradient Descent (SGD)
 * SGD with Momentum
 * Simulated Annealing
 * Genetic Algorithms (for training RNNs)
 * Contrastive Divergent (CD) Algorithm (for training RBMs)
 * Persistent CD (PCD)
 * Wake-Sleep Algorithm (for Stochastic ANNs)
 * Generative Stochastic Networks (GSN) for probabilistic models
 * Auto-Encoding Variational Bayes (AEVB) Algorithm
 * Activation Functions / Transfer Functions for deterministic units (must be differentiable)
 * Logistic
 * Rectifier (ReLU)
 * Softmax
 * Hyperbolic tangent
 * Swish
 * Cost Functions / Loss Functions / Objective Functions
 * Least-Squares
 * Cross-entropy
 * Relative Entropy / KL Divergence
 * Connectionist Temporal Classification (CTC)
 * Energy-Based Model (EBM)
 * Free energy (i.e. the contrastive term)
 * Regularization term
 * Loss Functionals / Loss Functions
 * Energy Loss
 * Generalized Perceptron Loss
 * Generalized Margin Losses
 * Negative Log-Likelihood Loss
 * Improve Generalization (to prevent overfitting)
 * Early stopping
 * Regularization / Weight decay
 * L1-regularization / Laplace prior
 * L2-regularization / Gaussian prior
 * Max norm constraints
 * Dropout
 * Mini-batch Learning
 * Ensemble Learning
 * Add noise
 * Theory of ANNs
 * Representation Theorem
 * Universal Approximation Theorem
 * Universal Turing Machine

Video Lectures

 * Neural Networks for Machine Learning by Geoffrey Hinton - Coursera
 * Neural networks class by Hugo Larochelle (Youtube )
 * Deep Learning and Neural Networks by Kevin Duh

Lecture Notes

 * Convolutional Neural Networks for Visual Recognition by Fei-Fei Li & Andrej Karpathy


 * CSC321: Introduction to Neural Networks and Machine Learning by Tijmen Tieleman - this might be a bit advanced for beginners
 * CSC2515: Introduction to Machine Learning by Geoffrey Hinton - this is very similar to the above
 * Cutting-Edge Trends in Deep Learning and Recognition by Svetlana Lazebnik
 * Neural Networks and Pattern Recognition by Ömer Cengiz ÇELEBİ

Books and Book Chapters
See Deep Learning Books.

Scholarly Articles

 * Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11).
 * Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
 * Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
 * Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation, 17(6), 1223-1263.
 * Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2451–2471.

Software
See Deep Learning Software.

General

 * artificial_neural_networks (Github)
 * What is the difference between CNNs, RBMs and Autoencoders - Stackexchange
 * Software Tools for RL, ANNs and Robotics - Python and MATLAB
 * Neural Networks and Deep Learning - free online book
 * Feedforward Neural Network - Metacademy
 * Backpropagation - Metacademy
 * Weight decay in Neural Networks - Metacademy
 * The Neural Network Zoo - Blog post with some of the ANN architectures
 * Neural Nets, Connectionism, Perceptrons, etc.  - Notebook
 * Neural Networks, Manifolds and Topology - Blog post
 * Bayesian Neural Network in PyMC3 (and Theano) - Blog post
 * Bayesian Neural Network in Edward (and Tensorflow) - Blog post
 * Hierarchical Bayesian ANN in PyMC3 (and Theano Lasagne) - Blog post
 * 10 Misconceptions about Neural Networks - Blog post
 * Artificial Neural Networks Explained - blog post
 * Pruning deep neural networks to make them fast and small - blog post
 * Neural Networks gone wild! They can sample from discrete distributions now! - blog post
 * keras-surgeon - Github
 * tensorpack - Github
 * Sequence Modeling With CTC - blog post
 * gumbel-softmax (Github) - code
 * bayesgan (Github) - code
 * Detectron - code
 * uTensor(Github) - AI inference library based on Mbed and TensorFlow
 * When is L2 weight decay better than L1 and when is max norm better? - Quora
 * CLR (GitHub) - code
 * pytorch-demos (GitHub) - code
 * Dropout in (Deep) Machine learning - blog post
 * Unsupervised Neural Networks Fight in a Minimax Game - blog post
 * memn2n (GitHub) - code
 * deepreplay (GitHub) - code
 * From Topological Data Analysis to Deep Learning: No Pain No Gain - blog post

TensorFlow

 * tensorflow-tutorial - Github
 * TensorFlow-Tutorials (Hvass-Labs) - Github
 * TensorFlow-Tutorials (nlintz) - Github
 * TensorFlow-Examples - Github
 * stanford-tensorflow-tutorials - Github
 * tensorlayer - Github
 * TensorFlow-Book - Github
 * Tensorflow-101 - Github
 * tensorflow-generative-model-collection - Github
 * caffe-tensorflow (GitHub) - code

RNN

 * Recurrent Neural Network - Metacademy
 * Advanced Recurrent Neural Networks - blog post
 * Recurrent Neural Networks for Language Modeling - blog post
 * Awesome-RNN (Github) - A curated list of resources dedicated to RNN
 * Predict Stock Prices Using RNN (Part 1, Part 2) - blog post
 * Introduction to LSTMs with TensorFlow - Blog post
 * Understanding LSTM Networks - Blog post
 * Understanding LSTM Networks by Example using Torch - Blog post
 * Time Series Forecasting with the Long Short-Term Memory Network in Python - blog post
 * How to Seed State for LSTMs for Time Series Forecasting in Python - blog post
 * Multivariate Time Series Forecasting with LSTMs in Keras - blog post
 * Time Series Prediction with LSTM Recurrent Neural Networks in Python with Keras - blog post
 * Multi-step Time Series Forecasting with Long Short-Term Memory Networks in Python - blog post
 * How to Develop Convolutional Neural Networks for Multi-Step Time Series Forecasting - blog post
 * How to Develop LSTM Models for Multi-Step Time Series Forecasting of Household Power Consumption - blog post
 * Unfolding RNNs (Part 1, Part 2) - blog post
 * LSTM implementation explained - blog post
 * Time Series Prediction Using LSTM Deep Neural Networks - blog post
 * Stock Market Predictions with LSTM in Python - blog post
 * Stock prediction LSTM using Keras (Kaggle)
 * Predict stock prices with LSTM (Kaggle)
 * New-York-Stock-Exchange-Predictions-RNN-LSTM (GitHub) - code
 * Vanilla Recurrent Neural Networks - blog post
 * How (not) to use Machine Learning for time series forecasting: Avoiding the pitfalls - blog post
 * Understanding Keras LSTMs - StackExchange
 * Recurrent Neural Networks in 10 minutes or less (slides, video)
 * Evolution: from vanilla RNN to GRU & LSTMs (slides, video)
 * Stateful LSTM in Keras - blog post