Artificial Neural Network

From Ioannis Kourouklides
Jump to navigation Jump to search

This page contains resources about Artificial Neural Networks.

For temporal (Time Series) and atemporal Sequential Data, please check Linear Dynamical Systems.


Subfields and Concepts[edit]

  • Feedforward Neural Network (FNN)
    • Single-Layer Perceptron (i.e. with no hidden layers)
    • Multi-Layer Perceptron (MLP) / Standard Neural Network (SNN)
    • Radial Basis Function (RBF) Network
    • Extreme Learning Machine (ELM)
    • Convolutional Neural Network (CNN or ConvNet)
    • Capsule Network (CapsNet)
  • Recurrent Neural Network (RNN)
    • Hopfield Network
    • Boltzmann Machine
    • Bidirectional RNN
    • Bidirectional associative memory (BAM)
    • Long short-term memory (LSTM)
    • Memory Network
    • Gated Rectified Unit RNN (GRU-RNN)
    • Vanilla Recurrent Network
    • Continuous Time RNN (CTRNN)
    • RNN-RBM
    • Echo State Network (ESN)
    • Unitary RNN (uRNN)
  • Stochastic Neural Network (i.e. with stochastic transfer function and units or stochastic weights)
    • Helmholtz Machine
    • Boltzmann Machine
    • Restricted Boltzmann Machine (RBM)
    • Conditional RBM (CRBM)
    • Autoassociative memory
    • Generative Stochastic Network
    • Generative Adversarial Network
    • Stochastic Feedforward Neural Network (with both stochastic and deterministic hidden units)
    • Stochastic Computation Graph
    • Variational Autoencoder (VAE)
    • Natural-Parameter Network
    • Variance Network
  • Kohonen Network / Self-organizing map (SOM) / Self-organising feature map (SOFM)
  • Probabilistic Nerual Network
    • Bayesian Neural Network (i.e. a Gaussian Process with finitely many weights)
      • Probabilistic Backpropagation
      • Bayes by Backprop
    • Bayesian Dark Knowledge (BDK)
    • Natural-Parameter Network (NPN) (i.e. distributions for both the weights and the neurons)
      • Gamma NPN
      • Gaussian NPN
      • Poisson NPN
  • Random Neural Network
  • Autoencoder (used for Dimensionality Reduction)
    • Linear Autoencoder (equivalent to PCA)
    • Stacked Denoising Autoencoder
    • Generalized Denoising Autoencoder
    • Sparse Autoencoder
    • Contractive Autoencoder (CAE)
    • Variational Autoencoder (VAE)
  • Deep Neural Network (i.e. more than two hidden layers)
    • Deep Multi-Layer Perceptron
    • Deep Belief Network (DBN)
    • Convolutional Deep Neural Network
    • Long short-term memory (LSTM)
    • Deep Autoencoder (i.e. two symmetrical DBN)
    • Neural Module Network (NMN)
  • HyperNetwork
    • HyperLSTM
  • Training
    • Automatic Differentiation
      • Backpropagation Algorithm
      • Backpropagation Through Time (for training RNNs)
      • Stochastic Backpropagation
    • Optimization
    • Contrastive Divergent (CD) Algorithm (for training RBMs)
      • Persistent CD (PCD)
    • Wake-Sleep Algorithm (for Stochastic ANNs)
    • Generative Stochastic Networks (GSN) for probabilistic models
    • Auto-Encoding Variational Bayes (AEVB) Algorithm
  • Activation Functions / Transfer Functions for deterministic units (must be differentiable)
    • Logistic
    • Rectifier (ReLU)
    • Softmax
    • Hyperbolic tangent
    • Swish
  • Cost Functions / Loss Functions / Objective Functions
    • Least-Squares
    • Cross-entropy
    • Relative Entropy / KL Divergence
    • Connectionist Temporal Classification (CTC)
  • Energy-Based Model (EBM)
    • Free energy (i.e. the contrastive term)
    • Regularization term
    • Loss Functionals / Loss Functions
      • Energy Loss
      • Generalized Perceptron Loss
      • Generalized Margin Losses
      • Negative Log-Likelihood Loss
  • Improve Generalization (to prevent overfitting)
    • Early stopping
    • Regularization / Weight decay
      • L1-regularization / Laplace prior
      • L2-regularization / Gaussian prior
      • Max norm constraints
    • Dropout
    • Mini-batch Learning
    • Ensemble Learning
    • Add noise
  • Theory of ANNs
    • Representation Theorem
    • Universal Approximation Theorem
    • Universal Turing Machine

Online Courses[edit]

Video Lectures[edit]


Lecture Notes[edit]

Books and Book Chapters[edit]

See Deep Learning Books.

Scholarly Articles[edit]

  • Hannun, A. (2017). Sequence Modeling with CTC. Distill, 2(11).
  • Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2015). Automatic differentiation in machine learning: a survey. arXiv preprint arXiv:1502.05767.
  • Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
  • Jacobsson, H. (2005). Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation17(6), 1223-1263.
  • Gers, F. A., Schmidhuber, J., & Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation, 12(10), 2451–2471.

Tutorials[edit]

Software[edit]

See Deep Learning Software.

See also[edit]

Other Resources[edit]

General[edit]

TensorFlow[edit]

RNN[edit]