Probability and Statistics



This page contains resources about Probability Theory and Statistics in general.

More specific information is included in each subfield.

A distinction should be made between Models and Methods (which might be applied on or using these Models).

Subfields and Concepts
See Category:Probability and Statistics for all its subfields.

Statistical Inference / Inferential Statistics

 * Frequentist Inference
 * Statistical Hypothesis Testing / Statistical Tests
 * Fisher's Null Hypothesis Testing
 * Neyman-Pearson Theory
 * Analysis of Variance (ANOVA)
 * Analysis of Covariance (ANCOVA)
 * Multivariate Analysis of Variance (MANOVA)
 * T-test
 * F-test
 * Tests of Goodness-of-Fit
 * Confidence Intervals
 * Bootstrapping
 * Bayesian Inference
 * Bayesian Testing: Bayes Factor
 * Bayesian Confidence Sets: Credible Intervals
 * Hierarchical Bayes
 * Empirical Bayes
 * Full Bayes
 * Computational Methods for Bayesian Inference (i.e. using Algorithmic Methods)
 * Exact Inference / Exact Marginalization
 * Approximate Inference
 * Deterministic / Structural: Variational Bayesian Inference (as Optimization)
 * Stochastic: Monte Carlo Inference / Sampling Inference / Particle-based Inference
 * Laplace Approximation
 * Inductive inference
 * Empirical Inference
 * Causal Inference
 * Interval Estimation
 * Estimation Theory / Point Estimation
 * Sufficiency, Minimality, Completeness and Variance Reduction Techniques (VRT)
 * Gauss-Markov Theorem
 * Lehmann–Scheffe Theorem
 * Factorization Theorem
 * Complete statistic
 * Minimal sufficient statistic
 * Ancillary statistic
 * Fisher information
 * Fisher information metric / Fisher–Rao metric
 * Scoring algorithm / Fisher's scoring
 * Score function
 * Cramer–Rao bound (CRB) / Cramer–Rao lower bound (CRLB)
 * Rao–Blackwell Theorem
 * Rao–Blackwellization
 * Rao–Blackwell estimator
 * Exponential family
 * Conjugate prior family
 * Decision Theory
 * Neyman-Pearson Theory
 * The Expected Loss Principle
 * Optimal decision rules
 * Bayesian Decision Theory / Bayes estimator
 * Cost function / Loss function
 * Risk function
 * Admissibility
 * Unbiasedness
 * Minimaxity
 * Algorithmic Information Theory
 * Kolmogorov Complexity / Algorithmic Complexity
 * Algorithmic Probability / Solomonoff Probability
 * Universal Search (by Levin)
 * Algorithmic Randomness (by Martin-Lof)
 * Solomonoff's Theory of Inductive Inference
 * Epicurus' Principle of Multiple Explanations
 * Occam's Razor
 * Bayes' rule
 * Minimum Description Length (MDL) principle
 * Minimum Message Length (MML)
 * Algorithmic Statistics
 * Model Selection and Evaluation
 * Akaike Information Criterion (AIC)
 * Bayesian Information Criterion (BIC)
 * Deviance Information Criterion (DIC)
 * Bayesian Predictive Information Criterion (BPIC)
 * Focused Information Criterion (FIC)
 * Minimum Description Length (MDL)
 * Minimum Message Length (MML)
 * Akaike Final Prediction Error (FPE)
 * Parzen's Criterion Autoregressive Transfer Function (CAT)
 * Bayesian Model Selection / Bayesian Model Comparison
 * Cross-Validation
 * Statistical Hypothesis Testing (for Multilevel Models / Nested Models only)
 * Lagrange multiplier test / Score test / Score Method
 * Likelihood-ratio test
 * Wald test
 * Model Evaluation Metrics (for Classification)
 * Confusion Matrix
 * Accuracy
 * F-measure / F1-score / F-score
 * Precision
 * Recall / Sensitivity / True Positive Rate
 * Specificity / True Negative Rate
 * False Positive Rate
 * False Negative Rate
 * Model Evaluation Metrics (for Regression)
 * Mean Square Error (MSE)
 * Root MSE (RMSE)
 * Mean Absolute Error (MAE)
 * R-Squared

Statistical Models

 * Regression Analysis
 * Linear Regression Model
 * Simple Linear Regression
 * Multiple Linear Regression (not to be confused with Multivariate Linear Regression)
 * General Linear Model / Multivariate Linear Model
 * Generalized Linear Model (GLM or GLIM)
 * Poisson Regression
 * Negative Binomial Regression
 * Logistic Regression Model / Logit Model
 * Multinomial Logistic Regression / Softmax Regression
 * Probit Model
 * Fixed Effects Model
 * Hierarchical Linear Models / Multilevel Models / Nested Data Models
 * Random Effects Model / Variance Components Model
 * Mixed Effects Models (not to be confused with Mixture Models)
 * Nonparametric Regression Models
 * Semi-parametric Regression Models
 * Nonlinear Regression Models
 * Robust Regression Models
 * Random sample consensus (RANSAC)
 * Least Squares Methods
 * Ordinary Least Squares / Linear Least Squares
 * Weighted Least Squares
 * Nonlinear Least Squares
 * L1-regularization / Least absolute shrinkage and selection operator (LASSO) / Laplace prior
 * L2-regularization / Ridge Regression / Tikhonov Regularization / Gaussian prior
 * Probabilistic Models
 * Stochastic Models (Stochastic Processes, Random Fields, ...)
 * Probabilistic Graphical Models
 * Latent Variable Models (i.e. Partially Observed Probabilistic Models)
 * Continuous Latent Variable Models
 * Discrete Latent Variable Models
 * State Space Models
 * Time Series Models
 * Reliability Engineering / Reliability Modelling
 * Survival Analysis
 * Reliability Theory
 * Risk Assessment
 * Hazard Function

Probability Theory

 * Random Variables
 * Continuous Random Variables
 * Probability Density Function
 * Discrete Random Variables
 * Probability Mass Function
 * Jointly Distributed Random Variables
 * Joint Density Function
 * Independent Random Variables
 * Uncorrelated Random Variables
 * Moments of a distribution
 * First Moment / Mean
 * Second Moment / Variance
 * Third Moment / Skewness
 * Fourth Moment / Kurtosis
 * Probabilistic Models
 * Stochastic Convergence
 * Probability Space
 * Measure Space
 * State Space
 * Theorem of Total Probability
 * Central Limit Theorem
 * Conditional Probability
 * Bayesian Probability Theory
 * Frequentist Probability Theory
 * Queueing Theory
 * Martingale Theory
 * Ergodic Theory
 * Decision Theory
 * Measure Theory
 * Utility Theory

Video Lectures

 * Probabilistic Systems Analysis and Applied Probability by John Tsitsiklis
 * Introduction to Probability - The Science of Uncertainty by edX  - very similar to the above
 * Probability by Salman Khan
 * Statistics by Salman Khan
 * Combinations - Counting Using Combinations

Lecture Notes

 * Introduction to Probability and Statistics by Dmitry Panchenko
 * Introduction to Probability and Statistics by Jeremy Orloff and Jonathan Bloom
 * Economic Theory I by Eric Zivot
 * Econometrics I by Rauli Susmel
 * AMS-310: Survey of Probability and Statistics by Xiaolin Li
 * Advanced Statistical Inference by Suhasini Subba Rao
 * Foundations of Statistical Inference by Julien Berestycki
 * ETC5410: Nonparametric smoothing methods by Rob J Hyndman
 * Regression III: Advanced Methods by William Jacoby
 * Statistical Theory I by Richard Lockhart
 * Class Notes in Statistics and Econometrics by Hans G. Ehrbar

Statistical Inference and Theory of Statistics

 * Bruce, P., & Bruce, A. (2017). Practical Statistics for Data Scientists: 50 Essential Concepts. O'Reilly Media.
 * Imbens, G. W., & Rubin D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction.
 * Ross, S. M. (2014). Introduction to probability models. 11th Ed. Academic Press.
 * Smith, R. C. (2013). Uncertainty quantification: theory, implementation, and applications. SIAM.
 * Gentle, J. E. (2013). Theory of statistics. (link)
 * DeGroot, M. H., & Schervish, M. J. (2012). Probability and statistics. 4th Ed. Pearson.
 * Abu-Mostafa, Y. S., Magdon-Ismail, M., & Lin, H. T. (2012). Learning From Data. AMLBook.
 * Diez, D. M., Barr, C. D., & Cetinkaya-Rundel, M. (2012). OpenIntro Statistics. CreateSpace.
 * Ramachandran, K. M., & Tsokos, C. P. (2012). Mathematical Statistics with Applications in R. Elsevier.
 * Liero, H., & Zwanzig, S. (2012). Introduction to the theory of statistical inference. CRC Press.
 * Wasserman, L. (2013). All of statistics: a concise course in statistical inference. Springer Science & Business Media.
 * Gentle, J. E. (2007). Matrix algebra: theory, computations, and applications in statistics. Springer Science & Business Media.
 * Rice, J. (2006). Mathematical statistics and data analysis. 3rd Ed. Duxbury Press.
 * Cox, D. R. (2006). Principles of statistical inference. Cambridge University Press.
 * Lavine, M. (2005). Introduction to Statistical Thought. Michael Lavine.
 * Young, G. A., & Smith, R. L. (2005). Essentials of statistical inference. Cambridge University Press.
 * Lehmann, E. L., & Casella, G. (2003). Theory of point estimation. Springer.
 * Bertsekas, D. P., & Tsitsiklis, J. N. (2002). Introduction to Probability. Athena scientific.
 * Casella, G., & Berger, R. L. (2002). Statistical inference. Cengage Learning.
 * Garthwaite, P. H., Jolliffe, I. T., & Jones, B. (2002). Statistical inference. Oxford University Press.
 * Shao, J. (2000). Mathematical Statistics. Springer.
 * Mukhopadhyay, N. (2000). Probability and statistical inference. CRC Press.
 * Schervish, M. J. (1995). Theory of statistics. Springer Science & Business Media.

Regression Analysis, Reliability and Generalized Linear Models

 * Greene, W. H. (2018). Econometric analysis. 8th Ed. Pearson.
 * Harrell, F. (2015). Regression modeling strategies. 2nd Ed. Springer.
 * Kroese, D. P., & Chan, J. C. (2016). Statistical modeling and computation. Springer.
 * Chatterjee, S., & Hadi, A. S. (2012). Regression analysis by example. 5th Ed. John Wiley & Sons.
 * Kaminskiy, M. P. (2012). Reliability models for engineers and scientists. CRC Press.
 * Goldstein, H. (2010). Multilevel statistical models. 4th Ed. John Wiley & Sons.
 * Tobias, P. A., & Trindade, D. (2011). Applied reliability. 3rd Ed. CRC Press.
 * Freedman, D. A. (2009). Statistical models: theory and practice. Cambridge University Press.
 * Dobson, A. J., & Barnett, A. (2008). An introduction to generalized linear models. 3rd Ed. CRC press.
 * Davison, A. C. (2003). Statistical models. Cambridge University Press.
 * Fox, J. (2008). Applied regression analysis and generalized linear models. 2nd Ed. Sage Publications.
 * Stapleton, J. H. (2007). Models for probability and statistical inference: theory and applications. John Wiley & Sons.
 * Li, Q., & Racine, J. S. (2007). Nonparametric Econometrics: Theory and Practice. Princeton University Press.
 * Birolini, A. (2007). Reliability engineering: theory and practice. 5th Ed. Springer.
 * Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press.
 * Faraway, J. J. (2005). Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models. CRC press.
 * Rausand, M., & Arnljot, H. A. (2004). System reliability theory: models, statistical methods, and applications. John Wiley & Sons.
 * Bazovsky, I. (2004). Reliability theory and practice. Courier Corporation.
 * Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). Semiparametric regression. Cambridge University Press.
 * Faraway, J. J. (2002). Practical regression and ANOVA using R. (link)
 * O'Connor, P., & Kleyner, A. (2002). Practical reliability engineering. 4th Ed. John Wiley & Sons.
 * Hayashi, F. (2000). Econometrics. Princeton University Press.
 * Elandt-Johnson, R. C., & Johnson, N. L. (1999). Survival models and data analysis. John Wiley & Sons.
 * Draper, N. R., & Smith, H. (1998). Applied regression analysis. 3rd Ed. John Wiley & Sons.
 * Long, J. S., & Freese, J. (1997). Regression models for categorical dependent variables. Sage Publications.
 * Leemis, L. M. (1995). Reliability: probabilistic models and statistical methods. Prentice Hall.
 * McCullagh, P., & Nelder, J. A. (1989). Generalized linear models. CRC press.

Counting and Probability

 * Shu, Z. (2016). Probability and Expectation (Volume 14). World Scientific
 * Zhou, X. (2015). Counting: Math for Gifted Students. CreateSpace.
 * Hollos, S. & Hollos, J. R. (2013). Probability Problems and Solutions. Abrazol Publishing.
 * Patrick, D. (2007). Introduction to Counting and Probability. 2nd Ed. AoPS Incorporated.
 * Hamming, R. W. (1993). The Art of Probability for Scientists and Engineers. CRC Press.

Software
See List of Statistical packages for a complete list.
 * The Lightspeed Matlab Toolbox
 * Statistics and Machine Learning Toolbox - MATLAB
 * Statistical functions (scipy.stats) - Python
 * Statistics (numpy) - Python
 * Statsmodels - Statistical Modeling and Econometrics in Python
 * revrand - Python
 * RandLib - C++

Other Resources

 * Probability and Statistics - Google Scholar Metrics (Top Publications)
 * Statistics - Nature
 * Video Tutorials - Youtube channel of 'Mathematical Monk'
 * Probability and Statistics by Khan Academy
 * Statistics by Wikibooks
 * Statistics by Wikiversity
 * Statistics - Notebook
 * Probability Theory - Notebook
 * Algorithmic Information Theory - Notebook
 * Bayesian statistics: a comprehensive course by Ox Educ - Youtube
 * Random - Lessons in Probability, Mathematical Statistics and Stochastic Processes
 * Learning Machine Learning — Probability Theory Fundamentals - Medium
 * Nuances of probability theory - Tom Minka