Editing Probability and Statistics

Jump to navigation Jump to search

Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.

The edit can be undone. Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.

Latest revision Your text
Line 4: Line 4:
  
 
More specific information is included in each subfield.
 
More specific information is included in each subfield.
 
A distinction should be made between ''Models'' and ''Methods'' (which might be applied on or using these Models).
 
  
 
==Subfields and Concepts==
 
==Subfields and Concepts==
Line 22: Line 20:
 
** Confidence Intervals
 
** Confidence Intervals
 
** Bootstrapping
 
** Bootstrapping
* Bayesian Inference
+
* [[Bayesian Machine Learning|Bayesian Inference]]
** Bayesian Testing: [[Bayesian Model Selection|Bayes Factor]]
+
** [[Bayesian Model Selection|Bayes Factor]]
** Bayesian Confidence Sets: Credible Intervals
+
** Credible Intervals
 +
** [[Variational Method|Variational Bayesian Inference]]
 +
** Empirical Bayes / Maximum Marginal Likelihood
 
** Hierarchical Bayes
 
** Hierarchical Bayes
** Empirical Bayes
+
** Bayesian Linear (Regression) Model
** Full Bayes
+
** [[Bayesian Nonparametrics|Bayesian Nonparametric Models]]
*  [[Bayesian Machine Learning|Computational Methods for Bayesian Inference]] (i.e. using [[Algorithm]]ic Methods)
 
** Exact Inference / Exact Marginalization
 
** Approximate Inference
 
*** Deterministic / Structural: [[Variational Method|Variational Bayesian Inference]] (as [[Optimization]])
 
*** Stochastic: [[Monte Carlo Method|Monte Carlo Inference / Sampling Inference / Particle-based Inference]]
 
*** Laplace Approximation
 
 
* Inductive inference
 
* Inductive inference
 
* [[Statistical Learning Theory|Empirical Inference]]
 
* [[Statistical Learning Theory|Empirical Inference]]
Line 39: Line 33:
 
* Interval Estimation
 
* Interval Estimation
 
* [[Statistical Signal Processing|Estimation Theory / Point Estimation]]
 
* [[Statistical Signal Processing|Estimation Theory / Point Estimation]]
* Sufficiency, Minimality, Completeness and Variance Reduction Techniques (VRT)
+
** Least Squares filters
** Gauss-Markov Theorem
+
** [[Kalman filter]]
** Lehmann–Scheffe Theorem
+
** Wiener filter
** Factorization Theorem
+
** [[Monte Carlo Method]]s
** Complete statistic
+
*** Particle filter
** Minimal sufficient statistic
+
** [[Mixture Model|Expectation-Maximization Algorithm]]
** Ancillary statistic
+
** Maximum Likelihood Estimator (MLE)
** Fisher information
+
** Maximum a posteriori (MAP) estimator
** Fisher information metric / Fisher–Rao metric
+
** [[Bayesian Parameter Estimation|Bayes Estimator]]
** Scoring algorithm / Fisher's scoring
+
*** Bayesian Decision Theory
** Score function
 
** Cramer–Rao bound (CRB) / Cramer–Rao lower bound (CRLB)
 
** Rao–Blackwell Theorem
 
*** Rao–Blackwellization
 
*** Rao–Blackwell estimator
 
** Exponential family
 
** Conjugate prior family
 
 
* Decision Theory
 
* Decision Theory
 
** Neyman-Pearson Theory
 
** Neyman-Pearson Theory
 
** The Expected Loss Principle
 
** The Expected Loss Principle
 
** Optimal decision rules
 
** Optimal decision rules
** Bayesian Decision Theory / Bayes estimator
+
** Bayesian Decision Theory / Bayesian Estimator
** [[Optimization|Cost function / Loss function]]
+
** Cost function / Loss function
 
** Risk function
 
** Risk function
 
** Admissibility
 
** Admissibility
Line 74: Line 61:
 
** Epicurus' Principle of Multiple Explanations
 
** Epicurus' Principle of Multiple Explanations
 
** Occam's Razor
 
** Occam's Razor
** Bayes' rule
+
** Bayes rule
 
** Minimum Description Length (MDL) principle
 
** Minimum Description Length (MDL) principle
 
** Minimum Message Length (MML)
 
** Minimum Message Length (MML)
** Algorithmic Statistics
 
 
* Model Selection and Evaluation
 
* Model Selection and Evaluation
 
** Akaike Information Criterion (AIC)
 
** Akaike Information Criterion (AIC)
Line 84: Line 70:
 
** Bayesian Predictive Information Criterion (BPIC)
 
** Bayesian Predictive Information Criterion (BPIC)
 
** Focused Information Criterion (FIC)
 
** Focused Information Criterion (FIC)
** Minimum Description Length (MDL)
+
** [[Bayesian Model Selection|Bayesian Model Selection / Bayesian Model Comparison]]
 +
*** Bayesian Model Averaging
 +
** [[Bayesian Parameter Estimation]]
 +
*** [[Bayesian Nonparametrics]]
 +
** Minimum Description Length (MDL) principle
 
** Minimum Message Length (MML)
 
** Minimum Message Length (MML)
 
** Akaike Final Prediction Error (FPE)
 
** Akaike Final Prediction Error (FPE)
 
** Parzen's Criterion Autoregressive Transfer Function (CAT)
 
** Parzen's Criterion Autoregressive Transfer Function (CAT)
** [[Bayesian Model Selection|Bayesian Model Selection / Bayesian Model Comparison]]
 
 
** Cross-Validation
 
** Cross-Validation
 
** Statistical Hypothesis Testing (for Multilevel Models / Nested Models only)
 
** Statistical Hypothesis Testing (for Multilevel Models / Nested Models only)
Line 94: Line 83:
 
*** Likelihood-ratio test
 
*** Likelihood-ratio test
 
*** Wald test
 
*** Wald test
** Model Evaluation Metrics (for Classification)
 
*** Confusion Matrix
 
*** Accuracy
 
*** F-measure / F1-score / F-score
 
*** Precision
 
*** Recall / Sensitivity / True Positive Rate
 
*** Specificity /  True Negative Rate
 
*** False Positive Rate
 
*** False Negative Rate
 
** Model Evaluation Metrics (for Regression)
 
*** Mean Square Error (MSE)
 
*** Root MSE (RMSE)
 
*** Mean Absolute Error (MAE)
 
*** R-Squared
 
  
 
===Statistical Models===
 
===Statistical Models===
Line 114: Line 89:
 
** Simple Linear Regression
 
** Simple Linear Regression
 
** Multiple Linear Regression ''(not to be confused with Multivariate Linear Regression)''
 
** Multiple Linear Regression ''(not to be confused with Multivariate Linear Regression)''
** General Linear Model / Multivariate Linear Model
+
** General Linear Model / Multivariate Linear Regression
 
** Generalized Linear Model (GLM or GLIM)
 
** Generalized Linear Model (GLM or GLIM)
 
** Poisson Regression
 
** Poisson Regression
** Negative Binomial Regression
+
** Least Squares Methods
 +
*** Ordinary Least Squares / Linear Least Squares
 +
*** Weighted Least Squares
 +
*** Nonlinear Least Squares
 
** Logistic Regression Model / Logit Model
 
** Logistic Regression Model / Logit Model
** Multinomial Logistic Regression / Softmax Regression
 
 
** Probit Model
 
** Probit Model
 
** Fixed Effects Model
 
** Fixed Effects Model
Line 126: Line 103:
 
*** Mixed Effects Models ''(not to be confused with Mixture Models)''
 
*** Mixed Effects Models ''(not to be confused with Mixture Models)''
 
** Nonparametric Regression Models
 
** Nonparametric Regression Models
** Semi-parametric Regression Models
 
 
** Nonlinear Regression Models
 
** Nonlinear Regression Models
 
** Robust Regression Models
 
** Robust Regression Models
 
** Random sample consensus (RANSAC)
 
** Random sample consensus (RANSAC)
** [[Optimization|Least Squares Methods]]
+
** [[Regularization]]
*** Ordinary Least Squares / Linear Least Squares
+
*** Ridge regression / Tikhonov regularization
*** Weighted Least Squares
+
*** Least absolute shrinkage and selection operator (LASSO)
*** Nonlinear Least Squares
+
*** Elastic Nets
*** L1-regularization / Least absolute shrinkage and selection operator (LASSO) / Laplace prior
 
*** L2-regularization / Ridge Regression / Tikhonov Regularization / Gaussian prior
 
 
* Probabilistic Models
 
* Probabilistic Models
 
** [[Stochastic Process|Stochastic Models (Stochastic Processes, Random Fields, ...)]]
 
** [[Stochastic Process|Stochastic Models (Stochastic Processes, Random Fields, ...)]]
Line 144: Line 118:
 
* [[Linear Dynamical System|State Space Models]]
 
* [[Linear Dynamical System|State Space Models]]
 
** Time Series Models
 
** Time Series Models
* Reliability Engineering / Reliability Modelling
 
** Survival Analysis
 
** Reliability Theory
 
** Risk Assessment
 
** Hazard Function
 
 
 
===Probability Theory===
 
===Probability Theory===
 
* Random Variables
 
* Random Variables
Line 172: Line 140:
 
* Theorem of Total Probability
 
* Theorem of Total Probability
 
* Central Limit Theorem
 
* Central Limit Theorem
* Conditional Probability
 
 
* Bayesian Probability Theory
 
* Bayesian Probability Theory
 
* Frequentist Probability Theory
 
* Frequentist Probability Theory
Line 193: Line 160:
 
===Lecture Notes===
 
===Lecture Notes===
 
*[http://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2005/lecture-notes/ Introduction to Probability and Statistics by Dmitry Panchenko]
 
*[http://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2005/lecture-notes/ Introduction to Probability and Statistics by Dmitry Panchenko]
*[https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/index.htm Introduction to Probability and Statistics by Jeremy Orloff and Jonathan Bloom]
 
 
*[http://faculty.washington.edu/ezivot/econ583/583syllabus.htm Economic Theory I by Eric Zivot]
 
*[http://faculty.washington.edu/ezivot/econ583/583syllabus.htm Economic Theory I by Eric Zivot]
 
*[https://www.bauer.uh.edu/rsusmel/phd/phdeconom1.htm Econometrics I by Rauli Susmel]
 
*[https://www.bauer.uh.edu/rsusmel/phd/phdeconom1.htm Econometrics I by Rauli Susmel]
Line 200: Line 166:
 
*[http://www.stats.ox.ac.uk/~berestyc/teaching/SB2a.html Foundations of Statistical Inference by Julien Berestycki]
 
*[http://www.stats.ox.ac.uk/~berestyc/teaching/SB2a.html Foundations of Statistical Inference by Julien Berestycki]
 
*[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.171.321&rep=rep1&type=pdf ETC5410: Nonparametric smoothing methods by Rob J Hyndman]
 
*[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.171.321&rep=rep1&type=pdf ETC5410: Nonparametric smoothing methods by Rob J Hyndman]
*[http://polisci.msu.edu/jacoby/icpsr/regress3/ Regression III: Advanced Methods by William Jacoby]
 
*[http://people.stat.sfu.ca/~lockhart/richard/830/13_3/ Statistical Theory I by Richard Lockhart]
 
*[http://content.csbs.utah.edu/~ehrbar/ecmet.pdf Class Notes in Statistics and Econometrics by Hans G. Ehrbar]
 
  
 
==Books==
 
==Books==
Line 217: Line 180:
 
* Ramachandran, K. M., & Tsokos, C. P. (2012). ''Mathematical Statistics with Applications in R''. Elsevier.
 
* Ramachandran, K. M., & Tsokos, C. P. (2012). ''Mathematical Statistics with Applications in R''. Elsevier.
 
* Liero, H., & Zwanzig, S. (2012). ''Introduction to the theory of statistical inference''. CRC Press.
 
* Liero, H., & Zwanzig, S. (2012). ''Introduction to the theory of statistical inference''. CRC Press.
* Wasserman, L. (2013). ''All of statistics: a concise course in statistical inference''. Springer Science & Business Media.
 
 
* Gentle, J. E. (2007). ''Matrix algebra: theory, computations, and applications in statistics''. Springer Science & Business Media.
 
* Gentle, J. E. (2007). ''Matrix algebra: theory, computations, and applications in statistics''. Springer Science & Business Media.
 
* Rice, J. (2006). ''Mathematical statistics and data analysis''. 3rd Ed. Duxbury Press.
 
* Rice, J. (2006). ''Mathematical statistics and data analysis''. 3rd Ed. Duxbury Press.
Line 231: Line 193:
 
* Schervish, M. J. (1995). ''Theory of statistics''. Springer Science & Business Media.
 
* Schervish, M. J. (1995). ''Theory of statistics''. Springer Science & Business Media.
  
=== Regression Analysis, Reliability and Generalized Linear Models ===
+
=== Regression Analysis and Generalized Linear Models ===
* Greene, W. H. (2018). ''Econometric analysis''. 8th Ed. Pearson.
+
* Greene, W. H. (2003). ''Econometric analysis''. Pearson.
 
* Harrell, F. (2015). ''Regression modeling strategies''. 2nd Ed. Springer.
 
* Harrell, F. (2015). ''Regression modeling strategies''. 2nd Ed. Springer.
 
* Kroese, D. P., & Chan, J. C. (2016). ''Statistical modeling and computation''. Springer.
 
* Kroese, D. P., & Chan, J. C. (2016). ''Statistical modeling and computation''. Springer.
 
* Chatterjee, S., & Hadi, A. S. (2012). ''Regression analysis by example''. 5th Ed. John Wiley & Sons.
 
* Chatterjee, S., & Hadi, A. S. (2012). ''Regression analysis by example''. 5th Ed. John Wiley & Sons.
* Kaminskiy, M. P. (2012). ''Reliability models for engineers and scientists''. CRC Press.
 
 
* Goldstein, H. (2010). ''Multilevel statistical models''. 4th Ed. John Wiley & Sons.
 
* Goldstein, H. (2010). ''Multilevel statistical models''. 4th Ed. John Wiley & Sons.
* Tobias, P. A., & Trindade, D. (2011). ''Applied reliability''. 3rd Ed. CRC Press.
 
 
* Freedman, D. A. (2009). ''Statistical models: theory and practice''. Cambridge University Press.
 
* Freedman, D. A. (2009). ''Statistical models: theory and practice''. Cambridge University Press.
 
* Dobson, A. J., & Barnett, A. (2008). ''An introduction to generalized linear models''. 3rd Ed. CRC press.
 
* Dobson, A. J., & Barnett, A. (2008). ''An introduction to generalized linear models''. 3rd Ed. CRC press.
Line 244: Line 204:
 
* Fox, J. (2008). ''Applied regression analysis and generalized linear models''. 2nd Ed. Sage Publications.
 
* Fox, J. (2008). ''Applied regression analysis and generalized linear models''. 2nd Ed. Sage Publications.
 
* Stapleton, J. H. (2007). ''Models for probability and statistical inference: theory and applications''. John Wiley & Sons.
 
* Stapleton, J. H. (2007). ''Models for probability and statistical inference: theory and applications''. John Wiley & Sons.
* Li, Q., & Racine, J. S. (2007). ''Nonparametric Econometrics: Theory and Practice''. Princeton University Press.
 
* Birolini, A. (2007). ''Reliability engineering: theory and practice''. 5th Ed. Springer.
 
 
* Gelman, A., & Hill, J. (2006). ''Data analysis using regression and multilevel/hierarchical models''. Cambridge University Press.
 
* Gelman, A., & Hill, J. (2006). ''Data analysis using regression and multilevel/hierarchical models''. Cambridge University Press.
 
* Faraway, J. J. (2005). ''Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models''. CRC press.
 
* Faraway, J. J. (2005). ''Extending the linear model with R: generalized linear, mixed effects and nonparametric regression models''. CRC press.
* Rausand, M., & Arnljot, H. A. (2004). ''System reliability theory: models, statistical methods, and applications''. John Wiley & Sons.
 
* Bazovsky, I. (2004). ''Reliability theory and practice''. Courier Corporation.
 
 
* Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). ''Semiparametric regression''. Cambridge University Press.
 
* Ruppert, D., Wand, M. P., & Carroll, R. J. (2003). ''Semiparametric regression''. Cambridge University Press.
 
* Faraway, J. J. (2002). Practical regression and ANOVA using R. ([http://www.ats.ucla.edu/stat/r/sk/books_pra.htm link])
 
* Faraway, J. J. (2002). Practical regression and ANOVA using R. ([http://www.ats.ucla.edu/stat/r/sk/books_pra.htm link])
* O'Connor, P., & Kleyner, A. (2002). ''Practical reliability engineering''. 4th Ed. John Wiley & Sons.
 
* Hayashi, F. (2000). ''Econometrics''. Princeton University Press.
 
* Elandt-Johnson, R. C., & Johnson, N. L. (1999). ''Survival models and data analysis''. John Wiley & Sons.
 
 
* Draper, N. R., & Smith, H. (1998). ''Applied regression analysis''. 3rd Ed. John Wiley & Sons.
 
* Draper, N. R., & Smith, H. (1998). ''Applied regression analysis''. 3rd Ed. John Wiley & Sons.
 
* Long, J. S., & Freese, J. (1997). ''Regression models for categorical dependent variables''. Sage Publications.
 
* Long, J. S., & Freese, J. (1997). ''Regression models for categorical dependent variables''. Sage Publications.
* Leemis, L. M. (1995). ''Reliability: probabilistic models and statistical methods''. Prentice Hall.
 
 
* McCullagh, P., & Nelder, J. A. (1989). ''Generalized linear models''. CRC press.
 
* McCullagh, P., & Nelder, J. A. (1989). ''Generalized linear models''. CRC press.
  
Line 265: Line 217:
 
* Hollos, S. & Hollos, J. R. (2013). ''Probability Problems and Solutions.'' Abrazol Publishing.
 
* Hollos, S. & Hollos, J. R. (2013). ''Probability Problems and Solutions.'' Abrazol Publishing.
 
* Patrick, D. (2007). ''Introduction to Counting and Probability''. 2nd Ed. AoPS Incorporated.
 
* Patrick, D. (2007). ''Introduction to Counting and Probability''. 2nd Ed. AoPS Incorporated.
 +
* Patrick, D. (2007). ''Intermediate Counting and Probability''. AoPS Incorporated.
 
* Hamming, R. W. (1993). ''The Art of Probability for Scientists and Engineers''. CRC Press.
 
* Hamming, R. W. (1993). ''The Art of Probability for Scientists and Engineers''. CRC Press.
  
Line 278: Line 231:
  
 
==See also==
 
==See also==
* [[Machine Learning]]
+
* [[Machine Learning|Computational Statistics / Machine Learning]]
 
* [[Statistical Learning Theory]]
 
* [[Statistical Learning Theory]]
 
* [[Statistical Signal Processing]]
 
* [[Statistical Signal Processing]]
 
* [[Information Theory]]
 
* [[Information Theory]]
 
* [[Optimization]]
 
* [[Optimization]]
* [[Computational Finance]]
 
 
* [[Combinatorics]]
 
* [[Combinatorics]]
 
* [[International Mathematical Olympiad]]
 
* [[International Mathematical Olympiad]]

Please note that all contributions to Ioannis Kourouklides are considered to be released under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) (see Ioannis Kourouklides:Copyrights for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource. Do not submit copyrighted work without permission!

Cancel Editing help (opens in new window)