\[ \DeclareMathOperator{\E}{E} \DeclareMathOperator{\mean}{mean} \DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Cov}{Cov} \DeclareMathOperator{\Cor}{Cor} \DeclareMathOperator{\Bias}{Bias} \DeclareMathOperator{\MSE}{MSE} \DeclareMathOperator{\RMSE}{RMSE} \DeclareMathOperator{\sd}{sd} \DeclareMathOperator{\se}{se} \DeclareMathOperator{\tr}{tr} \DeclareMathOperator{\median}{median} \DeclareMathOperator{\rank}{rank} \DeclareMathOperator*{\argmin}{arg\,min} \DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator{\logistic}{Logistic} \DeclareMathOperator{\logit}{Logit} \newcommand{\mat}[1]{\boldsymbol{#1}} \newcommand{\vec}[1]{\boldsymbol{#1}} \newcommand{\T}{'} % This follows BDA \newcommand{\dunif}{\mathsf{Uniform}} \newcommand{\dnorm}{\mathsf{Normal}} \newcommand{\dhalfnorm}{\mathrm{HalfNormal}} \newcommand{\dlnorm}{\mathsf{LogNormal}} \newcommand{\dmvnorm}{\mathsf{Normal}} \newcommand{\dgamma}{\mathsf{Gamma}} \newcommand{\dinvgamma}{\mathsf{InvGamma}} \newcommand{\dchisq}{\mathsf{ChiSquared}} \newcommand{\dinvchisq}{\mathsf{InvChiSquared}} \newcommand{\dexp}{\mathsf{Exponential}} \newcommand{\dlaplace}{\mathsf{Laplace}} \newcommand{\dweibull}{\mathsf{Weibull}} \newcommand{\dwishart}{\mathsf{Wishart}} \newcommand{\dinvwishart}{\mathsf{InvWishart}} \newcommand{\dlkj}{\mathsf{LkjCorr}} \newcommand{\dt}{\mathsf{StudentT}} \newcommand{\dhalft}{\mathsf{HalfStudentT}} \newcommand{\dbeta}{\mathsf{Beta}} \newcommand{\ddirichlet}{\mathsf{Dirichlet}} \newcommand{\dlogistic}{\mathsf{Logistic}} \newcommand{\dllogistic}{\mathsf{LogLogistic}} \newcommand{\dpois}{\mathsf{Poisson}} \newcommand{\dBinom}{\mathsf{Binomial}} \newcommand{\dmultinom}{\mathsf{Multinom}} \newcommand{\dnbinom}{\mathsf{NegativeBinomial}} \newcommand{\dnbinomalt}{\mathsf{NegativeBinomial2}} \newcommand{\dbetabinom}{\mathsf{BetaBinomial}} \newcommand{\dcauchy}{\mathsf{Cauchy}} \newcommand{\dhalfcauchy}{\mathsf{HalfCauchy}} \newcommand{\dbernoulli}{\mathsf{Bernoulli}} \newcommand{\R}{\mathbb{R}} \newcommand{\Reals}{\R} \newcommand{\RealPos}{\R^{+}} \newcommand{\N}{\mathbb{N}} \newcommand{\Nats}{\N} \newcommand{\cia}{\perp\!\!\!\perp} \DeclareMathOperator*{\plim}{plim} \DeclareMathOperator{\invlogit}{Inv-Logit} \DeclareMathOperator{\logit}{Logit} \DeclareMathOperator{\diag}{diag} \]

22 Annotated Bibliography

This is less an annotated and more of a citation and link dump while I move the references into the main text.

22.1 Textbooks

  • Kruschke (2015) Doing Bayesian data analysis (Kruschke 2015) Another accessible introduction aimed at psychology. Website with additional material.

  • McElreath (2016) Statistical rethinking (McElreath 2016) An accessible introduction to Bayesian stats; effectively an intro-stats/linear models course taught from a Bayesian perspective.

  • Lee (2012) Bayesian Statistics : An Introduction (Lee 2012)

  • Marin and Robert (2015) Bayesian Essentials with R (Marin and Robert 2014) and solutions manual

  • Robert and Casella. 2009. Introducing Monte Carlo Methods with R (Robert and Casella 2009)

  • Robert and Casella. 2004. Monte Carlo statistical methods (Robert and Casella 2004)

  • Albert (2009) Bayesian Computation with R (Albert 2009)

  • Jackman (2009) Bayesian Analysis for the Social Sciences (Jackman 2009) Covers commonly used models in the social sciences. Largely covers Gibbs sampling methods and

  • Hoff (2009) A First Course in Bayesian Statistical Methods (Hoff 2009)

  • Gelman, Carlin, Stern, Dunson, and Vehtari (2013) Bayesian data analysis (3rd Edition) (A. Gelman, Carlin, et al. 2013)

  • Gelman, and Hill (2007) Data analysis using regression and multilevel/hierarchical models (A. Gelman and Hill 2007) An accessible introduction to to linear models and multilevel models.

  • Efron and Hastie (2016) Computer Age Statistical Inference: Algorithms, Evidence, and Data Science This is a unique work that blends an overview of statistical methods with a history of statistics. (Efron and Hastie 2016)

  • Robert (2007) The Bayesian Choice A statistics graduate-level book on Bayesian statistics.

  • Berger (1993) Statistical Decision Theory and Bayesian Analysis (Berger 1993) The classic book on Bayesian inference and decision theory. The underlying statistical theory is still relevant even if its date makes the computational aspects less so.

  • Murphy (2012) Machine Learning: A Probabilistic Perspective (Murphy 2012) A machine learning book with a heavy Bayesian influence.

  • MacKay (2003) Information Theory, Inference, and Learning Algorithms URL. (MacKay 2003) On information theory, but combines it with Bayesian statistics, and is ultimately about learning and evidence. Lectures from the course are available here.

  • Gelman and Hill (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models (A. Gelman and Hill 2007)

  • Gelman, Carlin, Stern, Dunson, Vehtari, and Rubin (2013) Bayesian Data Analysis 3rd ed.

  • Jackman, Simon. 2009. Bayesian Analysis for the Social Sciences (Jackman 2009)

  • Lynch, Scott M. 2007. Introduction to Applied Bayesian Statistics and Estimation for Social Scientists

  • Lunn, Jackson, Best, Thomas, and Spiegelhalter (2012) The BUGS Book: A Practical Introduction to Bayesian Analysis (Lunn et al. 2012)

  • Peter Hoff. 2009. A First Course in Bayesian Statistical Methods (Hoff 2009)

  • Congdon. 2014. Applied Bayesian Modeling.

  • Marin and Roberts. 2014. Bayesian Essentials with R.

  • Robert and Casella. Introducing Monte Carlo Methods with R (Robert and Casella 2009)

22.2 Syllabi

  • Ryan Bakker and Johannes Karreth, “Introduction to Applied Bayesian Modeling” ICPSR. Summer 2016. Syllabus; code
  • Justin Esarey. “Advanced Topics in Political Methodology: Bayesian Statistics” Winter 2015. Syllabus; Lectures.
  • Kruschke. Doing Bayesian Data Analysis site.
  • Nick Beauchamp. “Bayesian Methods.” NYU. syllabus.
  • Alex Tanhk. “Bayesian Methods for the Social Sciences” U of Wisconsin. Spring 2017. syllabus.
  • MTH225 Statistics for Science Spring 2016. github website.
  • Ben Goodrich, “Bayesian Statistics for Social Sciences” Columbia University. Spring 2016.
  • Bakker. “Introduction to Applied Bayesian Analysis” University of Georgia. syllabus; site
  • Myimoto. “Advances in Quantitative Psychology: Bayesian Statistics, Modeling & Reasoning” U of Washington. Winter 2017. site
  • Neil Frazer. Bayesian Data Analysis. Hawaii. Spring 2017. syllabus
  • Lopes. 2016. Bayesian Statistical Learning: Readings in Statistics and Econometrics. syllabus.
  • Lopes. 2012 Simulation-based approaches to modern Bayesian econometrics. Short course.
  • Lopes. 2015. Bayesian Econometrics. syllabus.

22.3 Topics

22.4 Bayes’ Theorem

22.5 Article Length Introductions to Bayesian Statistics

22.5.1 Why Bayesian

22.5.2 Modern Statistical Workflow

22.5.3 Bayesian Philosophy

22.5.4 Bayesian Hypothesis Testing

22.5.5 Bayesian Frequentist Debates

22.5.6 Categorical

  • Agresti. Bayesian Inference for Categorical Data Analysis
  • Gelman. 2008. “A weakly informative default prior distribution for logistic and other regression models”
  • Rainey. 2016. “Dealing with Separation in Logistic Regression Models” Political Analysis
  • Wechsler, Izbicki, and Esteves (2013) “A Bayesian look at nonidentifiability: a simple example”"

22.5.7 Variable Selection

22.5.8 Multiple Testing

22.5.9 Rare Events

22.5.10 Identifiability

22.5.11 Shrinkage

22.6 Software

Software for general purpose Bayesian computation are called probabilistic programming languages.

  • Stan

  • BUGS modeling language. Models are specified in a different language.

    • NIMBLE A very new BUGS-like language that works with R.

    • JAGS Gibbs/MCMC based

    • WinBUGS Gibbs and MCMC based software. It was one of the first but is now obsolete and unmaintained. Use JAGS or Stan instead.

    • OpenBUGS The continuation of the WinBUGS project. Also no longer well maintained. Use JAGS or Stan instead.

  • R has multiple packages that implement some Bayesian methods. See the Bayesian Task View

  • Python

    • PyMC Very complete general-purpose Python package for Bayesian Analysis
    • The various Machine learning packages like scikit-learn.
  • Edward. By David Blei. Deep generative models, variational inference. Runs on TensorFlow. Implements variational and HMC methods, as well as optimization.

  • Church and Anglican are Lisp-based inference programs.

  • Stata: Since version 14 it can estimate some Bayesian models. It uses Metropolis-Hastings and Gibbs methods.

  • Julia

    • Mamba MCMC supporting multiple methods including Gibbs, MH, HMC, slice

22.6.1 Stan

Official Stan-dev R packages:

Others:

  • brms Bayesian generalized non-linear multilevel models using Stan
  • ggmcmc

22.6.2 Diagrams

22.6.2.1 DAGs and Plate Notation

See Plate notation

22.6.2.3 Venn Diagrams/Eikosograms

  • Oldford and W.H. Cherry. 2006. “Picturing Probability: the poverty of Venn diagrams, the richness of Eikosograms”

22.6.3 Priors

22.7 Bayesian Model Averaging

22.8 Multilevel Modeling

22.9 Mixture Models

22.10 Inference

22.10.1 Discussion of Bayesian Inference

  • Lindley. The Analysis of Experimental Data: The Appreciation of Tea and Wine

22.11 Model Checking

22.11.1 Posterior Predictive Checks

22.11.2 Prediction Criteria

22.12 Hierarchical Modeling

22.13 Shrinkage/Regularization

22.14 Empirical Bayes

22.15 History of Bayesian Statistics

22.16 Sampling Difficulties

22.17 Complicated Estimation and Testing

22.18 Pooling Polls

22.20 Bayesian point estimation / Decision

22.21 Stan Modeling Language

  • Ch 1–8 Introduction.
  • pay attention to Ch 1, 8. skim the rest. know where to look for help.
  • Ch 28. Optimizing Stan Code for Efficiency (Neal’s funnel, reparameterization, vectorization)
  • Ch 22. Reparameterization and change of variables
  • Ch 23. Customized
  • Ch 24. User-defined functions
  • Ch 25. problematic posteriors
  • Ch 29. Bayesian Data Analysis
  • Ch 30. Markov Chain Monte Carlo Sampling (R hat, ESS, convergence, thinning)
  • Ch 31. Penalized MLE
  • Ch 32. Bayesian Point Estimation
  • Ch 34. Hamiltonian Monte Carlo Sampling
  • Ch 35. Transformations of Constrained Variables - changes of variables.

22.22 Bayes Factors