11 Resistant: Outlier-resistant regression via the Student’s \(t\) distribution

Outlying data points can distort estimates of location, such as means or regression coefficients.9 Location estimates obtained via maximizing a iid normal likelihood over heavy tailed data will be sensitive to data in the tails (outliers). A popular alternative to normal errors in regression analyses is the Student’s \(t\) density, with an unknown degrees of freedom parameter. For low degrees of freedom, the Student’s \(t\) distribution has heavier tails than the normal, but tends to the normal as the degrees of freedom parameter increases. Treating the degrees of freedom parameter as an unknown parameter to be estimated thus provides a check on the appropriateness of the normal. By embedding a model with location parameters in the Student’s \(t\) density, we obtain outlier-resistant estimates of location parameters.

11.1 Data

This example uses data collected by Douglas Grob on incumbency advantage in American congressional elections, 1956-1994 (Jackman 2000).

The response variable is the proportion of the two-party vote won by the Democratic candidate in district \(i\) at election \(t\). Indicators for Democratic and Republican incumbency are the critical explanatory variables in the analysis. Coefficients on these indicators are regarded as estimates of incumbency advantage. A series of year-specific indicators (fixed effects) are also included in the specification.

\[ \begin{aligned}[t] y_i &\sim \mathsf{StudentT}(\nu, \mu_i, \sigma) \\ \mu_i &= \alpha + x_i \beta \end{aligned} \] The \(\alpha\), \(\beta\), and \(\sigma\) parameters are given weakly informative priors (assuming that all \(x\) are scaled to have mean 0, standard deviation 1): \[ \begin{aligned}[t] \alpha &\sim \mathsf{Normal}(\bar{y}, 10 s_y), \\ \beta &\sim \mathsf{Normal}(0, 2.5 s_y), \\ \sigma &\sim \mathsf{HalfCauchy}(0, 5 s_y) , \end{aligned} \] where \(\bar{y}\) is the mean of \(y\), and \(s_y\) is the standard deviation of \(y\). The degrees of freedom parameter in the Student’s \(t\) distribution is a parameter and given the weakly informative prior suggested by Juárez and Steel (2010), \[ \nu \sim \mathsf{Gamma}(2, 0.1) . \] The following code operationalizes this regression model. The conditional density of the vote proportions is \(t\), with unknown degrees of freedom, \(\nu.\)

  data {
  int N;
  vector[N] y;
  int K;
  matrix[N, K] X;
  int Y;
  int year[N];
  // priors
  real sigma_scale;
  vector[K] beta_loc;
  vector[K] beta_scale;
  real alpha_loc;
  real alpha_scale;
}
parameters {
  vector[Y] alpha;
  vector[K] beta;
  real nu;
  real sigma;
  real tau;
}
transformed parameters {
  vector[N] mu;
  for (i in 1:N) {
    mu[i] = alpha[year[i]] + X[i] * beta;
  }
}
model{
  // priors for error variance
  sigma ~ cauchy(0., sigma_scale);
  // priors for year intercepts
  alpha ~ normal(alpha_loc, alpha_scale);
    // priors for the regression coefficients
    beta ~ normal(beta_loc, beta_scale);
    // degrees of freedom
    nu ~ gamma(2, 0.1);
    // likelihood
    y ~ student_t(nu, mu, sigma);
}
generated quantities {
  real delta;
  delta = beta[3] + beta[4];
}

11.2 Reparameterization: standard deviation instead of scale

In the Student’s \(t\) distribution, the standard deviation is a function of the degrees of freedom. For degrees of freedom \(\nu > 2\), the variance is defined, and \[ \sigma^{*} = sd(y) = \sigma \sqrt{ \frac{\nu}{\nu - 2}} \] This makes the sampling of \(\nu\) and \(\sigma\) a priori dependent. Instead, we can place priors on the degrees of freedom \(\nu\) and the standard deviation \(\sigma^*\), and treat \(\sigma\) as a transformed parameter, \[ \begin{aligned} \sigma^{*} &\sim \mathsf{HalfCauchy}{(0, 5)} \\ \sigma &= \sigma^{*} \sqrt{\frac{\nu - 2}{\nu}} \\ \end{aligned} \]

  data {
  int N;
  vector[N] y;
  int K;
  matrix[N, K] X;
  int Y;
  int year[N];
  // priors
  real sigma_scale;
  vector[K] beta_loc;
  vector[K] beta_scale;
  real alpha_loc;
  real alpha_scale;
}
parameters {
  vector[Y] alpha;
  vector[K] beta;
  real nu;
  real sigma_raw;
  real tau;
}
transformed parameters {
  vector[N] mu;
  real sigma;
  for (i in 1:N) {
    mu[i] = alpha[year[i]] + X[i] * beta;
  }
  // paramterization so sigma and
  sigma = sigma_raw * sqrt((nu - 2) / nu);
}
model{
  // priors for the standard deviation
  sigma_raw ~ cauchy(0., sigma_scale);
  // priors for year intercepts
  alpha ~ normal(alpha_loc, alpha_scale);
    // priors for the regression coefficients
    beta ~ normal(beta_loc, beta_scale);
    // degrees of freedom
    nu ~ gamma(2, 0.1);
    // likelihood
    y ~ student_t(nu, mu, sigma);
}
generated quantities {
  real delta;
  delta = beta[3] + beta[4];
}

Questions

  1. How does using the Student-t distribution compare to using a normal distribution for the errors?
  2. Compare the effective sample size, \(\hat{R}\), and speed for \(\sigma\) and \(\nu\) when using the scale/degrees of freedom and standard deviation/degrees of freedom parameterizations.

References

Jackman, Simon. 2000. “Estimation and Inference Are Missing Data Problems: Unifying Social Science Statistics via Bayesian Simulation.” Political Analysis 8 (4). [Oxford University Press, Society for Political Methodology]: 307–32. http://www.jstor.org/stable/25791616.

Juárez, Miguel A., and Mark F. J. Steel. 2010. “Model-Based Clustering of Non-Gaussian Panel Data Based on Skew-t Distributions.” Journal of Business & Economic Statistics 28 (1). Informa UK Limited: 52–66. https://doi.org/10.1198/jbes.2009.07145.

  1. Example derived from Simon Jackman, “Resistant: Outlier-resistant regression via the t distribution,” 2007-07-24, URL.↩