Chapter 15 Assumptions in linear models

Before proceeding, we should talk about assumptions in linear models. And before that, we should clarify what we mean by linear models.

Linear models describe the response variable (the outcome of interest) as a sum of effects of one or more explanatory variables (factors that may influence the response), plus random variation. Linear models encompass various techniques, including t-tests, linear regression, analysis of variance (ANOVA), and analysis of covariance (ANCOVA).

All these techniques fall under the umbrella of linear models because the model is linear in its parameters: the predicted value is made by multiplying each explanatory variable by a coefficient and adding the results together (e.g. intercept + slope × x). For a t-test or ANOVA, the “explanatory variable” simply codes which group an observation belongs to, so the linearity is a little hard to visualise; for linear regression it is easy to see as a straight line.

Note that “linear in its parameters” does not mean that the relationship must be a straight line. A model that includes a squared term (intercept + b1 × x + b2 × x2), or that uses a transformed variable such as log(x), is still a linear model, even though it describes a curve. This is how linear models can handle many non-linear relationships.

In a nutshell, all the models we use in this course are linear models. Generalised Linear Models (GLMs), which we get to later, relax some of these assumptions. They keep the linear part of the model (intercept + slope × x and so on), but connect it to the mean of the response through a link function, and they generalise the model by allowing the errors to follow distributions other than the normal (e.g. Poisson for counts, binomial for proportions). With those distributions the variance is allowed to change with the mean, so GLMs relax the assumptions of normality and homoscedasticity (more on this later).

15.1 The assumptions

The assumptions of linear models are outlined below. The relevance of each assumption depends on the type of model. For example, the assumption of linearity is not important in a t-test. I should also emphasise that some assumptions are more important than others and that models/tests are quite robust to minor-to-moderate violations of some assumptions. Roughly in order of importance:

  1. The data can answer the question (the response measures what you are interested in, and the sample represents the population you want to generalise to). No statistical method can fix this.
  2. Independence, because violating it makes you far too confident in your results.
  3. Linearity (where there are continuous explanatory variables).
  4. Homoscedasticity (equal variance).
  5. Normality of the residuals, usually the least important, especially with larger samples.

Linearity: The assumption of linearity states that the relationship between the response variable and continuous explanatory variable(s) should be linear. This means that the effect of continuous explanatory variables on the response variable is additive and constant across all levels of the explanatory variable. For practical purposes, the assumption of linearity is only relevant when there is a continuous explanatory variable.

Independence: The observations in the dataset are assumed to be independent of each other. Common scenarios where this assumption may be violated are:

  1. Spatial clustering: Observations that are spatially close to each other may exhibit similarities. For example, measurements taken at nearby locations might have similar characteristics. In such cases, the assumption of independence is violated, and specialised spatial regression techniques may be required.

  2. Repeated measurements: When multiple measurements are collected from the same individual or unit, such as repeated measurements over time, the observations are likely to be correlated. For instance, if multiple blood pressure measurements are collected from each patient over time, measurements within each patient are expected to be more similar than measurements between different patients.

  3. Relatedness: When measurements are collected from individuals that are related via a pedigree (family tree) or phylogeny, the observations from closely-related individuals are likely to be more similar than those from distantly-related individuals.

  4. Time: Measurements taken close together in time (for example, on consecutive days) are often more similar than measurements taken far apart.

  5. Pseudoreplication: This is treating measurements as independent replicates when they are not, because they share the same experimental unit. For example, if a treatment is applied to whole fish tanks, then fish in the same tank are not independent replicates of the treatment: the tank is the unit of replication. Similarly, several measurements from the same nest box, or several nests in the same year, may share conditions that make them similar. Pseudoreplication makes the sample size look larger than it really is, and so makes p-values too small.

Homoscedasticity: The assumption of homoscedasticity (or homogeneity of variances) states that the variability of the residuals (the spread of the data around the fitted values) should be constant across all levels of the explanatory variables. In a linear regression, you can picture this as the cloud of data points being evenly dispersed like a ribbon of constant width running along the fitted line of the regression model. A common violation is a “funnel” shape, where the spread increases as the fitted values increase. For a t-test this assumption means that we expect the spread of data to be the same for both group 1 and group 2 (note that Welch’s t-test used by default in R with t.test relaxes this assumption).

Normality: The residuals of the model are assumed to follow a normal distribution with a mean of zero. Note that it is the residuals that are expected to be normally distributed, not the actual data. Estimates of means and differences are fairly robust to moderate non-normality, especially with larger samples.

15.2 Also check for

Two further problems are not strictly assumptions of the model, but can make the results misleading, so they are worth checking:

Multicollinearity: This occurs when explanatory variables are highly correlated with each other. It does not bias the estimates, but it makes them very imprecise (large standard errors) and hard to interpret, because the model cannot tell which of the correlated variables is responsible for an effect. It is desirable to have explanatory variables that are moderately or weakly correlated with each other. This is normally only an issue for multiple regression models.

Influential outliers: Outliers are extreme values. They are a problem when they are also influential, meaning that they have a large effect on the estimated coefficients, so that the results depend heavily on one or a few observations. Check whether such values are errors; if they are not, report the results both with and without them rather than simply deleting them.

15.3 How to check the assumptions

Most of these assumptions are checked after fitting the model, by looking at diagnostic plots of the residuals. You will meet these in the ANOVA chapter, where the autoplot() function from the ggfortify package draws them in one go:

  • Residuals vs Fitted: checks linearity (no curve in the points) and equal variance (no funnel shape).
  • Normal Q-Q: checks normality (points close to the straight line).
  • Scale-Location: another check of equal variance (no trend in the spread).
  • Residuals vs Leverage: identifies influential observations.

Independence cannot be checked from these plots. It has to be judged from how the data were collected, which is why the study design matters so much.

Finally, it is important to recognise that no model is perfect and that violations of assumptions are common in statistical analyses. While we strive to develop valid models, real-world data can deviate from idealised assumptions. This should not discourage us from using models; rather, it emphasises the need to fit the best model possible given the available data and circumstances. By carefully considering the assumptions and limitations of the chosen model, we can make informed decisions about its applicability and potential biases.

Moreover, it is critical to consider assumptions at the design stage of a study, because this allows for the collection of appropriate data and the implementation of strategies to minimise violations. By being mindful of the assumptions and continuously evaluating the model’s performance, we can enhance the reliability and validity of our analyses and draw meaningful conclusions from the data at hand.

Try it: Regression Diagnostics Lab (open the lab)

Break the assumptions one at a time.

  • Before you tick a violation, predict which of the four diagnostic panels will show it. Then check.
  • Which violation changes the estimated slope, and which mainly changes how uncertain it is?
  • With outliers switched on, are the outliers also influential (Cook’s distance)? Why not always?
Then try the same idea in R.