Two-way ANOVA: interaction, main effects, post hoc

A two-way ANOVA compares group means when the groups are defined by two categorical factors at once, for example watering regime and fertiliser, or drug and sex. Fitting it is easy. Reading it correctly is where people go wrong, because the output hands you three p-values and tempts you to start with the first one.

The default order is the opposite: look at the interaction first. When the interaction is material, report simple effects, meaning the effect of one factor at each level of the other, rather than main effects averaged over a factor that changes them. Post hoc testing comes last, and in a factorial design it has a specific meaning that differs from the one-way case.

What a two-way ANOVA tests

The model has two factors, call them A and B, and tests three null hypotheses at once:

• Main effect of A: the mean outcome is the same across the levels of A, averaged over B.
• Main effect of B: the same, for B averaged over A.
• Interaction A x B: the effect of A is the same at every level of B (equivalently, the effect of B is the same at every level of A).

The assumptions are those of the one-way ANOVA, applied to the cells of the design: independent observations, roughly normal residuals, and similar variance in every cell, where a cell is one combination of the two factors. For the one-way case and the choice between ANOVA, Welch’s version and a rank test, see one-way ANOVA, Welch ANOVA or Kruskal-Wallis.

Main effects and interactions

Suppose the interaction is absent. Then the factors act separately, each main effect has a plain meaning (“fertiliser B raises growth by this much, regardless of watering”), and you can report the main effects and stop.

Suppose the interaction is present. Now the effect of one factor depends on the level of the other, and a main effect is an average over levels where the effect differs, possibly in sign. The average can be near zero while the effect within each level is large. Reporting that main effect says nothing true about any actual condition. Rosnow and Rosenthal (1989) discuss this pattern at length and why interactions need to be read from the cell means rather than from the p-values alone.

The remedy is to describe simple effects: the effect of fertiliser at low watering, the effect of fertiliser at high watering, and so on. This is how a factorial design is meant to be read once the interaction is real.

Unbalanced designs: Type I, II or III sums of squares

When every cell has the same number of observations, the three kinds of sums of squares agree for the usual tests. When the cell counts differ, they do not, and software defaults differ too. Type I depends on the order in which the factors are entered. Type III tests each effect after adjusting for everything else, including the interaction, which makes main-effect tests depend on how factors are coded. Type II tests each main effect after the other main effect but not the interaction, and Langsrud (2003) argues that it is the sensible default when the interaction is negligible, with better power than Type III. Whichever type you use, report it. If you have the choice, a balanced design avoids the question.

Worked example: watering regime and fertiliser

A greenhouse trial grows plants under two watering regimes (low and high) with three fertilisers (A, B and C), eight plants per cell, 48 in all. The outcome is growth. The data are synthetic, generated in R 4.6.1, and built so that fertiliser B is best under low watering and worst under high watering.

Watering	Fertiliser A	Fertiliser B	Fertiliser C
Low	19.7	27.7	21.6
High	23.9	15.9	23.5

Averaged over watering, the three fertilisers look almost the same: 21.8, 21.8 and 22.6. The ANOVA (Type II sums of squares) says:

Term	F	df	p
Watering	6.30	1, 42	0.016
Fertiliser	0.46	2, 42	0.63
Watering x fertiliser	43.52	2, 42	less than 0.0001

The partial eta-squared of the interaction is 0.67, so two thirds of the variation not explained by watering or fertiliser alone is accounted for by their combination. Taken at face value, the fertiliser row says fertiliser makes no difference. That conclusion is wrong, and the interaction row shows why. Under low watering, B plants grow about 8 points more than A plants. Under high watering, B plants grow about 8 points less. The two effects cancel in the average, so the main effect of fertiliser disappears while the effect within each regime is large. Residuals were consistent with normality (Shapiro-Wilk p = 0.78) and the variances across the six cells were similar (Levene p = 0.54).

Post hoc tests after a two-way ANOVA

A significant F-test says that something differs, not what. A post hoc test, such as Tukey’s honestly significant difference (Tukey, 1949), compares pairs of means while holding the family-wise error rate at the stated level across all the pairs.

In a one-way design the question is simply which groups differ. In a two-way design there is a prior question: pairs of what? With a material interaction, comparing the six cells all against each other (15 pairs) answers many questions you did not ask, spends your error budget on them, and still does not isolate the effect you care about. The better plan is to compare within each level of the other factor.

In the example, Tukey comparisons of fertilisers within low watering give B above A by 7.95 and B above C by 6.08, both clearly different (adjusted p of 0.0001 or smaller), with A and C similar (adjusted p = 0.33). Within high watering the picture inverts: B is below A by 8.03 and below C by 7.65, again both with p < 0.0001. Each family has three comparisons, not fifteen. The procedures behind these adjustments, and when to prefer Holm or Benjamini-Hochberg to Tukey, are covered in multiple comparisons.

If the interaction is negligible, the simpler route is to test the main effects and, only when a factor with three or more levels is significant, follow with pairwise comparisons of its levels averaged over the other factor.

Which one to report

• Interaction is material: report the interaction test and a table or plot of cell means with intervals, then simple effects with an adjustment for the number of comparisons in each family. Do not report the main effects as if they were interpretable.
• Interaction is negligible: report the main effects, noting that the interaction was checked, plus the effect sizes. A partial eta-squared for each term is common; say which one you used.
• Unbalanced cells: state the type of sums of squares.
• Always: say which post hoc procedure you used and over which family of comparisons. “Tukey over the three fertiliser levels within each watering regime” is a complete statement, “Tukey post hoc” is not.

Do not decide that the interaction is negligible only because it is not significant: a small study may simply be unable to detect it. Look at the cell means before dropping it. For the wider decision tree, see how to choose a statistical test.

How Celeus handles this

For a two-way ANOVA Celeus fits both factors and their interaction with Type II sums of squares and reports the interaction test as the headline result, with the interaction’s partial eta-squared. It also reports the estimated mean of every combination of the two factors with a confidence interval, which is the table you read simple effects from, and it checks residual normality and equal variances across the cells.