ANOVA Tests and Alternatives: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of analysis of variance all the way through one-way, factorial, repeated measures, and mixed ANOVA designs, their non-parametric alternatives, post-hoc testing, effect sizes, and practical usage within the DataStatPro application. Whether you are encountering ANOVA for the first time or deepening your understanding of variance decomposition and group comparison, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is ANOVA?
3. The Mathematics Behind ANOVA
4. Assumptions of ANOVA
5. Types of ANOVA
6. Using the ANOVA Calculator Component
7. One-Way Between-Subjects ANOVA
8. Factorial Between-Subjects ANOVA
9. One-Way Repeated Measures ANOVA
10. Mixed ANOVA
11. Post-Hoc Tests and Planned Contrasts
12. Non-Parametric Alternatives
13. Advanced Topics
14. Worked Examples
15. Common Mistakes and How to Avoid Them
16. Troubleshooting
17. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into ANOVA, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 The Logic of Variance Decomposition

ANOVA is built upon one foundational insight: total variability in a dataset can be partitioned into components attributable to specific sources. For a one-way design:

$SS_{total} = SS_{between} + SS_{within}$

If the between-group variance is substantially larger than the within-group variance, it suggests that group membership explains a meaningful portion of the variability in scores — that is, the groups differ. This ratio of variances is the F-statistic.

1.2 The F-Distribution

The F-distribution arises from the ratio of two independent chi-squared distributions divided by their respective degrees of freedom:

$F = \frac{\chi^2_1 / \nu_1}{\chi^2_2 / \nu_2}$

In the ANOVA context, this becomes the ratio of two mean squares:

$F = \frac{MS_{between}}{MS_{within}} = \frac{SS_{between}/df_{between}}{SS_{within}/df_{within}}$

Under $H_0$ (all group means equal), this ratio equals approximately 1. Values much greater than 1 provide evidence against $H_0$.

The F-distribution is characterised by two parameters:

• $df_1$ = numerator degrees of freedom (between-groups)
• $df_2$ = denominator degrees of freedom (within-groups/error)

It is always non-negative and right-skewed.

1.3 Why Not Multiple t-Tests?

A natural question is: why not simply run multiple t-tests to compare groups? With $K$ groups, this would require $\binom{K}{2} = K(K-1)/2$ pairwise tests.

The familywise error rate (FWER) inflates:

$FWER = 1 - (1-\alpha)^m$

Where $m$ is the number of tests. For $K = 4$ groups: $m = 6$ pairwise tests; $FWER = 1 - (0.95)^6 = .265$ — far above the nominal $.05$.

ANOVA maintains the Type I error at $\alpha$ for the omnibus test that all group means are simultaneously equal.

1.4 The Relationship Between ANOVA and Regression

ANOVA and regression are mathematically equivalent. Both are special cases of the General Linear Model (GLM):

$Y_i = \mathbf{X}_i \boldsymbol{\beta} + \varepsilon_i, \quad \varepsilon_i \sim \mathcal{N}(0, \sigma^2)$

In ANOVA, the predictors $\mathbf{X}$ are categorical group membership variables (dummy or effect coded). Understanding this equivalence is essential for interpreting interactions and for moving to ANCOVA and mixed models.

1.5 Main Effects and Interactions

In factorial designs (multiple independent variables):

• A main effect is the effect of one independent variable (IV) averaged across all levels of the other IV(s).
• An interaction effect occurs when the effect of one IV depends on the level of another IV.

⚠️ When a significant interaction is present, main effects must be interpreted with extreme caution — the "average" effect of a variable may be misleading when the effect differs substantially across levels of the other variable.

1.6 Fixed vs. Random Effects

• Fixed effects: The levels of the IV are specifically chosen and are the only levels of interest (e.g., three specific drug dosages). Conclusions apply only to those levels.
• Random effects: The levels of the IV are a random sample from a larger population of possible levels (e.g., 10 randomly selected schools). Conclusions generalise to the population of levels.
• Mixed effects: Some factors are fixed, others are random. This is the basis of mixed models (also called hierarchical linear models).

Standard ANOVA assumes all factors are fixed. When factors are random, the denominator of the F-ratio changes.

1.7 Variance Explained: $\eta^2$, $\omega^2$, and $\varepsilon^2$

Effect sizes for ANOVA are variance-explained indices — they quantify what proportion of the total (or residual) variance is attributable to a given effect. These are reviewed in detail in the Mathematics section, but the key formulas are:

$\eta^2 = \frac{SS_{effect}}{SS_{total}}, \quad \omega^2 = \frac{SS_{effect}-(df_{effect})MS_{error}}{SS_{total}+MS_{error}}, \quad \varepsilon^2 = \frac{SS_{effect}-(df_{effect})MS_{error}}{SS_{total}}$

Both $\omega^2$ and $\varepsilon^2$ correct for the positive bias of $\eta^2$ in finite samples and are preferred for reporting.

2. What is ANOVA?

2.1 The Core Idea

Analysis of Variance (ANOVA) is a parametric inferential procedure for testing whether three or more population means differ simultaneously. Despite its name, ANOVA tests mean differences by comparing variances — specifically, by assessing whether the variability between groups is larger than expected given the variability within groups.

The general form of the F-statistic:

$F = \frac{\text{Variance between groups (signal)}}{\text{Variance within groups (noise)}} = \frac{MS_{between}}{MS_{within}}$

A large F indicates that the between-group differences are large relative to random sampling error — evidence that at least one group mean differs from the others.

2.2 What ANOVA Tests and Does Not Test

ANOVA tells you:

• Whether at least one group mean differs from the others (omnibus test).
• How much variance in the outcome is explained by group membership ($\eta^2$, $\omega^2$).

ANOVA does NOT tell you:

• Which specific groups differ from each other (requires post-hoc tests or planned contrasts).
• The direction or magnitude of specific pairwise differences.
• Whether the omnibus difference is practically meaningful (requires effect sizes with CIs).

2.3 The ANOVA Family

2.4 ANOVA in Context

The ANOVA test is one member of a broader family of procedures for comparing group means:

3. The Mathematics Behind ANOVA

3.1 One-Way ANOVA: Sum of Squares Decomposition

Consider $K$ groups with $n_j$ observations in group $j$ and total $N = \sum_{j=1}^K n_j$ observations. Let $\bar{x}_j$ be the mean of group $j$ and $\bar{x}_{..}$ be the grand mean.

Grand mean:

$\bar{x}_{..} = \frac{1}{N}\sum_{j=1}^K \sum_{i=1}^{n_j} x_{ij}$

Total sum of squares ($SS_{total}$): Total variability in the data.

$SS_{total} = \sum_{j=1}^K \sum_{i=1}^{n_j} (x_{ij} - \bar{x}_{..})^2$

$df_{total} = N - 1$

Between-groups sum of squares ($SS_{between}$): Variability due to group differences.

$SS_{between} = \sum_{j=1}^K n_j (\bar{x}_j - \bar{x}_{..})^2$

$df_{between} = K - 1$

Within-groups sum of squares ($SS_{within}$): Variability within groups (error).

$SS_{within} = \sum_{j=1}^K \sum_{i=1}^{n_j} (x_{ij} - \bar{x}_j)^2$

$df_{within} = N - K$

Verification: $SS_{total} = SS_{between} + SS_{within}$

3.2 Mean Squares and the F-Statistic

Mean squares are sums of squares divided by their degrees of freedom — they are variance estimates:

$MS_{between} = \frac{SS_{between}}{K-1}$

$MS_{within} = \frac{SS_{within}}{N-K}$

The F-statistic:

$F = \frac{MS_{between}}{MS_{within}}$

Under $H_0: \mu_1 = \mu_2 = \cdots = \mu_K$, the F-statistic follows an F-distribution with $(K-1, N-K)$ degrees of freedom. The p-value is:

$p = P(F_{K-1,\;N-K} \geq F_{obs})$

3.3 The Expected Mean Squares

Understanding why the F-ratio works requires examining the expected values of the mean squares under $H_0$ and $H_1$:

$E[MS_{within}] = \sigma^2$ (always an unbiased estimate of $\sigma^2$)

$E[MS_{between}] = \sigma^2 + \frac{\sum_{j=1}^K n_j(\mu_j - \mu)^2}{K-1}$

Under $H_0$ (all $\mu_j$ equal): $E[MS_{between}] = \sigma^2$, so $E[F] \approx 1$.

Under $H_1$ (some $\mu_j$ differ): $E[MS_{between}] > \sigma^2$, so $E[F] > 1$.

The non-centrality parameter of the F-distribution:

$\lambda = \frac{\sum_{j=1}^K n_j(\mu_j - \mu)^2}{\sigma^2}$

This links the population effect size to the expected F-statistic and is used for power analysis.

3.4 The ANOVA Source Table

The standard ANOVA output is presented in a source table:

3.5 Effect Sizes for One-Way ANOVA

Eta squared ($\eta^2$) — biased, but widely reported:

$\eta^2 = \frac{SS_{between}}{SS_{total}}$

Omega squared ($\omega^2$) — bias-corrected, preferred:

$\omega^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total} + MS_{within}}$

Epsilon squared ($\varepsilon^2$) — alternative bias correction:

$\varepsilon^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total}}$

Cohen's $f$ — for power analysis:

$f = \sqrt{\frac{\eta^2}{1-\eta^2}} \quad \text{or} \quad f = \sqrt{\frac{\omega^2}{1-\omega^2}}$

Relationship between effect sizes: $\omega^2 \leq \varepsilon^2 \leq \eta^2$

Cohen's (1988) benchmarks:

3.6 Confidence Intervals for ANOVA Effect Sizes

CIs for $\eta^2$ and $\omega^2$ use the non-central F-distribution. The observed F-statistic follows a non-central F-distribution with non-centrality parameter $\lambda$ related to the population effect:

$\lambda = \frac{\eta^2 \cdot N}{1-\eta^2}$

The 95% CI bounds for $\lambda$ are found numerically (inverting the non-central F CDF), then converted to $\eta^2$:

$\eta^2_L = \frac{\lambda_L}{\lambda_L + N}, \qquad \eta^2_U = \frac{\lambda_U}{\lambda_U + N}$

DataStatPro computes these exact CIs automatically using numerical iteration.

3.7 Factorial ANOVA: Partitioning Variance for Multiple Factors

For a two-factor (A $\times$ B) between-subjects ANOVA with $a$ levels of A, $b$ levels of B, and $n$ observations per cell:

$SS_{total} = SS_A + SS_B + SS_{A \times B} + SS_{within}$

$df_{total} = N - 1 = abn - 1$

Computing each SS:

Let $\bar{x}_{j.}$ = mean of level $j$ of factor A, $\bar{x}_{.k}$ = mean of level $k$ of factor B, $\bar{x}_{jk}$ = cell mean, $\bar{x}_{..}$ = grand mean.

$SS_A = bn\sum_{j=1}^a(\bar{x}_{j.} - \bar{x}_{..})^2$

$SS_B = an\sum_{k=1}^b(\bar{x}_{.k} - \bar{x}_{..})^2$

$SS_{A\times B} = n\sum_{j=1}^a\sum_{k=1}^b(\bar{x}_{jk} - \bar{x}_{j.} - \bar{x}_{.k} + \bar{x}_{..})^2$

$SS_{within} = \sum_{j=1}^a\sum_{k=1}^b\sum_{i=1}^n(x_{ijk} - \bar{x}_{jk})^2$

F-ratios (fixed effects model, all denominators are $MS_{within}$):

$F_A = \frac{MS_A}{MS_{within}}, \quad F_B = \frac{MS_B}{MS_{within}}, \quad F_{A\times B} = \frac{MS_{A\times B}}{MS_{within}}$

3.8 Partial Eta Squared ($\eta_p^2$) for Factorial Designs

In factorial ANOVA, partial eta squared isolates the effect of one factor after controlling for other effects:

$\eta_p^2 = \frac{SS_{effect}}{SS_{effect} + SS_{error}}$

⚠️ In factorial designs, the sum of all partial $\eta_p^2$ values can exceed 1.0. They are NOT proportions of total variance — only $\eta^2$ carries that interpretation. Always label the statistic precisely: write $\eta_p^2$, not $\eta^2$, for partial values.

Partial omega squared (preferred — bias-corrected):

$\omega_p^2 = \frac{SS_{effect} - df_{effect} \cdot MS_{error}}{SS_{total} + MS_{error}}$

3.9 Repeated Measures ANOVA: Within-Subjects Decomposition

For a one-way repeated measures design with $K$ conditions and $n$ participants:

$SS_{total} = SS_{between\text{-}subjects} + SS_{within\text{-}subjects}$

$SS_{within\text{-}subjects} = SS_{conditions} + SS_{error}$

The key feature: between-subjects variability is removed from the error term, which dramatically increases power when individual differences are large.

$F = \frac{MS_{conditions}}{MS_{error}} = \frac{SS_{cond}/(K-1)}{SS_{error}/((n-1)(K-1))}$

Generalised eta squared ($\eta_G^2$; Olejnik & Algina, 2003) is recommended for repeated measures designs because it is comparable across between-subjects and within- subjects designs:

$\eta_G^2 = \frac{SS_{conditions}}{SS_{conditions} + SS_{between\text{-}subjects} + SS_{error}}$

3.10 Sphericity and the Mauchly Test

Repeated measures ANOVA requires the sphericity assumption: the variances of the differences between all pairs of conditions are equal. Formally, for all pairs $(j, k)$:

$\text{Var}(x_j - x_k) = \text{constant}$

Mauchly's test evaluates this assumption:

• $H_0$: the sphericity assumption holds.
• A significant result ($p < .05$) indicates sphericity violation.

Epsilon ($\varepsilon$) corrections adjust the degrees of freedom when sphericity is violated. Two commonly used corrections:

Greenhouse-Geisser (GG) correction:

$\varepsilon_{GG} = \frac{\left(\sum_j \hat{\sigma}_{jj}\right)^2}{(K-1)\left(\sum_j\sum_k \hat{\sigma}_{jk}^2\right)}$

$0 < \varepsilon_{GG} \leq 1$; $\varepsilon_{GG} = 1$ means sphericity holds exactly.

Huynh-Feldt (HF) correction (less conservative than GG, preferred when $\varepsilon > 0.75$):

$\varepsilon_{HF} = \frac{n(K-1)\varepsilon_{GG} - 2}{(K-1)(n - 1 - (K-1)\varepsilon_{GG})}$

Corrected degrees of freedom:

$df_{conditions}^* = \varepsilon \cdot (K-1), \qquad df_{error}^* = \varepsilon \cdot (n-1)(K-1)$

Decision rule for epsilon corrections:

3.11 Mixed ANOVA: Between + Within Factors

A mixed ANOVA (also split-plot ANOVA) includes both between-subjects and within- subjects factors. For a design with one between factor (A, $a$ levels) and one within factor (B, $b$ levels) and $n$ participants per group:

Note the two separate error terms:

• Between-subjects effects (A) are tested against $MS_{S(A)}$ (between-subjects error).
• Within-subjects effects (B, A$\times$B) are tested against $MS_{B\times S(A)}$ (within- subjects error).

4. Assumptions of ANOVA

4.1 Normality of Residuals

ANOVA assumes that the residuals (differences between observed values and group means) are normally distributed within each population:

$\varepsilon_{ij} \sim \mathcal{N}(0, \sigma^2)$

How to check:

• Shapiro-Wilk test on residuals (most powerful for $n < 50$ per group).
• Q-Q plots of residuals: points should follow the diagonal.
• Histograms of residuals: should be approximately bell-shaped.
• Skewness ($|z| < 2$) and kurtosis ($|z| < 7$) of residuals.

Robustness: ANOVA is robust to mild normality violations, particularly when:

• Group sizes are equal (balanced design).
• $n \geq 15$–$20$ per group (CLT applies).
• Distributions are symmetric even if non-normal.

When violated: Use the Kruskal-Wallis test (independent groups) or the Friedman test (repeated measures) as non-parametric alternatives. Consider data transformations (log, square root, Box-Cox) for skewed distributions.

4.2 Homogeneity of Variance (Homoscedasticity)

Standard ANOVA assumes that all $K$ populations have equal variances:

$\sigma_1^2 = \sigma_2^2 = \cdots = \sigma_K^2$

This is the homoscedasticity assumption and is required for $MS_{within}$ to serve as a valid pooled estimate of the common population variance $\sigma^2$.

How to check:

• Levene's test (preferred — robust to non-normality): $H_0$: all group variances equal.
• Brown-Forsythe test (more robust, uses median rather than mean).
• Bartlett's test (powerful but sensitive to non-normality — avoid for non-normal data).
• Variance ratio rule: if $s^2_{max}/s^2_{min} > 4$, heterogeneity is concerning.

Robustness: ANOVA is relatively robust to heterogeneity when group sizes are equal. When group sizes are unequal AND variances are unequal, ANOVA can have severely inflated or deflated Type I error rates.

When violated: Use Welch's one-way ANOVA (with Games-Howell post-hoc tests), which does not assume equal variances and is recommended as the default for independent designs.

4.3 Independence of Observations

All observations must be independent of each other, both within and across groups. Dependence typically arises from:

• Clustered or nested data (students in classrooms, patients in hospitals).
• Repeated measurements on the same participant (use repeated measures ANOVA instead).
• Time series data.
• Family or matched data.

When violated: Use repeated measures ANOVA (for within-subjects data), mixed models (for nested or clustered data), or multilevel ANOVA (for hierarchical designs).

4.4 Sphericity (Repeated Measures Only)

As described in Section 3.10, repeated measures ANOVA additionally requires sphericity — that the variances of all pairwise difference scores are equal. This is a stronger assumption than homogeneity of variance.

When violated: Apply Greenhouse-Geisser or Huynh-Feldt corrections to the degrees of freedom, or use the multivariate approach (MANOVA on the repeated measures).

4.5 Interval Scale of Measurement

The dependent variable must be measured on at least an interval scale (equal-spaced intervals). Ordinal data (e.g., Likert scales) technically violate this assumption.

When violated: Use non-parametric alternatives (Kruskal-Wallis, Friedman) or analyse using ordinal regression.

4.6 Absence of Significant Outliers

ANOVA is based on means and is sensitive to extreme outliers, particularly in small samples. Outliers inflate $SS_{within}$ and $SS_{between}$ unpredictably.

How to check:

• Boxplots per group: values beyond $1.5 \times IQR$ are mild outliers; $3 \times IQR$ are extreme.
• Standardised residuals: $|z_i| > 3$ flags potential outliers.
• Studentised residuals from the ANOVA model.

When outliers present: Investigate the cause. Report analyses with and without outliers. Consider trimmed mean ANOVA or Kruskal-Wallis as robust alternatives.

4.7 Assumption Summary Table

5. Types of ANOVA

5.1 Classification by Design

By Number of Independent Variables

By Type of Factor

5.2 Choosing the Correct ANOVA Design

What is the number of independent variables?
├── 1 IV
│   └── Is the same participant in all conditions?
│       ├── NO (between-subjects)  → One-way between-subjects ANOVA
│       └── YES (within-subjects)  → One-way repeated measures ANOVA
└── 2+ IVs
    └── What type are the IVs?
        ├── All between-subjects   → Factorial between-subjects ANOVA
        ├── All within-subjects    → Fully within-subjects factorial ANOVA
        └── Mixed (some between, some within) → Mixed ANOVA

5.3 Type I, II, and III Sums of Squares

In unbalanced designs (unequal cell sizes), the partition of SS depends on the order in which effects are entered. Three conventions exist:

⚠️ For balanced designs (equal cell sizes), all three types give identical results. For unbalanced designs, Type III is the most commonly reported but requires full-rank parameterisation (effect coding or deviation coding, not dummy coding). Always specify which type was used when reporting factorial ANOVA results.

6. Using the ANOVA Calculator Component

The ANOVA Calculator component in DataStatPro provides a comprehensive tool for running, diagnosing, visualising, and reporting ANOVA designs and their alternatives.

Step-by-Step Guide

Step 1 — Select the ANOVA Design

Choose from the "ANOVA Type" dropdown:

• One-Way Between-Subjects ANOVA: One IV, independent groups.
• Factorial Between-Subjects ANOVA: Two or more IVs, independent groups.
• One-Way Repeated Measures ANOVA: One IV, same participants in all conditions.
• Mixed ANOVA: One or more between-subjects IVs and one or more within-subjects IVs.
• Welch's One-Way ANOVA: Robust to heterogeneity of variance.
• Kruskal-Wallis Test: Non-parametric one-way.
• Friedman Test: Non-parametric repeated measures.

Step 2 — Input Method

• Raw data: Upload or paste the dataset. DataStatPro performs all assumption checks automatically, computes effect sizes, and generates visualisations.
• Summary statistics: Enter group means, SDs, and $n$ values. Full assumption checks are not available but all inferential statistics and effect sizes are computed.
• ANOVA table values: Enter SS, df, and MS values from a published table to compute effect sizes, power, and CIs.

Step 3 — Specify the Design Structure

• Number of groups/levels for each factor.
• Factor names and level labels for clear output labelling.
• Cell sizes (equal or unequal — DataStatPro auto-detects balance).
• SS Type for factorial designs (Type I, II, or III — default: Type III).

Step 4 — Select Assumption Tests

DataStatPro automatically runs, with results displayed in a colour-coded panel:

• ✅ Shapiro-Wilk normality test on residuals (per group for small samples).
• ✅ Levene's test for homogeneity of variance (between-subjects designs).
• ✅ Mauchly's test for sphericity (repeated measures designs) with $\varepsilon_{GG}$ and $\varepsilon_{HF}$ automatically applied when sphericity is violated.
• ✅ Boxplots per group for outlier detection.

Step 5 — Select Post-Hoc Tests

When the omnibus F is significant, specify post-hoc tests:

• Tukey HSD — balanced designs, equal variances (controls FWER).
• Bonferroni — conservative; any design.
• Holm-Bonferroni — less conservative sequential procedure.
• Scheffé — most conservative; allows all possible contrasts.
• Games-Howell — unequal variances or unequal $n$ (recommended with Welch's ANOVA).
• Dunnett's test — comparing all groups to a single control group.
• Custom planned contrasts — specify weights for specific a priori comparisons.

Step 6 — Select Effect Sizes

• $\omega^2$ (preferred): Bias-corrected estimate for one-way ANOVA.
• $\omega_p^2$ (preferred for factorial): Partial omega squared.
• $\eta^2$ (common): Biased; provided for comparison.
• $\eta_p^2$ (common for factorial): Partial eta squared.
• $\eta_G^2$ (recommended for repeated measures): Generalised eta squared.
• Cohen's $f$: For power analysis.
• 95% CIs for all effect sizes via non-central F-distribution.

Step 7 — Select Display Options

• ✅ Full ANOVA source table with F-statistics and p-values.
• ✅ Descriptive statistics (mean, SD, SE, 95% CI) per group/cell.
• ✅ Effect size estimates with 95% CIs for each effect.
• ✅ Assumption test results panel.
• ✅ Post-hoc pairwise comparison table with adjusted p-values and effect sizes.
• ✅ Interaction plot (line plot of cell means) for factorial designs.
• ✅ Profile plots for repeated measures.
• ✅ Raincloud plots (half violin + boxplot + raw data) per group.
• ✅ Power analysis and required $n$ for each effect.
• ✅ APA 7th edition results paragraph (auto-generated).

Step 8 — Run the Analysis

Click "Run ANOVA". DataStatPro will:

1. Compute the full ANOVA source table.
2. Apply sphericity corrections automatically if Mauchly's test is significant.
3. Run all selected post-hoc tests and planned contrasts.
4. Compute effect sizes with exact CIs.
5. Generate all visualisations.
6. Output an APA-compliant results paragraph.

7. One-Way Between-Subjects ANOVA

7.1 Purpose and Design

The one-way between-subjects ANOVA tests whether the means of three or more independent groups differ significantly. It is the generalisation of the independent samples t-test (when $K = 2$, $F = t^2$) to $K \geq 3$ groups.

Common applications:

• Comparing exam scores across three teaching methods (Lecture, Flipped, Project-Based).
• Evaluating the effect of four drug dosage levels on pain rating.
• Assessing anxiety differences across five diagnostic categories.
• Comparing productivity across three management styles.

7.2 Full Procedure

Step 1 — State hypotheses

$H_0: \mu_1 = \mu_2 = \cdots = \mu_K$

$H_1: \text{At least one } \mu_j \text{ differs from the others}$

Step 2 — Compute grand mean and group means

$\bar{x}_{..} = \frac{\sum_{j=1}^K \sum_{i=1}^{n_j} x_{ij}}{N}, \qquad \bar{x}_j = \frac{1}{n_j}\sum_{i=1}^{n_j}x_{ij}$

Step 3 — Compute sums of squares

$SS_B = \sum_{j=1}^K n_j(\bar{x}_j - \bar{x}_{..})^2$

$SS_W = \sum_{j=1}^K \sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)^2$

$SS_T = SS_B + SS_W$

Step 4 — Compute degrees of freedom

$df_B = K-1, \quad df_W = N-K, \quad df_T = N-1$

Step 5 — Compute mean squares and F

$MS_B = SS_B/df_B, \quad MS_W = SS_W/df_W, \quad F = MS_B/MS_W$

Step 6 — Compute p-value and make a decision

$p = P(F_{K-1,\;N-K} \geq F_{obs})$

Reject $H_0$ if $p \leq \alpha$.

Step 7 — Compute effect sizes

$\eta^2 = SS_B/SS_T$

$\omega^2 = (SS_B - (K-1)MS_W)/(SS_T + MS_W)$

$f = \sqrt{\omega^2/(1-\omega^2)}$

Step 8 — Conduct post-hoc tests or planned contrasts

If $H_0$ is rejected, identify which groups differ (Section 11).

7.3 Computing $\omega^2$ and $\eta^2$ from F

When only the F-statistic, df, and $N$ are reported:

$\eta^2 = \frac{F \cdot df_B}{F \cdot df_B + df_W}$

$\omega^2 = \frac{(F-1) \cdot df_B}{F \cdot df_B + df_W + 1}$ (approximate)

7.4 Interpreting the Omnibus F-Test

The omnibus F-test is a global test. A significant result tells you only that:

1. At least one pair of group means differs significantly.
2. This difference is unlikely to have arisen by sampling error alone.

It does not tell you which groups differ, by how much, or in what direction. Post-hoc tests (Section 11) are required to answer these questions.

💡 When groups were theoretically predicted to differ in specific ways before data collection, use planned contrasts rather than (or in addition to) the omnibus F-test. Planned contrasts are more powerful and more informative than post-hoc tests.

8. Factorial Between-Subjects ANOVA

8.1 Purpose and Design

Factorial ANOVA simultaneously examines the effects of two or more IVs and their interactions on a continuous DV. It is more efficient than running separate one-way ANOVAs because it:

• Tests all main effects and interactions simultaneously.
• Controls FWER across all tests.
• Reveals interaction effects that separate one-way analyses cannot detect.
• Requires fewer participants than running separate experiments for each factor.

8.2 The Concept of Interaction

An interaction exists when the effect of one IV differs depending on the level of another IV. Interactions are the most important and often the most theoretically interesting finding in factorial designs.

Types of interactions:

Interpreting an interaction:

When a significant A$\times$B interaction is found:

1. Do not interpret main effects in isolation — they are averages that may be misleading when the interaction is substantial.
2. Probe the interaction with simple effects analysis: test the effect of A separately at each level of B (or vice versa).
3. Plot the interaction with a line plot: this is essential for understanding the pattern.

8.3 Simple Effects Analysis

Simple effects decompose the interaction by examining the effect of one IV at each level of the other IV. For a 2$\times$3 design:

• Simple effect of A at $B_1$: compare $\bar{x}_{A_1B_1}$ vs. $\bar{x}_{A_2B_1}$
• Simple effect of A at $B_2$: compare $\bar{x}_{A_1B_2}$ vs. $\bar{x}_{A_2B_2}$
• Simple effect of A at $B_3$: compare $\bar{x}_{A_1B_3}$ vs. $\bar{x}_{A_2B_3}$

Simple effects use $MS_{within}$ from the full factorial model as the error term (pooled error), which is more stable than separate-group estimates.

8.4 Effect Sizes in Factorial ANOVA

For factorial designs, report partial effect sizes for each effect:

Partial eta squared (common, biased):

$\eta_{p,A}^2 = \frac{SS_A}{SS_A + SS_{within}}$

$\eta_{p,B}^2 = \frac{SS_B}{SS_B + SS_{within}}$

$\eta_{p,A\times B}^2 = \frac{SS_{A\times B}}{SS_{A\times B} + SS_{within}}$

Partial omega squared (preferred, bias-corrected):

$\omega_{p,A}^2 = \frac{SS_A - df_A \cdot MS_{within}}{SS_{total} + MS_{within}}$

Generalised eta squared (recommended for between-subjects factorial designs, Olejnik & Algina, 2003):

$\eta_{G}^2 = \frac{SS_{effect}}{SS_{total}}$

For purely between-subjects designs, $\eta_G^2 = \eta^2$ for each effect.

9. One-Way Repeated Measures ANOVA

9.1 Purpose and Design

One-way repeated measures ANOVA tests whether means differ across $K$ conditions when the same participants are measured in all conditions. It is the generalisation of the paired t-test to $K \geq 3$ conditions.

Common applications:

• Measuring depression at three time points (pre, post, follow-up).
• Comparing cognitive performance across four task difficulty levels.
• Evaluating preference ratings for five product variants.
• Assessing physiological response across six stimulus intensities.

Advantages over one-way between-subjects ANOVA:

• Greater statistical power: between-subjects variability is removed from the error term.
• Fewer participants needed: each participant contributes $K$ observations.
• Controls individual difference confounds: the same person is compared across conditions.

Disadvantages:

• Carryover effects: experiencing one condition may affect performance in others (counterbalance with randomised condition order or include adequate wash-out periods).
• Sphericity assumption: more complex than the equal-variance assumption.
• Attrition: losing participants eliminates all their data from all conditions.

9.2 Full Procedure

Step 1 — Compute condition means and participant means

$\bar{x}_{.k}$ = mean of condition $k$; $\bar{x}_{i.}$ = mean of participant $i$; $\bar{x}_{..}$ = grand mean.

Step 2 — Compute sums of squares

$SS_{total} = \sum_{i=1}^n\sum_{k=1}^K (x_{ik} - \bar{x}_{..})^2$

$SS_{conditions} = n\sum_{k=1}^K(\bar{x}_{.k} - \bar{x}_{..})^2$

$SS_{subjects} = K\sum_{i=1}^n(\bar{x}_{i.} - \bar{x}_{..})^2$

$SS_{error} = SS_{total} - SS_{conditions} - SS_{subjects}$

Step 3 — Degrees of freedom

$df_{conditions} = K-1, \quad df_{subjects} = n-1, \quad df_{error} = (K-1)(n-1)$

Step 4 — Mean squares and F

$MS_{conditions} = SS_{conditions}/(K-1)$

$MS_{error} = SS_{error}/((K-1)(n-1))$

$F = MS_{conditions}/MS_{error}$

Step 5 — Sphericity check and correction

Run Mauchly's test. If violated, apply GG or HF correction:

$F_{corrected} = \frac{SS_{conditions}/(\varepsilon \cdot (K-1))}{SS_{error}/(\varepsilon \cdot (n-1)(K-1))} = \frac{MS_{conditions}}{MS_{error}}$

The F-statistic is unchanged; only the reference distribution (via corrected df) changes.

Step 6 — Effect size

Generalised eta squared ($\eta_G^2$) — recommended for repeated measures:

$\eta_G^2 = \frac{SS_{conditions}}{SS_{conditions} + SS_{subjects} + SS_{error}}$

Partial eta squared ($\eta_p^2$) — common but inflated:

$\eta_p^2 = \frac{SS_{conditions}}{SS_{conditions} + SS_{error}}$

Partial omega squared ($\omega_p^2$) — bias-corrected:

$\omega_p^2 = \frac{SS_{conditions} - (K-1)MS_{error}}{SS_{total} + MS_{error}}$

💡 Use $\eta_G^2$ for repeated measures when comparing effect sizes across studies using different designs (between-subjects vs. within-subjects), as it is the most design-comparable measure. For purely within-design comparisons, $\omega_p^2$ is the least-biased choice.

10. Mixed ANOVA

10.1 Purpose and Design

Mixed ANOVA combines at least one between-subjects factor and at least one within- subjects factor. It is among the most commonly used designs in psychology, medicine, and education because most longitudinal experiments involve:

• A between-subjects factor: Treatment group vs. control group.
• A within-subjects factor: Time (pre, post, follow-up).

The mixed ANOVA tests:

1. Main effect of the between-subjects factor (e.g., treatment vs. control, collapsed across time).
2. Main effect of the within-subjects factor (e.g., change over time, collapsed across groups).
3. Interaction (e.g., does the pattern of change over time differ between treatment and control?). This interaction is typically the primary research question.

10.2 The Primary Interaction: Time $\times$ Group

In a treatment evaluation study, the Time $\times$ Group interaction answers: "Does the treatment group change differently over time compared to the control group?" This is typically the most important test in a mixed ANOVA:

• If the interaction is significant: the time trajectories differ between groups — strong evidence of a treatment effect beyond any change in the control group.
• If the interaction is non-significant: the pattern of change does not differ between groups — the treatment does not appear to differentially affect change over time.

10.3 Probing a Significant Interaction

When the Group $\times$ Time interaction is significant:

Option 1 — Simple effects of Time within each Group: Conduct one-way repeated measures ANOVA (or paired t-tests with correction) separately for each group. This answers: "Did each group change significantly over time?"

Option 2 — Simple effects of Group at each Time point: Conduct independent t-tests (or one-way ANOVA) separately at each time point with Bonferroni correction. This answers: "At which time points do the groups differ?"

10.4 Sphericity in Mixed ANOVA

The sphericity assumption applies to the within-subjects factor and any interaction involving the within-subjects factor. Mauchly's test and GG/HF corrections apply specifically to:

• The main effect of the within-subjects factor (B).
• The interaction A$\times$B.

The between-subjects main effect (A) does not require sphericity but does require homogeneity of variance across groups (Levene's test).

10.5 Effect Sizes for Mixed ANOVA

For mixed ANOVA, generalised eta squared ($\eta_G^2$) is strongly recommended for all effects because it accounts for the different variance structures of between-subjects and within-subjects components:

$\eta_G^2 = \frac{SS_{effect}}{SS_{effect} + SS_{subjects} + SS_{error(between)} + SS_{error(within)}}$

This allows direct comparison of effect sizes from mixed designs with purely between- subjects or purely within-subjects designs.

11. Post-Hoc Tests and Planned Contrasts

11.1 The Need for Post-Hoc Testing

A significant omnibus F-test tells you only that some group means differ. Post-hoc tests are pairwise comparisons conducted after a significant F-test to determine which specific groups differ, while controlling the familywise error rate.

The key trade-off: Controlling the FWER requires more conservative critical values, which reduces power for individual comparisons. Choosing a post-hoc test involves balancing Type I and Type II error control.

11.2 Overview of Post-Hoc Tests

11.3 Tukey's HSD — Full Procedure

Tukey's Honestly Significant Difference (HSD) is the most commonly used post-hoc test for balanced designs with equal group variances. It controls the FWER at exactly $\alpha$ for all pairwise comparisons.

Critical value: The studentised range statistic $q_{K,\;N-K,\;\alpha}$, where $K$ is the number of groups.

Minimum significant difference (MSD):

$\text{HSD} = q_{K,\;N-K,\;\alpha} \times \sqrt{\frac{MS_{within}}{n}}$

For unequal group sizes (Tukey-Kramer method):

$\text{HSD}_{jk} = q_{K,\;N-K,\;\alpha} \times \sqrt{\frac{MS_{within}}{2}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

Declare groups $j$ and $k$ significantly different if $|\bar{x}_j - \bar{x}_k| > \text{HSD}_{jk}$.

95% CI for the pairwise difference $\mu_j - \mu_k$:

$(\bar{x}_j - \bar{x}_k) \pm q_{K,\;N-K,\;\alpha} \times \sqrt{\frac{MS_{within}}{2}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

11.4 Games-Howell Test — Unequal Variances

Games-Howell is the recommended post-hoc test when variances are unequal (Levene's significant) or group sizes differ substantially. It uses Welch-Satterthwaite df for each pairwise comparison:

Standard error for pair $(j, k)$:

$SE_{jk} = \sqrt{\frac{s_j^2}{n_j} + \frac{s_k^2}{n_k}}$

Test statistic:

$q_{jk} = \frac{|\bar{x}_j - \bar{x}_k|}{SE_{jk}}$

Degrees of freedom (Welch-Satterthwaite):

$\nu_{jk} = \frac{(s_j^2/n_j + s_k^2/n_k)^2}{(s_j^2/n_j)^2/(n_j-1) + (s_k^2/n_k)^2/(n_k-1)}$

Significance assessed against the studentised range distribution $q_{K,\;\nu_{jk},\;\alpha}$.

11.5 Planned Contrasts — A Priori Comparisons

Planned contrasts (a priori comparisons) are specific, theoretically motivated comparisons formulated before data collection. They are more powerful than post-hoc tests because:

• They do not require a significant omnibus F-test.
• They allow targeted tests of specific hypotheses.
• Fewer comparisons means a less severe FWER correction (or none, for orthogonal contrasts).

Contrast coefficients: A contrast is a weighted sum of group means $\psi = \sum_j c_j\mu_j$, where $\sum_j c_j = 0$.

Examples for $K = 4$ groups (Control, Drug A, Drug B, Drug C):

Orthogonal contrasts are statistically independent ($\sum_j c_{1j}c_{2j}/n_j = 0$). A set of $K-1$ orthogonal contrasts fully partitions $SS_{between}$ and does not require FWER correction.

Contrast F-statistic:

$SS_\psi = \frac{\left(\sum_j c_j \bar{x}_j\right)^2}{\sum_j c_j^2/n_j}, \qquad F_\psi = \frac{SS_\psi}{MS_{within}}, \quad \nu = (1, N-K)$

11.6 Effect Sizes for Pairwise Comparisons

After identifying which pairs of groups differ, report effect sizes for each significant pairwise comparison:

Cohen's $d$ for the pairwise comparison $(j, k)$:

$d_{jk} = \frac{\bar{x}_j - \bar{x}_k}{s_{pooled,jk}}$

Where $s_{pooled,jk}$ can be either the two-group pooled SD or $\sqrt{MS_{within}}$ from the full ANOVA model (recommended — more stable estimate).

Using $MS_{within}$ as the standardiser:

$d_{jk} = \frac{\bar{x}_j - \bar{x}_k}{\sqrt{MS_{within}}}$

Hedges' $g$ (bias-corrected):

$g_{jk} = d_{jk} \times \left(1 - \frac{3}{4(N-K)-1}\right)$

12. Non-Parametric Alternatives

12.1 When Non-Parametric Tests Are Appropriate

Non-parametric ANOVA alternatives are appropriate when:

• Data are ordinal (ranked or Likert-type treated as ordinal).
• Data are severely non-normally distributed and sample sizes are small.
• There are extreme outliers that distort mean-based statistics.
• The homogeneity of variance assumption is severely violated.

12.2 Kruskal-Wallis Test — Non-Parametric One-Way ANOVA

The Kruskal-Wallis H test is the non-parametric alternative to one-way between- subjects ANOVA. It tests whether the population distributions of $K \geq 3$ independent groups are identical (or equivalently, under the location-shift assumption, whether the groups have equal medians).

Procedure: