One-Way ANOVA: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of variance decomposition all the way through the mathematics, assumptions, effect sizes, post-hoc testing, non-parametric alternatives, interpretation, reporting, and practical usage of the One-Way ANOVA within the DataStatPro application. Whether you are encountering ANOVA for the first time or seeking a rigorous, unified understanding of between-groups inference, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is a One-Way ANOVA?
3. The Mathematics Behind One-Way ANOVA
4. Assumptions of One-Way ANOVA
5. Variants of One-Way ANOVA
6. Using the One-Way ANOVA Calculator Component
7. Full Step-by-Step Procedure
8. Effect Sizes for One-Way ANOVA
9. Post-Hoc Tests and Planned Contrasts
10. Confidence Intervals
11. Power Analysis and Sample Size Planning
12. Non-Parametric Alternative: Kruskal-Wallis Test
13. Advanced Topics
14. Worked Examples
15. Common Mistakes and How to Avoid Them
16. Troubleshooting
17. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into One-Way ANOVA, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 The Logic of Comparing Groups

When we measure a continuous outcome across three or more independent groups, we ask: "Are the observed differences in group means larger than what we would expect from random sampling alone?" One-Way ANOVA answers this question by comparing two sources of variability:

1. Between-groups variability: How much do the group means differ from each other? If groups have truly different population means, this variability should be large.
2. Within-groups variability: How much do observations within each group vary around their own group mean? This reflects pure random (sampling) error.

If the between-groups variability is substantially larger than the within-groups variability, we conclude that the groups differ beyond what chance alone would produce.

1.2 Why Not Multiple t-Tests?

With $K$ groups, one could run all $\binom{K}{2} = K(K-1)/2$ pairwise t-tests. However, this inflates the familywise error rate (FWER):

$FWER = 1 - (1-\alpha)^m$

Where $m$ is the number of tests. For $K = 4$ groups ($m = 6$ tests) at $\alpha = .05$:

$FWER = 1 - (0.95)^6 = .265$

Over 26% chance of at least one false positive. The one-way ANOVA omnibus test maintains the FWER at exactly $\alpha$ for the simultaneous test that all group means are equal.

1.3 Variance and Its Decomposition

The variance of a dataset measures the average squared deviation from the mean:

$s^2 = \frac{\sum_{i=1}^n(x_i - \bar{x})^2}{n-1}$

The key insight behind ANOVA is that total variance can be partitioned into meaningful components. For a one-way design:

$SS_{total} = SS_{between} + SS_{within}$

Each sum of squares (SS), when divided by its degrees of freedom, becomes a mean square (MS) — a variance estimate. The ratio of these variance estimates is the F-statistic.

1.4 The F-Distribution

The F-distribution arises from the ratio of two independent chi-squared variates divided by their degrees of freedom. In ANOVA:

$F = \frac{MS_{between}}{MS_{within}} \sim F_{K-1,\;N-K}$ under $H_0$

Properties of the F-distribution:

• Always non-negative (ratio of variances is always positive).
• Right-skewed; skewness decreases as df increase.
• Characterised by two df parameters: numerator ($df_1 = K-1$) and denominator ($df_2 = N-K$).
• Under $H_0$: $E[F] \approx 1$ (both MS estimate the same $\sigma^2$).
• Under $H_1$: $E[F] > 1$ (between-groups MS is inflated by true group differences).

1.5 The Expected Mean Squares

Understanding why the F-ratio works requires the expected values under $H_0$ and $H_1$:

$E[MS_{within}] = \sigma^2$ (always)

$E[MS_{between}] = \sigma^2 + \frac{\sum_{j=1}^K n_j(\mu_j - \mu)^2}{K-1}$

Under $H_0$ (all $\mu_j$ equal): $E[MS_{between}] = \sigma^2$, so $E[F] \approx 1$.

Under $H_1$: The second term is positive, so $E[MS_{between}] > \sigma^2$, giving $E[F] > 1$.

The non-centrality parameter:

$\lambda = \frac{\sum_{j=1}^K n_j(\mu_j - \mu)^2}{\sigma^2}$

links the true population effect to the power of the test.

1.6 Statistical Significance vs. Practical Significance

Like the t-test, the F-test answers: "Is the result unlikely under $H_0$?" It does not answer: "How large is the effect?"

With very large $N$, even trivially small group differences produce significant F-values. A study with $N = 1{,}000$ participants across five groups might find $F(4, 995) = 3.50$, $p = .008$, while $\omega^2 = 0.010$ — a statistically significant but practically negligible effect.

Always report:

1. The $F$-statistic, degrees of freedom, and p-value.
2. $\omega^2$ or $\varepsilon^2$ with 95% CI (practical effect size).
3. Group means and SDs.
4. Post-hoc comparisons with individual effect sizes.

1.7 The Relationship Between ANOVA and the t-Test

When $K = 2$, the one-way ANOVA F-statistic is exactly the square of the independent samples t-statistic:

$F_{1,\;N-2} = t_{N-2}^2$

The p-values are identical (both two-tailed). ANOVA generalises the independent samples t-test to $K \geq 3$ groups.

1.8 The Relationship Between ANOVA and Regression

ANOVA is a special case of the General Linear Model (GLM):

$Y_i = \mu + \tau_j + \varepsilon_i, \qquad \varepsilon_i \sim \mathcal{N}(0, \sigma^2)$

Where $\tau_j$ is the effect of group $j$ and $\sum_j \tau_j = 0$ (sum-to-zero constraint). In the regression framework, group membership is represented by dummy or effect-coded predictors. This equivalence is important because:

• ANOVA results can always be replicated using regression.
• Adding covariates (ANCOVA) is natural in the regression framework.
• Unbalanced designs are handled more flexibly in regression.

2. What is a One-Way ANOVA?

2.1 The Core Idea

One-Way Analysis of Variance (ANOVA) is a parametric inferential procedure for testing whether the means of three or more independent groups are simultaneously equal. It is called "one-way" because there is exactly one independent variable (IV) with $K \geq 3$ levels, and "between-subjects" because different participants appear in each group.

The general omnibus null hypothesis:

$H_0: \mu_1 = \mu_2 = \cdots = \mu_K$

$H_1: \text{At least one } \mu_j \text{ differs from the others}$

2.2 What One-Way ANOVA Tests and Does Not Test

One-Way ANOVA tells you:

• Whether the observed group mean differences are larger than expected by chance (omnibus test).
• How much of the total outcome variance is explained by group membership ($\omega^2$, $\eta^2$).

One-Way ANOVA does NOT tell you:

• Which specific groups differ from each other (requires post-hoc tests or planned contrasts).
• The direction or magnitude of specific pairwise differences.
• Whether the effect is practically meaningful (requires effect sizes with CIs).

2.3 Design Requirements

For one-way between-subjects ANOVA, the design must satisfy:

• One continuous DV (interval or ratio scale).
• One categorical IV with $K \geq 3$ levels (groups).
• Different participants in each group (independent samples).
• Each participant contributes exactly one score to exactly one group.

2.4 One-Way ANOVA in Context

2.5 Real-World Applications

3. The Mathematics Behind One-Way ANOVA

3.1 Notation

3.2 Sum of Squares Decomposition

Total Sum of Squares — total variability in the data:

$SS_{total} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_{..})^2$

$df_{total} = N - 1$

Between-Groups Sum of Squares — variability due to group membership:

$SS_{between} = \sum_{j=1}^K n_j(\bar{x}_j - \bar{x}_{..})^2$

$df_{between} = K - 1$

Within-Groups Sum of Squares — variability within each group (pure error):

$SS_{within} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_j)^2 = \sum_{j=1}^K(n_j-1)s_j^2$

$df_{within} = N - K$

Verification: $SS_{total} = SS_{between} + SS_{within}$; $df_{total} = df_{between} + df_{within}$

3.3 Mean Squares and the F-Ratio

Between-groups mean square:

$MS_{between} = \frac{SS_{between}}{K-1}$

Within-groups mean square (pooled error variance):

$MS_{within} = \frac{SS_{within}}{N-K}$

Note: $MS_{within}$ is the pooled estimate of the common population variance $\sigma^2$, assuming homogeneity of variance across groups.

The F-statistic:

$F = \frac{MS_{between}}{MS_{within}}$

Under $H_0$: $F \sim F_{K-1,\;N-K}$

p-value:

$p = P(F_{K-1,\;N-K} \geq F_{obs})$

3.4 The ANOVA Source Table

3.5 Computing the Grand Mean and Group Means

Grand mean (weighted by group sizes):

$\bar{x}_{..} = \frac{\sum_{j=1}^K n_j\bar{x}_j}{N} = \frac{\sum_{j=1}^K\sum_{i=1}^{n_j} x_{ij}}{N}$

For balanced designs (equal $n_j = n$):

$\bar{x}_{..} = \frac{1}{K}\sum_{j=1}^K \bar{x}_j$

3.6 The Pooled Standard Deviation

The pooled within-groups standard deviation $s_{pooled}$ is used for computing effect sizes for pairwise comparisons:

$s_{pooled} = \sqrt{MS_{within}} = \sqrt{\frac{\sum_{j=1}^K(n_j-1)s_j^2}{N-K}}$

This is a weighted average of the group standard deviations, using degrees of freedom as weights.

3.7 Computing SS from Summary Statistics

When only group means, SDs, and $n_j$ are available:

$SS_{between} = \sum_{j=1}^K n_j(\bar{x}_j - \bar{x}_{..})^2$

$SS_{within} = \sum_{j=1}^K(n_j-1)s_j^2$

$SS_{total} = SS_{between} + SS_{within}$

3.8 Computing $\omega^2$ and $\eta^2$ from F

When only the ANOVA table is reported (useful for computing effect sizes from published results):

Eta squared from F:

$\eta^2 = \frac{F \cdot df_B}{F \cdot df_B + df_W}$

Omega squared from F (approximate):

$\omega^2 \approx \frac{(F-1) \cdot df_B}{F \cdot df_B + df_W + 1}$

Exact omega squared from SS (preferred):

$\omega^2 = \frac{SS_B - (K-1)MS_W}{SS_T + MS_W}$

3.9 The Non-Central F-Distribution and Exact CIs

Under $H_1$, the F-statistic follows a non-central F-distribution with non-centrality parameter $\lambda$:

$\lambda = \frac{\sum_{j=1}^K n_j(\mu_j - \mu)^2}{\sigma^2} = \frac{\eta^2 \cdot N}{1-\eta^2}$

Exact 95% CI for $\omega^2$ (via non-central F):

Find $\lambda_L$ and $\lambda_U$ such that:

$P(F_{K-1,\;N-K}(\lambda_L) \geq F_{obs}) = 0.025$

$P(F_{K-1,\;N-K}(\lambda_U) \leq F_{obs}) = 0.025$

Then convert to $\eta^2$: $\eta^2_L = \lambda_L/(\lambda_L+N)$, $\eta^2_U = \lambda_U/(\lambda_U+N)$

And then to $\omega^2$ using the bias correction. DataStatPro performs this numerical computation automatically.

4. Assumptions of One-Way ANOVA

4.1 Normality of Residuals (Within-Group Normality)

One-Way ANOVA assumes that within each population, the observations are normally distributed. Equivalently, the residuals $e_{ij} = x_{ij} - \bar{x}_j$ should be normally distributed.

How to check:

• Shapiro-Wilk test on residuals (most powerful for $n < 50$ per group; $H_0$: residuals are normal). A significant result suggests departure.
• Shapiro-Wilk per group (preferred for small $n_j$): run separately for each group.
• Q-Q plot of all residuals: points should follow the diagonal reference line.
• Histograms per group: approximate bell shape.
• Skewness ($|z_{skew}| < 2$) and kurtosis ($|z_{kurt}| < 7$) of residuals.

Robustness: ANOVA is remarkably robust to mild non-normality, especially when:

• Group sizes are equal (balanced design).
• $n_j \geq 15$–$20$ per group (CLT applies to group means).
• The departure is mild skewness rather than heavy tails with extreme outliers.

When violated: Use the Kruskal-Wallis test as a non-parametric alternative (Section 12). Consider Box-Cox data transformations (log, square root) for right-skewed data. Report trimmed mean ANOVA for heavy-tailed distributions.

4.2 Homogeneity of Variance (Homoscedasticity)

One-Way ANOVA assumes all $K$ population variances are equal:

$\sigma_1^2 = \sigma_2^2 = \cdots = \sigma_K^2$

This assumption is required for $MS_{within}$ to serve as a valid pooled estimate of the common error variance $\sigma^2$.

How to check:

• Levene's test (preferred — robust to non-normality): Tests $H_0: \sigma_1^2 = \cdots = \sigma_K^2$. A significant result indicates heteroscedasticity.
• Brown-Forsythe test (more robust than Levene's for non-normal data, uses group medians rather than means).
• Bartlett's test (powerful but sensitive to non-normality — avoid for non-normal data).
• Variance ratio rule of thumb: If $s^2_{max}/s^2_{min} > 4$, heterogeneity is potentially problematic, especially with unequal $n_j$.

Robustness: ANOVA is relatively robust to heteroscedasticity when:

• Group sizes are equal ($n_j$ all equal) — equal $n_j$ is a strong protective factor.
• The larger variance is associated with the larger group (liberal; Type I error slightly inflated but manageable).

When $n_j$ are unequal AND variances are unequal, ANOVA Type I error can be severely inflated (or deflated depending on the pattern).

When violated: Use Welch's one-way ANOVA (Section 5), which does not assume equal variances. Follow with Games-Howell pairwise comparisons.

4.3 Independence of Observations

All observations must be independent of each other, both within and across groups. This is a design assumption — it cannot be tested statistically from the data.

Common violations:

• Participants from the same family, classroom, or hospital ward.
• Multiple measurements from the same participant treated as independent.
• Time series data with autocorrelated errors.
• Social networks where participants influence each other's scores.

When violated: Use multilevel models (participants nested within clusters), repeated measures ANOVA (repeated observations within participants), or time-series methods.

4.4 Interval Scale of Measurement

The dependent variable must be measured on at least an interval scale — equal spacing between values. Difference scores must be meaningful.

When violated: Use the Kruskal-Wallis test (for ordinal data), or ordinal regression for ordered categorical outcomes.

4.5 Absence of Influential Outliers

Extreme outliers distort both $SS_{between}$ and $SS_{within}$, producing unreliable F-statistics.

How to check:

• Boxplots per group: values beyond $1.5 \times IQR$ from the quartile are mild outliers; beyond $3 \times IQR$ are extreme.
• Standardised residuals: $|e_{ij}/s_{pooled}| > 3$ flags potential outliers.
• Studentised deleted residuals from the ANOVA model: $|t_i| > 3$.

When outliers present: Investigate the cause (data entry error? legitimate extreme score?). Report analyses with and without outliers. Consider the Kruskal-Wallis test or trimmed mean ANOVA as robust alternatives.

4.6 Balanced vs. Unbalanced Designs

While equal group sizes ($n_j$ all equal, balanced design) are not formally required, they are strongly preferred because:

• ANOVA is more robust to normality and homoscedasticity violations with equal $n_j$.
• Statistical power is maximised for a given total $N$.
• Effect size estimation is less biased.

Unbalanced designs (unequal $n_j$) are common in observational research. They require careful attention to variance heterogeneity and post-hoc test selection.

4.7 Assumption Summary Table

5. Variants of One-Way ANOVA

5.1 Standard One-Way ANOVA (Student's F-test)

The default one-way ANOVA assuming equal variances across groups. Uses the pooled $MS_{within}$ as the error term. This is appropriate when Levene's test is non-significant AND group sizes are approximately equal.

5.2 Welch's One-Way ANOVA

Welch's F-test (1951) is the recommended default for one-way between-subjects ANOVA. It does not assume homogeneity of variance. The statistic:

$F_W = \frac{\sum_{j=1}^K w_j(\bar{x}_j - \tilde{x})^2/(K-1)}{1 + \frac{2(K-2)}{K^2-1}\sum_{j=1}^K\frac{(1-w_j/\sum w_j)^2}{n_j-1}}$

Where $w_j = n_j/s_j^2$ and $\tilde{x} = \sum w_j\bar{x}_j/\sum w_j$ (weighted grand mean).

Welch-Satterthwaite df:

$\nu_W = \frac{K^2-1}{3\sum_{j=1}^K\frac{(1-w_j/\sum w_j)^2}{n_j-1}}$

Post-hoc: Use Games-Howell pairwise comparisons when Welch's ANOVA is significant.

💡 Just as Welch's t-test is the recommended default for two groups, Welch's one-way ANOVA is increasingly recommended as the default for $K \geq 3$ independent groups. The power loss when variances are truly equal is negligible, while the Type I error protection when variances differ is substantial. DataStatPro reports both standard and Welch's ANOVA by default.

5.3 Trimmed Mean ANOVA (Robust)

Trimmed mean ANOVA (Wilcox, 2017) replaces standard means with $\alpha$-trimmed means (e.g., 20% trimming from each tail). It is substantially more powerful than the Kruskal-Wallis test for symmetric heavy-tailed distributions while controlling Type I error under non-normality. Available in DataStatPro under "Robust ANOVA."

5.4 Brown-Forsythe F-test

An alternative to Welch's ANOVA that is more robust to certain distributional departures. Uses median-centred deviations for variance estimation. DataStatPro provides this as an additional output alongside Welch's F.

5.5 Choosing Between Variants

6. Using the One-Way ANOVA Calculator Component

The One-Way ANOVA Calculator in DataStatPro provides a comprehensive tool for running, diagnosing, visualising, and reporting one-way ANOVA and its alternatives.

Step-by-Step Guide

Step 1 — Select "One-Way Between-Subjects ANOVA"

From the "Test Type" dropdown, choose:

• One-Way ANOVA (Standard): Equal variance assumption; Student's F.
• One-Way ANOVA (Welch's): Recommended default; no equal variance assumption.
• One-Way ANOVA (Auto): Runs both; uses Welch's when Levene's is significant.
• Kruskal-Wallis Test: Non-parametric alternative.

Step 2 — Input Method

Choose how to provide the data:

• Raw data (long format): Two columns — one for the DV values, one for group membership. DataStatPro computes all statistics, runs all assumption checks, and generates full output automatically.
• Raw data (wide format): One column per group. DataStatPro converts to long format.
• Summary statistics: Enter $K$, $n_j$, $\bar{x}_j$, and $s_j$ for each group. Full assumption checks are not available; all inferential statistics and effect sizes are computed.
• ANOVA table values: Enter $SS_B$, $SS_W$, $df_B$, $df_W$ (from a published paper) to compute effect sizes, power, and CIs.

Step 3 — Specify Group Labels

Enter descriptive names for each group (e.g., "CBT," "BA," "Waitlist"). These labels appear in all output tables, plots, and the auto-generated APA paragraph.

Step 4 — Select Assumption Checks

DataStatPro automatically runs and displays:

• ✅ Shapiro-Wilk normality test on residuals (and per group for small $n_j$).
• ✅ Levene's test for homogeneity of variance.
• ✅ Brown-Forsythe test for homogeneity of variance (alongside Levene's).
• ✅ Boxplots per group for outlier detection.
• ✅ Q-Q plot of residuals for normality assessment.
• ✅ Variance ratio ($s^2_{max}/s^2_{min}$) with warning if $> 4$.

Step 5 — Select Post-Hoc Tests

When the omnibus F is significant, select post-hoc tests:

• Tukey HSD (default for balanced, equal variances).
• Tukey-Kramer (unbalanced, equal variances).
• Games-Howell (recommended when Levene's significant or Welch's used).
• Bonferroni (conservative; any design).
• Holm-Bonferroni (less conservative than Bonferroni).
• Dunnett's (all vs. one control group).
• Scheffé (all possible contrasts).
• Custom planned contrasts (specify contrast weights $c_j$).

Step 6 — Select Effect Sizes

• ✅ $\omega^2$ (bias-corrected; primary).
• ✅ $\varepsilon^2$ (alternative bias correction).
• ✅ $\eta^2$ (biased; provided for comparison and journal requirements).
• ✅ Cohen's $f$ (for power analysis).
• ✅ 95% CIs for $\omega^2$ and $\eta^2$ via non-central F-distribution.
• ✅ Cohen's $d_{jk}$ with 95% CI for each post-hoc pairwise comparison.

Step 7 — Select Display Options

• ✅ Full ANOVA source table with $F$, df, $p$.
• ✅ Descriptive statistics table ($n_j$, $\bar{x}_j$, $SD_j$, $SE_j$, 95% CI per group).
• ✅ Effect size table ($\omega^2$, $\varepsilon^2$, $\eta^2$, $f$) with CIs.
• ✅ Post-hoc comparison table (all pairs: difference, $SE$, adjusted $p$, $d_{jk}$, 95% CI for difference).
• ✅ Assumption test results panel (colour-coded: green/yellow/red).
• ✅ Raincloud plot per group (half violin + boxplot + raw data points).
• ✅ Means plot with 95% CIs and individual data points.
• ✅ Cohen's $d$ diagram for each significant pairwise comparison.
• ✅ Power curve: power vs. $n$ for observed $\omega^2$.
• ✅ APA 7th edition-compliant results paragraph (auto-generated).

Step 8 — Run the Analysis

Click "Run One-Way ANOVA". DataStatPro will:

1. Compute the full ANOVA source table.
2. Run all assumption tests and display colour-coded warnings.
3. Automatically switch to Welch's ANOVA if Levene's test is significant (when "Auto" selected).
4. Compute all effect sizes with exact non-central F-based CIs.
5. Run all selected post-hoc tests with adjusted p-values and individual $d_{jk}$.
6. Generate all visualisations.
7. Auto-generate the APA-compliant results paragraph.

7. Full Step-by-Step Procedure

7.1 Complete Computational Procedure

This section walks through every computational step for one-way ANOVA, from raw data to a complete APA-style conclusion.

Given: $K$ groups with observations $x_{ij}$ for $i = 1,\ldots,n_j$ and $j = 1,\ldots,K$. Total $N = \sum_j n_j$.

Step 1 — State the Hypotheses

$H_0: \mu_1 = \mu_2 = \cdots = \mu_K$

$H_1:$ At least one pair $\mu_j \neq \mu_k$ for $j \neq k$

Choose $\alpha$ (default: $.05$).

Step 2 — Compute Descriptive Statistics per Group

For each group $j$:

$\bar{x}_j = \frac{1}{n_j}\sum_{i=1}^{n_j}x_{ij}$

$s_j = \sqrt{\frac{\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)^2}{n_j-1}}$

$SE_j = s_j/\sqrt{n_j}$

Step 3 — Compute the Grand Mean

$\bar{x}_{..} = \frac{\sum_{j=1}^K n_j\bar{x}_j}{N}$

Step 4 — Check Assumptions

Normality: Run Shapiro-Wilk on residuals $e_{ij} = x_{ij} - \bar{x}_j$. If $p_{SW} < .05$ and $n_j < 30$: consider Kruskal-Wallis.

Homoscedasticity: Run Levene's test. If $p_{Levene} < .05$: use Welch's ANOVA.

Outliers: Inspect boxplots and standardised residuals.

Step 5 — Compute Sums of Squares

$SS_{between} = \sum_{j=1}^K n_j(\bar{x}_j - \bar{x}_{..})^2$

$SS_{within} = \sum_{j=1}^K(n_j-1)s_j^2$

$SS_{total} = SS_{between} + SS_{within}$

Verification: $SS_{total}$ can also be computed directly as $\sum_j\sum_i(x_{ij}-\bar{x}_{..})^2$ — both must agree.

Step 6 — Compute Degrees of Freedom

$df_{between} = K-1, \quad df_{within} = N-K, \quad df_{total} = N-1$

Step 7 — Compute Mean Squares

$MS_{between} = SS_{between}/(K-1)$

$MS_{within} = SS_{within}/(N-K)$

Step 8 — Compute the F-Statistic and p-value

$F = MS_{between}/MS_{within}$

$p = P(F_{K-1,\;N-K} \geq F_{obs})$

Reject $H_0$ if $p \leq \alpha$.

Step 9 — Compute Effect Sizes

Eta squared (biased):

$\eta^2 = SS_{between}/SS_{total}$

Omega squared (preferred, bias-corrected):

$\omega^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total} + MS_{within}}$

Epsilon squared (alternative correction):

$\varepsilon^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total}}$

Cohen's $f$:

$f = \sqrt{\omega^2/(1-\omega^2)}$

Step 10 — Compute 95% CI for $\omega^2$

Using the non-central F-distribution (computed numerically by DataStatPro).

Non-centrality parameter: $\hat{\lambda} = (F-1) \times df_{between}$

Find $\lambda_L$, $\lambda_U$ such that $P(F_{df_B,df_W}(\lambda) \geq F_{obs}) = 0.025$ for each bound.

$\omega^2_L = \frac{\lambda_L - df_B}{(\lambda_L - df_B) + N}$ (approximate)

$\omega^2_U = \frac{\lambda_U - df_B}{(\lambda_U - df_B) + N}$ (approximate)

Step 11 — Conduct Post-Hoc Tests (if $F$ significant)

Select the appropriate post-hoc test (Section 9). Compute pairwise differences, standard errors, adjusted p-values, and individual Cohen's $d_{jk}$ for each pair.

Step 12 — Interpret and Report

Combine all results into an APA-compliant report (Section 13.7).

8. Effect Sizes for One-Way ANOVA

8.1 Eta Squared ($\eta^2$) — Common but Biased

$\eta^2 = \frac{SS_{between}}{SS_{total}}$

$\eta^2$ is the proportion of total sample variance explained by group membership. It is the most commonly reported ANOVA effect size and appears as default output in SPSS.

Critical limitation: $\eta^2$ is positively biased — it systematically overestimates the true population effect size, especially in small samples and when $K$ is large relative to $N$. The bias magnitude:

$\text{Bias} = \eta^2 - \omega^2 \approx \frac{(K-1)(1-\eta^2)}{N-1}$

For $K = 3$, $N = 30$: Bias $\approx 2(1-\eta^2)/29$ — can be several percentage points. For $K = 3$, $N = 300$: Bias $\approx 2(1-\eta^2)/299$ — negligible.

⚠️ Report $\eta^2$ only when explicitly required by a journal or for historical comparison. Always report $\omega^2$ (or $\varepsilon^2$) as the primary effect size and label $\eta^2$ as "biased" in your manuscript.

8.2 Omega Squared ($\omega^2$) — Preferred

$\omega^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total} + MS_{within}}$

$\omega^2$ is a bias-corrected estimate of the population proportion of variance explained by the IV. It is the recommended primary effect size for one-way ANOVA.

Properties:

• Can be slightly negative in small samples when the true effect is zero or near zero — because the correction overshoots. Report as 0 by convention when negative.
• Always $\leq \eta^2$ (the correction always reduces or maintains the estimate).
• Converges to $\eta^2$ as $N \to \infty$.
• The population parameter estimated: $\omega^2_{pop} = \sigma^2_{between}/(\sigma^2_{between}+\sigma^2_{within})$

From F-statistic and df (approximate):

$\omega^2 \approx \frac{(F-1)(K-1)}{(F-1)(K-1) + N}$

8.3 Epsilon Squared ($\varepsilon^2$) — Alternative Correction

$\varepsilon^2 = \frac{SS_{between} - (K-1)MS_{within}}{SS_{total}}$

$\varepsilon^2$ uses the same numerator as $\omega^2$ but divides by $SS_{total}$ instead of $SS_{total} + MS_{within}$.

Properties:

• Always between $\omega^2$ and $\eta^2$: $\omega^2 \leq \varepsilon^2 \leq \eta^2$.
• Slightly less bias-correction than $\omega^2$.
• Computationally simpler than $\omega^2$ (no addition of $MS_{within}$ in denominator).
• Increasingly reported alongside $\omega^2$ in recent literature.

8.4 Cohen's $f$ — For Power Analysis

$f = \sqrt{\frac{\eta^2}{1-\eta^2}}$ or $f = \sqrt{\frac{\omega^2}{1-\omega^2}}$

Cohen's $f$ is used as the effect size input for ANOVA power analysis. It represents the ratio of between-groups SD to within-groups SD.

From group means and $\sigma$ (when population parameters are known):

$f = \frac{\sigma_{\mu}}{\sigma}$

Where $\sigma_{\mu} = \sqrt{\sum_j n_j(\mu_j-\mu)^2/N}$ is the SD of the group means.

Benchmarks: Small = 0.10, Medium = 0.25, Large = 0.40 (Cohen, 1988).

8.5 Comparison of Effect Size Estimates

For a dataset with $K = 3$, $N = 45$ (15 per group), and $F(2, 42) = 8.50$:

$\eta^2 = \frac{8.50 \times 2}{8.50 \times 2 + 42} = \frac{17.00}{59.00} = 0.288$

$\omega^2 \approx \frac{(8.50-1)\times 2}{8.50\times 2 + 42 + 1} = \frac{15.00}{60.00} = 0.250$

$\varepsilon^2 = \frac{SS_B - (K-1)MS_W}{SS_T}$ (requires full SS values)

The difference $\eta^2 - \omega^2 = 0.038$ (almost 4 percentage points of overestimation from $\eta^2$) — substantial and worth correcting.

8.6 Effect Sizes for Pairwise Comparisons

After the omnibus F-test, report individual effect sizes for each significant pairwise comparison using $\sqrt{MS_{within}}$ as the standardiser:

Cohen's $d_{jk}$ (using pooled within-groups SD):

$d_{jk} = \frac{\bar{x}_j - \bar{x}_k}{\sqrt{MS_{within}}}$

Hedges' $g_{jk}$ (bias-corrected):

$g_{jk} = d_{jk} \times \left(1 - \frac{3}{4(N-K)-1}\right)$

Using $\sqrt{MS_{within}}$ from the full ANOVA model (rather than just the two-group pooled SD) is preferred because it is based on all $K$ groups and is therefore a more stable estimate of the common population SD.

95% CI for the pairwise mean difference:

$(\bar{x}_j - \bar{x}_k) \pm t_{\alpha/2,\;N-K} \times \sqrt{MS_{within}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

8.7 Omega Squared vs. Partial Omega Squared

In one-way ANOVA with a single IV:

$\omega^2 = \omega_p^2$ (they are identical for one-factor designs)

$\eta^2 = \eta_p^2$ (they are identical for one-factor designs)

The partial and non-partial versions diverge only in factorial (multi-factor) designs.

9. Post-Hoc Tests and Planned Contrasts

9.1 The Need for Post-Hoc Testing

A significant omnibus F-test establishes that at least one group mean differs from the others. Post-hoc tests (also called multiple comparison procedures) identify which specific pairs of groups differ, while controlling the FWER.

The key trade-off: Controlling the FWER requires more conservative critical values, which reduces power for individual comparisons. The choice of post-hoc test involves balancing Type I error control and statistical power.

9.2 Tukey's HSD — Standard Pairwise Comparisons

Tukey's Honestly Significant Difference (HSD) is the most widely used post-hoc test for balanced designs with equal variances. It controls the FWER at exactly $\alpha$ for all pairwise comparisons simultaneously.

Critical value: The studentised range distribution $q_{K,\;N-K,\;\alpha}$.

Minimum Significant Difference:

$\text{HSD} = q_{K,\;N-K,\;\alpha} \times \sqrt{\frac{MS_{within}}{n}}$ (balanced)

For unequal group sizes (Tukey-Kramer):

$\text{HSD}_{jk} = q_{K,\;N-K,\;\alpha} \times \sqrt{\frac{MS_{within}}{2}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

Declare groups $j$ and $k$ significantly different if $|\bar{x}_j - \bar{x}_k| > \text{HSD}_{jk}$.

95% CI for pairwise difference $\mu_j - \mu_k$:

$(\bar{x}_j-\bar{x}_k) \pm q_{K,\;N-K,\;\alpha}/\sqrt{2} \times \sqrt{MS_{within}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

9.3 Games-Howell — Unequal Variances

When Levene's test is significant or Welch's ANOVA is used, Games-Howell is the recommended post-hoc procedure. It uses separate variance estimates per pair:

$t_{jk} = \frac{\bar{x}_j-\bar{x}_k}{\sqrt{s_j^2/n_j+s_k^2/n_k}}$

Compared against the studentised range distribution with Welch-Satterthwaite df:

$\nu_{jk} = \frac{(s_j^2/n_j+s_k^2/n_k)^2}{(s_j^2/n_j)^2/(n_j-1)+(s_k^2/n_k)^2/(n_k-1)}$

9.4 Bonferroni and Holm-Bonferroni Corrections

Bonferroni correction (simplest, most conservative):

Compare each pairwise p-value to $\alpha^* = \alpha/m$ where $m = K(K-1)/2$.

Holm-Bonferroni sequential procedure (less conservative, same FWER control):

1. Sort the $m$ p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$.
2. Compare $p_{(i)}$ to $\alpha/(m-i+1)$.
3. Reject all $H_{0(i)}$ for which $p_{(j)} \leq \alpha/(m-j+1)$ for all $j \leq i$.

Holm-Bonferroni is uniformly more powerful than Bonferroni and should be preferred in all cases.

9.5 Dunnett's Test — All Groups vs. One Control

When comparing $K-1$ experimental groups to a single control group (and not making comparisons among experimental groups), Dunnett's test provides optimal power while controlling FWER.

$m = K-1$ comparisons (each experimental group vs. control)

Compared against Dunnett's distribution (not the studentised range) with parameters $(K-1, N-K)$.

9.6 Planned Contrasts — A Priori Comparisons

Planned contrasts are specific, theoretically motivated comparisons formulated before data collection. They are more powerful than post-hoc tests because:

• They do not require a significant omnibus F-test (though conducting the F-test first is still recommended).
• Fewer comparisons means less severe FWER correction.
• Orthogonal planned contrasts do not require any correction.

Contrast specification: A contrast is a linear combination $\psi = \sum_{j=1}^K c_j\mu_j$ with the constraint $\sum_{j=1}^K c_j = 0$.

Contrast SS and F:

$SS_\psi = \frac{\left(\sum_j c_j\bar{x}_j\right)^2}{\sum_j c_j^2/n_j}$

$F_\psi = SS_\psi/MS_{within}$, compared to $F_{1,\;N-K}$

Orthogonality condition (two contrasts $c$ and $c'$ are orthogonal if):

$\sum_{j=1}^K \frac{c_j c_j'}{n_j} = 0$

A set of $K-1$ mutually orthogonal contrasts fully partitions $SS_{between}$:

$\sum_{\psi=1}^{K-1} SS_\psi = SS_{between}$

Example for $K = 4$ groups (Control, Drug A, Drug B, Drug C):

These three contrasts are mutually orthogonal (for equal $n_j$) and decompose $SS_{between}$ into three orthogonal components — no FWER correction needed.

10. Confidence Intervals

10.1 95% CI for Each Group Mean

The 95% CI for the population mean $\mu_j$:

$\bar{x}_j \pm t_{\alpha/2,\;N-K} \times \sqrt{MS_{within}/n_j}$

Note: This CI uses $MS_{within}$ from the full ANOVA model (not $s_j$), producing a more stable estimate that borrows strength from all groups (valid under homoscedasticity).

10.2 95% CI for Pairwise Mean Differences

The 95% CI for $\mu_j - \mu_k$:

$(\bar{x}_j-\bar{x}_k) \pm t_{\alpha/2,\;N-K} \times \sqrt{MS_{within}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

Using Tukey-adjusted critical values (simultaneous 95% CI for all pairs):

$(\bar{x}_j-\bar{x}_k) \pm \frac{q_{K,\;N-K,\;\alpha}}{\sqrt{2}} \times \sqrt{MS_{within}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

Tukey-adjusted CIs are wider but simultaneously valid for all $\binom{K}{2}$ pairs.

10.3 95% CI for $\omega^2$

The exact CI uses the non-central F-distribution (DataStatPro computes this numerically). The CI communicates the precision of the effect size estimate and is required for complete reporting.

CI width as a function of $N$ and $K$ (for $\omega^2 = 0.10$):

⚠️ With only 30 participants ($n = 10$/group), the 95% CI for $\omega^2 = 0.10$ spans approximately $[0.00, 0.24]$ — essentially uninformative about the true effect magnitude. Always report the CI alongside the point estimate.

11. Power Analysis and Sample Size Planning

11.1 A Priori Power Analysis

A priori power analysis determines the required sample size before data collection to achieve desired power $1-\beta$ at significance level $\alpha$ for a hypothesised effect of size $f$.

Non-centrality parameter: $\lambda = f^2 \times N = f^2 \times Kn$ (balanced design)

Power computation (exact, using non-central F):

$\text{Power} = P\!\left(F_{K-1,\;K(n-1)}(\lambda) > F_{crit}\right)$

Where $F_{crit} = F_{\alpha,\;K-1,\;K(n-1)}$ and $\lambda = f^2 \times Kn$.

No closed form exists — DataStatPro uses numerical integration of the non-central F-distribution.

Approximate $n$ per group:

$n \approx \frac{(z_{1-\alpha/(K-1)} + z_{1-\beta})^2}{f^2} + 1$

Required $n$ per group for 80% power ($\alpha = .05$, two-sided):

11.2 Determining $f$ from Prior Literature

When prior studies report $\eta^2$ or $\omega^2$:

$f = \sqrt{\frac{\omega^2_{prior}}{1-\omega^2_{prior}}}$

When prior studies report group means and a common SD estimate:

$f = \frac{\text{SD of group means}}{\text{estimated within-group SD}} = \frac{\sqrt{\sum_j(\mu_j-\mu)^2/K}}{\sigma}$

When only a t-statistic from a pilot study with two groups is available:

$f = \frac{|d|}{2}$ (approximate, for a two-group pilot)

11.3 Sensitivity Analysis

The minimum detectable $f$ for a given $N$, $K$, and target power: