ANCOVA: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of covariance adjustment all the way through the mathematics, assumptions, effect sizes, post-hoc testing, non-parametric alternatives, interpretation, reporting, and practical usage of the Analysis of Covariance (ANCOVA) within the DataStatPro application. Whether you are encountering ANCOVA for the first time or seeking a rigorous, unified understanding of covariate-adjusted between-groups inference, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is ANCOVA?
3. The Mathematics Behind ANCOVA
4. Assumptions of ANCOVA
5. Variants of ANCOVA
6. Using the ANCOVA Calculator Component
7. Full Step-by-Step Procedure
8. Effect Sizes for ANCOVA
9. Post-Hoc Tests and Planned Contrasts
10. Confidence Intervals
11. Power Analysis and Sample Size Planning
12. Non-Parametric Alternative: Quade and Ranked ANCOVA
13. Advanced Topics
14. Worked Examples
15. Common Mistakes and How to Avoid Them
16. Troubleshooting
17. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into ANCOVA, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 From One-Way ANOVA to ANCOVA

One-Way ANOVA tests whether group means on a dependent variable (DV) differ beyond what chance alone would produce. However, ANOVA assumes that groups are equivalent on all variables except the independent variable (IV) — an assumption satisfied by random assignment in experiments, but rarely in observational research.

ANCOVA extends ANOVA by:

1. Statistically controlling for one or more continuous covariates (CVs) that are correlated with the DV.
2. Removing covariate-related variance from the error term, increasing statistical power.
3. Adjusting group means to what they would be if all groups had identical covariate scores — producing adjusted means (also called estimated marginal means).

1.2 What is a Covariate?

A covariate (also called a concomitant variable) is a continuous variable that:

• Is correlated with the DV.
• Is not manipulated by the researcher (it is measured, not assigned).
• Is measured before the treatment or is logically prior to the treatment effect.

Examples:

• Pre-test score as a covariate when the DV is a post-test score.
• Age as a covariate when the DV is a cognitive outcome.
• Baseline anxiety when the DV is post-treatment anxiety.
• IQ when the DV is academic achievement.

The covariate should be chosen on theoretical grounds, not selected post-hoc because it "improves" the results. Including irrelevant covariates reduces power by consuming degrees of freedom.

1.3 The Two Goals of ANCOVA

ANCOVA serves two distinct but related purposes, and understanding which goal applies to your study is critical for correct interpretation:

Goal 1 — Increase Statistical Power (Experimental Designs)

In randomised experiments, groups are equal on average at baseline (including on the covariate). Including a covariate reduces $MS_{error}$ by removing variance explained by the covariate from the residual. This shrinks the denominator of the F-ratio, increasing power to detect treatment effects.

Goal 2 — Statistical Control (Quasi-Experimental and Observational Designs)

In non-randomised designs, groups may differ on the covariate at baseline. ANCOVA adjusts group means to a common covariate value, providing a partial statistical control for pre-existing differences. However, this control is imperfect and cannot fully substitute for randomisation.

⚠️ These two goals have different interpretational requirements. For Goal 1 (randomised experiments), ANCOVA assumptions are easily met and interpretation is straightforward. For Goal 2 (observational designs), the assumption of covariate independence from group membership is violated by design, requiring careful interpretational caveats about residual confounding.

1.4 The Regression Foundation of ANCOVA

ANCOVA is a special case of the General Linear Model (GLM):

$Y_i = \mu + \tau_j + \beta(X_i - \bar{X}) + \varepsilon_i$

Where:

• $Y_i$ is the DV score for participant $i$.
• $\tau_j$ is the effect of group $j$ ($\sum_j \tau_j = 0$).
• $X_i$ is the covariate score for participant $i$.
• $\bar{X}$ is the grand mean of the covariate.
• $\beta$ is the common within-group regression coefficient (slope) of $Y$ on $X$.
• $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ is the residual error.

This is equivalent to a multiple regression model with group dummy codes and the covariate as predictors. The F-test for the group effect in ANCOVA tests whether groups differ after partialling out the covariate.

1.5 Variance Partitioning in ANCOVA

ANCOVA partitions the total sum of squares differently from ANOVA:

$SS_{total} = SS_{between(adj)} + SS_{covariate} + SS_{within(adj)}$

The covariate "absorbs" variance from the error term. The adjusted within-groups SS ($SS_{within(adj)}$) is smaller than the unadjusted $SS_{within}$, leading to a smaller $MS_{error}$ and greater power — provided the covariate is genuinely correlated with the DV.

1.6 Adjusted Means

The core output of ANCOVA is the adjusted group mean — the estimated group mean after removing the linear effect of the covariate:

$\bar{Y}_{j(adj)} = \bar{Y}_j - \hat{\beta}(\bar{X}_j - \bar{X}_{..})$

Where:

• $\bar{Y}_j$ is the observed (unadjusted) mean of group $j$.
• $\hat{\beta}$ is the pooled within-group regression coefficient.
• $\bar{X}_j$ is the mean covariate score in group $j$.
• $\bar{X}_{..}$ is the grand mean of the covariate.

The adjusted mean represents what the group mean would have been if all groups had the same average covariate score ($\bar{X}_{..}$). These are also called Estimated Marginal Means (EMMs) and are the primary means for interpretation and post-hoc testing in ANCOVA.

1.7 The Homogeneity of Regression Slopes Assumption

Unlike ANOVA, ANCOVA carries a critical additional assumption: the within-group regression slope of $Y$ on $X$ must be the same in all $K$ groups. If the slope varies across groups, the covariate adjustment is non-uniform, the ANCOVA F-test is invalid, and an interaction model (covariate × group) is more appropriate. This is the most commonly violated and overlooked ANCOVA assumption.

1.8 ANCOVA vs. Gain Score Analysis

A common alternative to ANCOVA for pre-post designs is to compute gain scores (post − pre) and run a one-way ANOVA. The choice between ANCOVA and gain score analysis depends on the reliability and variability of the pre-test:

• ANCOVA is preferred when the pre-test is highly reliable and when groups have different pre-test means (common in quasi-experimental designs).
• Gain score ANOVA is preferred when pre-test scores are unreliable (Lord's paradox arises under certain conditions).
• Both approaches are valid for fully randomised experiments; ANCOVA generally has more power.

2. What is ANCOVA?

2.1 The Core Idea

Analysis of Covariance (ANCOVA) is a parametric inferential procedure that combines one-way ANOVA with linear regression. It tests whether the adjusted means of $K \geq 2$ independent groups are simultaneously equal, after statistically controlling for one or more continuous covariates.

The ANCOVA omnibus null hypothesis:

$H_0: \mu_{1(adj)} = \mu_{2(adj)} = \cdots = \mu_{K(adj)}$

$H_1: \text{At least one adjusted mean } \mu_{j(adj)} \text{ differs from the others}$

The adjusted means are population means evaluated at the grand mean of the covariate:

$\mu_{j(adj)} = \mu_j - \beta(\mu_{X_j} - \mu_X)$

2.2 What ANCOVA Tests and Does Not Test

ANCOVA tells you:

• Whether adjusted group mean differences are larger than expected by chance, after accounting for the covariate (omnibus test).
• How much of the covariate-adjusted outcome variance is explained by group membership ($\omega^2_p$, $\eta^2_p$).
• The direction and statistical significance of the covariate's relationship with the DV.
• What group means would look like if groups were equated on the covariate.

ANCOVA does NOT tell you:

• Which specific adjusted group means differ (requires post-hoc tests on adjusted means).
• Whether the treatment caused the DV change in non-randomised designs (residual confounding may remain).
• The effect size for individual pairwise adjusted mean differences (requires Cohen's $d_{adj}$).
• Whether the homogeneity of regression slopes assumption holds (must be tested separately).

2.3 Design Requirements

For a one-way between-subjects ANCOVA, the design must satisfy:

• One continuous DV (interval or ratio scale).
• One categorical IV with $K \geq 2$ levels (groups).
• One or more continuous covariates (interval or ratio scale), measured prior to or independently of the treatment.
• Different participants in each group (independent samples).
• Each participant contributes exactly one score to exactly one group.

2.4 ANCOVA in Context

2.5 Real-World Applications

3. The Mathematics Behind ANCOVA

3.1 Notation

3.2 The Within-Group Regression Coefficient

ANCOVA uses the pooled within-group regression coefficient $\hat{\beta}$, computed from the pooled within-group sums of cross-products:

Within-group sum of squares for the covariate:

$SS_{XX(W)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_j)^2$

Within-group sum of cross-products (covariate × DV):

$SP_{XY(W)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_j)(y_{ij} - \bar{y}_j)$

Pooled within-group regression coefficient:

$\hat{\beta} = \frac{SP_{XY(W)}}{SS_{XX(W)}}$

This slope represents the average linear relationship between the covariate and DV within groups, pooled across all $K$ groups. It is equivalent to the slope obtained from a regression of $Y$ on $X$ with all between-group variance removed.

3.3 Adjusted Group Means

The adjusted group mean for group $j$:

$\bar{y}_{j(adj)} = \bar{y}_j - \hat{\beta}(\bar{x}_j - \bar{x}_{..})$

Interpretation: The adjusted mean is the estimated group mean when the group's covariate mean equals the grand mean of the covariate. It answers: "What would Group $j$'s mean DV score be if they had, on average, the same covariate score as the entire sample?"

Adjusted grand mean:

$\bar{y}_{..(adj)} = \frac{\sum_{j=1}^K n_j \bar{y}_{j(adj)}}{N} = \bar{y}_{..}$

The adjusted grand mean equals the unadjusted grand mean (covariate adjustment preserves the overall mean).

3.4 Sum of Squares Decomposition in ANCOVA

ANCOVA involves computing adjusted sums of squares by removing the linear effect of the covariate from both the total and within-group SS.

Total sum of squares for DV (unadjusted):

$SS_{YY(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(y_{ij} - \bar{y}_{..})^2$

Within-group sum of squares for DV (unadjusted):

$SS_{YY(W)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(y_{ij} - \bar{y}_j)^2 = \sum_{j=1}^K(n_j-1)s_{y_j}^2$

Total sum of squares for covariate:

$SS_{XX(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_{..})^2$

Total sum of cross-products:

$SP_{XY(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_{..})(y_{ij} - \bar{y}_{..})$

Adjusted within-groups SS (error after covariate removal):

$SS_{within(adj)} = SS_{YY(W)} - \frac{SP_{XY(W)}^2}{SS_{XX(W)}}$

The second term is the reduction in error SS due to the covariate. It equals $\hat{\beta}^2 \times SS_{XX(W)}$ — the within-group regression of $Y$ on $X$.

Adjusted total SS:

$SS_{total(adj)} = SS_{YY(T)} - \frac{SP_{XY(T)}^2}{SS_{XX(T)}}$

Adjusted between-groups SS:

$SS_{between(adj)} = SS_{total(adj)} - SS_{within(adj)}$

Verification:

$SS_{between(adj)} = SS_{total(adj)} - SS_{within(adj)}$

Note: In ANCOVA, $SS_{total(adj)} \neq SS_{between(adj)} + SS_{within(adj)}$ if computed from raw unadjusted SS — it is the adjusted versions that sum correctly.

3.5 Degrees of Freedom

The key difference from ANOVA: the error $df$ is reduced by $p$ (one per covariate), because each covariate costs one degree of freedom to estimate its slope. This is why including irrelevant covariates (low correlation with DV) reduces power despite removing some variance.

Power gain from covariate requires:

$\frac{r^2_{XY}(N - K - p)}{1} > \frac{p}{1}$

Simplified: the covariate must explain more variance than the $df$ it consumes. For a single covariate ($p = 1$), any $|r_{XY}| > 0$ provides power gain when $N$ is large enough.

Break-even correlation for a single covariate:

$r^2_{break-even} = \frac{1}{N - K}$

For $N = 60$, $K = 3$: $r_{break-even} = \sqrt{1/57} = 0.133$. Any $|r_{XY}| > 0.133$ within groups yields higher power from ANCOVA than from one-way ANOVA.

3.6 Mean Squares and the F-Ratio

Adjusted between-groups mean square:

$MS_{between(adj)} = \frac{SS_{between(adj)}}{K-1}$

Adjusted within-groups mean square (adjusted error variance):

$MS_{error(adj)} = \frac{SS_{within(adj)}}{N - K - p}$

The ANCOVA F-statistic:

$F = \frac{MS_{between(adj)}}{MS_{error(adj)}}$

Under $H_0$: $F \sim F_{K-1,\;N-K-p}$

p-value:

$p = P(F_{K-1,\;N-K-p} \geq F_{obs})$

3.7 The F-Test for the Covariate

ANCOVA also produces an F-test for the covariate itself:

$F_{cov} = \frac{SS_{covariate}/p}{MS_{error(adj)}}$

Where:

$SS_{covariate} = \hat{\beta}^2 \times SS_{XX(W)} = \frac{SP_{XY(W)}^2}{SS_{XX(W)}}$

This tests whether the pooled within-group regression slope $\hat{\beta}$ is significantly different from zero. A non-significant covariate F-test suggests the covariate is not linearly related to the DV within groups, and including it may reduce power.

3.8 The ANCOVA Source Table

3.9 Computing the Pooled Within-Group Correlation

The pooled within-group correlation between the covariate and DV:

$r_{XY(W)} = \frac{SP_{XY(W)}}{\sqrt{SS_{XX(W)} \times SS_{YY(W)}}}$

This correlation quantifies the linear relationship between covariate and DV within groups, pooled across groups. It determines how much variance the covariate removes from the error term.

Percentage of within-group variance explained by covariate:

$r^2_{XY(W)} = \frac{SP_{XY(W)}^2}{SS_{XX(W)} \times SS_{YY(W)}}$

Percentage reduction in error SS:

$\frac{SS_{YY(W)} - SS_{within(adj)}}{SS_{YY(W)}} = r^2_{XY(W)}$

3.10 Adjusted Standard Error for Adjusted Means

The standard error of an adjusted group mean:

$SE_{j(adj)} = \sqrt{MS_{error(adj)}\left(\frac{1}{n_j} + \frac{(\bar{x}_j - \bar{x}_{..})^2}{SS_{XX(W)}}\right)}$

The second term inside the square root captures additional uncertainty from the fact that the covariate mean of group $j$ may deviate from the grand mean. When $\bar{x}_j = \bar{x}_{..}$, this term vanishes and $SE_{j(adj)} = \sqrt{MS_{error(adj)}/n_j}$.

3.11 Multiple Covariates

With $p$ covariates, the ANCOVA model becomes:

$Y_i = \mu + \tau_j + \beta_1(X_{1i} - \bar{X}_1) + \beta_2(X_{2i} - \bar{X}_2) + \cdots + \beta_p(X_{pi} - \bar{X}_p) + \varepsilon_i$

The adjusted SS are computed using matrix algebra (the GLM framework):

$SS_{within(adj)} = SS_{YY(W)} - \mathbf{b}'\mathbf{S}_{XX(W)}\mathbf{b}$

Where $\mathbf{b} = \mathbf{S}_{XX(W)}^{-1}\mathbf{s}_{XY(W)}$ is the vector of pooled within-group regression coefficients, $\mathbf{S}_{XX(W)}$ is the pooled within-group covariance matrix of covariates, and $\mathbf{s}_{XY(W)}$ is the pooled within-group covariance vector between covariates and DV.

DataStatPro handles multiple covariates automatically using matrix algebra in its GLM engine.

3.12 Computing Effect Sizes from the ANCOVA Table

Partial eta squared (from F):

$\eta^2_p = \frac{F \cdot df_B}{F \cdot df_B + df_{error(adj)}}$

Partial omega squared (bias-corrected, preferred):

$\omega^2_p \approx \frac{(F-1)(K-1)}{(F-1)(K-1) + N}$

Exact partial omega squared (from SS):

$\omega^2_p = \frac{SS_{B(adj)} - (K-1)MS_{error(adj)}}{SS_{T(adj)} + MS_{error(adj)}}$

4. Assumptions of ANCOVA

ANCOVA carries all the assumptions of one-way ANOVA plus three additional covariate-specific assumptions. Violating the covariate assumptions is more consequential than violating standard ANOVA assumptions.

4.1 Normality of Residuals

The adjusted residuals $e_{ij} = y_{ij} - \bar{y}_{j(adj)} - \hat{\beta}(x_{ij} - \bar{x}_j)$ must be normally distributed within each group.

How to check:

• Shapiro-Wilk test on adjusted residuals ($H_0$: residuals are normal).
• Q-Q plot of adjusted residuals: points should follow the diagonal.
• Histograms per group of adjusted residuals.
• Skewness and kurtosis of adjusted residuals: $|z_{skew}| < 2$, $|z_{kurt}| < 7$.

Robustness: ANCOVA is robust to mild non-normality, especially with balanced designs and $n_j \geq 20$ per group. The robustness applies to the F-test for the group effect; tests on the covariate coefficient are also robust with moderate $N$.

When violated: Use the Quade test (non-parametric ANCOVA) or ranked ANCOVA as described in Section 12. Consider log or square root transformations for right-skewed DVs.

4.2 Homogeneity of Variance (Homoscedasticity)

The adjusted within-group variances must be equal across all $K$ groups:

$\sigma^2_{1(adj)} = \sigma^2_{2(adj)} = \cdots = \sigma^2_{K(adj)}$

This applies to the residuals after removing the covariate effect.

How to check:

• Levene's test on the adjusted residuals (most commonly used).
• Brown-Forsythe test on adjusted residuals (more robust to non-normality).
• Variance ratio rule: If $s^2_{max(adj)}/s^2_{min(adj)} > 4$, heterogeneity is potentially problematic.

When violated: Use a heteroscedastic ANCOVA (Welch-type adjustment) or non-parametric alternatives. Report the violation and the robust alternative results alongside standard ANCOVA results.

4.3 Independence of Observations

All observations must be independent within and across groups. This is a design assumption that cannot be tested from data.

Common violations:

• Clustered data (students nested in classrooms).
• Repeated measurements from the same participant.
• Social network dependencies among participants.

When violated: Use multilevel ANCOVA (covariate in mixed-effects model) or repeated measures ANCOVA.

4.4 Interval Scale of Measurement

Both the DV and the covariate(s) must be measured on at least an interval scale. The covariate must be continuous or at minimum have many ordered categories.

When violated for DV: Use the Quade test or ranked ANCOVA.

When violated for covariate: If the covariate is binary (0/1), include it as a categorical factor rather than a continuous covariate (use two-way ANOVA or ANCOVA with the binary variable as a blocking factor).

4.5 Homogeneity of Regression Slopes ⭐ CRITICAL

The most important ANCOVA-specific assumption: the within-group regression slope of $Y$ on $X$ must be the same in all $K$ groups:

$\beta_1 = \beta_2 = \cdots = \beta_K = \beta$

Why this matters: ANCOVA uses a single pooled slope $\hat{\beta}$ to adjust all group means. If the true slopes differ across groups, the single adjustment is incorrect for at least some groups — the adjusted means are meaningless.

Conceptually: The homogeneity of regression slopes assumption requires that the covariate-DV relationship is parallel across groups. Heterogeneous slopes indicate a covariate × group interaction — the effect of the covariate on the DV differs depending on group membership.

How to check:

• Test the covariate × group interaction in a separate model:

  $Y_i = \mu + \tau_j + \beta(X_i - \bar{X}) + \gamma_j(X_i - \bar{X}) + \varepsilon_i$

  where $\gamma_j$ represents the deviation of group $j$'s slope from the common slope.

• The interaction $F$-test ($H_0$: all $\gamma_j = 0$) tests homogeneity of slopes.

  $F_{interaction} = \frac{(SS_{within(adj,\; homogeneous)} - SS_{within(adj,\; heterogeneous)})/(K-1)}{MS_{error(heterogeneous)}}$

• Rule: If the interaction $F$ is significant at $p < .05$, the homogeneity of regression slopes assumption is violated and standard ANCOVA should not be used.

• Visual check: Plot the regression lines of $Y$ on $X$ separately for each group. Lines should be approximately parallel.

When violated:

• Do not use standard ANCOVA.
• Use Johnson-Neyman analysis (see Section 13.3) to identify regions of the covariate where groups differ significantly.
• Use moderated regression with the group × covariate interaction term.
• Report the interaction as a finding in its own right.

4.6 Independence of Covariate and Treatment (Group Membership)

For ANCOVA to provide valid adjusted means, the covariate must be independent of (i.e., not caused by) the treatment. Specifically:

• In randomised experiments: The covariate must be measured before random assignment. A pre-test score measured before treatment satisfies this assumption. Post-randomisation covariates (measured after treatment begins) may be influenced by the treatment, making adjustment inappropriate.

• In observational studies: Groups systematically differ on the covariate by design (e.g., younger vs. older participants). ANCOVA adjusts for this, but the adjustment is conditional on the model being correctly specified.

How to check:

• Verify that the covariate was measured before treatment.
• Check whether groups differ significantly on the covariate: run a one-way ANOVA with the covariate as the DV and the IV as the grouping factor.
  • In randomised experiments: this test should be non-significant (covariate balance is expected from randomisation). A significant result may indicate randomisation failure.
  • In observational studies: this test will typically be significant (groups differ on the covariate by design). ANCOVA provides statistical control, not a substitute for randomisation.

When violated (covariate influenced by treatment):

• Removing the variance explained by a post-treatment covariate may remove part of the treatment effect itself — over-controlling bias.
• Use causal diagram analysis (DAG) to determine appropriate covariate adjustment.

4.7 Linearity of Covariate-DV Relationship

ANCOVA assumes the relationship between the covariate and DV is linear within each group. Non-linear relationships are not fully removed by the linear adjustment, leaving residual covariate variance in the error term.

How to check:

• Scatterplots of $Y$ vs. $X$ within each group: relationship should appear linear.
• Residual plots: Plot adjusted residuals vs. covariate scores. A U-shaped or inverted-U pattern indicates non-linearity.
• Polynomial test: Add $X^2$ to the ANCOVA model and test whether it contributes significantly. If $F(X^2)$ is significant, the linear assumption may be violated.

When violated:

• Add a quadratic term $X^2$ to the ANCOVA model (polynomial ANCOVA).
• Use spline regression for flexible non-linear adjustment.
• Apply a transformation to the covariate (e.g., log $X$ for right-skewed covariates).

4.8 Reliability of the Covariate

ANCOVA assumes the covariate is measured without error. In practice, all psychological and behavioural measures contain measurement error. Measurement error in the covariate causes incomplete adjustment — residual confounding remains even after ANCOVA.

Consequences of covariate unreliability:

• Under-adjustment: The adjusted means do not fully reflect what group means would be at a common true covariate score.
• In randomised experiments: small downward bias in power (measurement error in the covariate leaves some adjustable variance in the error term).
• In observational studies: systematic bias in adjusted means — groups that score higher on the unreliable covariate will be over-adjusted; lower-scoring groups will be under-adjusted. This can produce misleading conclusions about treatment effects.

Remedy:

• Use reliability-corrected ANCOVA (Porter & Raudenbush, 1987).
• Report the reliability (Cronbach's $\alpha$, test-retest $r$) of the covariate.
• Acknowledge unreliability as a limitation when it is low ($\alpha < 0.80$).

4.9 Absence of Influential Outliers

Outliers on the covariate or DV can distort $\hat{\beta}$ and the adjusted means substantially.

How to check:

• Cook's distance for each observation in the ANCOVA regression model: $D_i > 1$ flags influential observations.
• Leverage values ($h_{ii}$): High leverage points have extreme covariate scores that pull the regression slope.
• Studentised deleted residuals: $|t_i| > 3$ flags potential outliers on the DV after covariate adjustment.
• Scatterplots of $Y$ vs. $X$ with group labels to visually identify extreme points.

4.10 Assumption Summary Table

5. Variants of ANCOVA

5.1 Standard One-Way ANCOVA

The default ANCOVA with a single continuous covariate, assuming homogeneity of regression slopes, normality of adjusted residuals, and homoscedasticity. Uses the pooled within-group regression slope for adjustment. This is appropriate when all assumptions are met.

5.2 ANCOVA with Multiple Covariates

When two or more covariates are available and each contributes unique variance to the DV, including all of them in ANCOVA maximises power. The mathematical extension uses multiple regression within the GLM framework (Section 3.11).

Guidelines for multiple covariates:

• Include only theoretically motivated covariates chosen before data collection.
• Avoid including covariates that are highly correlated with each other (multicollinearity degrades slope estimation).
• Each additional covariate costs one $df_{error}$; the power benefit must exceed this cost ($r^2_{X_p Y(W)} > 1/(N-K-p+1)$).
• With $p$ covariates and small $N$, ANCOVA can become over-parameterised.

5.3 Welch-Type Heteroscedastic ANCOVA

Analogous to Welch's one-way ANOVA, this variant relaxes the assumption of equal adjusted within-group variances. It uses group-specific variance estimates in the F-test denominator, with Welch-Satterthwaite degrees of freedom correction.

DataStatPro implements heteroscedastic ANCOVA using the HC3 heteroscedasticity- consistent variance estimator for the group effect test.

Use when: Levene's test on adjusted residuals is significant (especially with unequal $n_j$).

5.4 ANCOVA with Categorical Covariate (Blocking Factor)

When the "covariate" is categorical (e.g., site, school, gender), it functions as a blocking factor rather than a continuous covariate. This is handled as a two-way ANOVA (main effect of group + main effect of block) rather than ANCOVA.

DataStatPro handles this automatically: selecting a categorical variable as a covariate prompts the user to reclassify it as a blocking factor in a factorial ANOVA design.

5.5 ANCOVA for Pre-Post Designs (Pre-Test as Covariate)

The most common ANCOVA application in clinical and educational research:

• DV = post-test score
• Covariate = pre-test score (same measure, administered before treatment)
• IV = treatment group

This design removes pre-existing individual differences (captured by the pre-test) from the error term, isolating treatment effects on change from baseline while controlling for regression to the mean.

Advantages over gain score analysis:

• More power when the pre-post correlation is moderate to high ($r > 0.50$).
• Corrects for regression to the mean more effectively than gain scores.
• Adjusted means are interpretable as "what the post-test would be if groups had equal pre-test scores."

5.6 Johnson-Neyman ANCOVA (Heterogeneous Slopes)

When the homogeneity of regression slopes assumption is violated (significant group × covariate interaction), Johnson-Neyman analysis identifies the region of the covariate where the group difference is statistically significant and the region where it is not.

The Johnson-Neyman boundary point(s) are the covariate values at which the adjusted group difference transitions from significant to non-significant. Between $K = 2$ groups:

$X_{JN} = \bar{X} \pm \frac{t_{crit}\sqrt{SE^2_{b_1-b_2} \cdot F_{crit} - (\bar{X}_1 - \bar{X}_2)^2 \cdot \text{something}}}{b_1 - b_2}$

DataStatPro computes Johnson-Neyman regions numerically and displays them as a floodlight plot (significance region shaded along the covariate axis).

5.7 Choosing Between Variants

6. Using the ANCOVA Calculator Component

The ANCOVA Calculator in DataStatPro provides a comprehensive tool for running, diagnosing, visualising, and reporting ANCOVA and its alternatives.

Step-by-Step Guide

Step 1 — Select "ANCOVA"

From the "Test Type" dropdown, choose:

• ANCOVA (Standard): Equal regression slopes and equal variances assumed.
• ANCOVA (Heteroscedastic): HC3 variance-corrected; no equal variance assumption.
• ANCOVA (Auto): Runs standard ANCOVA, then switches to heteroscedastic if Levene's test on adjusted residuals is significant.
• Quade Test: Non-parametric ANCOVA alternative.

Step 2 — Input Method

Choose how to provide the data:

• Raw data (long format): Three or more columns — DV values, group membership, and covariate(s). DataStatPro computes all statistics, runs all assumption checks, and generates full output automatically.
• Raw data (wide format): One column per group for DV, plus covariate column(s). DataStatPro converts to long format.
• Summary statistics: Enter $K$, $n_j$, $\bar{y}_j$, $s_{y_j}$, $\bar{x}_j$, $s_{x_j}$, and within-group covariance $s_{xy_j}$ for each group. Full assumption checks (especially homogeneity of slopes) are limited; inferential statistics and effect sizes are computed.

Step 3 — Specify Variables

• Dependent Variable (DV): Select the continuous outcome column.
• Independent Variable (IV / Group): Select the categorical grouping column.
• Covariate(s): Select one or more continuous covariate columns. For multiple covariates, select all relevant columns; DataStatPro enters them simultaneously in the GLM.
• Group Labels: Enter descriptive names for each group level.

Step 4 — Select Assumption Checks

DataStatPro automatically runs and displays:

• ✅ Homogeneity of regression slopes test (group × covariate interaction F-test) — most critical ANCOVA assumption.
• ✅ Shapiro-Wilk test on adjusted residuals.
• ✅ Levene's test on adjusted residuals for homoscedasticity.
• ✅ Brown-Forsythe test on adjusted residuals.
• ✅ Q-Q plot of adjusted residuals.
• ✅ Scatterplot of DV vs. covariate with separate regression lines per group (visual check for parallel slopes).
• ✅ Linearity check: Residual vs. covariate plot; optional polynomial test.
• ✅ Cook's distance and leverage plot for influential observations.
• ✅ Covariate balance test (one-way ANOVA with covariate as DV).
• ✅ Variance ratio ($s^2_{max(adj)}/s^2_{min(adj)}$) with warning if $> 4$.

Step 5 — Select Post-Hoc Tests

When the omnibus F is significant, select post-hoc tests on adjusted means:

• Tukey HSD on adjusted means (balanced designs, equal variances; default).
• Tukey-Kramer on adjusted means (unbalanced, equal variances).
• Games-Howell on adjusted means (unequal variances; use with heteroscedastic ANCOVA).
• Bonferroni on adjusted means (conservative; any design).
• Holm-Bonferroni on adjusted means (less conservative than Bonferroni).
• Dunnett's on adjusted means (all vs. one control group).
• Scheffé on adjusted means (all possible contrasts).
• Custom planned contrasts on adjusted means (specify contrast weights $c_j$).

Step 6 — Select Effect Sizes

• ✅ Partial $\omega^2_p$ (bias-corrected; primary for group effect).
• ✅ Partial $\varepsilon^2_p$ (alternative bias correction).
• ✅ Partial $\eta^2_p$ (biased; provided for comparison and journal requirements).
• ✅ Cohen's $f_p$ (for power analysis).
• ✅ $\eta^2_p$ for covariate (variance explained by covariate, partialling group).
• ✅ 95% CIs for $\omega^2_p$ and $\eta^2_p$ via non-central F-distribution.
• ✅ Cohen's $d_{adj,jk}$ with 95% CI for each post-hoc pairwise comparison on adjusted means.

Step 7 — Select Display Options

• ✅ Full ANCOVA source table with $F$, df, $p$ (covariate row + group row + error row).
• ✅ Unadjusted and adjusted means table ($n_j$, $\bar{y}_j$, $\bar{y}_{j(adj)}$, $SE_{j(adj)}$, 95% CI per group).
• ✅ Covariate statistics ($\hat{\beta}$, $SE_\beta$, $t$, $p$, $r_{XY(W)}$).
• ✅ Effect size table ($\omega^2_p$, $\varepsilon^2_p$, $\eta^2_p$, $f_p$) with CIs.
• ✅ Post-hoc comparison table (adjusted mean differences, $SE$, adjusted $p$, $d_{adj,jk}$, 95% CI).
• ✅ Assumption test results panel (colour-coded: green/yellow/red).
• ✅ Scatterplot with per-group regression lines (parallel slopes check).
• ✅ Adjusted means plot with 95% CIs (EMM plot).
• ✅ Q-Q plot of adjusted residuals.
• ✅ Cook's distance plot.
• ✅ Unadjusted vs. adjusted means comparison plot.
• ✅ Johnson-Neyman floodlight plot (if slopes are heterogeneous).
• ✅ Power curve: power vs. $n$ for observed $\omega^2_p$.
• ✅ APA 7th edition-compliant results paragraph (auto-generated).

Step 8 — Run the Analysis

Click "Run ANCOVA". DataStatPro will:

1. Test homogeneity of regression slopes; warn if violated and offer Johnson-Neyman analysis as an alternative.
2. Compute the full ANCOVA source table (adjusted SS, adjusted MS, F-ratios, p-values).
3. Run all assumption tests and display colour-coded warnings.
4. Compute all adjusted group means ($\bar{y}_{j(adj)}$) and their standard errors.
5. Compute all effect sizes with exact non-central F-based CIs.
6. Run all selected post-hoc tests on adjusted means with adjusted p-values and individual $d_{adj,jk}$.
7. Generate all visualisations.
8. Auto-generate the APA-compliant results paragraph.

7. Full Step-by-Step Procedure

7.1 Complete Computational Procedure

This section walks through every computational step for ANCOVA, from raw data to a complete APA-style conclusion. A single covariate ($p = 1$) is assumed.

Given: $K$ groups, DV $y_{ij}$, covariate $x_{ij}$, $i = 1,\ldots,n_j$, $j = 1,\ldots,K$. Total $N = \sum_j n_j$.

Step 1 — State the Hypotheses

$H_0: \mu_{1(adj)} = \mu_{2(adj)} = \cdots = \mu_{K(adj)}$

$H_1:$ At least one adjusted population mean $\mu_{j(adj)}$ differs from the others.

Choose $\alpha$ (default: $.05$).

Step 2 — Compute Descriptive Statistics per Group

For each group $j$, compute for both $y_{ij}$ and $x_{ij}$:

$\bar{y}_j = \frac{1}{n_j}\sum_{i=1}^{n_j}y_{ij}, \quad s_{y_j} = \sqrt{\frac{\sum_{i=1}^{n_j}(y_{ij}-\bar{y}_j)^2}{n_j-1}}$

$\bar{x}_j = \frac{1}{n_j}\sum_{i=1}^{n_j}x_{ij}, \quad s_{x_j} = \sqrt{\frac{\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_j)^2}{n_j-1}}$

$r_{xy_j} = \frac{\sum_i(x_{ij}-\bar{x}_j)(y_{ij}-\bar{y}_j)}{(n_j-1)s_{x_j}s_{y_j}}$

Grand means:

$\bar{y}_{..} = \frac{\sum_j n_j\bar{y}_j}{N}, \quad \bar{x}_{..} = \frac{\sum_j n_j\bar{x}_j}{N}$

Step 3 — Check Assumption: Homogeneity of Regression Slopes

Fit the full interaction model:

$Y_i = \mu + \tau_j + \beta X_i + \gamma_j X_i + \varepsilon_i$

Test $H_0$: all $\gamma_j = 0$ using the interaction F-test.

If $p_{interaction} < .05$: stop standard ANCOVA; use Johnson-Neyman or moderated regression instead. Report this finding.

If $p_{interaction} \geq .05$: proceed with standard ANCOVA.

Step 4 — Compute Within-Group Sums of Squares and Cross-Products

$SS_{YY(W)} = \sum_{j=1}^K(n_j-1)s_{y_j}^2$

$SS_{XX(W)} = \sum_{j=1}^K(n_j-1)s_{x_j}^2$

$SP_{XY(W)} = \sum_{j=1}^K(n_j-1)s_{x_j}s_{y_j}r_{xy_j}$

Step 5 — Compute Total Sums of Squares and Cross-Products

$SS_{YY(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(y_{ij}-\bar{y}_{..})^2$

$SS_{XX(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_{..})^2$

$SP_{XY(T)} = \sum_{j=1}^K\sum_{i=1}^{n_j}(x_{ij}-\bar{x}_{..})(y_{ij}-\bar{y}_{..})$

Step 6 — Compute Between-Group Sums of Squares and Cross-Products

$SS_{YY(B)} = SS_{YY(T)} - SS_{YY(W)}$

$SS_{XX(B)} = SS_{XX(T)} - SS_{XX(W)}$

$SP_{XY(B)} = SP_{XY(T)} - SP_{XY(W)}$

Step 7 — Compute the Pooled Within-Group Regression Coefficient

$\hat{\beta} = \frac{SP_{XY(W)}}{SS_{XX(W)}}$

Step 8 — Compute Adjusted Sums of Squares

Adjusted within-groups SS (error):

$SS_{within(adj)} = SS_{YY(W)} - \frac{SP_{XY(W)}^2}{SS_{XX(W)}} = SS_{YY(W)} - \hat{\beta} \cdot SP_{XY(W)}$

Adjusted total SS:

$SS_{total(adj)} = SS_{YY(T)} - \frac{SP_{XY(T)}^2}{SS_{XX(T)}}$

Adjusted between-groups SS:

$SS_{between(adj)} = SS_{total(adj)} - SS_{within(adj)}$

Covariate SS:

$SS_{covariate} = SS_{YY(W)} - SS_{within(adj)} = \hat{\beta} \cdot SP_{XY(W)}$

Step 9 — Compute Degrees of Freedom

$df_{between} = K-1$

$df_{covariate} = 1$ (for single covariate; $p$ in general)

$df_{error} = N - K - 1$ (for single covariate; $N-K-p$ in general)

$df_{total} = N - 2$ (for single covariate; $N-1-p$ in general)

Step 10 — Compute Mean Squares and F-Ratios

$MS_{between(adj)} = SS_{between(adj)}/(K-1)$

$MS_{error(adj)} = SS_{within(adj)}/(N-K-1)$

$MS_{covariate} = SS_{covariate}/1$

$F_{group} = MS_{between(adj)}/MS_{error(adj)}$ with $df = (K-1,\;N-K-1)$

$F_{covariate} = MS_{covariate}/MS_{error(adj)}$ with $df = (1,\;N-K-1)$

Step 11 — Compute Adjusted Group Means

$\bar{y}_{j(adj)} = \bar{y}_j - \hat{\beta}(\bar{x}_j - \bar{x}_{..})$

$SE_{j(adj)} = \sqrt{MS_{error(adj)}\left(\frac{1}{n_j} + \frac{(\bar{x}_j-\bar{x}_{..})^2}{SS_{XX(W)}}\right)}$

Step 12 — Check Remaining Assumptions

• Run Shapiro-Wilk on adjusted residuals.
• Run Levene's test on adjusted residuals.
• Run linearity check (residual vs. covariate plot).
• Inspect Cook's distances for influential observations.
• Run covariate balance test (ANOVA with covariate as DV).

Step 13 — Compute Effect Sizes

Partial eta squared:

$\eta^2_p = \frac{SS_{between(adj)}}{SS_{between(adj)} + SS_{within(adj)}}$

Partial omega squared (bias-corrected, preferred):

$\omega^2_p = \frac{SS_{between(adj)} - (K-1)MS_{error(adj)}}{SS_{between(adj)} + SS_{within(adj)} + MS_{error(adj)}}$

Partial epsilon squared:

$\varepsilon^2_p = \frac{SS_{between(adj)} - (K-1)MS_{error(adj)}}{SS_{between(adj)} + SS_{within(adj)}}$

Cohen's $f_p$:

$f_p = \sqrt{\frac{\omega^2_p}{1-\omega^2_p}}$

Step 14 — Compute 95% CI for $\omega^2_p$

Using the non-central F-distribution with $df_1 = K-1$ and $df_2 = N-K-1$. DataStatPro performs this computation numerically (see Section 8.5 for details).

Step 15 — Conduct Post-Hoc Tests on Adjusted Means (if F significant)

Select the appropriate post-hoc test (Section 9). Compute pairwise differences of adjusted means, standard errors, adjusted p-values, and individual Cohen's $d_{adj,jk}$ for each pair.

Step 16 — Interpret and Report

Combine all results into an APA-compliant report (Section 13.8).

8. Effect Sizes for ANCOVA

8.1 Partial vs. Total Effect Sizes in ANCOVA

In ANCOVA, effect sizes are partial — they express the proportion of variance explained by group membership after removing the variance explained by the covariate. Partial effect sizes are appropriate because the covariate is not the substantive effect of interest; it is only a control variable.

Important distinction:

• Total $\eta^2$ (non-partial): Proportion of total variance in $Y$ explained by group membership. This ignores the covariate and is equivalent to the ANOVA effect size calculated as if the covariate were not in the model.
• Partial $\eta^2_p$: Proportion of DV variance not explained by the covariate that is explained by group membership. Always $\geq$ total $\eta^2$.
• Always report partial effect sizes for ANCOVA and label them explicitly as partial.

8.2 Partial Eta Squared ($\eta^2_p$) — Common but Biased

$\eta^2_p = \frac{SS_{between(adj)}}{SS_{between(adj)} + SS_{within(adj)}}$

$\eta^2_p$ is the proportion of adjusted DV variance accounted for by group membership. It is the most commonly reported ANCOVA effect size (default in SPSS and most software).

Critical limitation: $\eta^2_p$ is positively biased, overestimating the true population partial effect, particularly in small samples with many groups.

From F (direct computation):

$\eta^2_p = \frac{F \cdot df_B}{F \cdot df_B + df_{error(adj)}}$

⚠️ Report $\eta^2_p$ only when explicitly required by a journal or for historical comparison. Always report $\omega^2_p$ (or $\varepsilon^2_p$) as the primary effect size and label $\eta^2_p$ as "biased" in your manuscript.

8.3 Partial Omega Squared ($\omega^2_p$) — Preferred

$\omega^2_p = \frac{SS_{between(adj)} - (K-1)MS_{error(adj)}}{SS_{between(adj)} + SS_{within(adj)} + MS_{error(adj)}}$

$\omega^2_p$ is the bias-corrected estimate of the population partial proportion of variance explained. It is the recommended primary effect size for ANCOVA.

Properties:

• Can be slightly negative in small samples when the true effect is zero — report as 0 by convention.
• Always $\leq \eta^2_p$.
• Converges to $\eta^2_p$ as $N \to \infty$.
• Accounts for the reduction in $df_{error}$ due to the covariate.

From F (approximate):

$\omega^2_p \approx \frac{(F-1)(K-1)}{(F-1)(K-1) + N}$

8.4 Partial Epsilon Squared ($\varepsilon^2_p$) — Alternative Correction

$\varepsilon^2_p = \frac{SS_{between(adj)} - (K-1)MS_{error(adj)}}{SS_{between(adj)} + SS_{within(adj)}}$

Properties:

• Always: $\omega^2_p \leq \varepsilon^2_p \leq \eta^2_p$.
• Slightly less bias-correction than $\omega^2_p$.
• Computationally simpler (no $MS_{error(adj)}$ in denominator).

8.5 Cohen's $f_p$ — For Power Analysis

$f_p = \sqrt{\frac{\eta^2_p}{1-\eta^2_p}}$ or $f_p = \sqrt{\frac{\omega^2_p}{1-\omega^2_p}}$