Kruskal-Wallis Test: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of non-parametric inference for multiple independent groups all the way through the mathematics, assumptions, effect sizes, post-hoc testing, interpretation, reporting, and practical usage of the Kruskal-Wallis Test within the DataStatPro application. Whether you are encountering the Kruskal-Wallis Test for the first time or seeking a rigorous understanding of rank-based multi-group comparison, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is the Kruskal-Wallis Test?
3. The Mathematics Behind the Kruskal-Wallis Test
4. Assumptions of the Kruskal-Wallis Test
5. Variants of the Kruskal-Wallis Test
6. Using the Kruskal-Wallis Test Calculator Component
7. Full Step-by-Step Procedure
8. Effect Sizes for the Kruskal-Wallis Test
9. Post-Hoc Tests and Pairwise Comparisons
10. Confidence Intervals
11. Power Analysis and Sample Size Planning
12. Advanced Topics
13. Worked Examples
14. Common Mistakes and How to Avoid Them
15. Troubleshooting
16. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into the Kruskal-Wallis Test, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 Parametric vs. Non-Parametric Inference for Multiple Groups

Parametric tests such as the one-way ANOVA assume that observations within each group come from normally distributed populations with equal variances. When these assumptions are met, parametric tests are optimal — they use the maximum amount of information from the data and achieve the highest possible statistical power.

Non-parametric tests replace raw data values with their ranks and make minimal assumptions about the shape of population distributions. They are more robust to violations of normality and the presence of outliers. The Kruskal-Wallis Test is the leading non-parametric alternative to the one-way between-subjects ANOVA for comparing three or more independent groups.

1.2 The Concept of Ranks and Rank Sums

Ranking transforms raw data values into their ordered positions. Given $N$ observations combined across all $K$ groups, rank from 1 (smallest) to $N$ (largest):

• Assign midranks (average ranks) to tied observations.
• The sum of all ranks: $\sum_{i=1}^N R_i = N(N+1)/2$.

By working with ranks rather than raw values, the Kruskal-Wallis Test:

• Is insensitive to extreme outliers (they simply receive the highest or lowest ranks).
• Does not require normally distributed populations.
• Is applicable to ordinal data where arithmetic operations on raw values are not meaningful.

Example:

Corrected example:

1.3 The Chi-Squared Distribution

For large samples, the Kruskal-Wallis $H$ statistic follows a chi-squared distribution with $K-1$ degrees of freedom:

$H \sim \chi^2_{K-1}$ (approximately, for $n_j \geq 5$ per group)

The chi-squared distribution:

• Is always non-negative (sum of squared standard normal variates).
• Is right-skewed; becomes more symmetric as df increases.
• Has mean $= df$ and variance $= 2 \times df$.
• Is the asymptotic distribution of many test statistics derived from rank data.

Critical values for $\chi^2_{K-1}$ at $\alpha = .05$:

1.4 The Null and Alternative Hypotheses

Under the location-shift (stochastic equivalence) model:

$H_0:$ All $K$ population distributions are identical.

$H_1:$ At least one population distribution is stochastically different from at least one other (tends to produce larger or smaller values).

More precisely (without the location-shift assumption):

$H_0: P(X_j > X_k) = 0.5$ for all pairs $j \neq k$ (stochastic equality)

$H_1:$ At least one $P(X_j > X_k) \neq 0.5$ for some pair $(j, k)$

When the population distributions have the same shape but potentially different locations (medians), the Kruskal-Wallis test is equivalent to testing equality of medians:

$H_0: \theta_1 = \theta_2 = \cdots = \theta_K$ (where $\theta_j$ is the median of group $j$)

1.5 Why Not Multiple Mann-Whitney Tests?

With $K$ groups, one could run all $\binom{K}{2}$ pairwise Mann-Whitney U tests. However, this inflates the familywise error rate (FWER):

$FWER = 1-(1-\alpha)^m$

For $K = 4$ groups ($m = 6$ pairwise tests) at $\alpha = .05$: $FWER = 1-(0.95)^6 = .265$

The Kruskal-Wallis omnibus test maintains the FWER at $\alpha$ for the simultaneous test of all group differences, after which post-hoc procedures control pairwise comparisons.

1.6 The Asymptotic Relative Efficiency

The Asymptotic Relative Efficiency (ARE) of the Kruskal-Wallis test relative to one-way ANOVA is $3/\pi \approx 0.955$ for normally distributed data — a negligible efficiency loss of approximately 5%. For non-normal distributions, the Kruskal-Wallis test can be substantially more powerful:

💡 The ARE of 0.955 means that for normally distributed data, the Kruskal-Wallis test requires approximately $1/0.955 \approx 1.047$ times as many observations as one-way ANOVA to achieve the same power — a cost of only about 5%. In exchange, the test is robust to any departures from normality. This makes it a safe default when normality is uncertain.

1.7 Statistical Significance vs. Practical Significance

Like the one-way ANOVA F-test, the Kruskal-Wallis test answers: "Is the observed rank-based difference across groups larger than what chance alone would produce?" It does not answer: "How large is the effect?"

Always report:

1. The $H$ statistic (tie-corrected), degrees of freedom, and p-value.
2. $\eta^2_H$ (or $\epsilon^2_H$) as an effect size measure.
3. Group medians and interquartile ranges.
4. Post-hoc pairwise comparisons with individual effect sizes ($r_{rb}$).

2. What is the Kruskal-Wallis Test?

2.1 The Core Idea

The Kruskal-Wallis Test (Kruskal & Wallis, 1952) is a non-parametric inferential procedure for testing whether $K \geq 3$ independent groups come from the same population distribution. It is the natural extension of the Mann-Whitney U test to three or more groups, and the non-parametric analogue of the one-way between-subjects ANOVA.

Rather than comparing group means (as ANOVA does), the Kruskal-Wallis test:

1. Combines all $N$ observations across groups and ranks them from 1 to $N$.
2. Computes the mean rank $\bar{R}_j$ for each group.
3. Tests whether the mean ranks differ more than expected by chance under $H_0$ (which states all groups have the same distribution).
4. Summarises the evidence in the $H$ statistic, which follows a $\chi^2$ distribution under $H_0$ for large samples.

Under $H_0$, if all groups have the same distribution, each group should have a mean rank close to the overall mean rank $(N+1)/2$. Large deviations of group mean ranks from the overall mean rank produce a large $H$ statistic, providing evidence against $H_0$.

2.2 When to Use the Kruskal-Wallis Test

The Kruskal-Wallis Test is appropriate when:

• The dependent variable is ordinal (e.g., Likert items, pain ratings, rankings).
• The DV is continuous but severely non-normally distributed within groups, especially with small $n_j < 15$–$20$.
• There are extreme outliers that cannot be explained or removed and would distort the ANOVA F-statistic.
• The homogeneity of variance assumption is severely violated and even Welch's ANOVA may be inappropriate.
• The data represent count data or skewed positive data (reaction times, response latencies) with small samples.
• The research question concerns whether one group tends to produce higher values than others rather than specifically about mean differences.

2.3 The Kruskal-Wallis Test vs. Related Procedures

2.4 What the Kruskal-Wallis Test Tests

Under the standard location-shift assumption (all distributions have the same shape but potentially different locations), the Kruskal-Wallis test is a test of:

Equal population medians (or equivalently, equal location parameters).

Without the location-shift assumption (which should be checked — see Section 4.1), the test is more correctly described as a test of stochastic equality: whether one group tends to produce systematically larger values than another.

⚠️ A common misstatement is that the Kruskal-Wallis test always tests for equal medians. This is only true under the location-shift assumption (same shape across groups). If group distributions have different shapes, the test may reject $H_0$ even if all group medians are equal. Always state which interpretation applies based on the data.

2.5 Real-World Applications

3. The Mathematics Behind the Kruskal-Wallis Test

3.1 Notation

3.2 Step 1 — Ranking All Observations

Combine all $N$ observations from all $K$ groups into a single dataset and rank from 1 (smallest) to $N$ (largest).

For tied values: Assign the average rank (midrank) to all tied observations:

If values at positions $r, r+1, \ldots, r+t-1$ are all equal, each receives rank $(r + r+1 + \cdots + r+t-1)/t = r + (t-1)/2$.

Verification: The sum of all ranks must equal $N(N+1)/2$.

$\sum_{j=1}^K W_j = \frac{N(N+1)}{2}$

3.3 Step 2 — Computing Group Rank Sums and Mean Ranks

For each group $j$:

$W_j = \sum_{i=1}^{n_j} R_{ij}$ (sum of ranks assigned to observations in group $j$)

$\bar{R}_j = W_j/n_j$ (mean rank for group $j$)

The overall mean rank is:

$\bar{R} = \frac{N+1}{2}$

Under $H_0$ (all groups from same distribution), $E[\bar{R}_j] = (N+1)/2$ for all $j$.

3.4 Step 3 — The Kruskal-Wallis H Statistic

Basic H statistic (no ties):

$H = \frac{12}{N(N+1)}\sum_{j=1}^K \frac{W_j^2}{n_j} - 3(N+1)$

Equivalent computational form:

$H = \frac{12}{N(N+1)}\sum_{j=1}^K n_j\left(\bar{R}_j - \frac{N+1}{2}\right)^2$

This second form makes the logic transparent: $H$ is a weighted sum of squared deviations of group mean ranks from the overall mean rank $(N+1)/2$, scaled by $12/(N(N+1))$ to produce a statistic that follows a $\chi^2$ distribution.

Key properties:

• $H \geq 0$ always.
• $H = 0$ when all group mean ranks are identical (maximum similarity).
• $H$ is large when group mean ranks differ substantially.
• Under $H_0$: $H \sim \chi^2_{K-1}$ asymptotically.

3.5 Step 4 — The Tie Correction

When tied values exist, the basic $H$ statistic is slightly underestimated. The tie-corrected version is always used in practice:

$H_c = \frac{H}{C}$

Where the correction factor $C$ is:

$C = 1 - \frac{\sum_{m=1}^{g}(t_m^3 - t_m)}{N^3 - N}$

And:

• $g$ = number of distinct tied groups (groups of equal values).
• $t_m$ = number of observations in the $m$-th tied group.
• The sum is over all tied groups (including ties of size 1 contributes 0 to the sum, since $1^3 - 1 = 0$, so only actual ties matter).

Properties of $C$:

• $C = 1$ when there are no ties (correction has no effect).
• $0 < C < 1$ when ties exist (correction increases $H$, making it more conservative to reject $H_0$ — wait, actually the correction increases $H$ since we divide by a number less than 1: $H_c = H/C > H$ when $C < 1$).
• $C$ is close to 1 when ties are few or $N$ is large.

The tie correction is increasingly important when:

• Many observations share the same value.
• The measurement scale is coarse (e.g., integer ratings 1–5).
• $N$ is relatively small.

3.6 Step 5 — The p-value

For large samples ($n_j \geq 5$ per group):

$p = P(\chi^2_{K-1} \geq H_c)$

This asymptotic chi-squared approximation is generally accurate for $n_j \geq 5$.

For small samples ($n_j < 5$):

Use exact tables (available in statistical references) or the permutation distribution computed by DataStatPro. The exact p-value is based on all possible ways to assign $N$ ranks to $K$ groups of sizes $n_1, n_2, \ldots, n_K$.

Exact p-value (small samples):

$p = \frac{\text{Number of rank assignments giving } H \geq H_{obs}}{\text{Total number of possible rank assignments}}$

Total possible assignments $= N!/(n_1!\times n_2!\times\cdots\times n_K!)$

DataStatPro automatically uses the exact distribution for small samples ($n_j < 5$) and the chi-squared approximation (with tie correction) for larger samples.

3.7 The Relationship Between H and the ANOVA F-Statistic

The Kruskal-Wallis $H$ statistic is mathematically related to the ANOVA $F$-statistic applied to the ranks. Specifically, if we replaced the raw data with their ranks and ran a standard one-way ANOVA:

$F_{ranks} = \frac{MS_{B,ranks}}{MS_{W,ranks}}$

Then:

$H = \frac{(N-1) \times F_{ranks}}{N - 1 - MS_{B,ranks}/MS_{T,ranks} \times (K-1)} \approx F_{ranks} \times \frac{K-1}{1}$ (for large $N$)

More precisely, $H$ and $F_{ranks}$ are monotonically related — large $F_{ranks}$ always corresponds to large $H$. This equivalence shows that the Kruskal-Wallis test is essentially ANOVA on the ranks.

3.8 The Exact Distribution of H for Small Samples

For $K = 3$ with very small group sizes, Kruskal and Wallis (1952) tabulated the exact distribution. Selected critical values for the exact test ($\alpha = .05$):

For $n_j \geq 5$, the chi-squared approximation is generally adequate.

3.9 Decomposition: H as a Sum of Pairwise Contrasts

The total $H$ statistic can be decomposed into contributions from individual pairs of groups. For the pairwise comparison of groups $j$ and $k$:

$H_{jk} = \frac{12n_jn_k}{N(N+1)}\left(\bar{R}_j - \bar{R}_k\right)^2$

These pairwise contributions do not sum exactly to $H$ (because the ranks are shared across the full dataset), but they are useful for understanding which group pairs drive the overall significant result.

The standard Dunn post-hoc test (Section 9) uses the pairwise differences in mean ranks $(\bar{R}_j - \bar{R}_k)$ to construct post-hoc z-statistics.

4. Assumptions of the Kruskal-Wallis Test

4.1 Same Shape Across Groups (Location-Shift Assumption)

The Kruskal-Wallis Test's standard interpretation (as a test of equal medians/locations) requires that all $K$ population distributions have the same shape — they may differ only in location (median). This is the location-shift or stochastic dominance assumption.

Why it matters: If the distributions have different shapes (e.g., one group is symmetric and another is right-skewed), the Kruskal-Wallis test may reject $H_0$ even when all medians are equal — it is then detecting a difference in dispersion or shape, not location.

How to check:

• Density plots or histograms per group: do they have roughly the same shape?
• Boxplots per group: are the interquartile ranges (IQRs) similar across groups?
• Levene's test or Brown-Forsythe test (adapted for scale differences): test whether spread differs across groups.
• Q-Q plots comparing group distributions to each other.

When violated:

• If groups differ only in location (shift), the Kruskal-Wallis test tests medians. ✅
• If groups differ in both location and scale, the Kruskal-Wallis test mixes these effects. Use the Brunner-Munzel test (pairwise) or Fligner-Killeen test (for scale differences only) instead.
• Report descriptive statistics for both location (median) and spread (IQR) to help readers assess which aspect of the distribution differs.

4.2 Independence of Observations

All $N$ observations must be independent of each other, both within and across groups. Each participant or experimental unit must contribute exactly one observation to exactly one group.

Common violations:

• Repeated measurements on the same participant (use the Friedman Test instead).
• Clustered data (participants from the same family, classroom, or hospital).
• Time series with autocorrelated observations.

When violated: Use the Friedman test (for repeated measures), multilevel models, or time-series methods.

4.3 Ordinal Measurement (Rankable Data)

The Kruskal-Wallis Test requires that observations can be meaningfully ranked — there must be a natural ordering such that one value can be identified as greater than, less than, or equal to another. This is satisfied for:

• Interval and ratio-scale data (continuous measures).
• Ordinal data where values have a clear order (Likert scales, pain ratings, letter grades).

When violated: If data are purely nominal (categories with no natural order), use chi-squared tests or Fisher's exact test.

4.4 Random Sampling

Observations within each group should constitute a random sample from the respective population, or at least be exchangeable under $H_0$. This is required for the p-value to be valid.

4.5 Minimum Sample Size per Group

The chi-squared approximation for the p-value requires $n_j \geq 5$ per group for adequate accuracy. For smaller groups:

• Use the exact permutation distribution of $H$ (DataStatPro computes this automatically when $n_j < 5$).
• Be aware that exact small-sample tables exist for $K = 3$ and small $n_j$.

4.6 Absence of Excessive Ties

While the Kruskal-Wallis test handles ties through the correction factor $C$, excessive ties reduce statistical power and may distort the chi-squared approximation.

Types of ties and their impact:

• Ties within a group: Reduce the precision of rank information for that group.
• Ties across groups: The tie correction adjusts for these but power is still reduced.
• Extreme ties (many observations at the same value): Consider whether the data are truly ordinal, and whether the permutation version of the test is more appropriate.

How to check: Compute the correction factor $C$ — values of $C < 0.95$ indicate substantial ties.

4.7 Assumption Summary Table

5. Variants of the Kruskal-Wallis Test

5.1 Standard Kruskal-Wallis with Chi-Squared Approximation

The default implementation: compute $H_c$ using the tie correction and compare to $\chi^2_{K-1}$. Appropriate for $n_j \geq 5$ per group with few or moderate ties.

5.2 Exact Permutation Version

For small samples ($n_j < 5$ per group) or when ties are extensive, the exact permutation test generates the null distribution of $H$ by enumerating all possible rank assignments to the $K$ groups. DataStatPro automatically uses this for small samples.

Permutation algorithm:

1. Compute $H_{obs}$ from the observed data.
2. Enumerate (or randomly sample $B = 10{,}000$ times for large $N$) all possible assignments of $N$ combined ranks to groups of sizes $n_1, n_2, \ldots, n_K$.
3. Compute $H^{(b)}$ for each permutation.
4. $p =$ proportion of permutations with $H^{(b)} \geq H_{obs}$.

5.3 Jonckheere-Terpstra Test — Ordered Alternatives

When the $K$ groups represent an ordered quantitative variable (e.g., increasing drug dose: 0, 10, 20, 40 mg) and the alternative hypothesis is that the response is monotonically ordered across groups, the Jonckheere-Terpstra (JT) test is more powerful than the Kruskal-Wallis test:

$H_1: \theta_1 \leq \theta_2 \leq \cdots \leq \theta_K$ (at least one strict inequality)

The JT statistic counts the number of concordant pairs across ordered groups:

$J = \sum_{j < k} U_{jk}$

Where $U_{jk}$ is the Mann-Whitney $U$ statistic for groups $j$ and $k$ (counting how many observations in group $k$ exceed observations in group $j$).

DataStatPro provides the Jonckheere-Terpstra test under "Ordered Kruskal-Wallis."

5.4 Welch-Type Robust Kruskal-Wallis

The standard Kruskal-Wallis test assumes that the within-group rank dispersions are equal (analogous to the equal variance assumption). The robust Kruskal-Wallis extends Welch's approach to rank-based inference, providing better Type I error control when group scale parameters differ substantially.

5.5 Steel-Dwass Test — Non-Parametric All-Pairs Comparison

The Steel-Dwass test (also called Steel-Dwass-Critchlow-Fligner) is a non-parametric analogue of Tukey's HSD that uses pairwise Mann-Whitney statistics with a studentised range correction. It provides FWER control for all pairwise non-parametric comparisons without requiring the Kruskal-Wallis omnibus test to be significant first.

5.6 Choosing Between Variants

6. Using the Kruskal-Wallis Test Calculator Component

The Kruskal-Wallis Test Calculator in DataStatPro provides a comprehensive tool for running, diagnosing, visualising, and reporting the Kruskal-Wallis test and post-hoc pairwise comparisons.

Step-by-Step Guide

Step 1 — Select "Kruskal-Wallis Test"

From the "Test Type" dropdown, choose:

• Kruskal-Wallis Test (Standard): Chi-squared approximation with tie correction.
• Kruskal-Wallis Test (Exact): Permutation-based exact p-value (for $n_j < 5$ or many ties).
• Jonckheere-Terpstra Test: For ordered group alternatives.

💡 DataStatPro automatically suggests the Kruskal-Wallis test when the normality check on residuals from a one-way ANOVA is significant, or when the user selects an ordinal DV. A blue information banner appears in the One-Way ANOVA component with a direct "Switch to Kruskal-Wallis" button.

Step 2 — Input Method

• Raw data (long format): Two columns — one for DV values, one for group labels. DataStatPro computes all ranks, statistics, assumption diagnostics, and outputs automatically.
• Raw data (wide format): One column per group. Automatically reformatted to long.
• Summary statistics (medians + $n_j$): Limited output — only descriptive statistics and a note that inferential tests require raw data.
• Published $H$ statistic: Enter $H_c$, $K$, $N$, and any available tie information to compute p-values and effect sizes from a published result.

Step 3 — Specify Group Labels

Enter descriptive names for each group. These appear in all output tables, rank tables, and the auto-generated APA paragraph.

Step 4 — Select Assumption Diagnostics

DataStatPro automatically runs and displays:

• ✅ Density plots and histograms per group (for shape/location-shift assessment).
• ✅ Boxplots per group with medians, IQRs, and outlier identification.
• ✅ Tie correction factor $C$ — displayed with a warning if $C < 0.95$.
• ✅ Observations per group $n_j$ — warning if any $n_j < 5$.
• ✅ Levene's test on raw data (to assess shape differences alongside the KW test).
• ✅ Shapiro-Wilk per group (to contextualise why KW was chosen over ANOVA).

Step 5 — Select Post-Hoc Tests

When the omnibus $H$ is significant, choose from:

• Dunn test + Holm-Bonferroni correction (default; recommended for most applications).
• Dunn test + Bonferroni correction (more conservative).
• Dunn test + Benjamini-Hochberg (FDR control) (for exploratory analyses).
• Steel-Dwass test (non-parametric equivalent of Tukey HSD).
• Conover-Iman test (more powerful than Dunn; valid after significant $H$).
• Pairwise Mann-Whitney U tests + Holm correction (most powerful; recommended when $n_j$ per pair is adequate).

Step 6 — Select Effect Sizes

• ✅ $\eta^2_H$ (primary effect size; computed from $H_c$).
• ✅ $\epsilon^2_H$ (alternative less-biased effect size).
• ✅ 95% CI for $\eta^2_H$ (bootstrap).
• ✅ $r_{rb,jk}$ (rank-biserial correlation for each pairwise comparison).
• ✅ 95% CI for each $r_{rb,jk}$ (bootstrap or Fisher $z$-transform).

Step 7 — Select Display Options

• ✅ Kruskal-Wallis $H_c$, df, $p$-value, and decision.
• ✅ Tie correction factor $C$ and tie summary.
• ✅ Descriptive statistics: $n_j$, median, IQR, mean rank $\bar{R}_j$ per group.
• ✅ Full rank table: individual $x_{ij}$, $R_{ij}$, group assignment.
• ✅ Effect size table: $\eta^2_H$, $\epsilon^2_H$, with 95% CIs.
• ✅ Post-hoc comparison table: $z_{jk}$, $p_{adj}$, $r_{rb,jk}$, 95% CI per pair.
• ✅ Assumption diagnostic plots (density, boxplot, Shapiro-Wilk results).
• ✅ Raincloud plot per group (half violin + boxplot + raw points).
• ✅ Mean rank plot with 95% CI bands.
• ✅ Pairwise $r_{rb}$ heatmap (for large $K$).
• ✅ Power curve: power vs. $n$ for observed $\eta^2_H$.
• ✅ Comparison with one-way ANOVA results (runs both; flags discrepancies).
• ✅ APA 7th edition-compliant results paragraph (auto-generated).

Step 8 — Run the Analysis

Click "Run Kruskal-Wallis Test". DataStatPro will:

1. Rank all $N$ observations combined, applying midranks for ties.
2. Compute $W_j$, $\bar{R}_j$, $H$, $C$, and $H_c$.
3. Compute exact p-value (small samples) or chi-squared approximation (large samples).
4. Compute $\eta^2_H$ and $\epsilon^2_H$ with bootstrap 95% CIs.
5. Run all selected post-hoc tests with adjusted p-values and $r_{rb,jk}$.
6. Generate all selected visualisations.
7. Auto-generate the APA-compliant results paragraph.

7. Full Step-by-Step Procedure

7.1 Complete Computational Procedure

This section walks through every step for the Kruskal-Wallis test, from raw data to a complete APA-style conclusion.

Given: $K$ independent groups with observations $x_{ij}$ for $i = 1, \ldots, n_j$ and $j = 1, \ldots, K$. Total $N = \sum_{j=1}^K n_j$.

Step 1 — State the Hypotheses and Design

$H_0:$ All $K$ population distributions are identical (same location).

$H_1:$ At least one population distribution has a different location from at least one other.

State: the sign convention for differences (which group expected to be higher), the significance level $\alpha$ (default $.05$), and whether the p-value will be exact or asymptotic (based on $n_j$).

Step 2 — Collect and Arrange the Data

Arrange all observations in a table indicating group membership. Verify:

• Each participant contributes exactly one observation to exactly one group.
• No systematic pairing or matching across groups (use Friedman if paired).
• The DV is at least ordinal (rankable).

Step 3 — Check Assumption: Shape Similarity Across Groups

Produce density plots or histograms for each group. Assess whether the distributions have approximately the same shape (symmetry, spread) and differ mainly in location. If shapes differ substantially, note this in the results and interpret the test as a test of stochastic equality rather than equal medians.

Step 4 — Rank All Observations Combined

Create a new column with the combined ranks of all $N$ observations:

1. List all $N$ values together with their group labels.
2. Sort by value (ascending).
3. Assign ranks 1 to $N$.
4. For tied values, compute and assign the midrank.
5. Return to original order.

Verification: $\sum_j W_j = \sum_j\sum_i R_{ij} = N(N+1)/2$.

Step 5 — Compute Group Rank Sums and Mean Ranks

For each group $j$:

$W_j = \sum_{i=1}^{n_j} R_{ij}$

$\bar{R}_j = W_j/n_j$

$\bar{R} = (N+1)/2$ (overall mean rank, same for all groups under $H_0$)

Step 6 — Compute the H Statistic

$H = \frac{12}{N(N+1)}\sum_{j=1}^K \frac{W_j^2}{n_j} - 3(N+1)$

Or equivalently:

$H = \frac{12}{N(N+1)}\sum_{j=1}^K n_j\left(\bar{R}_j - \frac{N+1}{2}\right)^2$

Step 7 — Apply the Tie Correction

Identify all groups of tied absolute values and compute:

$C = 1 - \frac{\sum_{m=1}^g (t_m^3 - t_m)}{N^3 - N}$

$H_c = H/C$

If there are no ties: $C = 1$ and $H_c = H$.

Step 8 — Compute the p-value

If all $n_j \geq 5$: Compare $H_c$ to $\chi^2_{K-1}$:

$p = P(\chi^2_{K-1} \geq H_c)$

If any $n_j < 5$: Use the exact permutation distribution (DataStatPro computes this).

Reject $H_0$ if $p \leq \alpha$.

Step 9 — Compute Effect Sizes

Eta squared for Kruskal-Wallis:

$\eta^2_H = \frac{H_c - K + 1}{N - K}$

Epsilon squared (alternative, less biased):

$\epsilon^2_H = \frac{H_c}{N(N+1)/\overline{n} - 1}$

Where $\overline{n}$ is the harmonic mean of group sizes: $\overline{n} = K/\sum_j(1/n_j)$.

For balanced designs ($n_j = n$): $\epsilon^2_H = H_c/(N-1)$.

Step 10 — Conduct Post-Hoc Tests (if $H$ significant)

When $H_c$ is significant at level $\alpha$, identify which specific pairs of groups differ using Dunn's test or pairwise Mann-Whitney tests with appropriate FWER control (Section 9). Report pairwise z-statistics, adjusted p-values, and rank-biserial correlations $r_{rb,jk}$.

Step 11 — Compute Descriptive Statistics per Group

For each group $j$:

• $n_j$ (group size)
• Median = middle value of $x_{1j}, x_{2j}, \ldots, x_{n_j,j}$
• IQR = Q3 − Q1
• $\bar{R}_j$ (mean rank)
• Min, Max (range)

Step 12 — Interpret and Report

Combine all results into a complete APA-compliant report (Section 12.7).

8. Effect Sizes for the Kruskal-Wallis Test

8.1 Eta Squared for Kruskal-Wallis ($\eta^2_H$)

$\eta^2_H$ is the primary effect size for the Kruskal-Wallis test. It estimates the proportion of variance in the ranks explained by group membership:

$\eta^2_H = \frac{H_c - K + 1}{N - K}$

Equivalent formula from the ANOVA-on-ranks perspective:

$\eta^2_H = \frac{SS_{B,ranks}}{SS_{T,ranks}}$

where $SS_{B,ranks}$ and $SS_{T,ranks}$ are computed from the ranked data using standard ANOVA formulas.

Properties:

• Range: $[0, 1]$ (but can be slightly negative in small samples when the true effect is zero; report as 0 by convention).
• Interpretation: the proportion of rank variability attributable to group differences.
• Comparable to $\eta^2$ from one-way ANOVA (uses the same Cohen benchmarks).
• Slightly positively biased (analogous to $\eta^2$ being biased in ANOVA).

Approximate formula from $H_c$ alone:

$\eta^2_H \approx \frac{H_c}{N-1}$ (for balanced designs or as a rough approximation)

8.2 Epsilon Squared ($\epsilon^2_H$) — Less-Biased Alternative

$\epsilon^2_H$ (Kelley, 1935; adapted for Kruskal-Wallis) provides a less-biased estimate of the population effect size:

$\epsilon^2_H = \frac{H_c}{(N^2-1)/(N+1)} = \frac{H_c(N+1)}{N^2-1} = \frac{H_c}{N-1}$

Wait — the correct formula for $\epsilon^2_H$ adapted for the KW context:

$\epsilon^2_H = \frac{H_c - (K-1)}{N - K}$

For balanced designs with equal $n_j$, this simplifies closely to $\eta^2_H$ with a small correction for the $K-1$ term. DataStatPro reports both $\eta^2_H$ and $\epsilon^2_H$.

💡 For practical purposes, $\eta^2_H$ and $\epsilon^2_H$ are usually very similar. Use $\eta^2_H$ for comparability with published literature (it is more widely reported) and $\epsilon^2_H$ when you want a less-biased estimate. Always specify which was computed.

8.3 Cohen's Benchmarks for $\eta^2_H$

Since $\eta^2_H$ is interpreted as a proportion of explained variance (in ranks), the same benchmarks as for ANOVA's $\eta^2$ apply:

⚠️ Cohen's (1988) benchmarks are rough guidelines. Always contextualise within your domain — an $\eta^2_H = 0.10$ may be large in some fields (e.g., social psychology field studies) and small in others (e.g., laboratory-controlled cognitive tasks).

8.4 Rank-Biserial Correlation ($r_{rb,jk}$) for Pairwise Comparisons

For each significant pairwise comparison identified in post-hoc testing, report the rank-biserial correlation as the pairwise effect size:

$r_{rb,jk} = \frac{2z_{jk}}{\sqrt{n_j + n_k}}$

Where $z_{jk}$ is the z-statistic from the Dunn test for the pair $(j, k)$.

Or, directly from Mann-Whitney $U_{jk}$ (the preferred approach):

$r_{rb,jk} = 1 - \frac{2U_{jk}}{n_j n_k}$

Interpretation: $r_{rb,jk} = 0.5$ means that 75% of observations in group $j$ exceed observations in group $k$ (a large effect).

Cohen's benchmarks for $r_{rb}$ (same as Pearson $r$):

8.5 Converting Between Effect Size Metrics

8.6 The Probability of Superiority Interpretation

The rank-biserial correlation $r_{rb,jk}$ is directly related to the probability of superiority — the probability that a randomly selected observation from group $j$ exceeds a randomly selected observation from group $k$:

$PS_{jk} = P(X_j > X_k) = \frac{1 + r_{rb,jk}}{2}$

Examples:

This probability of superiority interpretation is accessible to non-statistical audiences and is the recommended supplementary reporting alongside $r_{rb,jk}$.

9. Post-Hoc Tests and Pairwise Comparisons

9.1 Why Post-Hoc Tests Are Needed

A significant Kruskal-Wallis test establishes that at least one group tends to produce systematically different values from at least one other. It does not identify which specific pairs of groups differ. Post-hoc procedures address this while controlling the FWER.

⚠️ When the omnibus Kruskal-Wallis test is non-significant, do not run pairwise post-hoc comparisons (except for pre-planned contrasts). Fishing for significant pairs after a non-significant omnibus test inflates the FWER and constitutes p-hacking.

9.2 Dunn's Test — Standard Post-Hoc for Kruskal-Wallis

Dunn's test (Dunn, 1964) is the most widely used post-hoc procedure following a significant Kruskal-Wallis test. It uses the ranks from the original Kruskal-Wallis analysis (not re-ranked pairwise).

For each pair of groups $(j, k)$:

Test statistic:

$z_{jk} = \frac{\bar{R}_j - \bar{R}_k}{SE_{jk}}$

Standard error with tie correction:

$SE_{jk} = \sqrt{\frac{N(N+1)}{12}\left(\frac{1}{n_j}+\frac{1}{n_k}\right) - \frac{\sum_m(t_m^3-t_m)}{12(N-1)}\left(\frac{1}{n_j}+\frac{1}{n_k}\right)}$

Simplified (common form):

$SE_{jk} = \sqrt{\frac{N(N+1)}{12}\cdot\frac{n_j+n_k}{n_j n_k} - \frac{\sum_m(t_m^3-t_m)(n_j+n_k)}{12N(N-1)n_j n_k}}$

Two-tailed p-value:

$p_{jk} = 2[1-\Phi(|z_{jk}|)]$

FWER correction: Apply Holm-Bonferroni (recommended) or Bonferroni to the $m = K(K-1)/2$ pairwise p-values.

Effect size per pair:

$r_{rb,jk} = \frac{2z_{jk}}{\sqrt{n_j+n_k}}$

9.3 Holm-Bonferroni Correction (Recommended)

For $m = K(K-1)/2$ pairwise comparisons:

1. Sort p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$.
2. Compare $p_{(i)}$ to $\alpha^*_{(i)} = \alpha/(m-i+1)$.
3. Starting from the smallest p-value, reject $H_{0,(i)}$ if $p_{(i)} \leq \alpha^*_{(i)}$.
4. Stop rejecting when the first non-rejection is encountered; all subsequent pairs are also non-significant.

Holm-Bonferroni provides the same FWER control as Bonferroni but is uniformly more powerful. It should always be preferred over simple Bonferroni.

9.4 Bonferroni Correction

Each comparison uses $\alpha^* = \alpha/m$. More conservative than Holm but simpler to compute manually:

$p_{adj,jk} = \min(1, p_{jk} \times m)$

Compare $p_{adj,jk}$ to $\alpha$.

9.5 Benjamini-Hochberg FDR Control (For Exploratory Research)

For exploratory analyses where controlling the false discovery rate (FDR) rather than FWER is acceptable:

1. Sort p-values: $p_{(1)} \leq \cdots \leq p_{(m)}$.
2. Find the largest $i$ such that $p_{(i)} \leq i\alpha/m$.
3. Reject all $H_{0,(1)}, \ldots, H_{0,(i)}$.

FDR control allows more discoveries than FWER control but accepts a higher rate of false positives among rejected hypotheses. Use this approach only for hypothesis generation, not confirmation.

9.6 Conover-Iman Test — More Powerful Alternative to Dunn

The Conover-Iman test (Conover & Iman, 1979) is more powerful than Dunn's test because it uses the t-distribution rather than the z-distribution for the pairwise comparisons, and it is valid only after a significant Kruskal-Wallis test.

Test statistic:

$t_{jk} = \frac{\bar{R}_j - \bar{R}_k}{\sqrt{MS_W \cdot (N-1-H_c)/(N-K) \cdot (1/n_j+1/n_k)}}$

Where $MS_W = \frac{1}{N-1}\left[\frac{N(N+1)(2N+1)}{6} - \sum_j\frac{W_j^2}{n_j}\right]$ is computed from the ranks.

This $t_{jk}$ statistic follows a t-distribution with $N-K$ df approximately, giving slightly smaller critical values (more power) than the normal approximation in Dunn's test.

9.7 Pairwise Mann-Whitney U Tests — Most Powerful Option

When post-hoc comparisons are planned in advance, pairwise Mann-Whitney U tests with Holm-Bonferroni correction provide the most powerful approach:

For each pair $(j, k)$:

1. Run a Mann-Whitney U test using only the $n_j + n_k$ observations from those two groups (not the full-dataset ranks).
2. Compute the rank-biserial correlation $r_{rb,jk}$ directly from $U_{jk}$.
3. Apply Holm-Bonferroni correction to the $m$ pairwise p-values.

Why this is more powerful than Dunn's test: Dunn's test uses the full-dataset ranks (which dilute the pairwise signal), while pairwise Mann-Whitney uses only the two groups' data (giving sharper discrimination).

Limitation: The pairwise Mann-Whitney approach does not use a common error term across pairs (unlike Dunn), which means it is slightly less efficient when the assumption of equal group dispersions holds.

9.8 Steel-Dwass Test — Non-Parametric Tukey HSD Analogue

The Steel-Dwass test (also Critchlow & Fligner, 1991) provides simultaneous confidence intervals and a test that controls the FWER without requiring the omnibus Kruskal-Wallis to be significant first. It is the non-parametric counterpart of Tukey's HSD.

DataStatPro provides the Steel-Dwass test under "Advanced Post-Hoc Options."

9.9 Planned Contrasts (Non-Parametric)

When specific comparisons are theoretically motivated before data collection, a priori contrasts can be specified. For non-parametric designs, planned contrasts use the same Dunn or Mann-Whitney approach but without FWER correction (or with a less conservative correction such as Holm applied only to the planned tests).

Linear trend contrast (Jonckheere-Terpstra): For ordered groups, this is more powerful than any pairwise approach.

9.10 Post-Hoc Selection Guide

10. Confidence Intervals

10.1 CI for the Effect Size $\eta^2_H$

The exact CI for $\eta^2_H$ does not have a closed-form solution. DataStatPro computes it via bootstrap when raw data are available:

1. Resample $N$ observations with replacement from the combined dataset, maintaining group sizes $n_1, n_2, \ldots, n_K$.
2. Compute $H_c^{(b)}$ and $\eta^2_{H,(b)}$ for each of $B = 10{,}000$ bootstrap samples.
3. The 95% CI is the 2.5th and 97.5th percentiles of the bootstrap distribution of $\eta^2_{H,(b)}$.

An approximate CI based on the non-central chi-squared distribution:

The exact non-central chi-squared CI for the ANOVA $F$-test extends to the KW $H$ statistic. Find $\lambda_L$ and $\lambda_U$ such that:

$P(\chi^2_{K-1}(\lambda_L) \geq H_c) = 0.025$ and $P(\chi^2_{K-1}(\lambda_U) \leq H_c) = 0.025$

$\eta^2_{H,L} = \lambda_L/(N-1)$; $\eta^2_{H,U} = \lambda_U/(N-1)$

DataStatPro provides both bootstrap and chi-squared-based CIs.

10.2 CI for Pairwise Rank-Biserial Correlations

The 95% CI for each pairwise $r_{rb,jk}$ uses the Fisher $z$-transformation:

$z_{r} = \text{arctanh}(r_{rb,jk}), \quad SE_{z_r} = \frac{1}{\sqrt{n_j+n_k-3}}$

$95\%\text{ CI for } z_r: z_r \pm 1.96/\sqrt{n_j+n_k-3}$

Back-transform: $r_{rb} = \tanh(z_r)$

Or via bootstrap when raw data are available (more accurate for small $n_j + n_k$).

10.3 Confidence Intervals for Group Medians

The 95% CI for each group's population median is based on order statistics:

For group $j$ with $n_j$ observations, the CI bounds are determined by:

$L_j = \lfloor n_j/2 - z_{\alpha/2}\sqrt{n_j}/2 \rfloor$; $U_j = n_j - L_j + 1$

The CI is $(x_{(L_j)}, x_{(U_j)})$ where $x_{(k)}$ is the $k$-th order statistic.

DataStatPro computes these exact binomial-based CIs for each group median.

10.4 CI Width and Precision

Width of 95% CI for $\eta^2_H = 0.10$ as a function of $N$ (bootstrap):

⚠️ With only 30 total observations ($n_j = 10$ per group), the 95% CI for $\eta^2_H = 0.10$ spans approximately $[0.00, 0.23]$ — essentially uninformative. Always report the CI. Studies with small samples can achieve statistical significance only for large true effects, but the CI reveals the inherent imprecision.

11. Power Analysis and Sample Size Planning

11.1 Power of the Kruskal-Wallis Test

Power analysis for the Kruskal-Wallis test is more complex than for ANOVA because power depends on the entire distribution of the data, not just means and variances. Three approaches are used in practice:

Approach 1 — Use ARE relative to one-way ANOVA (normal data):

$n'_{KW} \approx n_{ANOVA} \times \frac{\pi}{3} \approx 1.047 \times n_{ANOVA}$

This gives the required $n$ per group for the Kruskal-Wallis test when data are approximately normal — add approximately 5% to the ANOVA-based sample size.

Approach 2 — Direct simulation (DataStatPro Monte Carlo power module):

Specify the distribution (normal, logistic, exponential), effect size $\eta^2_H$ (or group medians and spread), $K$, $\alpha$, and desired power. DataStatPro simulates power via Monte Carlo.

Approach 3 — Use the non-central chi-squared approximation:

Power $\approx P(\chi^2_{K-1}(\lambda) > \chi^2_{K-1,\alpha})$

Where $\lambda = (N-1)\eta^2_H$ for the non-centrality parameter.

11.2 Required Sample Size per Group (80% Power, $\alpha = .05$)

Based on ARE adjustment from one-way ANOVA (normal data):

All values are $n$ per group. Total $N$ = $n \times K$. Values are $\approx 5\%$ larger than corresponding ANOVA requirements for normal data.

11.3 Sensitivity Analysis

Minimum detectable $\eta^2_H$ for 80% power ($\alpha = .05$):

$\eta^2_{H,min} \approx \frac{\chi^2_{\alpha,K-1} + 2\sqrt{\chi^2_{\alpha,K-1}}}{N}$ (rough approximation)

More precisely, using the non-central chi-squared:

$\lambda_{min} =$ non-centrality for 80% power $\approx \chi^2_{K-1}(\alpha)+ 1.28\sqrt{2\chi^2_{K-1}(\alpha)}$

$\eta^2_{H,min} \approx \lambda_{min}/(N-1)$

11.4 Power Advantage Under Non-Normal Distributions

When data are non-normal, the Kruskal-Wallis test's power advantage over ANOVA increases:

💡 For data from any distribution other than the normal, the Kruskal-Wallis test requires fewer observations than one-way ANOVA to achieve the same power. This makes it a safe and often optimal choice when normality is uncertain.

12. Advanced Topics

12.1 Relationship Between Kruskal-Wallis H and ANOVA F

The Kruskal-Wallis test is precisely one-way ANOVA applied to the ranks. If we replace each $x_{ij}$ with its rank $R_{ij}$ and run a standard one-way ANOVA, the resulting $F_{ranks}$ statistic is monotonically related to $H_c$ by: