Effect Size Calculator: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of Effect Sizes all the way through advanced estimation, interpretation, reporting, and practical usage within the DataStatPro application. Whether you are encountering effect sizes for the first time or looking to deepen your understanding of practical significance in research, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is an Effect Size?
3. The Mathematics Behind Effect Sizes
4. Assumptions of Effect Size Estimation
5. Types of Effect Sizes
6. Using the Effect Size Calculator Component
7. Effect Sizes for Mean Differences
8. Effect Sizes for Variance Explained
9. Effect Sizes for Associations and Categorical Data
10. Model Fit and Evaluation
11. Advanced Topics
12. Worked Examples
13. Common Mistakes and How to Avoid Them
14. Troubleshooting
15. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into effect sizes, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 Statistical Significance vs. Practical Significance

A p-value answers the question: "If the null hypothesis were true, how likely is it that we would observe data at least as extreme as what we actually observed?"

A small p-value tells us the result is unlikely under $H_0$ — but it does not tell us how large the effect is or whether it matters in practice.

Consider two studies, both with $p < .001$:

• Study A: $n = 10{,}000$, mean difference = 0.2 points on a 100-point scale.
• Study B: $n = 40$, mean difference = 15 points on a 100-point scale.

Study A has a highly significant but trivially small effect. Study B has a large, practically meaningful effect. Effect sizes quantify the magnitude of an effect independently of sample size — they answer the question: "How big is the effect?"

1.2 Standard Deviation and Variance

The standard deviation $\sigma$ (population) or $s$ (sample) measures the spread of a distribution:

$s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2}$

Most effect sizes for mean differences are standardised by dividing the raw difference by a standard deviation. This makes the effect size unit-free and comparable across studies using different measurement scales.

1.3 The Normal Distribution

Many effect size formulas assume that data come from normally distributed populations. The standard normal distribution $Z \sim \mathcal{N}(0, 1)$ is used to convert effect sizes into probabilities such as the common language effect size and probability of superiority.

The relationship between an effect size $d$ and the area of non-overlap between two normal distributions is fundamental to interpreting effect sizes in terms of real-world probabilities.

1.4 Correlation and Covariance

The Pearson correlation coefficient $r$ is a standardised measure of linear association:

$r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{(n-1)s_x s_y}$

It ranges from $-1$ to $+1$ and is itself an effect size for the strength of a linear relationship between two continuous variables.

1.5 Variance Decomposition

Many effect sizes for ANOVA and regression are ratios of variances:

$\eta^2 = \frac{SS_{effect}}{SS_{total}}$

Understanding the decomposition of variance into between-group (explained) and within-group (unexplained) components is essential for interpreting these effect sizes.

1.6 Confidence Intervals

A confidence interval (CI) for an effect size gives a range of plausible values for the true population effect, given the sample data. A 95% CI means that if we repeated the study 100 times, approximately 95 of the resulting intervals would contain the true population effect size.

Always report effect sizes with confidence intervals — a point estimate alone is insufficient because it conveys no information about precision or uncertainty.

1.7 The Non-Central Distributions

Effect sizes such as Cohen's $d$ and $\eta^2$ follow non-central distributions in finite samples — their sampling distributions are not symmetric, especially when the true population effect is non-zero.

• Cohen's $d$ follows a non-central $t$-distribution.
• $\eta^2$ and $\omega^2$ follow non-central F-distributions.

Confidence intervals for these effect sizes must account for the non-centrality of the sampling distribution, which is why exact CIs require iterative numerical methods rather than simple $\pm z \times SE$ formulas.

2. What is an Effect Size?

2.1 The Core Idea

An effect size is a standardised, scale-free numerical index that quantifies the magnitude of a phenomenon — how large a difference is, how strong an association is, or how much variance is explained. Effect sizes are:

• Unitless: Not expressed in the original measurement units, enabling comparison across studies.
• Sample-size independent: Unlike $p$-values, effect sizes do not change simply because the sample size changes.
• Interpretable: Translate statistical results into practically meaningful statements.
• Meta-analytically combinable: Effect sizes from multiple studies can be aggregated in a meta-analysis.

2.2 Why Effect Sizes Are Essential

The limitations of p-values alone:

1. With large samples, even trivially small effects produce significant p-values.
2. With small samples, even large effects may be non-significant.
3. A p-value carries no information about the size or direction of an effect.
4. p-values cannot be meaningfully compared across studies with different sample sizes.

What effect sizes add:

• Quantify the practical significance of a finding.
• Enable power analysis for planning future studies.
• Facilitate meta-analysis by providing a common metric.
• Allow assessment of clinical significance in applied settings.
• Provide context for evaluating whether a finding is worth acting upon.

2.3 The Effect Size Framework

Every effect size belongs to one of three broad families:

2.4 Real-World Applications

2.5 Statistical Significance vs. Effect Size: A Unified View

The relationship between sample size, effect size, and statistical significance can be summarised by the power equation. For a $t$-test:

$t = d \cdot \sqrt{\frac{n}{2}} \quad \text{(independent samples)}$

This shows that the $t$-statistic (and therefore p-value) is a joint function of both the effect size $d$ AND the sample size $n$. A non-significant result could mean:

• The true effect is zero or negligible, OR
• The sample is too small to detect a real effect (low power).

A significant result could mean:

• There is a genuine, meaningful effect, OR
• The sample is so large that even a trivially small effect becomes significant.

Effect sizes disentangle magnitude from sample size.

3. The Mathematics Behind Effect Sizes

3.1 Cohen's $d$ — The Fundamental Standardised Mean Difference

Cohen's $d$ is the cornerstone effect size for comparing two means. It expresses the difference between two means in standard deviation units.

For two independent groups:

$d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}$

Where the pooled standard deviation is:

$s_{pooled} = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$

For a one-sample design (comparing a sample mean to a known population value $\mu_0$):

$d = \frac{\bar{x} - \mu_0}{s}$

For a paired/repeated-measures design:

$d_{paired} = \frac{\bar{d}}{s_d}$

Where $\bar{d}$ is the mean of the difference scores and $s_d$ is the standard deviation of the difference scores.

Interpretation: $d = 1.0$ means the two group means are 1 standard deviation apart — a group with mean 50 differs from a group with mean 60 if both have $s = 10$.

3.2 Hedges' $g$ — Bias-Corrected Cohen's $d$

Cohen's $d$ is slightly positively biased in small samples — it overestimates the true population effect size. Hedges' $g$ applies a correction factor $J$ to remove this bias:

$g = d \times J$

Where the correction factor is:

$J = 1 - \frac{3}{4\nu - 1}$

With $\nu = n_1 + n_2 - 2$ degrees of freedom (for independent samples) or $\nu = n - 1$ (for one-sample or paired designs).

A more precise version uses the gamma function:

$J = \frac{\Gamma(\nu/2)}{\sqrt{\nu/2} \cdot \Gamma((\nu-1)/2)}$

The bias is negligible for $n > 20$ per group but can be substantial ($> 5\%$) for very small samples ($n < 10$).

3.3 Glass's $\Delta$ — Using the Control Group SD

When the two groups have different variances (especially in pre-post or treatment-control designs where the treatment may change variability), Glass's $\Delta$ standardises by only the control group standard deviation:

$\Delta = \frac{\bar{x}_{treatment} - \bar{x}_{control}}{s_{control}}$

This makes the effect size interpretable as "how many standard deviation units above (or below) the control group distribution is the average treatment participant?"

3.4 Confidence Intervals for $d$ Using the Non-Central $t$-Distribution

The exact 95% CI for Cohen's $d$ uses the non-central $t$-distribution. The observed $t$-statistic has a non-central $t$-distribution with non-centrality parameter:

$\lambda = d \sqrt{\frac{n_1 n_2}{n_1 + n_2}}$

The confidence limits for $\lambda$ are found by solving:

$P(t_\nu(\lambda_L) \geq t_{obs}) = 0.025$ and $P(t_\nu(\lambda_U) \leq t_{obs}) = 0.025$

Then converting back to $d$:

$d_{L} = \lambda_L \sqrt{\frac{n_1 + n_2}{n_1 n_2}}, \quad d_{U} = \lambda_U \sqrt{\frac{n_1 + n_2}{n_1 n_2}}$

This requires numerical iteration (no closed form) and is computed automatically by DataStatPro.

An approximate 95% CI (adequate for $n > 20$ per group) uses:

$d \pm 1.96 \times SE_d, \quad SE_d \approx \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1 + n_2)}}$

3.5 Eta Squared ($\eta^2$) — Proportion of Variance Explained

Eta squared is the proportion of total variance in the dependent variable attributable to the independent variable (group membership in ANOVA):

$\eta^2 = \frac{SS_{effect}}{SS_{total}}$

For a one-way ANOVA:

$SS_{total} = SS_{between} + SS_{within}$

$\eta^2 = \frac{SS_{between}}{SS_{between} + SS_{within}}$

Relationship to Cohen's $d$ (two groups only):

$\eta^2 = \frac{d^2}{d^2 + 4}, \quad d = \frac{2\sqrt{\eta^2}}{\sqrt{1 - \eta^2}}$

Limitation: $\eta^2$ is biased upward — it overestimates the population effect because it uses the total sum of squares from the sample. It should not be reported for multi-factor ANOVA (use partial $\eta^2$ or $\omega^2$ instead).

3.6 Partial Eta Squared ($\eta_p^2$) — Controlling for Other Effects

In factorial ANOVA (multiple IVs), partial eta squared estimates the proportion of variance explained by one effect after removing the variance attributable to other effects:

$\eta_p^2 = \frac{SS_{effect}}{SS_{effect} + SS_{error}}$

Note that in a one-way ANOVA (single IV), $\eta_p^2 = \eta^2$. In multi-factor ANOVA, $\eta_p^2 > \eta^2$ for every effect, and the sum of all partial $\eta^2$ values can exceed 1.0.

⚠️ Because partial $\eta^2$ values can sum to more than 1.0 across all effects in a factorial design, they should never be compared to the "proportion of total variance explained" — that interpretation applies only to $\eta^2$, not $\eta_p^2$.

3.7 Omega Squared ($\omega^2$) — Unbiased Variance-Explained Effect Size

Omega squared ($\omega^2$) is a bias-corrected version of $\eta^2$ that better estimates the population proportion of variance explained:

For one-way ANOVA:

$\omega^2 = \frac{SS_{between} - (K-1) \cdot MS_{within}}{SS_{total} + MS_{within}}$

Where:

• $K$ = number of groups.
• $MS_{within} = SS_{within}/(n - K)$ = mean square within groups.

Partial omega squared for factorial designs:

$\omega_p^2 = \frac{SS_{effect} - df_{effect} \cdot MS_{error}}{SS_{total} + MS_{error}}$

$\omega^2$ is generally preferred over $\eta^2$ because it does not inflate with small samples and provides a less biased estimate of the population effect.

3.8 Epsilon Squared ($\varepsilon^2$) — Another Unbiased Estimate

Epsilon squared is an alternative to omega squared, computationally simpler:

$\varepsilon^2 = \frac{SS_{between} - (K-1) \cdot MS_{within}}{SS_{total}}$

Like $\omega^2$, $\varepsilon^2$ corrects for positive bias and can be slightly negative in small samples when the true population effect is near zero.

3.9 Cohen's $f$ and $f^2$ — Effect Size for ANOVA and Regression

Cohen's $f$ converts variance-explained effect sizes into a ratio suitable for power analysis:

$f = \sqrt{\frac{\eta^2}{1 - \eta^2}}$

Or from $\omega^2$:

$f = \sqrt{\frac{\omega^2}{1 - \omega^2}}$

Cohen's $f^2$ is used for multiple regression and includes several variants:

Global $f^2$ (overall model fit):

$f^2_{global} = \frac{R^2}{1 - R^2}$

Local $f^2$ (effect of a specific predictor or set of predictors, controlling for others):

$f^2_{local} = \frac{R^2_{full} - R^2_{reduced}}{1 - R^2_{full}} = \frac{\Delta R^2}{1 - R^2_{full}}$

3.10 Pearson's $r$ and $r^2$ — Correlation and Coefficient of Determination

Pearson's $r$ is the effect size for the linear relationship between two continuous variables:

$r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n(x_i-\bar{x})^2 \cdot \sum_{i=1}^n(y_i-\bar{y})^2}}$

$r^2$ (the coefficient of determination) is the proportion of variance in $Y$ explained by $X$:

$r^2 = 1 - \frac{SS_{residual}}{SS_{total}}$

Confidence interval for $r$ using Fisher's $z$-transformation:

$z_r = \frac{1}{2}\ln\left(\frac{1+r}{1-r}\right) = \text{arctanh}(r)$

$SE_{z_r} = \frac{1}{\sqrt{n-3}}$

95% CI for $z_r$:

$z_r \pm 1.96 \cdot \frac{1}{\sqrt{n-3}}$

Converting CI bounds back to $r$:

$r = \frac{e^{2z_r} - 1}{e^{2z_r} + 1} = \tanh(z_r)$

3.11 Odds Ratio, Risk Ratio, and Number Needed to Treat

For binary outcomes (event vs. no event) in two groups, the primary effect sizes are:

$2 \times 2$ Contingency Table:

Risk (Probability) in each group:

$p_1 = \frac{a}{a+b}, \quad p_2 = \frac{c}{c+d}$

Absolute Risk Difference (ARD):

$\text{ARD} = p_1 - p_2$

Risk Ratio (Relative Risk, RR):

$\text{RR} = \frac{p_1}{p_2} = \frac{a/(a+b)}{c/(c+d)}$

Odds Ratio (OR):

$\text{OR} = \frac{p_1/(1-p_1)}{p_2/(1-p_2)} = \frac{a/b}{c/d} = \frac{ad}{bc}$

Number Needed to Treat (NNT):

$\text{NNT} = \frac{1}{\lvert \text{ARD} \rvert} = \frac{1}{\lvert p_1 - p_2 \rvert}$

NNT is the number of patients who must receive the treatment for one additional patient to benefit (or be harmed, if NNT is expressed as NNH — Number Needed to Harm).

95% CI for the log Odds Ratio:

$\ln(\widehat{\text{OR}}) \pm 1.96 \times SE_{\ln(OR)}$

Where:

$SE_{\ln(OR)} = \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}}$

Back-transforming:

$\text{OR}_{95\%\text{CI}} = \left[e^{\ln(\widehat{\text{OR}}) - 1.96 \cdot SE}, \; e^{\ln(\widehat{\text{OR}}) + 1.96 \cdot SE}\right]$

3.12 Effect Sizes for Categorical Association

Phi coefficient ($\phi$) for $2 \times 2$ tables:

$\phi = \frac{ad - bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}$

Equivalent to Pearson $r$ for two binary variables. Ranges from $-1$ to $+1$.

Cramér's $V$ for $r \times c$ contingency tables ($r$ rows, $c$ columns):

$V = \sqrt{\frac{\chi^2}{n \cdot \min(r-1, c-1)}}$

Ranges from 0 (no association) to 1 (perfect association).

Cohen's $w$ for goodness-of-fit and $\chi^2$ tests:

$w = \sqrt{\sum_{i=1}^{k} \frac{(P_{0i} - P_{1i})^2}{P_{0i}}}$

Where $P_{0i}$ are the null (expected) proportions and $P_{1i}$ are the alternative (observed/hypothesised) proportions.

3.13 Rank-Biserial Correlation

The rank-biserial correlation ($r_{rb}$) is the effect size for the Mann-Whitney U test (non-parametric alternative to Cohen's $d$ when normality is not assumed):

$r_{rb} = 1 - \frac{2U}{n_1 n_2} = \frac{2U}{n_1 n_2} - 1$

Or equivalently:

$r_{rb} = \frac{\bar{R}_1 - \bar{R}_2}{n/2}$

Where $\bar{R}_1$ and $\bar{R}_2$ are the mean ranks of the two groups, and $n = n_1 + n_2$.

Ranges from $-1$ to $+1$. $r_{rb} = 0.5$ means that 75% of observations in Group 1 exceed those in Group 2.

4. Assumptions of Effect Size Estimation

4.1 Correct Scale and Direction of Variables

Effect sizes are only meaningful when variables are measured on an appropriate scale and when the direction of differences is clearly defined.

Why it matters: Reversing the direction of scoring (e.g., higher score = worse outcome vs. higher score = better outcome) changes the sign of the effect size. Ambiguous scoring leads to misinterpretation.

How to check: Before computing any effect size, clearly state:

• What constitutes a positive vs. negative effect.
• Whether higher or lower scores are better.
• What the reference/baseline condition is.

4.2 Normally Distributed Populations (for $d$-family)

Cohen's $d$, Hedges' $g$, and Glass's $\Delta$ assume that the observed scores come from normally distributed populations. Violations of normality can distort the pooled standard deviation and produce misleading effect size estimates.

How to check:

• Shapiro-Wilk test for small samples ($n < 50$).
• Q-Q plots and histograms for larger samples.
• Skewness ($\lvert z \rvert < 2$) and kurtosis ($\lvert z \rvert < 7$).

When violated: Use rank-biserial correlation ($r_{rb}$) for the Mann-Whitney test or common language effect size (CL) as alternatives that do not assume normality.

4.3 Equal or Known Population Variances

Cohen's $d$ uses the pooled standard deviation, which implicitly assumes homogeneity of variance. When population variances differ substantially:

• Glass's $\Delta$ is preferred (uses only the control group SD).
• Use Welch's $d$ (sometimes called $d_s$), which uses only the denominator group's SD.
• Report both groups' variances alongside the effect size.

Variance ratio rule of thumb: If $s^2_{larger}/s^2_{smaller} > 4$, consider using Glass's $\Delta$ rather than Cohen's $d$.

4.4 Independence of Observations

Effect sizes based on means (Cohen's $d$), correlations ($r$), and variance-explained measures ($\eta^2$) all assume that observations are independent of each other.

When violated:

• Paired designs: Use paired $d$ (based on difference scores) rather than independent samples $d$.
• Clustered data: Use multilevel effect sizes (e.g., intraclass correlation for between-cluster effects).
• Repeated measures: Report generalised $\eta^2$ or partial $\omega^2$ from the correct ANOVA model.

4.5 Adequate Sample Size for Stable Estimates

Effect size estimates are very unstable in small samples. The sampling variability of $d$ can be enormous with $n < 10$ per group:

This table shows that with only 5 observations per group, the 95% CI for $d = 0.5$ spans nearly 3 standard deviation units — essentially uninformative. Effect sizes require adequate sample sizes to be interpretable.

4.6 No Selective Reporting (Publication Bias)

When effect sizes are extracted from published literature, they are subject to publication bias — studies with larger, significant effects are more likely to be published than those with small, non-significant effects. This means that the average published effect size overestimates the true population effect.

Remedies:

• Pre-register studies to reduce selective reporting.
• In meta-analysis, use funnel plots and Egger's test to detect publication bias.
• Use trim-and-fill methods to correct for publication bias.

5. Types of Effect Sizes

5.1 The Three Families of Effect Sizes

The $d$-Family (Standardised Mean Differences)

These effect sizes express the difference between means in standard deviation units.

The $r$-Family (Variance-Explained and Correlation)

These effect sizes express how much of the total variance is explained by the effect.

The Risk/Odds Family

These effect sizes are appropriate for binary outcomes.

5.2 Choosing the Right Effect Size

The table below provides a quick reference for selecting the appropriate effect size measure based on your statistical test:

6. Using the Effect Size Calculator Component

The Effect Size Calculator component in DataStatPro provides a comprehensive tool for computing, visualising, and interpreting effect sizes across all major statistical designs.

Step-by-Step Guide

Step 1 — Select the Effect Size Family

Choose from the "Effect Size Type" dropdown:

• Mean Difference (d-family): For comparing means from $t$-tests or similar.
• Variance Explained (r-family): For ANOVA, regression, and correlation.
• Categorical / Binary Outcomes: For $\chi^2$ tests, risk ratios, odds ratios.
• Non-Parametric: For Mann-Whitney U, Wilcoxon tests.

Step 2 — Select the Specific Effect Size

Based on your design, select the specific effect size:

• Independent samples: Cohen's $d$, Hedges' $g$, Glass's $\Delta$.
• One-sample or paired: Cohen's $d_{paired}$, $d_z$.
• ANOVA: $\eta^2$, $\eta_p^2$, $\omega^2$, $\omega_p^2$, $\varepsilon^2$, Cohen's $f$.
• Regression: $R^2$, adjusted $R^2$, Cohen's $f^2$ (global or local).
• Correlation: Pearson $r$, $r^2$.
• Binary outcomes: OR, RR, ARD, NNT, $\phi$, Cramér's $V$.
• Non-parametric: Rank-biserial $r_{rb}$.

💡 Recommendation: For ANOVA, always compute and report $\omega^2$ (or $\omega_p^2$ for factorial designs) in addition to or instead of $\eta^2$. Omega squared is less biased and is increasingly required by journals.

Step 3 — Input Method

Choose how to provide the data:

• Raw data: Provide the actual dataset; the app computes summary statistics and effect sizes automatically.
• Summary statistics: Enter means, SDs, and $n$ values directly.
• Test statistics: Enter the $t$, $F$, or $\chi^2$ statistic with degrees of freedom.

💡 Tip: When computing effect sizes from published papers that only report test statistics, use the "From test statistics" input method. For example, $d = t\sqrt{1/n_1 + 1/n_2}$ and $\eta^2 = F \cdot df_{between} / (F \cdot df_{between} + df_{within})$.

Step 4 — Specify Design Details

• Number of groups (for ANOVA).
• Factorial structure (for multi-factor designs — specify main effects and interactions).
• Sample sizes per group.
• Degrees of freedom (if computing from test statistics).

Step 5 — Select Confidence Level

Choose the confidence level for intervals (default: 95%). The application computes:

• Exact CIs based on non-central $t$ and $F$ distributions for $d$, $\eta^2$, $\omega^2$.
• Fisher $z$-transformation CIs for Pearson $r$.
• Wald CIs for OR, RR, ARD (on the log scale, back-transformed).
• Bootstrap CIs when raw data are provided.

Step 6 — Select Benchmarks

Choose the benchmark system for interpreting the magnitude:

• Cohen (1988) — the original benchmarks (small/medium/large).
• Field (2013) — discipline-specific benchmarks.
• Funder & Ozer (2019) — benchmarks based on social science effect size distributions.
• Sawilowsky (2009) — extended benchmarks for very small and very large effects.
• Custom — enter your own domain-specific thresholds.

⚠️ Important: Cohen's benchmarks were intended as rough conventions when no better information is available. Always prioritise domain-specific benchmarks and contextual interpretation over generic small/medium/large labels.

Step 7 — Display Options

Select which outputs and visualisations to display:

• ✅ Effect size point estimate with 95% CI.
• ✅ Benchmark classification (small / medium / large) with the selected system.
• ✅ Visualisation of the two distributions with overlap.
• ✅ Common Language Effect Size (CL) percentage.
• ✅ Probability of Superiority.
• ✅ Number Needed to Treat (for binary outcomes).
• ✅ Variance overlap diagram ($U_1$, $U_2$, $U_3$ statistics).
• ✅ Forest plot (when multiple effect sizes are provided).
• ✅ Conversion to other effect size types.
• ✅ Power analysis based on observed effect size.

Step 8 — Run the Calculation

Click "Calculate Effect Size". The application will:

1. Compute the requested effect size(s) from the provided data or statistics.
2. Construct confidence intervals using the appropriate method.
3. Classify the magnitude using the selected benchmark system.
4. Generate all selected visualisations.
5. Provide an interpretation paragraph in plain language.

7. Effect Sizes for Mean Differences

7.1 Cohen's $d$ for Independent Samples — Full Procedure

Step 1 — Compute group means and standard deviations

$\bar{x}_1 = \frac{1}{n_1}\sum_{i=1}^{n_1} x_{1i}, \quad \bar{x}_2 = \frac{1}{n_2}\sum_{i=1}^{n_2} x_{2i}$

$s_1 = \sqrt{\frac{1}{n_1-1}\sum_{i=1}^{n_1}(x_{1i}-\bar{x}_1)^2}, \quad s_2 = \sqrt{\frac{1}{n_2-1}\sum_{i=1}^{n_2}(x_{2i}-\bar{x}_2)^2}$

Step 2 — Compute pooled standard deviation

$s_{pooled} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}$

Step 3 — Compute Cohen's $d$

$d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}$

Step 4 — Apply Hedges' correction (especially if $n < 20$ per group)

$g = d \times \left(1 - \frac{3}{4(n_1 + n_2 - 2) - 1}\right)$

Step 5 — Compute the 95% CI (exact, via non-central $t$)

The exact CI is computed numerically. The approximate 95% CI is:

$d \pm 1.96 \times \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1 + n_2 - 2)}}$

Step 6 — Compute the Common Language Effect Size (CL)

The Common Language Effect Size (McGraw & Wong, 1992) is the probability that a randomly selected person from Group 1 scores higher than a randomly selected person from Group 2:

$CL = \Phi\left(\frac{d}{\sqrt{2}}\right)$

Where $\Phi$ is the standard normal CDF.

$CL = 0.50$ means 50% probability of superiority (no effect); $CL = 0.75$ means 75% of the time, a person from Group 1 outscores a person from Group 2.

7.2 Cohen's Benchmark Classification for $d$ and $g$

Cohen (1988) proposed the following conventions, intended as rough guides only:

⚠️ Cohen himself warned against mechanical application of these benchmarks. He stated: "The effect size conventions are offered as conventions of last resort, to be used only when no better basis for setting the ES is available." Always contextualise effect sizes within your specific research domain.

Extended benchmarks (Sawilowsky, 2009):

7.3 Variance Overlap Statistics (Cohen's $U$)

To complement Cohen's $d$, three overlap statistics provide intuitive, probabilistic interpretations of the separation between two normal distributions:

$U_1$: The proportion of the combined distributions that is NOT overlapping:

$U_1 = 2\Phi\left(\frac{\lvert d \rvert}{2}\right) - 1$

$U_2$: The proportion of one distribution that exceeds the same proportion in the other distribution (percentage of the non-treatment distribution exceeded by the median of the treatment distribution):

$U_2 = \Phi\left(\frac{\lvert d \rvert}{2}\right) \times 100\%$

$U_3$ (Cohen's $U_3$): The proportion of the treatment distribution that exceeds the median of the control distribution:

$U_3 = \Phi(\lvert d \rvert) \times 100\%$

Example for $d = 0.50$:

$U_3 = \Phi(0.50) = 0.691 \to 69.1\%$

Interpretation: 69.1% of the treatment group scores above the median of the control group.

7.4 Computing $d$ from Common Test Statistics

When raw data are unavailable, $d$ can be computed from reported test statistics:

From an independent samples $t$-test:

$d = t \sqrt{\frac{1}{n_1} + \frac{1}{n_2}} = \frac{t\sqrt{n_1 + n_2}}{\sqrt{n_1 n_2}}$

From a one-sample or paired $t$-test:

$d_z = \frac{t}{\sqrt{n}}$

From an $F$-ratio (two-group ANOVA, $df_1 = 1$):

$d = \sqrt{F \left(\frac{1}{n_1} + \frac{1}{n_2}\right)}$

From a $\chi^2$ statistic (for $\phi$ or Cramér's $V$):

$\phi = \sqrt{\frac{\chi^2}{n}}, \quad V = \sqrt{\frac{\chi^2}{n \cdot \min(r-1,c-1)}}$

Converting between effect size families:

$r = \frac{d}{\sqrt{d^2 + \frac{(n_1+n_2)^2}{n_1 n_2}}}, \quad d = \frac{2r}{\sqrt{1-r^2}}$ (for equal group sizes)

$\eta^2 = \frac{d^2}{d^2+4}$ (for equal group sizes)

8. Effect Sizes for Variance Explained

8.1 $\eta^2$, $\omega^2$, and $\varepsilon^2$ Comparison

For a one-way ANOVA with $K$ groups and $n$ total observations:

Given the ANOVA table:

Eta squared:

$\eta^2 = \frac{SS_B}{SS_T}$

Omega squared (preferred):

$\omega^2 = \frac{SS_B - (K-1)MS_W}{SS_T + MS_W}$

Epsilon squared:

$\varepsilon^2 = \frac{SS_B - (K-1)MS_W}{SS_T}$

Relationship: $\omega^2 \leq \varepsilon^2 \leq \eta^2$

All three measure the same thing (proportion of variance explained) but $\omega^2$ and $\varepsilon^2$ are corrected for the positive bias of $\eta^2$ in finite samples.

8.2 Benchmark Interpretations for Variance-Explained Effect Sizes

Cohen (1988) benchmarks:

Note on $\eta^2$ benchmarks: These were established when $\eta^2$ was the standard report. Since $\omega^2$ and $\varepsilon^2$ are systematically smaller than $\eta^2$, the same verbal benchmarks do not directly transfer. Use the $f$ or $f^2$ conversions for power analysis regardless of which variance-explained index you report.

8.3 Generalised Eta Squared ($\eta_G^2$)

Generalised eta squared ($\eta_G^2$; Olejnik & Algina, 2003) is designed for between-subjects comparison across studies by distinguishing between:

• Manipulated variables (experimental factors set by the researcher, e.g., treatment).
• Measured variables (participant characteristics, e.g., age, sex).

$\eta_G^2 = \frac{SS_{effect}}{SS_{effect} + \sum_{m} SS_{measured} + SS_{error}}$

Where the summation is over all measured (non-manipulated) variables in the design.

$\eta_G^2$ is more comparable across different experimental designs (between-subjects, within-subjects, mixed) than either $\eta^2$ or $\eta_p^2$ and is increasingly recommended for factorial and mixed ANOVA designs.

8.4 $R^2$ and Adjusted $R^2$ for Regression

The coefficient of determination $R^2$ is the proportion of variance in $Y$ explained by the regression model:

$R^2 = 1 - \frac{SS_{residual}}{SS_{total}} = \frac{SS_{regression}}{SS_{total}}$

Adjusted $R^2$ corrects for the number of predictors $p$ in the model:

$R^2_{adj} = 1 - (1-R^2)\frac{n-1}{n-p-1}$

Adjusted $R^2$ can be negative when the model fits worse than a horizontal line.

$R^2$ change ($\Delta R^2$) for evaluating the increment from adding predictors:

$\Delta R^2 = R^2_{full} - R^2_{reduced}$

Cohen's $f^2$ for the increment:

$f^2_{local} = \frac{\Delta R^2}{1 - R^2_{full}}$

8.5 Confidence Intervals for $\eta^2$ and $\omega^2$

CIs for variance-explained effect sizes use the non-central F-distribution. The observed $F$-ratio has a non-central $F$ distribution:

$F \sim F'(df_{between}, df_{within}, \lambda)$

Where $\lambda$ is the non-centrality parameter:

$\lambda = \frac{\eta^2 \cdot n}{1 - \eta^2}$

The CI bounds for $\lambda$ are found numerically, then converted to $\eta^2$:

$\eta^2_L = \frac{\lambda_L}{\lambda_L + n}, \quad \eta^2_U = \frac{\lambda_U}{\lambda_U + n}$

For $\omega^2$ and $\varepsilon^2$, a transformation approach is used: first compute the CI for $\eta^2$ (or $f$), then convert to the desired effect size.

9. Effect Sizes for Associations and Categorical Data

9.1 Pearson $r$ — Correlation Effect Size

The Pearson correlation $r$ is simultaneously a descriptive statistic and an effect size. It requires no additional calculation — the correlation coefficient itself IS the standardised effect size for the strength of a linear relationship.

Benchmarks for $r$:

💡 Funder & Ozer (2019) argued that Cohen's benchmarks are too conservative for social/behavioural science, where $r = 0.30$ is actually a large effect in practice. Consider the base rates in your field when applying benchmarks.

9.2 Interpreting the Odds Ratio

The Odds Ratio (OR) is the most common effect size in case-control studies and logistic regression.