t-Tests and Alternatives: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of hypothesis testing all the way through advanced t-test variants, non-parametric alternatives, interpretation, reporting, and practical usage within the DataStatPro application. Whether you are encountering t-tests for the first time or seeking a deeper understanding of when and how to apply parametric and non-parametric tests for comparing means, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What is a t-Test?
3. The Mathematics Behind t-Tests
4. Assumptions of t-Tests
5. Types of t-Tests
6. Using the t-Test Calculator Component
7. One-Sample t-Test
8. Independent Samples t-Test
9. Paired Samples t-Test
10. Welch's t-Test — Unequal Variances
11. Non-Parametric Alternatives
12. Advanced Topics
13. Worked Examples
14. Common Mistakes and How to Avoid Them
15. Troubleshooting
16. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into t-tests, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 Populations, Samples, and Parameters

A population is the complete set of individuals or observations of interest. A sample is a subset drawn from the population. Parameters describe population characteristics (e.g., $\mu$, $\sigma$), while statistics describe sample characteristics (e.g., $\bar{x}$, $s$).

The t-test is an inferential procedure — it uses sample statistics to draw conclusions about unknown population parameters. The fundamental question in every t-test is: "Is the difference between observed means large enough to conclude that the true population means differ?"

1.2 The Sampling Distribution of the Mean

If we repeatedly drew samples of size $n$ from a population with mean $\mu$ and standard deviation $\sigma$, the distribution of sample means $\bar{x}$ would itself be a distribution — the sampling distribution of the mean. By the Central Limit Theorem (CLT):

$\bar{x} \sim \mathcal{N}\!\left(\mu,\; \frac{\sigma^2}{n}\right) \quad \text{as } n \to \infty$

The standard error of the mean is:

$SE_{\bar{x}} = \frac{\sigma}{\sqrt{n}}$

When the population $\sigma$ is unknown (as in virtually all real applications), it is estimated by the sample standard deviation $s$, giving the estimated standard error:

$\widehat{SE}_{\bar{x}} = \frac{s}{\sqrt{n}}$

This substitution is what necessitates the use of the $t$-distribution rather than the standard normal distribution.

1.3 The t-Distribution

The Student's t-distribution was derived by William Sealy Gosset (publishing under the pseudonym "Student") in 1908. It arises whenever we estimate a normally distributed population's mean using a small sample and an unknown variance.

The t-distribution is characterised by a single parameter: degrees of freedom $\nu$. As $\nu \to \infty$, the t-distribution converges to the standard normal $\mathcal{N}(0,1)$.

Key properties:

• Symmetric and bell-shaped, centred at 0.
• Has heavier tails than the standard normal (more probability in the extremes).
• Heavier tails for smaller $\nu$ — reflecting greater uncertainty from estimating $\sigma$.
• Requires looking up critical values $t_{\alpha/2, \nu}$ for hypothesis testing.

Critical values for common $\alpha$ levels:

1.4 Hypothesis Testing Framework

Every t-test operates within the Neyman-Pearson hypothesis testing framework:

Step 1 — State the hypotheses:

• $H_0$ (null hypothesis): The parameter equals a specified value (e.g., $\mu_1 = \mu_2$).
• $H_1$ (alternative hypothesis): The parameter differs from that value.

Step 2 — Choose $\alpha$: The significance level is the acceptable Type I error rate (conventionally $\alpha = .05$). It is the probability of rejecting $H_0$ when it is true.

Step 3 — Compute the test statistic: The t-statistic measures how many standard errors the observed result is from the null hypothesis value.

Step 4 — Compute the p-value: The probability of observing a t-statistic at least as extreme as the one obtained, assuming $H_0$ is true.

Step 5 — Make a decision: Reject $H_0$ if $p \leq \alpha$; fail to reject $H_0$ if $p > \alpha$.

Step 6 — Compute and report the effect size with CI: Statistical significance alone is insufficient. Always accompany the t-test result with Cohen's $d$ (or equivalent) and its 95% confidence interval.

1.5 Type I and Type II Errors

• Type I error ($\alpha$): Concluding there is an effect when there is none (false positive).
• Type II error ($\beta$): Missing a true effect (false negative).
• Power ($1-\beta$): The probability of correctly detecting a true effect.

1.6 One-Tailed vs. Two-Tailed Tests

A two-tailed test places the rejection region in both tails of the distribution and is appropriate when the direction of the effect is not specified in advance:

$H_1: \mu_1 \neq \mu_2$

A one-tailed test places the entire rejection region in one tail, appropriate only when a directional prediction is made before data collection on strong theoretical grounds:

$H_1: \mu_1 > \mu_2 \quad \text{or} \quad H_1: \mu_1 < \mu_2$

⚠️ One-tailed tests should be pre-registered and theoretically justified before data collection. Using a one-tailed test post-hoc to achieve significance is p-hacking. In the absence of a strong directional prediction, always use a two-tailed test.

1.7 Confidence Intervals and Their Relationship to t-Tests

A $(1-\alpha) \times 100\%$ confidence interval for the mean difference is directly related to the two-tailed t-test at significance level $\alpha$: the null hypothesis $H_0: \mu_1 - \mu_2 = 0$ is rejected at level $\alpha$ if and only if $0$ lies outside the $(1-\alpha) \times 100\%$ CI.

The CI provides strictly more information than the p-value — it communicates both the direction and precision of the estimate and enables assessment of practical significance.

2. What is a t-Test?

2.1 The Core Idea

A t-test is a parametric inferential statistical test used to determine whether there is a statistically significant difference between means. The general form of the t-statistic is:

$t = \frac{\text{Observed difference} - \text{Null hypothesis value}}{\text{Estimated standard error of the difference}}$

The denominator — the standard error — is the key: it scales the observed difference by the sampling variability, allowing us to determine whether the difference is larger than what we would typically expect from sampling variation alone.

2.2 When to Use a t-Test

A t-test is appropriate when:

• The outcome variable is continuous (interval or ratio scale).
• You are comparing one or two group means.
• The data are approximately normally distributed (or $n$ is large enough for CLT).
• Observations within each group are independent (for independent t-tests).

2.3 The Three Versions of the t-Test

2.4 The t-Test in Context

The t-test is one member of a broader family of inferential tests:

2.5 Statistical Significance vs. Practical Significance

A t-test answers: "Is the observed mean difference larger than expected by chance?" It does not answer: "Is the difference large enough to matter in practice?"

With large samples, trivially small differences become statistically significant. A study comparing two teaching methods with $n = 5{,}000$ per group might find $t(9998) = 3.20$, $p = .001$, for a mean difference of 0.3 points on a 100-point scale — significant but practically meaningless.

Always report:

1. The t-statistic and p-value (statistical significance).
2. Cohen's $d$ or equivalent effect size (practical significance).
3. The 95% CI for the mean difference and for the effect size.

3. The Mathematics Behind t-Tests

3.1 The One-Sample t-Statistic

The one-sample t-test tests whether a sample mean $\bar{x}$ differs significantly from a hypothesised population mean $\mu_0$:

$t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}$

Where:

• $\bar{x}$ = sample mean
• $\mu_0$ = hypothesised population mean (null value)
• $s$ = sample standard deviation
• $n$ = sample size

Under $H_0: \mu = \mu_0$, this statistic follows a t-distribution with $\nu = n - 1$ degrees of freedom.

The 95% CI for the population mean:

$\bar{x} \pm t_{\alpha/2,\; n-1} \cdot \frac{s}{\sqrt{n}}$

3.2 The Independent Samples t-Statistic (Student's)

The independent samples t-test (Student's version) tests whether two population means are equal, assuming homogeneity of variance:

$t = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}\sqrt{\dfrac{1}{n_1} + \dfrac{1}{n_2}}}$

Where the pooled standard deviation is:

$s_{pooled} = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$

Degrees of freedom: $\nu = n_1 + n_2 - 2$

The 95% CI for the mean difference $(\mu_1 - \mu_2)$:

$(\bar{x}_1 - \bar{x}_2) \pm t_{\alpha/2,\; n_1+n_2-2} \cdot s_{pooled}\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$

3.3 Welch's t-Statistic — Unequal Variances

Welch's t-test does not assume equal population variances. It computes a separate variance estimate for each group:

$t_W = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}}}$

The degrees of freedom are approximated by the Welch-Satterthwaite equation:

$\nu_W = \frac{\left(\dfrac{s_1^2}{n_1} + \dfrac{s_2^2}{n_2}\right)^2}{\dfrac{\left(s_1^2/n_1\right)^2}{n_1-1} + \dfrac{\left(s_2^2/n_2\right)^2}{n_2-1}}$

Note: $\nu_W$ is generally non-integer and is typically rounded down. Welch's df are always $\leq n_1 + n_2 - 2$ (i.e., always fewer or equal df than Student's t-test, making it a more conservative test).

The 95% CI for the mean difference:

$(\bar{x}_1 - \bar{x}_2) \pm t_{\alpha/2,\; \nu_W} \cdot \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$

3.4 The Paired Samples t-Statistic

The paired t-test treats the data as a set of $n$ difference scores $d_i = x_{1i} - x_{2i}$ computed for each pair. It tests whether the mean difference $\bar{d}$ is significantly different from zero:

$t = \frac{\bar{d}}{s_d / \sqrt{n}}$

Where:

• $\bar{d} = \frac{1}{n}\sum_{i=1}^n d_i$ = mean of the difference scores
• $s_d = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (d_i - \bar{d})^2}$ = SD of the difference scores
• $n$ = number of pairs

Degrees of freedom: $\nu = n - 1$

The 95% CI for the mean difference:

$\bar{d} \pm t_{\alpha/2,\; n-1} \cdot \frac{s_d}{\sqrt{n}}$

The relationship between the paired t-statistic, the correlation $r_{12}$ between paired measurements, and the independent samples t-statistic:

$s_d^2 = s_1^2 + s_2^2 - 2r_{12}s_1 s_2$

This shows that when $r_{12} > 0$ (paired measurements are positively correlated), the paired test has a smaller denominator (less error variance) and thus greater statistical power than the independent samples test for the same data.

3.5 The p-value

The p-value is computed from the t-statistic and degrees of freedom using the cumulative distribution function (CDF) of the t-distribution:

Two-tailed p-value:

$p = 2 \times P(T_\nu \geq \lvert t_{obs} \rvert) = 2 \times [1 - F_{t,\nu}(\lvert t_{obs} \rvert)]$

One-tailed p-value (upper tail):

$p = P(T_\nu \geq t_{obs}) = 1 - F_{t,\nu}(t_{obs})$

One-tailed p-value (lower tail):

$p = P(T_\nu \leq t_{obs}) = F_{t,\nu}(t_{obs})$

Where $F_{t,\nu}$ is the CDF of the t-distribution with $\nu$ degrees of freedom.

3.6 Computing Effect Sizes from t-Statistics

When raw data are unavailable, effect sizes can be computed directly from the reported t-statistic:

Cohen's $d$ from independent samples t-test:

$d = t\sqrt{\frac{n_1 + n_2}{n_1 n_2}} = \frac{t\sqrt{n_1+n_2}}{\sqrt{n_1 n_2}}$

For equal group sizes ($n_1 = n_2 = n$):

$d = \frac{2t}{\sqrt{2n}} = t\sqrt{\frac{2}{n}}$

Cohen's $d_z$ from one-sample or paired t-test:

$d_z = \frac{t}{\sqrt{n}}$

Pearson $r$ from any t-statistic:

$r = \sqrt{\frac{t^2}{t^2 + \nu}}$

Where $\nu$ is the degrees of freedom. Note: this $r$ is equivalent to the point-biserial correlation between the binary group variable and the continuous outcome.

3.7 The Non-Central t-Distribution and Exact CIs for $d$

Under the alternative hypothesis (when a true effect exists), the t-statistic does not follow a central t-distribution — it follows a non-central t-distribution with non-centrality parameter:

$\lambda = \frac{\mu_1 - \mu_2}{\sigma\sqrt{1/n_1 + 1/n_2}} = d \cdot \sqrt{\frac{n_1 n_2}{n_1 + n_2}}$

This non-centrality parameter links the population effect size $d$ to the expected t-statistic. Exact 95% CIs for Cohen's $d$ invert this relationship numerically (no closed form exists) — a computation performed automatically by DataStatPro.

3.8 Statistical Power of the t-Test

Power is the probability that the t-test correctly rejects $H_0$ when a true effect $d$ exists:

$\text{Power} = P\!\left(T_\nu(\lambda) > t_{crit}\right)$

Where $T_\nu(\lambda)$ is the non-central t-distribution with non-centrality parameter:

$\lambda = d\sqrt{\frac{n_1 n_2}{n_1 + n_2}}$ (independent) $\quad$ or $\quad$ $\lambda = d\sqrt{n}$ (one-sample or paired)

For the independent samples t-test with equal groups, the approximate required sample size for power $1-\beta$ at two-sided level $\alpha$:

$n_{per\;group} \approx \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2}$

4. Assumptions of t-Tests

4.1 Normality of the Sampling Distribution

The t-test assumes that the sampling distribution of the mean difference is normal. This is satisfied when either:

• The population from which data are drawn is normally distributed, OR
• The sample size is sufficiently large for the CLT to ensure approximate normality of $\bar{x}$ (generally $n \geq 30$ per group, though skewed distributions may require larger $n$).

How to check:

• Shapiro-Wilk test for $n < 50$ (most powerful normality test for small samples).
• Kolmogorov-Smirnov or Lilliefors test for $n \geq 50$.
• Q-Q (quantile-quantile) plots: points should fall approximately on the diagonal line.
• Histograms and density plots: assess approximate bell shape.
• Skewness ($\lvert z_{skew} \rvert < 2$) and kurtosis ($\lvert z_{kurt} \rvert < 7$).

Robustness: The t-test is remarkably robust to mild non-normality, especially for larger samples. For moderate non-normality with $n \geq 20$ per group, the t-test's Type I error rate remains close to the nominal $\alpha$.

When violated: Use the Mann-Whitney U test (independent) or Wilcoxon signed-rank test (paired) as non-parametric alternatives. Consider data transformation (log, square root) if the distribution is strongly skewed.

4.2 Homogeneity of Variance (for Independent Samples t-Test)

Student's independent t-test assumes that the two populations have equal variances ($\sigma_1^2 = \sigma_2^2$). This assumption is required for the pooled standard deviation to be a valid common estimator.

How to check:

• Levene's test (preferred — robust to non-normality): $H_0$: equal variances.
• Brown-Forsythe test: more robust variant of Levene's for non-normal data.
• Variance ratio rule of thumb: if $s^2_{larger}/s^2_{smaller} > 4$, heterogeneity is potentially problematic.
• $F$-test of equality of variances (sensitive to non-normality — use with caution).

⚠️ A statistically significant Levene's test does not automatically invalidate Student's t-test for large equal-sized samples (the test is robust). However, when groups are unequal in size AND have unequal variances, Student's t-test can be severely anti-conservative (inflated Type I error). In this case, always use Welch's t-test.

When violated: Use Welch's t-test, which does not assume equal variances and is generally recommended as the default for independent samples comparisons (see Section 10).

4.3 Independence of Observations

Within each group, all observations must be independent — the score of one participant must not influence the score of any other. This is an assumption about the study design, not about the data, and cannot be tested statistically.

Common violations:

• Students sampled from the same classroom (classroom effect).
• Patients sampled from the same hospital ward.
• Animals from the same litter.
• Repeated measurements on the same participant (use paired t-test instead).
• Family members in the same study.

When violated: For clustered data, use multilevel models. For repeated measures within the same participant, use the paired t-test or repeated measures ANOVA.

4.4 Scale of Measurement

t-Tests assume the dependent variable is measured on at least an interval scale — that is, the differences between values are meaningful and equal across the scale.

When violated: If the outcome is ordinal (ranked categories) or continuous but severely non-normal, use non-parametric alternatives (Mann-Whitney U, Wilcoxon signed-rank).

4.5 Random Sampling

For inferential conclusions to generalise to the population, the sample should be randomly selected from the population of interest. In practice, many research samples are convenience samples; this limits the generalisability of conclusions but does not invalidate the mathematical procedure of the t-test itself.

4.6 Absence of Influential Outliers

Extreme outliers can dramatically distort the mean and standard deviation, leading to inflated or deflated t-statistics. The t-test is sensitive to outliers, particularly in small samples.

How to check:

• Boxplots: inspect for values more than $1.5 \times IQR$ beyond the quartiles.
• Standardised scores: flag $\lvert z \rvert > 3$ as potential outliers.
• Grubbs' test or generalised ESD method for formal outlier detection.

When outliers are present: Investigate whether outliers represent data entry errors, measurement errors, or genuine extreme values. Report analyses with and without outliers. Consider using the trimmed mean t-test or a robust alternative.

4.7 Assumption Summary Table

5. Types of t-Tests

5.1 Decision Flowchart for Test Selection

The following logic guides selection of the appropriate t-test or alternative:

Is the outcome variable continuous (interval/ratio)?
├── NO  → Use chi-squared / Fisher's exact (categorical outcomes)
└── YES → How many groups?
          ├── MORE THAN 2 → Use ANOVA (or Kruskal-Wallis)
          └── 1 OR 2 → Are observations independent or paired?
                        ├── PAIRED (same units, two conditions)
                        │   ├── Normal differences? → Paired t-test
                        │   └── Non-normal?         → Wilcoxon signed-rank
                        └── INDEPENDENT (different participants)
                            ├── One group vs. known value?
                            │   ├── Normal?     → One-sample t-test
                            │   └── Non-normal? → Wilcoxon signed-rank (one-sample)
                            └── Two independent groups
                                ├── Normal + equal variances → Student's t-test
                                ├── Normal + unequal variances → Welch's t-test ✅ (recommended default)
                                └── Non-normal or ordinal → Mann-Whitney U

5.2 Choosing Between Student's and Welch's t-Test

A persistent question in applied statistics is whether to use Student's t-test (assuming equal variances) or Welch's t-test (not assuming equal variances) for independent samples.

The consensus recommendation: Use Welch's t-test as the default for independent samples comparisons:

Simulation studies (Ruxton, 2006; Delacre et al., 2017) consistently show that Welch's t-test controls Type I error across all conditions, whereas Student's t-test fails when $n$ and $\sigma$ are both unequal. The loss of power from using Welch's when variances are truly equal is negligible.

💡 The recommendation to default to Welch's t-test is supported by simulation evidence and is increasingly standard practice. DataStatPro reports both Student's and Welch's results by default, with Welch's highlighted as the recommended result.

6. Using the t-Test Calculator Component

The t-Test Calculator component in DataStatPro provides a comprehensive tool for running, diagnosing, visualising, and reporting t-tests and their alternatives.

Step-by-Step Guide

Step 1 — Select the Test Type

Choose from the "Test Type" dropdown:

• One-Sample t-Test: Compare a sample mean to a known or hypothesised value.
• Independent Samples t-Test: Compare means from two unrelated groups.
• Paired Samples t-Test: Compare means from two related measurements.
• Welch's t-Test: Independent samples without the equal variance assumption.
• Mann-Whitney U Test: Non-parametric independent samples comparison.
• Wilcoxon Signed-Rank Test: Non-parametric paired or one-sample comparison.

Step 2 — Input Method

Choose how to provide the data:

• Raw data: Upload or paste the dataset directly. DataStatPro computes all statistics automatically, checks assumptions, and flags potential issues.
• Summary statistics: Enter $n$, $\bar{x}$, and $s$ for each group. Full assumption checks are not available but all test statistics and effect sizes are computed.
• Test statistic + df: Enter the $t$-statistic and degrees of freedom to compute p-values, effect sizes, and CIs from a published result.

💡 When using raw data, DataStatPro automatically runs Shapiro-Wilk tests for normality and Levene's test for equality of variances, and displays the results alongside the main output with colour-coded warnings for violations.

Step 3 — Specify the Null Hypothesis Value

• One-sample: Enter $\mu_0$ (default: 0). This is the population mean under $H_0$.
• Independent/Paired: Default is $H_0: \mu_1 - \mu_2 = 0$. Specify a non-zero value for non-inferiority or superiority testing (e.g., $H_0: \mu_1 - \mu_2 \leq -\Delta$).

Step 4 — Select the Alternative Hypothesis

• Two-tailed (default): $H_1: \mu_1 \neq \mu_2$ — most common, requires no directional prediction.
• Upper one-tailed: $H_1: \mu_1 > \mu_2$ — pre-registered directional prediction.
• Lower one-tailed: $H_1: \mu_1 < \mu_2$ — pre-registered directional prediction.

Step 5 — Choose the Significance Level

Select $\alpha$ (default: $.05$). DataStatPro also provides results for $\alpha = .01$ and $\alpha = .001$ simultaneously for reference.

Step 6 — Select the Variance Assumption

For independent samples tests:

• Equal variances (Student's): Uses pooled SD; $\nu = n_1 + n_2 - 2$.
• Unequal variances (Welch's) — Recommended: Uses Welch-Satterthwaite df.
• Auto: Run both; flag discrepancies based on Levene's test result.

Step 7 — Select Display Options

Choose which outputs to display:

• ✅ t-statistic, df, p-value, and decision.
• ✅ Means, SDs, and standard errors for each group.
• ✅ Mean difference with 95% CI (exact).
• ✅ Cohen's $d$ (or $d_z$) and Hedges' $g$ with 95% CI.
• ✅ Common Language Effect Size (CL %).
• ✅ Assumption test results (Shapiro-Wilk, Levene's).
• ✅ Distribution visualisation (two overlapping density curves with shaded overlap).
• ✅ Effect size visualisation (Cohen's $d$ diagram with $U_1$, $U_2$, $U_3$).
• ✅ Power analysis and required $n$ for 80%, 90%, 95% power.
• ✅ APA-style results paragraph.
• ✅ Equivalence test (TOST) for assessing negligibility of the effect.

Step 8 — Run the Analysis

Click "Run t-Test". DataStatPro will:

1. Compute the t-statistic, degrees of freedom, and p-value.
2. Construct the 95% CI for the mean difference.
3. Compute Cohen's $d$, Hedges' $g$, and their exact CIs.
4. Run all selected assumption tests and display warnings.
5. Generate all selected visualisations.
6. Generate an APA 7th edition-compliant results paragraph.

7. One-Sample t-Test

7.1 Purpose and Design

The one-sample t-test answers the question: "Is the mean of my sample significantly different from a specific, theoretically or practically meaningful value $\mu_0$?"

Common applications:

• Testing whether a sample's mean IQ differs from the population mean of 100.
• Testing whether a sample's mean reaction time differs from a published normative value.
• Quality control: testing whether a machine produces items with a target weight.
• Testing whether a clinical sample's mean score differs from the healthy population norm.

7.2 Full Procedure

Given: A sample of $n$ observations with mean $\bar{x}$ and standard deviation $s$. Test $H_0: \mu = \mu_0$ vs. $H_1: \mu \neq \mu_0$.

Step 1 — Compute the sample mean and SD

$\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i, \qquad s = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (x_i - \bar{x})^2}$

Step 2 — Compute the standard error

$SE = \frac{s}{\sqrt{n}}$

Step 3 — Compute the t-statistic

$t = \frac{\bar{x} - \mu_0}{SE} = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$

Step 4 — Determine degrees of freedom

$\nu = n - 1$

Step 5 — Compute the p-value

$p = 2 \times P(T_{n-1} \geq \lvert t \rvert)$

Compare to $\alpha$. Reject $H_0$ if $p \leq \alpha$.

Step 6 — Compute the 95% CI for $\mu$

$\bar{x} \pm t_{\alpha/2,\; n-1} \cdot \frac{s}{\sqrt{n}}$

Step 7 — Compute Cohen's $d$

$d = \frac{\bar{x} - \mu_0}{s}$

Hedges' $g$ (bias-corrected):

$g = d \times \left(1 - \frac{3}{4(n-1) - 1}\right)$

7.3 Interpreting the One-Sample t-Test

8. Independent Samples t-Test

8.1 Purpose and Design

The independent samples t-test answers: "Do two independent groups have the same population mean?" It requires that the two groups are composed of entirely different participants with no systematic pairing or matching.

Common applications:

• Comparing test scores between a treatment group and a control group.
• Comparing anxiety levels between males and females.
• Comparing response times between two experimental conditions (between-subjects design).
• Comparing customer satisfaction between two product versions.

8.2 Full Procedure (Student's)

Given: Group 1 with $n_1$ observations ($\bar{x}_1$, $s_1$) and Group 2 with $n_2$ observations ($\bar{x}_2$, $s_2$). Test $H_0: \mu_1 = \mu_2$.

Step 1 — Compute summary statistics

$\bar{x}_j = \frac{1}{n_j}\sum_{i=1}^{n_j} x_{ji}, \qquad s_j = \sqrt{\frac{1}{n_j-1}\sum_{i=1}^{n_j}(x_{ji}-\bar{x}_j)^2}, \quad j \in \{1, 2\}$

Step 2 — Check variance homogeneity (Levene's test)

Run Levene's test. If $p_{Levene} \leq .05$, favour Welch's t-test (Section 10). Regardless, reporting both is best practice.

Step 3 — Compute pooled standard deviation

$s_{pooled} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1+n_2-2}}$

Step 4 — Compute the t-statistic

$t = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}\sqrt{1/n_1 + 1/n_2}}$

Step 5 — Degrees of freedom and p-value

$\nu = n_1 + n_2 - 2$

$p = 2 \times P(T_\nu \geq \lvert t \rvert)$

Step 6 — 95% CI for the mean difference

$(\bar{x}_1 - \bar{x}_2) \pm t_{\alpha/2,\;\nu} \cdot s_{pooled}\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$

Step 7 — Effect sizes

$d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}, \qquad g = d \times \left(1 - \frac{3}{4(n_1+n_2-2)-1}\right)$

Common Language Effect Size:

$CL = \Phi\!\left(\frac{d}{\sqrt{2}}\right)$

8.3 APA Reporting Template

"An independent samples t-test revealed [a significant / no significant] difference in [DV] between [Group 1] ($M =$, $SD =$) and [Group 2] ($M =$, $SD =$), $t(\nu) =$, $p =$, $d =$ [95% CI: , ]. This represents a [small / medium / large] effect according to Cohen's (1988) benchmarks."

Example: "An independent samples Welch's t-test revealed a significant difference in anxiety scores between the CBT group ($M = 12.3$, $SD = 4.1$) and the waitlist control group ($M = 18.7$, $SD = 5.2$), $t(57.4) = -5.62$, $p < .001$, $d = 1.38$ [95% CI: 0.87, 1.88]. This represents a large treatment effect."

9. Paired Samples t-Test

9.1 Purpose and Design

The paired samples t-test (also: dependent samples, matched pairs, or repeated measures t-test) answers: "Do two related measurements differ significantly from each other?"

When observations are paired:

• Pre-post designs: The same participant measured before and after an intervention.
• Matched pairs: Participants matched on key characteristics (age, sex, IQ) and randomised to different conditions.
• Within-subjects designs: Each participant experiences both conditions.
• Natural pairs: Twins, left hand vs. right hand, matched siblings.

Advantage over independent t-test: By comparing within-person differences, the paired design removes between-person variability from the error term, substantially increasing power when the within-person correlation $r_{12}$ is positive.

9.2 Full Procedure

Given: $n$ pairs of observations $(x_{1i}, x_{2i})$.

Step 1 — Compute difference scores

$d_i = x_{1i} - x_{2i}, \qquad i = 1, 2, \ldots, n$

Step 2 — Compute mean and SD of differences

$\bar{d} = \frac{1}{n}\sum_{i=1}^n d_i, \qquad s_d = \sqrt{\frac{1}{n-1}\sum_{i=1}^n(d_i - \bar{d})^2}$

Step 3 — Compute the standard error of the mean difference

$SE_{\bar{d}} = \frac{s_d}{\sqrt{n}}$

Step 4 — Compute the t-statistic

$t = \frac{\bar{d}}{SE_{\bar{d}}} = \frac{\bar{d}}{s_d/\sqrt{n}}$

Step 5 — Degrees of freedom and p-value

$\nu = n - 1$

$p = 2 \times P(T_{n-1} \geq \lvert t \rvert)$

Step 6 — 95% CI for the mean difference

$\bar{d} \pm t_{\alpha/2,\; n-1} \cdot \frac{s_d}{\sqrt{n}}$

Step 7 — Effect sizes

Cohen's $d_z$ (most commonly reported for paired designs):

$d_z = \frac{\bar{d}}{s_d} = \frac{t}{\sqrt{n}}$

Cohen's $d_{rm}$ (repeated measures $d$, accounting for the pre-post correlation):

$d_{rm} = \frac{\bar{d}}{s_{av}} \times \sqrt{2(1 - r_{12})}$

Where $s_{av} = (s_1 + s_2)/2$ and $r_{12}$ is the correlation between the two measurements. Note that $d_{rm}$ is more comparable to $d$ from independent samples designs than $d_z$ is.

Cohen's $d_{av}$ (standardised by the average SD):

$d_{av} = \frac{\bar{d}}{s_{av}} = \frac{\bar{x}_1 - \bar{x}_2}{(s_1+s_2)/2}$

9.3 Comparing Paired and Independent t-Tests for the Same Data

When data are paired (pre-post), computing the incorrect independent t-test is a serious error. The relationship between the two t-statistics is:

$t_{paired} = t_{independent} \cdot \sqrt{\frac{2(1-r_{12})}{1}} \cdot \sqrt{n}$

More precisely, the t-statistics are related through the within-pair correlation $r_{12}$:

$SE_{paired} = SE_{independent} \cdot \sqrt{2(1-r_{12})}$

When $r_{12} > 0$ (typical for repeated measures): $SE_{paired} < SE_{independent}$, so $t_{paired} > t_{independent}$ — the paired test is more powerful. When $r_{12} = 0$ (independence), both tests are equivalent. When $r_{12} < 0$ (rare), the independent test is more powerful.

⚠️ Never apply an independent samples t-test to paired data. Doing so ignores the within-pair correlation, produces an inflated standard error, and loses statistical power. Conversely, applying a paired t-test to genuinely independent data violates the independence assumption of the difference scores.

10. Welch's t-Test — Unequal Variances

10.1 Why Welch's is Preferred

Welch's t-test (1947) is a modification of Student's t-test that does not assume equal population variances. It is the recommended default for independent samples comparisons for three reasons:

1. Robustness: It maintains correct Type I error rates regardless of whether variances are equal or unequal.
2. Negligible power loss: When variances are truly equal, Welch's test loses very little power compared to Student's.
3. Correct coverage: The CI from Welch's has the correct nominal coverage probability across all variance ratio conditions.

10.2 Full Procedure

Step 1 — Compute group summary statistics

$\bar{x}_1, s_1, n_1$ and $\bar{x}_2, s_2, n_2$

Step 2 — Compute separate variance estimates

$v_1 = \frac{s_1^2}{n_1}, \qquad v_2 = \frac{s_2^2}{n_2}$

Step 3 — Compute Welch's t-statistic

$t_W = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{v_1 + v_2}}$

Step 4 — Compute Welch-Satterthwaite degrees of freedom

$\nu_W = \frac{(v_1 + v_2)^2}{\dfrac{v_1^2}{n_1-1} + \dfrac{v_2^2}{n_2-1}}$

Round $\nu_W$ down to the nearest integer for conservative inference.

Step 5 — p-value

$p = 2 \times P(T_{\nu_W} \geq \lvert t_W \rvert)$

Step 6 — 95% CI for mean difference

$(\bar{x}_1 - \bar{x}_2) \pm t_{\alpha/2,\; \nu_W} \cdot \sqrt{v_1 + v_2}$

Step 7 — Effect size (Glass's $\Delta$ or Welch's $d$)

When variances are unequal, the appropriate standardiser for Cohen's $d$ is debated. Options include:

Pooled SD (ignores heterogeneity — caution):

$d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}$

Glass's $\Delta$ (control group SD as standardiser — recommended for treatment/control):

$\Delta = \frac{\bar{x}_1 - \bar{x}_2}{s_2}$

Average SD (unbiased when neither group is the reference):

$d_{av} = \frac{\bar{x}_1 - \bar{x}_2}{(s_1 + s_2)/2}$

💡 DataStatPro reports all three standardisers alongside Welch's t-test, with Glass's $\Delta$ highlighted when one group is a designated control, and $d_{av}$ highlighted when neither group is a natural reference.

10.3 Student's vs. Welch's: A Direct Comparison

11. Non-Parametric Alternatives

11.1 When to Use Non-Parametric Tests

Non-parametric tests (also called distribution-free tests) are appropriate when:

• Data are ordinal (ranks, Likert items treated as ordinal).
• Data are severely non-normally distributed and sample sizes are small.
• There are extreme outliers that cannot be removed and that distort parametric statistics.
• The research question concerns ranks or medians rather than means.

Trade-off: Non-parametric tests are more robust to assumption violations but have lower statistical power than their parametric counterparts when parametric assumptions ARE met. When normality holds, using a non-parametric test discards information.

💡 Non-parametric does not mean "assumption-free." The Mann-Whitney U test assumes that the two distributions have the same shape (just shifted); violation of this shape assumption means U tests the combined null of equal location AND equal shape, not just equal medians.

11.2 Mann-Whitney U Test (Non-Parametric Independent Samples)

The Mann-Whitney U test (also Wilcoxon rank-sum test) is the non-parametric alternative to the independent samples t-test. It tests whether the distributions of two independent groups are identical (or, under the shape assumption, whether one group tends to have higher ranks than the other).

Procedure:

Step 1 — Rank all observations

Combine all $n_1 + n_2$ observations and assign ranks from 1 (smallest) to $N = n_1 + n_2$. For tied values, assign the average of the tied ranks.

Step 2 — Compute the rank sums

$W_1 = \sum_{i=1}^{n_1} R_i \quad \text{(sum of ranks for Group 1)}$

$W_2 = \sum_{j=1}^{n_2} R_j \quad \text{(sum of ranks for Group 2)}$

Check: $W_1 + W_2 = N(N+1)/2$

Step 3 — Compute U statistics

$U_1 = n_1 n_2 + \frac{n_1(n_1+1)}{2} - W_1$

$U_2 = n_1 n_2 + \frac{n_2(n_2+1)}{2} - W_2$

Check: $U_1 + U_2 = n_1 n_2$

The test statistic is $U = \min(U_1, U_2)$.

Step 4 — Compute z-approximation (for $n > 10$)

Under $H_0$, with continuity correction:

$z = \frac{U - n_1 n_2/2}{\sqrt{n_1 n_2(n_1+n_2+1)/12}}$

For ties, the variance requires a correction factor:

$z = \frac{U - n_1 n_2/2}{\sqrt{\dfrac{n_1 n_2}{12}\!\left(n+1 - \dfrac{\sum_{k}(t_k^3 - t_k)}{n(n-1)}\right)}}$

Where $t_k$ is the number of observations in the $k$-th tied group.

Step 5 — Effect size: Rank-biserial correlation

$r_{rb} = 1 - \frac{2U}{n_1 n_2} = \frac{U_1 - U_2}{n_1 n_2}$

Or equivalently:

$r_{rb} = \frac{2z}{\sqrt{n_1 + n_2}}$

Interpretation: $r_{rb} = 0.5$ means that 75% of observations in Group 1 exceed those in Group 2.

Cohen's benchmarks for $r_{rb}$ (same as $r$):

11.3 Wilcoxon Signed-Rank Test (Non-Parametric Paired)

The Wilcoxon signed-rank test is the non-parametric alternative to the paired t-test. It tests whether the distribution of difference scores is symmetric about zero.

Procedure:

Step 1 — Compute and rank absolute differences

Compute $d_i = x_{1i} - x_{2i}$. Remove pairs where $d_i = 0$. Let $n'$ = number of non-zero differences.

Rank $\lvert d_i \rvert$ from 1 (smallest) to $n'$ (largest), assigning average ranks for ties.

Step 2 — Sum positive and negative ranks

$W^+ = \sum_{d_i > 0} R_i \quad \text{(sum of ranks of positive differences)}$

$W^- = \sum_{d_i < 0} R_i \quad \text{(sum of ranks of negative differences)}$

Check: $W^+ + W^- = n'(n'+1)/2$

The test statistic is $W = \min(W^+, W^-)$.

Step 3 — z-approximation (for $n' > 10$)

$z = \frac{W^+ - n'(n'+1)/4}{\sqrt{n'(n'+1)(2n'+1)/24}}$

With tie correction:

$z = \frac{W^+ - n'(n'+1)/4}{\sqrt{\dfrac{n'(n'+1)(2n'+1)}{24} - \dfrac{\sum_k(t_k^3-t_k)}{48}}}$

Step 4 — Effect size

$r_W = \frac{z}{\sqrt{n'}}$

Or, the matched-pairs rank-biserial correlation:

$r_{rb} = 1 - \frac{4W^+}{n'(n'+1)}$

11.4 One-Sample Wilcoxon Signed-Rank Test

The one-sample version tests $H_0$: the population median equals $\theta_0$. Compute $d_i = x_i - \theta_0$ and apply the Wilcoxon signed-rank procedure as above.

11.5 Comparing Parametric and Non-Parametric Tests

⚠️ The Asymptotic Relative Efficiency (ARE) of the Mann-Whitney U test relative to the t-test is $3/\pi \approx 0.955$ for normal data — meaning you only need about 5% more observations with Mann-Whitney to achieve the same power as the t-test. This near-equality of efficiency makes Mann-Whitney a safe choice when normality is questionable.

11.6 Brunner-Munzel Test — Handling Unequal Shapes

When the two distributions have different shapes (not just different locations), the Mann-Whitney test actually tests a compound null of equal location AND equal spread. The Brunner-Munzel test (Brunner & Munzel, 2000) is a robust alternative that tests only the stochastic equality of the two groups without the shape assumption:

$H_0: P(X_1 > X_2) + 0.5 \cdot P(X_1 = X_2) = 0.5$

The test statistic uses ranked data with separate within-group rankings:

$t_{BM} = \frac{n_1 n_2 (\bar{R}_1^{(int)} - \bar{R}_2^{(int)})}{N\sqrt{n_1\hat{S}_1^2 + n_2\hat{S}_2^2}}$

Where $\bar{R}_j^{(int)}$ are internal group ranks. DataStatPro reports the Brunner-Munzel test as an option under the Non-Parametric menu when the distribution shape assumption may be violated.

12. Advanced Topics

12.1 Robust t-Tests — Trimmed Means and Winsorisation

Trimmed mean t-tests use the $\alpha$-trimmed mean (removing the top and bottom $\alpha$ proportion of observations) as the measure of central tendency. Yuen's (1974) $t$-test for trimmed means:

$t_{trim} = \frac{\bar{x}_{t1} - \bar{x}_{t2}}{SE_{trim}}$

With 20% trimming from each tail:

$\bar{x}_{t} = \frac{\sum_{i=h+1}^{n-h} x_{(i)}}{n - 2h}, \quad h = \lfloor \alpha n \rfloor$

$SE_{trim} = \sqrt{\frac{W_1}{h_1(h_1-1)} + \frac{W_2}{h_2(h_2-1)}}$

Where $W_j$ are the Winsorised sum of squared deviations, and $h_j = n_j - 2\lfloor \alpha n_j \rfloor$.

Trimmed mean t-tests are substantially more powerful than rank-based tests for heavy-tailed symmetric distributions while maintaining good Type I error control.

12.2 Bootstrap t-Tests

The bootstrap t-test (Efron & Tibshirani, 1994) makes no parametric distributional assumptions. It constructs the null distribution of the t-statistic empirically:

Procedure:

1. Compute the observed t-statistic $t_{obs}$.
2. Centre both samples on a common mean: $x_i^* = x_i - \bar{x}_j + \bar{x}_{grand}$.
3. Draw $B$ bootstrap samples (typically $B = 10{,}000$) from the centred samples with replacement and compute $t^*_b$ for each.
4. The p-value is the proportion of bootstrap t-statistics exceeding $\lvert t_{obs} \rvert$.

Percentile bootstrap CI for the mean difference:

Resample from the original data and compute the mean difference for each bootstrap sample. The 95% CI is the 2.5th and 97.5th percentiles of the bootstrap distribution.

The bootstrap is particularly valuable for small, non-normal samples where both parametric and asymptotic approximations may be poor.

12.3 Bayesian t-Tests

The Bayesian t-test (Rouder et al., 2009; Jeffreys, 1961) quantifies evidence for both $H_0$ (no effect) and $H_1$ (an effect exists) using the Bayes Factor ($BF_{10}$):

$BF_{10} = \frac{P(\text{data} \mid H_1)}{P(\text{data} \mid H_0)}$

$BF_{10}$ represents how many times more likely the data are under $H_1$ than $H_0$.

For the default Bayesian t-test (Rouder et al., 2009), the prior on effect size under $H_1$ is a Cauchy distribution with scale $r$:

$\delta \sim \text{Cauchy}(0, r)$

The default scale $r = \sqrt{2}/2 \approx 0.707$ represents a "medium" effect size prior.

Interpreting Bayes Factors:

Advantages of Bayesian t-tests:

• Quantify evidence for $H_0$ (null results can be informative).
• Avoid the optional stopping problem (p-values are invalid for sequential testing).
• Provide a single, direct statement about the relative evidence for each hypothesis.

Limitations: Sensitive to the choice of prior. Results should always be reported with the prior specification and checked for sensitivity to alternative priors.

12.4 Equivalence Testing with TOST

Standard null hypothesis testing only allows rejection of $H_0: d = 0$. When the goal is to demonstrate absence of a meaningful effect (e.g., showing that a generic drug is bioequivalent to a brand-name drug), the Two One-Sided Tests (TOST) procedure is required.

Specify equivalence bounds $[-\Delta_L, \Delta_U]$ (e.g., $d = \pm 0.20$, corresponding to a "trivially small" effect).

TOST procedure:

Test $H_{01}: \mu_1 - \mu_2 \leq -\Delta_L$ using upper one-tailed test. Test $H_{02}: \mu_1 - \mu_2 \geq \Delta_U$ using lower one-tailed test.

Equivalence is concluded at level $\alpha$ when both one-tailed tests reject their respective nulls — equivalently, when the $90\%$ CI (for $\alpha = .05$) for the mean difference falls entirely within $(-\Delta_L, \Delta_U)$.

💡 Note that TOST uses a 90% CI (not 95%) when $\alpha = .05$, because each one-tailed test is at the $\alpha = .05$ level. The 90% CI corresponds to two one-tailed 5% tests.

Minimum detectable equivalence with $n$ per group:

$\Delta_{min} = t_{\alpha,\; n-1} \cdot \sqrt{\frac{2s^2}{n}} + t_{\beta,\; n-1} \cdot \sqrt{\frac{2s^2}{n}}$

12.5 Sequential t-Tests and Alpha Spending

Traditional t-tests do not allow for interim analyses — looking at the data multiple times inflates the Type I error rate. Sequential approaches address this:

Sequential Probability Ratio Test (SPRT): Compute a likelihood ratio $\Lambda$ after each observation. Stop when $\Lambda \leq B$ (accept $H_0$) or $\Lambda \geq A$ (reject $H_0$), where $A = (1-\beta)/\alpha$ and $B = \beta/(1-\alpha)$.

Alpha spending functions (O'Brien-Fleming, Pocock): Pre-specify how the total $\alpha$ budget is distributed across planned interim and final analyses.

Bayesian sequential testing: Use Bayes Factors to monitor evidence continuously. Unlike frequentist sequential testing, Bayesian sequential testing is valid at any stopping point without an alpha correction.

12.6 Multiple Comparisons and t-Tests

When multiple t-tests are conducted within the same study, the familywise error rate (FWER) — the probability of at least one Type I error — inflates:

$FWER = 1 - (1-\alpha)^k$

Where $k$ is the number of tests. For $k = 5$ independent tests at $\alpha = .05$: $FWER = 1 - (0.95)^5 = .226$.

Correction methods:

⚠️ Corrections for multiple comparisons should be planned before data collection and applied to the entire family of tests. Post-hoc correction of selected tests is not valid. When tests are planned contrasts from a theoretically derived framework, no correction may be necessary — this should be justified explicitly.

12.7 Effect Sizes for t-Tests: Choosing the Right $d$ Variant

Multiple variants of Cohen's $d$ exist for the paired design, and choosing the wrong one leads to incomparable effect sizes across studies:

Which to use:

• Report $d_z$ when comparing paired designs to other paired designs.
• Report $d_{av}$ when comparing a paired design to a between-subjects design.
• Always specify which variant was used.

12.8 Reporting t-Tests According to APA 7th Edition

The APA Publication Manual (7th ed.) requires:

1. Test statistic: $t(\nu) =$ value
2. p-value: $p =$ value (report exact value; use $p < .001$ only when below .001)
3. Effect size with 95% CI: Cohen's $d =$ [LB, UB] or Hedges' $g$
4. Group means and standard deviations
5. Whether Welch's correction was applied (for independent t-tests)
6. Whether the CI is for the mean difference or the standardised effect size

Full APA template:

"[Group 1] ($M =$, $SD =$, $n =$) and [Group 2] ($M =$, $SD =$, $n =$) were compared using [Student's / Welch's] independent samples t-test. The test revealed [a significant / no significant] mean difference, $t(\nu) =$, $p =$, $d =$ [95% CI: , ], indicating a [small / medium / large] effect."

13. Worked Examples

Example 1: One-Sample t-Test — Comparing Response Time to a Normative Standard

A cognitive neuroscience researcher measures simple reaction times (in ms) for $n = 25$ adults diagnosed with ADHD. The published population norm for neurotypical adults is $\mu_0 = 250$ ms. The researcher tests whether the ADHD sample has a significantly different mean reaction time.

Data summary:

$n = 25, \quad \bar{x} = 281.4 \text{ ms}, \quad s = 42.8 \text{ ms}$

Step 1 — Standard error:

$SE = \frac{42.8}{\sqrt{25}} = \frac{42.8}{5} = 8.56 \text{ ms}$