Sample Size and Power Analysis: Zero to Hero Tutorial

This comprehensive tutorial takes you from the foundational concepts of statistical power and sample size determination all the way through advanced interpretation, reporting, assumption checking, and practical usage within the DataStatPro application. Whether you are planning a new study for the first time or deepening your understanding of how to design adequately powered research, this guide builds your knowledge systematically from the ground up.

Table of Contents

1. Prerequisites and Background Concepts
2. What are Sample Size and Power Analysis?
3. The Mathematics Behind Power Analysis
4. Considerations and Planning Checklist
5. Power Analysis for Common Statistical Tests
6. Using the Sample Size and Power Analysis Calculator Component
7. Step-by-Step Procedure
8. Interpreting the Output
9. Visualising Power and Sample Size
10. Sensitivity Analysis and Robustness Checks
11. Advanced Topics
12. Worked Examples
13. Common Mistakes and How to Avoid Them
14. Troubleshooting
15. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before diving into sample size and power analysis, it is essential to be comfortable with the following foundational statistical concepts. Each is briefly reviewed below.

1.1 Hypothesis Testing Framework

All power analyses are grounded in the hypothesis testing framework. A statistical test evaluates the evidence in a dataset against a null hypothesis:

• Null hypothesis ($H_0$): The default position — typically, that no effect exists, no difference is present, or variables are unassociated.
• Alternative hypothesis ($H_1$): The research hypothesis — that an effect exists, a difference is present, or variables are associated.

The test produces a test statistic (e.g., $t$, $F$, $\chi^2$, $z$) and a corresponding p-value. If $p \leq \alpha$, we reject $H_0$ in favour of $H_1$.

1.2 The Four Outcomes of a Hypothesis Test

Every hypothesis test results in one of four possible outcomes, two of which are correct decisions and two of which are errors:

The probabilities associated with each outcome:

1.3 The Significance Level ($\alpha$)

The significance level $\alpha$ is the maximum acceptable probability of a Type I error — that is, the probability of declaring a significant result when the null hypothesis is actually true. The researcher chooses $\alpha$ before collecting data.

Conventional values:

1.4 The p-Value

The p-value is the probability of observing a test statistic as extreme as or more extreme than the one obtained, assuming $H_0$ is true:

$p = P(\text{test statistic} \geq t_{obs} \mid H_0)$

A small p-value (below $\alpha$) means the observed result is unlikely under $H_0$ and constitutes evidence against $H_0$. Crucially, the p-value does not tell you the probability that $H_0$ is true, nor does it measure the size or practical importance of an effect.

1.5 Effect Size

An effect size is a standardised, scale-free measure of the magnitude of a phenomenon. It is the single most important input to any power analysis. Common effect size measures by test type:

1.6 Cohen's Conventions for Effect Size

Jacob Cohen (1988) proposed widely used benchmarks for effect size magnitudes across common statistical tests. These are conventions of last resort — domain knowledge always supersedes them:

1.7 The Normal and Non-Central Distributions

Power analysis relies on understanding how test statistics are distributed under two scenarios:

• Under $H_0$: The test statistic follows a central distribution (e.g., standard normal $\mathcal{N}(0,1)$, central t, central $\chi^2$, central F).
• Under $H_1$: The test statistic follows a non-central distribution — the same family but shifted by a non-centrality parameter $\lambda$, which quantifies the true size of the effect and the sample size:

$\lambda = \delta \times \sqrt{n} \quad \text{(for z/t-tests, simplified form)}$

Power is the probability that a non-centrally distributed test statistic exceeds the critical value derived from the central distribution.

1.8 Directionality: One-Tailed vs. Two-Tailed Tests

The directionality of a test affects the critical region and therefore the power:

• Two-tailed test: The critical region is split across both tails of the distribution. Rejects $H_0$ for both very large and very small test statistics. Critical value: $z_{\alpha/2}$ (e.g., $1.96$ for $\alpha = .05$).
• One-tailed test: The critical region is entirely in one tail. More powerful for detecting effects in the predicted direction but cannot detect effects in the opposite direction. Critical value: $z_{\alpha}$ (e.g., $1.645$ for $\alpha = .05$).

For a given effect size and sample size, a one-tailed test has greater power than a two-tailed test. However, one-tailed tests require a strong directional a priori justification and are vulnerable to criticism if the actual effect is in the opposite direction.

⚠️ Most journals and reporting guidelines recommend two-tailed tests unless there is a strong, pre-registered, directional theoretical justification. DataStatPro defaults to two-tailed tests for all procedures.

2. What are Sample Size and Power Analysis?

2.1 The Core Questions

Power analysis and sample size determination address four deeply interconnected questions in research design:

1. Power analysis (post-hoc): Given my sample size, effect size, and $\alpha$, what was the probability of detecting a true effect?
2. Sample size calculation (a priori): Given a desired power, effect size, and $\alpha$, how many participants do I need?
3. Minimum detectable effect: Given my sample size and $\alpha$, what is the smallest effect I have adequate power to detect?
4. Sensitivity analysis: Given my sample size and $\alpha$, how does power vary across a range of plausible effect sizes?

These four questions form the four modes of power analysis. A priori sample size calculation — performed before data collection — is the most important and is the focus of most of this tutorial.

2.2 Why Power Analysis Matters

2.3 The Four Elements of Power Analysis

Every power analysis involves exactly four quantities, any one of which can be computed from the other three:

Power (1 - β)  ←─────────────────────────────┐
         │                                   │
Sample size (N) ─────────────────────►       │
         ↕                                   │
Effect size (ES) ───────────────────►         │
         │                                   │
Significance level (α) ─────────────►         │
         └───────────────────────────────────┘

Specify any three → solve for the fourth.

2.4 Four Modes of Power Analysis

⚠️ Post-hoc power analysis computed using the observed effect size is widely regarded as uninformative and should be avoided. When $p > \alpha$, observed power will typically be low — but this is a mathematical consequence of the non-significant result, not an independent finding. Instead of post-hoc power, report the 95% confidence interval for the effect size and a sensitivity power analysis.

2.5 Desired Power: Choosing $1 - \beta$

The conventional target for statistical power is 0.80 (80%), implying a 20% Type II error rate. Higher power targets are increasingly recommended:

The ratio of Type I to Type II error rates is also informative:

• At $\alpha = .05$ and power $= .80$: ratio $= \beta/\alpha = .20/.05 = 4$ (Type II errors are 4× more likely than Type I errors).
• At $\alpha = .05$ and power $= .95$: ratio $= .05/.05 = 1$ (equal error rates).

2.6 Real-World Applications

3. The Mathematics Behind Power Analysis

3.1 The General Power Framework

For any test with test statistic $T$ and critical value $t_{crit}$:

$\text{Power} = 1 - \beta = P(T \geq t_{crit} \mid H_1 \text{ is true})$

Under $H_1$, $T$ follows a non-central distribution with non-centrality parameter $\lambda$. Power is the probability that this non-centrally distributed test statistic exceeds the critical value determined under $H_0$.

The critical value $t_{crit}$ is determined by $\alpha$ and the test type:

• Two-tailed: $t_{crit} = z_{1-\alpha/2}$ (e.g., $1.960$ for $\alpha = .05$)
• One-tailed: $t_{crit} = z_{1-\alpha}$ (e.g., $1.645$ for $\alpha = .05$)

3.2 Power for the One-Sample z-Test

The simplest case: testing whether a population mean $\mu$ equals a known value $\mu_0$, with known population SD $\sigma$ and sample size $n$.

Non-centrality parameter:

$\lambda = \frac{(\mu_1 - \mu_0)}{\sigma / \sqrt{n}} = d \times \sqrt{n}$

Where $d = (\mu_1 - \mu_0)/\sigma$ is Cohen's $d$ for the one-sample case.

Power (two-tailed):

$1 - \beta = 1 - \Phi(z_{\alpha/2} - \lambda) + \Phi(-z_{\alpha/2} - \lambda)$

For practical purposes (when $\lambda$ is not near zero):

$1 - \beta \approx \Phi(\lambda - z_{\alpha/2})$

Where $\Phi$ is the standard normal CDF.

Sample size formula (solving for $n$):

$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2$

Where $z_{1-\beta}$ is the critical value for the desired power (e.g., $0.842$ for $1-\beta = 0.80$; $1.282$ for $1-\beta = 0.90$; $1.645$ for $1-\beta = 0.95$).

3.3 Power for the Two-Sample Independent t-Test

Testing whether two population means differ ($H_1: \mu_1 \neq \mu_2$), with equal group sizes $n_1 = n_2 = n$ and pooled SD $\sigma_{pooled}$.

Cohen's $d$ (standardised mean difference):

$d = \frac{\mu_1 - \mu_2}{\sigma_{pooled}}$

Non-centrality parameter:

$\lambda = \frac{d}{\sqrt{1/n_1 + 1/n_2}} = d \times \sqrt{\frac{n}{2}}$

Sample size per group (approximate):

$n_{per\;group} = \frac{2(z_{\alpha/2} + z_{1-\beta})^2}{d^2}$

Unequal group sizes: Let $n_2 = k \times n_1$ (allocation ratio $k$). The total sample size is minimised when $k = 1$ (equal groups). For unequal allocation:

$n_1 = \frac{(z_{\alpha/2} + z_{1-\beta})^2(1 + 1/k)}{d^2}, \qquad n_2 = k \times n_1$

⚠️ Unequal group sizes are less efficient than equal groups for a given total $N$. Unless there is a compelling reason (e.g., one group is more expensive to recruit, or a 2:1 allocation is ethically required), equal groups maximise power per participant.

3.4 Power for the Paired t-Test

Testing whether the mean of paired differences $\mu_D = \mu_1 - \mu_2$ equals zero.

Cohen's $d_z$ (based on the SD of differences):

$d_z = \frac{\mu_D}{\sigma_D}$

Where $\sigma_D = \sqrt{\sigma_1^2 + \sigma_2^2 - 2\rho\sigma_1\sigma_2}$ and $\rho$ is the correlation between paired measurements.

The relationship to Cohen's $d$ for independent groups:

$d_z = \frac{d}{\sqrt{2(1-\rho)}}$

This shows that paired designs are more powerful when $\rho > 0$ — the higher the correlation between paired measurements, the greater the efficiency gain over an independent design.

Sample size (number of pairs):

$n_{pairs} = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d_z}\right)^2$

3.5 Power for One-Way ANOVA

Testing whether $k$ group means are equal ($H_1$: at least two means differ).

Cohen's $f$ (standardised SD of group means):

$f = \frac{\sigma_m}{\sigma_{within}} = \sqrt{\frac{\sum_{j=1}^k n_j(\mu_j - \mu)^2 / N}{\sigma_{within}^2}}$

Where $\sigma_m$ is the SD of the group means and $\sigma_{within}$ is the common within-group SD.

Relationship to $\eta^2$ (eta-squared, the proportion of variance explained):

$f = \sqrt{\frac{\eta^2}{1 - \eta^2}}$

Non-centrality parameter:

$\lambda = f^2 \times N$

Under $H_1$, the F-statistic follows a non-central F distribution with numerator df $= k - 1$, denominator df $= N - k$, and non-centrality parameter $\lambda$.

Sample size per group (equal groups):

$n_{per\;group} = \frac{\lambda_{required}}{k \times f^2}$

Where $\lambda_{required}$ is the non-centrality parameter needed to achieve the desired power at the specified $\alpha$, $df_1 = k-1$, and $df_2 = k(n-1)$ (solved iteratively as $df_2$ depends on $n$).

3.6 Power for Pearson Correlation

Testing whether the population correlation $\rho = 0$ ($H_1: \rho \neq 0$).

Effect size: The population correlation coefficient $\rho$ itself.

Fisher's z-transformation: To stabilise the variance:

$z_r = \frac{1}{2}\ln\!\left(\frac{1+r}{1-r}\right) = \text{arctanh}(r), \qquad SE_{z_r} = \frac{1}{\sqrt{n-3}}$

Non-centrality parameter:

$\lambda = z_\rho \times \sqrt{n - 3}$

Where $z_\rho = \text{arctanh}(\rho)$.

Sample size:

$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{z_\rho}\right)^2 + 3$

3.7 Power for the Chi-Square Test of Association

Testing whether two categorical variables are independent. See also the chi-square tutorial for additional detail.

Effect size — Cohen's $w$:

$w = \sqrt{\sum_{i}\sum_{j} \frac{(P_{ij} - P_{i\cdot}P_{\cdot j})^2}{P_{i\cdot}P_{\cdot j}}}$

For $2 \times 2$ tables, $w = \phi$ (phi coefficient); for larger tables, $w$ is related to Cramér's $V$:

$w = V \times \sqrt{\min(r-1,\; c-1)}$

Non-centrality parameter:

$\lambda = w^2 \times N$

Under $H_1$, $\chi^2$ follows a non-central chi-square distribution with $df = (r-1)(c-1)$ and non-centrality parameter $\lambda$.

Sample size:

$N = \frac{\lambda_{required}}{w^2}$

3.8 Power for Proportion Tests

3.8.1 One-Sample Proportion Test

Testing $H_0: \pi = \pi_0$ vs. $H_1: \pi \neq \pi_0$.

Cohen's $h$ (arcsine difference):

$h = 2\arcsin(\sqrt{\pi_1}) - 2\arcsin(\sqrt{\pi_0})$

Sample size:

$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{h}\right)^2$

3.8.2 Two-Sample Proportion Test

Testing $H_0: \pi_1 = \pi_2$ vs. $H_1: \pi_1 \neq \pi_2$.

$h = 2\arcsin(\sqrt{\pi_1}) - 2\arcsin(\sqrt{\pi_2})$

$n_{per\;group} = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{h}\right)^2$

3.9 Power for Multiple Regression

Testing whether a set of predictors explains a meaningful proportion of variance in an outcome, or whether a specific predictor contributes above and beyond others.

Cohen's $f^2$ (for the overall model or incremental $R^2$):

$f^2 = \frac{R^2}{1 - R^2}$

For testing an increment in $R^2$ when adding $u$ new predictors:

$f^2 = \frac{R^2_{full} - R^2_{reduced}}{1 - R^2_{full}}$

Non-centrality parameter:

$\lambda = f^2 \times N$

Under $H_1$, the F-statistic for testing $u$ predictors follows a non-central F distribution with $df_1 = u$ and $df_2 = N - p - 1$ (where $p$ is the total number of predictors in the full model).

Sample size:

$N = \frac{\lambda_{required}}{f^2} + p + 1$

3.10 Summary of Key Sample Size Formulae

3.11 The Power Curve

The power curve plots power ($1 - \beta$) as a function of one of the four analysis inputs while holding the others constant. The most common power curve plots:

• Power vs. $N$: Shows how power increases as sample size grows. For most tests, power rises steeply at first and then plateaus. Used to identify the point of diminishing returns.
• Power vs. effect size: Shows how power changes across a range of effect sizes for a fixed $N$. Used for sensitivity analysis.
• Power vs. $\alpha$: Shows the trade-off between Type I and Type II error rates.

The power curve always satisfies:

• $1 - \beta \to \alpha$ as $N \to 0$ or effect size $\to 0$ (cannot do better than chance).
• $1 - \beta \to 1$ as $N \to \infty$ or effect size $\to \infty$.

4. Considerations and Planning Checklist

4.1 Specifying the Effect Size: The Critical Decision

The effect size is the single most consequential input to a power analysis. Poorly specified effect sizes lead to either seriously underpowered or wastefully overpowered studies. Use the following hierarchy of evidence for effect size specification:

⚠️ Do not use the effect size from a pilot study directly. Pilot effect sizes are estimated from small samples and are highly unstable. The pilot effect size will often overestimate the true effect (publication bias in miniature). Use pilot data to confirm feasibility and estimate nuisance parameters (e.g., SD, ICC), not to determine the effect size for the power calculation.

4.2 Choosing the Significance Level ($\alpha$)

The choice of $\alpha$ should be deliberate and justified:

4.3 Choosing the Power Target ($1 - \beta$)

The power target should balance the cost of missing a real effect against the cost of increasing sample size:

4.4 Identifying the Primary Outcome and Test

Power analysis is conducted for a single primary hypothesis and its associated primary test. Secondary hypotheses should have their own power analyses if they are to be formally tested.

Before beginning the analysis, clearly specify:

1. What is the primary outcome variable (and its scale)?
2. What is the primary comparison or test (e.g., mean difference, correlation)?
3. What is the statistical test that will be applied?
4. Is the test one-tailed or two-tailed?
5. What are the key assumptions (e.g., equal variances, paired vs. independent)?

4.5 Accounting for Anticipated Attrition and Missing Data

In longitudinal studies or clinical trials, participants drop out or produce missing data. The required sample size at enrollment must account for this:

$N_{enroll} = \frac{N_{analysis}}{1 - r_{attrition}}$

Where $r_{attrition}$ is the expected attrition rate (as a proportion).

Example: A study needs $N_{analysis} = 120$ completers and expects 15% attrition:

$N_{enroll} = \frac{120}{1 - 0.15} = \frac{120}{0.85} \approx 142$

For multi-wave longitudinal studies, apply the attrition correction at each wave or use the cumulative attrition rate.

4.6 Accounting for Stratification and Clustering

In studies with complex sampling designs:

• Stratified designs: Power analysis proceeds separately within strata, then sample sizes are combined.
• Clustered designs (e.g., school classes, clinical sites): The design effect (DEFF) inflates the required sample size to account for within-cluster correlation (intraclass correlation, ICC):

$DEFF = 1 + (m - 1) \times ICC$

$N_{cluster} = N_{simple} \times DEFF$

Where $m$ is the average cluster size and $ICC$ is the intraclass correlation coefficient. For clustered randomised trials (CRTs):

$n_{clusters} = \frac{N_{cluster}}{m}$

4.7 Multiple Testing Corrections

When testing $k$ hypotheses simultaneously, control the familywise error rate (FWER) or false discovery rate (FDR):

Bonferroni correction (FWER):

$\alpha' = \frac{\alpha}{k}$

For each test, use $\alpha'$ in the power analysis. This increases the required sample size substantially for large $k$.

Holm-Bonferroni (less conservative): Apply a sequential correction; compute power for the $j$-th most significant test using $\alpha' = \alpha / (k - j + 1)$.

Benjamini-Hochberg (FDR): Controls the expected proportion of false positives among significant results. Less conservative than Bonferroni for large-scale testing.

4.8 Reporting the Power Analysis: Documentation Standards

A complete power analysis report must include:

5. Power Analysis for Common Statistical Tests

5.1 One-Sample t-Test

Research question: Does the population mean differ from a known or hypothesised value $\mu_0$?

Effect size: $d = (\mu_1 - \mu_0) / \sigma$

Key inputs:

• $\mu_0$: Hypothesised value under $H_0$
• $\mu_1$: Expected true mean under $H_1$
• $\sigma$: Population or estimated SD

Power formula: Based on non-central t-distribution with $df = n - 1$ and non-centrality parameter $\lambda = d\sqrt{n}$.

5.2 Two-Sample Independent t-Test

Research question: Do two independent group means differ?

Effect size: $d = (\mu_1 - \mu_2) / \sigma_{pooled}$

Key inputs:

• $\mu_1$, $\mu_2$: Expected means for each group
• $\sigma_{pooled}$: Pooled within-group SD
• $k$: Allocation ratio $n_2 / n_1$ (default $k = 1$, equal groups)

Assumptions for power analysis:

• Equal variances (if Welch correction is planned, add $\approx 10\%$ to $N$)
• Normally distributed outcomes within groups

5.3 Paired t-Test

Research question: Does the mean of within-subject or matched-pair differences differ from zero?

Effect size: $d_z = \mu_D / \sigma_D$

Key additional input:

• $\rho$: Expected correlation between paired measurements (used to derive $\sigma_D$ from $\sigma_1$ and $\sigma_2$)

Efficiency gain over independent t-test:

$\text{Required pairs} = \frac{\text{Required per group (independent)}}{2} \times (1 - \rho)$

A within-subjects design with $\rho = 0.50$ requires approximately half the total participants of an independent-groups design for the same power.

5.4 One-Way ANOVA (Fixed Effects)

Research question: Do $k$ group means differ?

Effect size: $f = \sigma_m / \sigma_{within}$

Key inputs:

• $k$: Number of groups
• $\mu_1, \ldots, \mu_k$: Expected group means (or $\sigma_m$, the SD of means)
• $\sigma_{within}$: Common within-group SD
• $n_{per\;group}$: Equal or specified group sizes

Important: ANOVA power analysis requires specifying the pattern of means (which groups differ by how much), not just the overall effect size. The same $f$ can arise from very different mean patterns.

5.5 Factorial ANOVA

Research question: Do main effects and/or interactions exist in a factorial design?

Each effect (main effect A, main effect B, interaction A×B) has its own effect size $f$ and its own power analysis. Key additional considerations:

• Power for interaction effects is typically much lower than for main effects of the same nominal magnitude.
• Interaction effect sizes should be estimated directly (not derived from main effects).
• For a $2 \times 2$ factorial design, the interaction effect size for a crossover interaction is often set to half the main effect size as a conservative estimate.

5.6 Repeated Measures ANOVA

Research question: Does a measured variable change across $k$ time points or conditions within subjects?

Effect size: Cohen's $f$ based on within-person variance.

Key additional input:

• $\rho$: Average correlation among repeated measures (intraclass correlation). Higher $\rho$ → greater power advantage of repeated measures over independent groups.

Non-centrality parameter adjustment for repeated measures:

$\lambda_{RM} = \frac{f^2 \times n \times k}{1 - \rho}$

5.7 Pearson Correlation

Research question: Is there a linear relationship between two continuous variables?

Effect size: $\rho$ (population correlation coefficient)

Sample size:

$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{z_\rho}\right)^2 + 3$

Where $z_\rho = \text{arctanh}(\rho) = \frac{1}{2}\ln\frac{1+\rho}{1-\rho}$.

Note: Power for correlation tests is low for small correlations. Detecting $\rho = 0.30$ with 80% power at $\alpha = .05$ requires $n \approx 84$.

5.8 Multiple Regression

Research question: Does a set of $p$ predictors explain variance in the outcome? Or does adding $u$ predictors significantly improve prediction?

Effect size: $f^2 = R^2 / (1 - R^2)$ (overall model) or $f^2 = \Delta R^2 / (1 - R^2_{full})$ (incremental)

Key inputs:

• $u$: Number of predictors being tested
• $p$: Total predictors in the full model
• $R^2$ or $\Delta R^2$: Expected variance explained

Important: Power in multiple regression depends on the number of predictors being tested ($u$), not on the total model. Testing a single predictor in a model with many covariates uses $u = 1$; testing the full model uses $u = p$.

5.9 Chi-Square Test of Association

Research question: Are two categorical variables associated?

Effect size: Cohen's $w$ (related to Cramér's $V$: $w = V\sqrt{\min(r-1, c-1)}$)

Key inputs:

• Table dimensions: $r$ rows, $c$ columns
• $df = (r-1)(c-1)$
• Expected cell proportions under $H_1$

Important: For $2 \times 2$ tables, $w = \phi$ and the formula simplifies to the two-proportion case. For larger tables, specifying the full expected cell proportion matrix provides the most accurate power estimate.

5.10 Test Type Comparison Table

6. Using the Sample Size and Power Analysis Calculator Component

The Sample Size and Power Analysis Calculator in DataStatPro provides a comprehensive tool for conducting, visualising, and reporting power analyses for all common statistical tests.

Step-by-Step Guide

Step 1 — Navigate to the Component

Go to Study Design → Sample Size and Power Analysis.

Step 2 — Select the Analysis Mode

Choose one of the four analysis modes:

• A priori: Compute required sample size.
• Post-hoc: Compute achieved power.
• Sensitivity: Compute minimum detectable effect size.
• Criterion: Compute required $\alpha$ (advanced use).

Step 3 — Select the Statistical Test

Choose the test family and specific test from the hierarchical menu:

• Mean Tests
  • One-Sample t-Test
  • Two-Sample Independent t-Test
  • Paired t-Test
  • One-Way ANOVA
  • Factorial ANOVA (Two-Way, Three-Way)
  • Repeated Measures ANOVA
• Association Tests
  • Pearson Correlation
  • Spearman Correlation
  • Multiple Regression (Linear)
• Categorical Tests
  • Chi-Square Test of Association
  • Chi-Square Goodness-of-Fit
  • One-Sample Proportion Test
  • Two-Sample Proportion Test
• Survival Analysis
  • Log-Rank Test
  • Cox Proportional Hazards
• Advanced
  • Equivalence Test (TOST)
  • Non-Inferiority Test
  • Clustered Design (CRT)
  • Generic Non-Central Distribution

Step 4 — Specify Effect Size

Choose your effect size specification method:

• Direct entry: Enter the effect size measure directly (e.g., $d = 0.50$).
• From parameters: Enter the raw parameters (e.g., $\mu_1 = 100$, $\mu_2 = 95$, $\sigma = 15$) and DataStatPro computes the effect size automatically.
• From proportions: Enter $\pi_1$ and $\pi_2$ for proportion tests; DataStatPro computes Cohen's $h$ automatically.
• From expected table: Enter the full expected cell proportion matrix for chi-square tests.
• Effect size calculator: Use DataStatPro's built-in effect size converter to transform between $d$, $r$, $f$, $\eta^2$, $\omega^2$, OR, and RR.

Step 5 — Specify Remaining Parameters

Depending on the analysis mode, enter the known quantities:

• Significance level ($\alpha$): Default $.05$; specify $.01$ or $.001$ if needed.
• Desired power ($1 - \beta$): Default $.80$; options $.80$, $.85$, $.90$, $.95$, $.99$, or custom.
• Directionality: Two-tailed (default) or one-tailed.
• Number of groups / predictors: As applicable to the selected test.
• Allocation ratio $k = n_2/n_1$: For two-group tests (default $k = 1$).
• ICC and cluster size: For clustered designs.
• Attrition rate: For enrollment adjustment.

Step 6 — Set Display Options

• ✅ Primary result: Required $N$ (or power, or MDE) with exact formula.
• ✅ Enrollment $N$ (attrition-adjusted).
• ✅ Per-group $n$ breakdown.
• ✅ Power curve: Power vs. $N$ for current effect size and $\alpha$.
• ✅ Sensitivity curve: Power vs. effect size for current $N$ and $\alpha$.
• ✅ Power contour plot: Power as a function of both $N$ and effect size.
• ✅ Non-centrality parameter $\lambda$ and critical value.
• ✅ Type I error rate ($\alpha$), Type II error rate ($\beta$), power ($1-\beta$).
• ✅ Summary table: Power at $N \pm 10\%$, $N \pm 25\%$, $N \pm 50\%$.
• ✅ Design effect and ICC-adjusted $N$ (for clustered designs).
• ✅ APA 7th edition power analysis paragraph (auto-generated).

Step 7 — Run the Analysis

Click "Compute Sample Size / Power". DataStatPro will:

1. Convert effect size inputs to the required format (apply transformations if needed).
2. Solve for the requested output using exact non-central distribution methods.
3. Apply attrition, ICC, and multiple testing corrections if specified.
4. Generate power curve, sensitivity curve, and contour plot.
5. Produce the APA-compliant power analysis reporting paragraph.

7. Step-by-Step Procedure

7.1 Full Manual Procedure for A Priori Sample Size Calculation

Step 1 — Identify the Primary Research Question

State the primary outcome, the comparison of interest, and the direction of the hypothesised effect. Confirm the appropriate statistical test.

Step 2 — Choose the Significance Level and Directionality

State $\alpha$ and justify the choice. Specify whether the test is one-tailed or two-tailed, with justification.

Identify $z_{\alpha/2}$ (two-tailed) or $z_\alpha$ (one-tailed) from the standard normal distribution:

Step 3 — Choose the Power Target

State the desired power $1 - \beta$ and justify the choice. Identify $z_{1-\beta}$:

Step 4 — Specify and Justify the Effect Size

State the effect size measure, its numerical value, and the source or rationale. Convert raw parameters to a standardised effect size using the appropriate formula (Section 3).

Step 5 — Apply the Sample Size Formula

Substitute the values of $z_{\alpha/2}$, $z_{1-\beta}$, and the effect size into the appropriate formula from Section 3.10. Round up to the nearest whole number.

⚠️ Always round the required $N$ UP, never down. Rounding down results in a study with slightly less power than targeted.

Step 6 — Adjust for Unequal Groups (If Applicable)

For two-group designs with unequal allocation (ratio $k \neq 1$):

$n_1 = \left\lceil \frac{(z_{\alpha/2} + z_{1-\beta})^2(1 + 1/k)}{d^2} \right\rceil, \qquad n_2 = \lceil k \times n_1 \rceil$

Verify that the total $N = n_1 + n_2$ achieves the target power with exact non-central distribution methods.

Step 7 — Adjust for Attrition

$N_{enroll} = \left\lceil \frac{N_{analysis}}{1 - r_{attrition}} \right\rceil$

Step 8 — Adjust for Clustering (If Applicable)

$DEFF = 1 + (m - 1) \times ICC, \qquad N_{cluster} = \lceil N_{simple} \times DEFF \rceil$

Number of clusters: $n_{clusters} = \lceil N_{cluster} / m \rceil$

Step 9 — Adjust for Multiple Testing (If Applicable)

Replace $\alpha$ with $\alpha' = \alpha/k$ in the sample size formula (Bonferroni), where $k$ is the number of primary hypotheses.

Step 10 — Verify with Power Curve

Using the computed $N$, confirm the achieved power with exact non-central distribution calculations. Plot the power curve to show power at values of $N$ above and below the target. Confirm the achieved power is at or above the target.

Step 11 — Conduct Sensitivity Analysis

Report the minimum detectable effect size (MDE) at the computed $N$:

$d_{MDE} = \frac{z_{\alpha/2} + z_{1-\beta}}{\sqrt{n/2}}$ (two-sample t, per group)

This tells stakeholders the smallest effect the study is designed to detect.

Step 12 — Document and Report

Compile all inputs and outputs into a complete power analysis report (APA format provided in Section 15). Retain all working for audit and reproducibility.

8. Interpreting the Output

8.1 Reading the Required Sample Size

8.2 Understanding the Power Curve

8.3 Sensitivity Output: Minimum Detectable Effect

The minimum detectable effect (MDE) is the smallest effect the study has the specified power to detect:

Report the MDE in original units (not just in standardised form) to make it interpretable to domain experts who may not be familiar with Cohen's $d$.

8.4 The Non-Centrality Parameter ($\lambda$)

The non-centrality parameter $\lambda$ summarises the total signal in the study — it captures both the effect size and the sample size:

$\lambda = ES^2 \times N \quad \text{(approximately, for many tests)}$

8.5 Interpreting Achieved Post-Hoc Power

Post-hoc power (computed after data collection using the observed effect size) has a deterministic relationship with the p-value:

Because of this mathematical relationship, post-hoc power using the observed effect size adds no information beyond the p-value. Instead, report:

• The 95% CI for the effect size.
• A sensitivity analysis: "What is the power for a range of plausible true effect sizes?"

8.6 Interpreting the Contour Plot

The power contour plot displays power as a function of both $N$ and effect size simultaneously, with contour lines at specific power levels (e.g., 0.60, 0.70, 0.80, 0.90, 0.95):

9. Visualising Power and Sample Size

9.1 Power Curve (Power vs. Sample Size)

The power curve is the primary visualisation for a priori power analysis. It plots the statistical power ($y$-axis) as a function of sample size ($x$-axis) for fixed $\alpha$ and effect size.

Key annotations on the DataStatPro power curve:

• A horizontal dashed line at the target power level (e.g., 0.80).
• A vertical dashed line at the required $N$.
• The intersection point highlighted and labelled with exact $N$ and power.
• Shaded region below the target power: underpowered zone.
• Optional: Multiple curves for different effect sizes or $\alpha$ levels.

Best practices:

• Set the x-axis range to display at least $[1, 3 \times N_{required}]$ to show the full shape of the curve.
• Annotate the MDE at the required $N$.
• Use a logarithmic x-axis when the required $N$ is very large to avoid compression of the curve at small $N$.

9.2 Sensitivity Curve (Power vs. Effect Size)

The sensitivity curve plots power ($y$-axis) as a function of effect size ($x$-axis) for a fixed $N$ and $\alpha$.

Use cases:

• Assessing robustness: "What happens to power if the true effect is smaller than anticipated?"
• Reporting minimum detectable effect: The effect size at which the curve crosses the target power line.
• Communicating uncertainty about the effect size to stakeholders.

Best practices:

• Mark Cohen's small/medium/large benchmarks as vertical reference lines.
• Annotate the MDE (the effect size where the curve intersects the target power line).
• Shade the "adequately powered" region (to the right of the MDE).

9.3 Power Contour Plot (Power as a Function of $N$ and Effect Size)

The contour plot provides the most comprehensive two-dimensional view of how power depends on both sample size and effect size. Contour lines connect combinations of $(N, ES)$ that yield equal power.

Reading the contour plot:

• The study's planned $(N, ES)$ combination is marked.
• Power target contour (e.g., 0.80) divides the plot into adequate and inadequate power regions.
• Researchers can identify the trade-off: increasing the effect size estimate by a given amount allows reducing $N$ by a corresponding amount while maintaining power.

9.4 Error Rate Trade-Off Plot

The error rate trade-off plot visualises the relationship between $\alpha$ (Type I error rate) and $\beta$ (Type II error rate = $1 -$ power) for a fixed $N$ and effect size:

• As $\alpha$ decreases (stricter threshold), $\beta$ increases (lower power).
• The optimal trade-off depends on the relative costs of Type I and Type II errors in the specific application.

Useful for:

• Choosing between $\alpha = .05$ and $\alpha = .01$ given limited sample size.
• Demonstrating to reviewers the implications of changing the significance threshold.

9.5 G*Power-Style Distribution Plot

DataStatPro generates the classic two-distribution diagram showing:

• The central distribution of the test statistic under $H_0$ (blue curve).
• The non-central distribution of the test statistic under $H_1$ (orange curve).
• The critical value ($t_{crit}$) as a vertical dashed line.
• The $\alpha$ region (critical region under $H_0$, right tail of blue curve).
• The power ($1 - \beta$) region (area under the orange curve beyond $t_{crit}$).
• The $\beta$ region (area under the orange curve to the left of $t_{crit}$).

This plot is highly effective for teaching the concept of power and for communicating results to non-statistician audiences.

9.6 Sample Size Comparison Table Plot

For multiple scenarios (e.g., small/medium/large effect; power = 0.80/0.90/0.95), DataStatPro generates a bubble chart or heatmap where:

• Rows represent power targets.
• Columns represent effect sizes.
• Cell values (or bubble sizes) represent the required $N$.

This provides a rapid overview of how sample size requirements vary across the range of plausible inputs, supporting scenario planning.

9.7 Attrition-Adjusted Recruitment Funnel

For longitudinal studies or clinical trials, DataStatPro generates a funnel diagram showing:

• Enrollment target (accounting for attrition).
• Expected completers at each wave.
• Final analytic sample.
• Required sample vs. expected completers — highlighting any shortfall.

10. Sensitivity Analysis and Robustness Checks

10.1 What Is a Sensitivity Analysis in Power Planning?

A sensitivity analysis for power examines how the required sample size (or achieved power) changes as input parameters vary within plausible ranges. It answers: "How robust is my power calculation to uncertainty in the assumed effect size, standard deviation, or other inputs?"

10.2 Varying the Effect Size

The most important sensitivity analysis varies the effect size across a range defined by:

• The SESOI (lower bound — the smallest effect that matters).
• The expected effect from prior literature (central estimate).
• A larger, optimistic effect (upper bound).

Report $N_{required}$ for each scenario:

Decision rule: Plan for the scenario that produces the largest $N$ to ensure adequate power across all plausible effect sizes.

10.3 Varying the Standard Deviation

For mean-based tests, the effect size $d = \Delta\mu / \sigma$ depends on $\sigma$. If $\sigma$ is estimated from a pilot study or literature with uncertainty, sensitivity analysis should vary $\sigma$ across a plausible range (e.g., $\sigma \pm 20\%$):

$d_{lower} = \frac{\Delta\mu}{\sigma_{upper}}, \qquad d_{upper} = \frac{\Delta\mu}{\sigma_{lower}}$

Report $N_{required}$ for $d_{lower}$ and $d_{upper}$ as the worst and best cases.

10.4 The "What If" Power Table

A comprehensive "What If" power table reports power for a grid of $N$ values and effect sizes, enabling researchers and reviewers to assess robustness:

(Two-sample independent t-test, $\alpha = .05$, two-tailed)

10.5 Bayesian Power Analysis

Classical power analysis assumes a fixed, known effect size. Bayesian power analysis incorporates uncertainty about the effect size by averaging power over a prior distribution of effect sizes:

$\overline{Power} = \int_0^\infty (1 - \beta(d)) \times p(d) \; dd$