Sample Size and Power Analysis: Comprehensive Reference Guide

This comprehensive guide covers power analysis fundamentals, sample size calculations for various study designs, effect size determination, and practical considerations for planning statistical studies with detailed mathematical formulations and interpretation guidelines.

Overview

Sample size and power analysis are crucial components of study design that determine the ability to detect meaningful effects and ensure adequate statistical power. Proper planning prevents underpowered studies and resource waste while maintaining scientific rigor.

Power Analysis Fundamentals

1. Statistical Errors

Type I Error (α):

• Probability of rejecting true null hypothesis
• False positive rate
• Typically set at 0.05 (5%)

Type II Error (β):

• Probability of failing to reject false null hypothesis
• False negative rate
• Typically set at 0.10 or 0.20

Statistical Power (1-β):

• Probability of correctly rejecting false null hypothesis
• Ability to detect true effect
• Typically desired at 0.80 (80%) or 0.90 (90%)

2. Effect Size

Definition: Standardized measure of the magnitude of difference or association.

Cohen's Conventions:

• Small effect: d = 0.2, r = 0.1, f = 0.1
• Medium effect: d = 0.5, r = 0.3, f = 0.25
• Large effect: d = 0.8, r = 0.5, f = 0.4

Cohen's d (standardized mean difference): $d = \frac{\mu_1 - \mu_2}{\sigma}$

Correlation coefficient (r): $r = \frac{\text{covariance}}{\sigma_x \sigma_y}$

Cohen's f (ANOVA effect size): $f = \sqrt{\frac{\eta^2}{1-\eta^2}}$

3. Factors Affecting Power

1. Effect size: Larger effects easier to detect
2. Sample size: Larger samples increase power
3. Significance level (α): Lower α decreases power
4. Variability: Lower variability increases power
5. Study design: More efficient designs increase power

Sample Size Calculations for Different Study Designs

1. One-Sample Tests

One-Sample t-test (mean): $n = \frac{(z_{\alpha/2} + z_\beta)^2 \sigma^2}{(\mu_1 - \mu_0)^2}$

One-Sample z-test (proportion): $n = \frac{(z_{\alpha/2} + z_\beta)^2 p_0(1-p_0)}{(p_1 - p_0)^2}$

Where:

• $\mu_0$ = null hypothesis mean
• $\mu_1$ = alternative hypothesis mean
• $p_0$ = null hypothesis proportion
• $p_1$ = alternative hypothesis proportion

2. Two-Sample Tests

Independent samples t-test (equal variances): $n = \frac{2(z_{\alpha/2} + z_\beta)^2 \sigma^2}{(\mu_1 - \mu_2)^2}$

Independent samples t-test (unequal variances): $n_1 = \frac{(z_{\alpha/2} + z_\beta)^2(\sigma_1^2 + \sigma_2^2/k)}{(\mu_1 - \mu_2)^2}$

Where k = $n_2/n_1$ (allocation ratio)

Two-sample z-test (proportions): $n = \frac{(z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_\beta\sqrt{p_1(1-p_1) + p_2(1-p_2)})^2}{(p_1 - p_2)^2}$

Where $\bar{p} = (p_1 + p_2)/2$

Paired t-test: $n = \frac{(z_{\alpha/2} + z_\beta)^2 \sigma_d^2}{\mu_d^2}$

Where:

• $\sigma_d$ = standard deviation of differences
• $\mu_d$ = mean difference

3. ANOVA Designs

One-way ANOVA: $n = \frac{(F_{\alpha,k-1,\infty} + F_{\beta,k-1,\infty})^2}{f^2}$

Simplified formula: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{f^2} + 1$

Two-way ANOVA: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{f^2 \times df_{effect}} + c$

Where:

• f = Cohen's f effect size
• c = correction factor based on design
• $df_{effect}$ = degrees of freedom for effect

Repeated Measures ANOVA: $n = \frac{(z_{\alpha/2} + z_\beta)^2(1 + (k-1)\rho)}{k \times f^2}$

Where:

• k = number of repeated measures
• ρ = correlation between repeated measures

4. Factorial Designs

2×2 Factorial Design: $n = \frac{4(z_{\alpha/2} + z_\beta)^2 \sigma^2}{(\text{main effect})^2}$

For interaction effect: $n = \frac{4(z_{\alpha/2} + z_\beta)^2 \sigma^2}{(\text{interaction effect})^2}$

General factorial design: $n = \frac{(z_{\alpha/2} + z_\beta)^2 \sigma^2 \times \text{design factor}}{(\text{effect size})^2}$

Correlation and Regression Studies

1. Correlation Analysis

Sample size for correlation: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{(\frac{1}{2}\ln(\frac{1+r}{1-r}))^2} + 3$

Fisher's z-transformation: $z_r = \frac{1}{2}\ln\left(\frac{1+r}{1-r}\right)$

Power for given sample size: $Power = \Phi\left(\frac{|z_r|\sqrt{n-3} - z_{\alpha/2}}{1}\right)$

2. Linear Regression

Simple linear regression: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{f^2} + u + 1$

Where:

• u = number of predictors
• $f^2 = \frac{R^2}{1-R^2}$ (effect size)

Multiple regression: $n = \frac{(z_{\alpha/2} + z_\beta)^2(1-R^2)}{R^2} + u + 1$

Logistic regression: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{p(1-p)(\ln(OR))^2}$

Where:

• p = proportion of events
• OR = odds ratio

Non-Parametric Test Sample Sizes

1. Mann-Whitney U Test

Asymptotic relative efficiency (ARE) = 0.955: $n_{nonparametric} = \frac{n_{parametric}}{0.955}$

Direct formula: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{12(\Phi^{-1}(P(X > Y)) - 0.5)^2}$

2. Wilcoxon Signed-Rank Test

ARE = 0.955 relative to paired t-test: $n_{Wilcoxon} = \frac{n_{t-test}}{0.955}$

3. Kruskal-Wallis Test

ARE = 0.864 relative to one-way ANOVA: $n_{KW} = \frac{n_{ANOVA}}{0.864}$

Survival Analysis Sample Size Calculations

1. Log-Rank Test

Formula: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{p_1 p_2 (\ln(HR))^2}$

Where:

• $p_1, p_2$ = proportions in each group
• HR = hazard ratio

With censoring: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{p_1 p_2 (\ln(HR))^2 \times P_{event}}$

Where $P_{event}$ = probability of observing event

2. Cox Proportional Hazards

Number of events needed: $E = \frac{(z_{\alpha/2} + z_\beta)^2}{(\ln(HR))^2}$

Total sample size: $n = \frac{E}{P_{event}}$

3. Exponential Survival

Equal allocation: $n = \frac{2(z_{\alpha/2} + z_\beta)^2(\lambda_1 + \lambda_2)^2}{(\lambda_1 - \lambda_2)^2 T}$

Where:

• λ = hazard rates
• T = study duration

Cluster Randomized Trials and Multilevel Studies

1. Cluster Randomized Trials

Design effect: $DE = 1 + (m-1)\rho$

Where:

• m = average cluster size
• ρ = intracluster correlation coefficient

Adjusted sample size: $n_{cluster} = n_{individual} \times DE$

Number of clusters: $c = \frac{n_{cluster}}{m}$

2. Multilevel Models

Two-level design: $n_{level2} = \frac{(z_{\alpha/2} + z_\beta)^2 \sigma^2_{total}}{(\text{effect size})^2 \times (1-\rho)}$

Three-level design: $n = \frac{(z_{\alpha/2} + z_\beta)^2}{(\text{effect size})^2} \times \text{variance inflation factor}$

3. Stepped Wedge Designs

Sample size adjustment: $n_{SW} = n_{parallel} \times \frac{3(1-\rho)}{2T\rho}$

Where:

• T = number of time periods
• ρ = intracluster correlation

Post-Hoc Power Analysis Considerations

1. Observed Power

Problems with observed power:

• Circular reasoning when non-significant
• Misleading interpretation
• Not useful for study interpretation

Formula: $\text{Observed Power} = \Phi\left(\frac{|t_{observed}| - t_{\alpha/2}}{\sqrt{1}}\right)$

2. Confidence Intervals

Preferred approach:

• Report confidence intervals instead of post-hoc power
• Provides information about precision
• Indicates practical significance

Relationship to power: $CI = \text{estimate} \pm t_{\alpha/2} \times SE$

3. Effect Size Estimation

Retrospective effect size: $d = \frac{\bar{x}_1 - \bar{x}_2}{s_{pooled}}$

Confidence interval for effect size: $CI_d = d \pm t_{\alpha/2} \times SE_d$

Software Recommendations and Practical Guidelines

1. Software Options

Specialized Software:

• G*Power (free, comprehensive)
• PASS (commercial, extensive)
• nQuery (commercial, clinical trials)
• SAS/PROC POWER
• R packages (pwr, PowerTOST)

General Statistical Software:

• SPSS (limited power analysis)
• Stata (sampsi, power commands)
• SAS (PROC POWER)
• R (multiple packages)

2. Practical Considerations

Planning Phase:

1. Define primary endpoint clearly
2. Specify effect size of interest
3. Consider feasibility constraints
4. Plan for attrition/dropout
5. Consider multiple comparisons

Effect Size Determination:

• Literature review
• Pilot studies
• Clinical significance
• Regulatory guidelines
• Expert opinion

Sample Size Inflation:

• Dropout rate: multiply by 1/(1-dropout rate)
• Non-compliance: adjust for dilution effect
• Multiple comparisons: Bonferroni or other corrections

3. Reporting Guidelines

Essential Elements:

• Primary hypothesis and endpoint
• Effect size and justification
• Power and significance level
• Sample size calculation method
• Assumptions made
• Software used

Example: "Sample size was calculated for a two-sided t-test comparing mean scores between groups. Assuming a medium effect size (Cohen's d = 0.5), α = 0.05, and power = 0.80, a total sample size of 128 participants (64 per group) was required. Accounting for 20% attrition, we aimed to recruit 160 participants."

4. Common Mistakes

Avoid These Errors:

• Using post-hoc power analysis for interpretation
• Ignoring multiple comparisons
• Unrealistic effect size assumptions
• Inadequate consideration of dropout
• Confusing statistical and clinical significance

This comprehensive guide provides the foundation for understanding and conducting proper sample size and power analysis for various study designs and statistical tests.