Experimental Design Principles: Zero to Hero Tutorial

This tutorial takes you from the core logic of experimental design to practical planning, randomization, power, analysis, reporting, and quality checks in DataStatPro. It is written for researchers who need more than a checklist: you will learn why each design decision matters, how it affects causal interpretation, and how to build experiments that can survive peer review, replication, and real-world constraints.

Table of Contents

1. Prerequisites and Background Concepts
2. What Is an Experiment?
3. The Causal Logic of Experiments
4. Core Principles of Experimental Design
5. Types of Experimental Designs
6. Planning an Experiment Step by Step
7. Randomization, Blocking, and Blinding
8. Sample Size and Power Planning
9. Choosing the Statistical Analysis
10. Using DataStatPro in the Design Workflow
11. Worked Examples
12. Common Mistakes and How to Avoid Them
13. Troubleshooting
14. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before designing an experiment, you should be comfortable with:

• Outcome variable: The response you measure, such as test score, blood pressure, yield, conversion rate, or reaction time.
• Treatment or intervention: The condition you deliberately manipulate.
• Factor: An independent variable in an experiment.
• Level: A specific value of a factor, such as low dose, medium dose, and high dose.
• Experimental unit: The smallest unit that can be independently assigned to a treatment.
• Confounder: A variable related to both treatment assignment and outcome.
• Power: The probability of detecting a true effect, usually written as $1-\beta$.
• Type I error: The probability of a false positive, usually written as $\alpha$.

A useful starting question is:

What is the smallest unit that will actually receive the intervention independently?

If students are randomized by classroom, the classroom is the experimental unit for the intervention even if outcomes are measured for individual students. This distinction affects sample size, analysis, and interpretation.

2. What Is an Experiment?

An experiment is a study in which the researcher deliberately assigns units to one or more conditions and then measures outcomes. The central goal is causal inference:

Would the same unit have had a different outcome under a different treatment?

Because each unit can only be observed under one treatment condition at a time, experiments use design principles such as randomization, control groups, replication, and blinding to make this unobserved counterfactual comparison credible.

2.1 Experiment vs. Observational Study

2.2 What Experiments Are Best For

Experiments are strongest when the research question asks whether an intervention causes a change:

• Does a new teaching method improve exam scores?
• Does a fertilizer increase crop yield?
• Does a reminder message increase vaccination uptake?
• Does a pricing change affect purchase intent?
• Does a training program improve job performance?

3. The Causal Logic of Experiments

For unit $i$, define:

• $Y_i(1)$ = outcome if the unit receives treatment.
• $Y_i(0)$ = outcome if the unit receives control.

The individual causal effect is:

$\tau_i = Y_i(1) - Y_i(0)$

The average treatment effect is:

$ATE = E[Y(1) - Y(0)]$

In a randomized experiment, treatment assignment $T$ is independent of the potential outcomes:

$T \perp \{Y(1), Y(0)\}$

That independence is the reason the difference in group means can estimate a causal effect:

$\hat{\tau} = \bar{Y}_T - \bar{Y}_C$

Randomization does not guarantee that every covariate is perfectly balanced in a specific sample, but it makes systematic bias unlikely and allows statistical uncertainty to be quantified.

4. Core Principles of Experimental Design

4.1 Control

Control means reducing alternative explanations. It can be achieved through:

• A comparison group.
• Standardized procedures.
• Eligibility criteria.
• Baseline measurements.
• Holding important conditions constant.

4.2 Randomization

Randomization assigns units to treatment conditions by chance. It protects against both known and unknown confounders.

4.3 Replication

Replication means having enough independent units to estimate variability. Repeating a treatment on many units allows the treatment effect to be separated from random noise.

4.4 Blocking

Blocking groups similar units before randomization. For example, randomize within school, clinic, farm plot, or baseline severity group.

Blocking can improve precision when the blocking variable is strongly related to the outcome.

4.5 Blinding

Blinding prevents expectations from influencing treatment delivery, participant behaviour, measurement, or analysis.

5. Types of Experimental Designs

5.1 Completely Randomized Design

Units are randomly assigned to groups without restrictions.

Use when units are relatively homogeneous and sample size is adequate.

5.2 Randomized Block Design

Units are grouped into blocks, then randomized within each block.

Example: Randomize students separately within each school.

5.3 Factorial Design

Two or more factors are manipulated simultaneously. A $2 \times 2$ design has two factors, each with two levels.

Factorial designs estimate main effects and interactions:

$Y_{ijk} = \mu + \alpha_j + \beta_k + (\alpha\beta)_{jk} + \varepsilon_{ijk}$

5.4 Repeated-Measures Design

The same units are measured under multiple conditions or time points.

Use when individual differences are large and carryover can be controlled.

5.5 Crossover Design

Each participant receives multiple treatments in different sequences, separated by a washout period.

Crossover designs are powerful but unsuitable when treatments have permanent effects.

5.6 Cluster Randomized Design

Groups rather than individuals are randomized.

Examples include schools, hospitals, villages, clinics, farms, and work teams.

The design must account for intraclass correlation:

$DE = 1 + (m - 1)\rho$

where $DE$ is the design effect, $m$ is average cluster size, and $\rho$ is the intraclass correlation coefficient.

6. Planning an Experiment Step by Step

Step 1: Define the Research Question

A strong experimental question names:

• Population.
• Intervention.
• Comparator.
• Outcome.
• Time frame.

Example:

Among first-year university students, does interactive teaching compared with standard lecture improve final exam score after one semester?

Step 2: Specify Hypotheses

For a two-group mean comparison:

$H_0: \mu_T - \mu_C = 0$

$H_1: \mu_T - \mu_C \neq 0$

For a directional claim:

$H_1: \mu_T - \mu_C > 0$

Use directional hypotheses only when justified before data collection.

Step 3: Define the Experimental Unit

Ask:

• Who or what receives the treatment?
• Can two observations influence each other?
• Is assignment individual-level or group-level?

Misidentifying the experimental unit is a common cause of false precision.

Step 4: Choose the Design

Step 5: Define Outcomes

Choose one primary outcome and a small number of secondary outcomes.

For each outcome specify:

• Measurement instrument.
• Measurement time point.
• Scoring rule.
• Direction of improvement.
• Clinically or practically meaningful difference.

Step 6: Plan the Analysis Before Data Collection

Pre-specify:

• Primary analysis model.
• Treatment contrast.
• Covariates.
• Missing data strategy.
• Outlier rules.
• Subgroup or interaction analyses.

7. Randomization, Blocking, and Blinding

7.1 Simple Randomization

Each unit has a fixed probability of assignment to each group.

Use for large studies where accidental imbalance is unlikely.

7.2 Block Randomization

Block randomization keeps group sizes balanced over time.

For block size 4 with two groups, possible blocks include:

7.3 Stratified Randomization

Stratify when a baseline factor is both important and likely to be imbalanced.

Examples:

• Study site.
• Disease severity.
• Grade level.
• Sex.
• Baseline score category.

7.4 Allocation Concealment

Allocation concealment means the next assignment cannot be predicted before enrollment. It is different from blinding and is essential for preventing selection bias.

8. Sample Size and Power Planning

Experimental design and power planning are inseparable. A beautifully randomized study can still fail if it is too small to detect the planned effect.

8.1 Two-Group Continuous Outcome

For equal group sizes, a useful approximation is:

$n_{\text{per group}} \approx \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2\sigma^2}{\Delta^2}$

where:

• $\Delta$ is the minimum meaningful mean difference.
• $\sigma$ is the expected within-group standard deviation.
• $\alpha$ is the Type I error rate.
• $1-\beta$ is power.

8.2 Effect Size

Cohen's $d$ is:

$d = \frac{\mu_T - \mu_C}{\sigma}$

Use prior literature, pilot data, or a smallest effect size of interest. Do not choose an effect size only because it produces a convenient sample size.

8.3 Attrition Adjustment

If expected attrition is $r$, inflate sample size:

$n_{\text{recruited}} = \frac{n_{\text{required}}}{1-r}$

For 15% attrition and $n=128$ required:

$n_{\text{recruited}} = \frac{128}{0.85} = 151$

9. Choosing the Statistical Analysis

Analysis should match the design, not just the shape of the final dataset.

10. Using DataStatPro in the Design Workflow

Use DataStatPro at three stages:

10.1 Before Data Collection

• Use Sample Size and Power tools to estimate required sample size.
• Use effect-size tutorials to define the smallest meaningful effect.
• Use assumption tutorials to understand analysis requirements.

10.2 During Data Preparation

• Use Data Management tools to screen for missingness, impossible values, and coding errors.
• Verify treatment labels, block identifiers, and time-point variables.

10.3 During Analysis

• Use the analysis module that matches the design.
• Report effect sizes and confidence intervals, not only p-values.
• Export publication-ready tables and plots for transparent reporting.

11. Worked Examples

Example 1: Educational Intervention

Question: Does interactive teaching improve exam scores compared with lecture?

Primary model:

$Y_i = \beta_0 + \beta_1T_i + \beta_2X_i + \varepsilon_i$

where $T_i$ is treatment and $X_i$ is baseline score.

Example 2: Agriculture Factorial Experiment

Question: Do fertilizer type and irrigation level jointly affect yield?

This is a $3 \times 2$ factorial design. The key analysis is two-way ANOVA with interaction:

$Yield = Fertilizer + Irrigation + Fertilizer \times Irrigation + Error$

If the interaction is large, interpret simple effects rather than only main effects.

Example 3: Workplace Training Trial

Question: Does a training program improve productivity after 8 weeks?

Because employees work in teams and share managers, randomizing individuals may cause contamination. A cluster randomized design by team is more appropriate.

The sample size should be inflated by the design effect:

$DE = 1 + (m - 1)\rho$

12. Common Mistakes and How to Avoid Them

13. Troubleshooting

Groups Are Imbalanced After Randomization

Small random imbalances can occur by chance. Report baseline characteristics and use pre-specified covariate adjustment for important predictors.

Participants Did Not Follow Assigned Treatment

Plan both:

• Intention-to-treat analysis: Analyze by assigned group.
• Per-protocol or complier analysis: Sensitivity analysis for adherence.

Outcome Data Are Missing

Assess whether missingness differs by group. Use transparent missing-data rules and avoid complete-case analysis when missingness is substantial or related to outcomes.

The Interaction Is Significant

Do not interpret main effects alone. Examine simple effects, estimated marginal means, and interaction plots.

14. Quick Reference Cheat Sheet

Design Selection

Key Formulas

Reporting Checklist

• Research question and primary hypothesis.
• Experimental unit and randomization method.
• Treatment and comparator definitions.
• Primary and secondary outcomes.
• Sample size justification.
• Allocation concealment and blinding.
• Analysis model and effect size.
• Missing-data and noncompliance handling.

Next Steps

After designing the experiment, continue with:

• Sample Size and Power Analysis for planning recruitment.
• t-Tests and Alternatives for two-group mean comparisons.
• ANOVA Tests and Alternatives for multi-group designs.
• ANCOVA for covariate-adjusted experimental analysis.
• Effect Size Calculator for practical significance reporting.