Survey Design and Sampling Methods: Zero to Hero Tutorial

This tutorial takes you from the foundations of survey research to questionnaire design, sampling frames, sample size planning, bias control, weighting, analysis, and reporting in DataStatPro. It is built for researchers who want survey results that are readable, statistically defensible, and useful for real decisions.

Table of Contents

1. Prerequisites and Background Concepts
2. What Is Survey Research?
3. The Survey Design Workflow
4. Types of Survey Designs
5. Questionnaire Design
6. Response Scale Design
7. Sampling Foundations
8. Probability Sampling Methods
9. Non-Probability Sampling Methods
10. Sample Size Planning
11. Bias, Missingness, and Weighting
12. Using DataStatPro for Survey Analysis
13. Worked Examples
14. Common Mistakes and How to Avoid Them
15. Troubleshooting
16. Quick Reference Cheat Sheet

1. Prerequisites and Background Concepts

Before designing a survey, you should understand:

• Population: The full group you want to describe or compare.
• Sampling frame: The practical list or mechanism used to reach population members.
• Sample: The respondents or units actually observed.
• Sampling error: Random difference between sample estimate and population value.
• Coverage error: Error from missing parts of the population in the sampling frame.
• Nonresponse error: Bias when respondents differ from nonrespondents.
• Measurement error: Error caused by question wording, scales, recall, or response mode.
• Weighting: Adjusting respondent contributions to better match the target population.

The most important survey design question is:

Who exactly should this survey represent?

If the answer is vague, every later decision becomes fragile.

2. What Is Survey Research?

Survey research systematically collects standardized information from a sample of people, households, organizations, records, or other units.

Surveys are commonly used to:

• Estimate prevalence, satisfaction, knowledge, attitudes, or behaviours.
• Compare subgroups.
• Track change over time.
• Study relationships among variables.
• Support policy, product, clinical, educational, or operational decisions.

2.1 What Surveys Can and Cannot Do

Surveys do not become reliable simply because the sample is large. A large biased sample can be worse than a smaller well-designed probability sample.

3. The Survey Design Workflow

A strong survey moves through seven stages:

1. Define the target population.
2. Define the survey objectives and primary estimates.
3. Choose the survey mode.
4. Build or select the sampling frame.
5. Choose the sampling method.
6. Design, test, and revise the questionnaire.
7. Plan analysis, weighting, reporting, and quality checks.

3.1 Define Primary Estimates

Write primary estimates before drafting questions.

Examples:

• Proportion of students satisfied with advising.
• Mean patient experience score.
• Difference in vaccine hesitancy by age group.
• Association between job stress and turnover intention.

For a proportion:

$\hat{p} = \frac{x}{n}$

For a mean:

$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$

4. Types of Survey Designs

4.1 Cross-Sectional Survey

Data are collected at one point in time.

Use for:

• Prevalence estimates.
• Satisfaction snapshots.
• Population description.
• Initial exploratory relationships.

4.2 Repeated Cross-Sectional Survey

Different samples from the same population are surveyed at multiple time points.

Use for tracking population-level trends without following the same individuals.

4.3 Panel Survey

The same respondents are followed over time.

Use for studying individual-level change, but plan for attrition.

4.4 Cohort Survey

A defined cohort is followed, such as students entering university in a specific year.

Use when age, exposure period, or life stage matters.

4.5 Mixed-Mode Survey

Data are collected through more than one mode, such as online plus phone follow-up.

Mixed-mode designs can improve coverage but may introduce mode effects.

5. Questionnaire Design

5.1 Start With Constructs

Do not begin by writing questions. Begin by defining constructs.

5.2 Write Clear Questions

Good survey questions are:

• Specific.
• Short.
• Neutral.
• Answerable from the respondent's perspective.
• Linked to the analysis plan.

Poor:

Do you regularly exercise and eat healthy food?

Better:

In the past 7 days, on how many days did you exercise for at least 30 minutes?

5.3 Avoid Double-Barreled Questions

Double-barreled questions ask about two things at once.

Poor:

How satisfied are you with salary and benefits?

Better:

How satisfied are you with your salary?

How satisfied are you with your benefits?

5.4 Avoid Leading or Loaded Wording

Poor:

How much do you support this unfair policy?

Better:

What is your opinion of this policy?

5.5 Use Recall Windows

Ask about a defined period:

• "In the past 7 days..."
• "During the last semester..."
• "Since your most recent appointment..."

Recall windows reduce ambiguity and improve comparability.

6. Response Scale Design

6.1 Common Scale Types

6.2 Number of Scale Points

Use the same direction consistently. Switching between positive-to-negative and negative-to-positive scales causes avoidable response errors.

6.3 Label the Scale

Fully labelled scales reduce interpretation differences.

7. Sampling Foundations

7.1 Target Population vs. Sampling Frame

The target population is who you want to represent. The sampling frame is who can actually be sampled.

Coverage error occurs when the frame does not adequately cover the population.

7.2 Sampling Error

For a simple random sample proportion:

$SE(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

Approximate 95% margin of error:

$ME \approx 1.96 \times SE(\hat{p})$

For the conservative case $\hat{p}=0.50$:

$n \approx \frac{1.96^2 \times 0.50(1-0.50)}{ME^2}$

8. Probability Sampling Methods

Probability sampling means every population unit has a known, non-zero chance of selection.

8.1 Simple Random Sampling

Every unit has equal selection probability.

Use when:

• A complete frame exists.
• The population is not strongly clustered.
• Direct contact is feasible.

8.2 Systematic Sampling

Select every $k$th unit after a random start:

$k = \frac{N}{n}$

Avoid systematic sampling when the frame has a hidden periodic pattern.

8.3 Stratified Sampling

Divide the population into strata, then sample within each stratum.

Use when subgroup precision matters.

8.4 Cluster Sampling

Sample clusters first, then units within clusters.

Use when listing all individuals is difficult but clusters are available.

Cluster sampling often increases variance because people within clusters are similar.

8.5 Multistage Sampling

Sample in stages, such as district -> school -> classroom -> student.

This is common in national surveys, education studies, and household surveys.

9. Non-Probability Sampling Methods

Non-probability samples do not give every population member a known selection probability.

Non-probability sampling can be useful, but report limitations clearly.

10. Sample Size Planning

10.1 Estimating a Proportion

For desired margin of error $ME$:

$n = \frac{z_{1-\alpha/2}^2 p(1-p)}{ME^2}$

If $p$ is unknown, use $p=0.50$ for the largest required sample.

10.2 Estimating a Mean

$n = \left(\frac{z_{1-\alpha/2}\sigma}{ME}\right)^2$

where $\sigma$ is the expected population standard deviation.

10.3 Finite Population Correction

When sampling a meaningful fraction of a finite population:

$n_{adj} = \frac{n}{1 + (n-1)/N}$

10.4 Design Effect

For clustered or weighted surveys:

$n_{effective} = \frac{n}{DEFF}$

If $DEFF = 1.5$, a sample of 600 has an effective sample size of 400.

10.5 Nonresponse Inflation

If expected response rate is $RR$:

$n_{invited} = \frac{n_{completed}}{RR}$

For 500 completed surveys and 40% response:

$n_{invited} = \frac{500}{0.40} = 1250$

11. Bias, Missingness, and Weighting

11.1 Main Sources of Survey Error

11.2 Missing Data

Plan how to handle:

• Item nonresponse.
• Unit nonresponse.
• "Prefer not to answer."
• Skip logic missingness.

Do not automatically treat all blanks as zero.

11.3 Survey Weights

A basic weight is the inverse of selection probability:

$w_i = \frac{1}{\pi_i}$

Weights may also adjust for nonresponse and post-stratification.

Weighted mean:

$\bar{x}_w = \frac{\sum_i w_i x_i}{\sum_i w_i}$

Weighted proportion:

$\hat{p}_w = \frac{\sum_i w_i y_i}{\sum_i w_i}$

where $y_i=1$ for the category of interest and $0$ otherwise.

12. Using DataStatPro for Survey Analysis

Use DataStatPro after designing the survey to:

• Clean and validate categorical and numeric responses.
• Summarize frequencies, proportions, means, and standard deviations.
• Create confidence intervals for proportions and means.
• Compare groups with chi-square tests, t-tests, ANOVA, or non-parametric tests.
• Model outcomes with regression when adjustment is needed.
• Build publication-ready charts and tables.

Recommended workflow:

1. Import survey data.
2. Check variable coding and missing values.
3. Produce descriptive summaries.
4. Estimate primary outcomes with confidence intervals.
5. Compare planned subgroups.
6. Export tables and plots for reporting.

13. Worked Examples

Example 1: Student Satisfaction Survey

Objective: Estimate the proportion of students satisfied with academic advising.

If expected satisfaction is unknown and desired margin of error is 5%:

$n = \frac{1.96^2 \times 0.50(1-0.50)}{0.05^2} = 384.16$

Round up to 385 completed responses before nonresponse inflation.

Example 2: Employee Engagement Panel

Objective: Track change in engagement over four quarters.

Use a panel survey if individual-level change is important. Analyze repeated measurements with paired comparisons or longitudinal models.

Key risk: attrition. Report whether dropouts differ from retained respondents.

Example 3: Public Health Household Survey

Objective: Estimate vaccination coverage in a city.

Use multistage cluster sampling when a full household list is unavailable.

Report:

• Cluster selection method.
• Household selection method.
• Response rate.
• Weighting method.
• Design effect or effective sample size.

14. Common Mistakes and How to Avoid Them

15. Troubleshooting

Response Rate Is Low

Improve invitations, reminders, survey length, mobile usability, trust signals, and contact mode. Compare early and late responders as a nonresponse diagnostic.

Subgroup Sample Sizes Are Too Small

Consider stratified oversampling, collapsing categories only when defensible, or reporting wide intervals instead of overinterpreting noisy estimates.

Respondents Skip Sensitive Questions

Move sensitive items later, explain confidentiality, provide "prefer not to answer," and avoid unnecessary identifiers.

Results Differ From Known Population Margins

Check coverage, response patterns, and weighting. Differences may reveal bias or real population change; do not assume either without evidence.

16. Quick Reference Cheat Sheet

Design Selection

Sampling Selection

Key Formulas

Reporting Checklist

• Target population and sampling frame.
• Survey mode and field period.
• Sampling method and selection probabilities, if applicable.
• Questionnaire development and pilot testing.
• Response rate and missing-data handling.
• Weighting or adjustment method.
• Primary estimates with confidence intervals.
• Clear limitations on generalizability.

Next Steps

After designing the survey, continue with:

• Sample Size and Power Analysis for response target planning.
• Categorical Descriptives for response distributions.
• Confidence Interval Calculators for proportions and means.
• Chi-Square Test for categorical subgroup comparisons.
• Data Quality and Validation for survey cleaning.