Multiple Comparison Adjustment: Which Method Should You Use?

Multiple comparison adjustment limits false-positive findings when a study evaluates several related hypotheses. The correct method depends first on what counts as one family of tests, then on whether the study needs familywise error rate (FWER) control or false discovery rate (FDR) control, and finally on the dependence among the tests.

For many confirmatory analyses, Holm is a practical default because it strongly controls FWER under general dependence and is less conservative than plain Bonferroni. For exploratory analyses with many hypotheses, Benjamini-Hochberg often provides a more useful balance between discovery and error control when its dependence conditions are reasonable.

The DataStatPro Multiple Comparison Adjustment tool calculates eight adjusted p-value methods side by side, identifies which results remain significant, summarizes method differences, and provides exportable results.

Why do multiple comparisons increase false positives?

A significance level of 0.05 limits the Type I error probability for one test under its null hypothesis. It does not usually limit the probability of at least one false positive across a larger family of tests.

If m independent null hypotheses are tested at alpha = 0.05, the probability of at least one false positive is:

1 - (1 - 0.05)^m

This probability is approximately:

These values assume independence and that all null hypotheses are true. Real studies can have correlated outcomes and a mixture of true and false null hypotheses, but the table illustrates why a list of unadjusted p values can overstate the evidence.

Adjustment is relevant when testing multiple outcomes, treatment groups, time points, subgroups, model terms, correlations, biomarkers, genes, survey items, or follow-up contrasts. It should be planned together with the analysis rather than added only after inconvenient results appear.

What is a family of hypotheses?

A hypothesis family is the set of statistical claims for which the researcher wants a joint error guarantee. Defining this family is a scientific and design decision, not a calculation performed by software.

Tests may belong to one family when they address the same primary question, support one combined claim, or are selected from the same opportunity for discovery. Examples include all pairwise comparisons following one omnibus test, several primary outcomes used to establish efficacy, or thousands of genes screened in one experiment.

Do not split one natural family into many small groups merely to obtain smaller adjustments. At the same time, unrelated questions do not automatically belong to a single family just because they appear in the same dataset. The protocol, estimand, decision rule, and reporting plan should explain the grouping.

What is the difference between FWER and FDR?

FWER is the probability of making one or more Type I errors within the defined family. FWER control is appropriate when even one false claim would be costly, such as a confirmatory primary analysis, a safety decision, or a small set of planned comparisons.

FDR is the expected proportion of false discoveries among the hypotheses declared significant. FDR control allows a greater chance that at least one false positive occurs, but it can provide substantially more power when many discoveries are being screened. It is common in genomics, neuroimaging, high-dimensional surveys, and other exploratory settings.

An FDR level of 0.05 does not mean that each significant result has a 5% probability of being false. It describes the long-run expected proportion of false rejections under the procedure and its assumptions.

Which multiple comparison method should you choose?

DataStatPro provides six FWER methods and two FDR methods. The method should be selected before comparing which column gives the most favorable result.

Bonferroni versus Holm

Bonferroni multiplies each p value by the number of tests, with adjusted values capped at 1. It is simple and easy to audit, but it can lose considerable power as the family grows.

Holm orders the p values and applies a sequential adjustment. It provides strong FWER control and is uniformly at least as powerful as Bonferroni. When a protocol does not specifically require Bonferroni, Holm is often the more efficient general-purpose choice.

Hochberg, Hommel, Sidak, and Holm-Sidak

These methods can offer additional power, but their dependence conditions matter. Independence should not be assumed merely because tests were calculated in separate rows. Outcomes measured on the same participant, related biomarkers, repeated time points, and overlapping model terms are often correlated.

Hochberg is a step-up FWER procedure. Hommel is a closed-testing approach related to Simes testing. Sidak and Holm-Sidak use the joint probability structure of independent tests. Use these procedures when their assumptions and the study's confirmatory plan support them.

Benjamini-Hochberg versus Benjamini-Yekutieli

Benjamini-Hochberg controls FDR under independence and certain forms of positive dependence. It is widely used because it retains more discoveries than FWER procedures.

Benjamini-Yekutieli extends FDR control to arbitrary dependence by adding a harmonic-number factor. This protection can be much more conservative, particularly when the number of tests is large. The choice between BH and BY should reflect the dependence structure and the consequences of false discoveries, not simply the number of significant results.

How do you use DataStatPro for multiple comparison adjustment?

The calculator organizes the process into five tabs: Input, Results, Summary, History, and Methods Guide.

Step 1: Define the family and alpha level

Write down which hypotheses belong to the family and why. Then select alpha = 0.10, 0.05, 0.01, or 0.001. The conventional 0.05 level is not mandatory; the threshold should follow the study protocol and decision context.

Step 2: Enter or import the p values

Provide at least two uniquely named hypotheses and a valid p value from 0 to 1 for each. Rows can be entered manually or pasted as name and p-value pairs separated by tabs, commas, or semicolons. The tool also includes example datasets for genomics, psychology, and clinical trials.

Use the raw p values from the same prespecified family. Do not mix already adjusted p values with raw p values, and do not enter repeated versions of one analysis selected after examining the results without documenting that selection process.

Step 3: Select methods and calculate

Choose which of the eight methods should appear, then select Apply All Adjustments. DataStatPro ranks the original p values, calculates the selected adjusted p values, enforces the required monotonic ordering for stepwise methods, and marks values at or below the chosen alpha level.

Displaying several methods is useful for understanding sensitivity. The primary interpretation should still use the method justified by the study plan rather than whichever procedure creates the largest rejection count.

Step 4: Compare the Results and Summary tabs

The Results table shows the hypothesis name, original p value, rank, adjusted p value for each selected method, and significance status. Columns can be sorted to inspect specific methods.

The Summary tab compares the number and percentage of rejected hypotheses across methods. It also identifies whether each method controls FWER or FDR, states its dependence guidance, and provides a copyable reporting paragraph.

Step 5: Export and document the analysis

Download the full text report or CSV results. Retain the hypothesis family definition, raw p values, selected alpha, chosen adjustment, software version, adjusted results, and rationale for the method. The History tab can restore calculations during the current session, but durable research documentation should use the exported files.

Worked example: six related outcomes

Suppose a study tests six related outcomes with raw p values of .001, .008, .021, .043, .120, and .310. At alpha = .05, four results are significant without adjustment.

Selected adjusted values are:

Bonferroni and Holm each retain two results at .05. Benjamini-Hochberg retains three because it controls FDR rather than the probability of any false positive. This does not make BH the better method automatically. If these were confirmatory primary outcomes, FWER control might be required. If they were candidates in a discovery screen, FDR control might be appropriate.

The example also shows why adjusted p values are useful for reporting: each can be compared with the original alpha level. An equivalent decision can sometimes be made using adjusted alpha thresholds, but reports should clearly state which approach was used.

How should adjusted p values be reported?

Name the correction, define the family, report the number of tests, and distinguish adjusted from unadjusted values. When relevant, report effect sizes and confidence intervals alongside multiplicity-adjusted inference. Statistical significance does not communicate effect magnitude or practical importance.

An example methods sentence is:

Six prespecified outcome hypotheses were treated as one family. Holm adjustment was used to control the familywise error rate at .05.

An example results sentence is:

Four outcomes had unadjusted p values below .05; two remained significant after Holm adjustment (adjusted p values = .006 and .040).

For broader guidance, see how to report p values, confidence intervals, and effect sizes in APA 7. If the uncertainty begins earlier in the workflow, use the statistical test selection guide before adjusting the resulting p values.

Common mistakes to avoid

• Defining the hypothesis family only after seeing the p values.
• Choosing the correction that produces the most significant findings.
• Treating FDR control as if it were FWER control.
• Assuming tests are independent because they have different names.
• Applying a second correction to p values that were already adjusted.
• Correcting p values while ignoring multiplicity in confidence intervals or decision rules.
• Reporting only adjusted significance labels without effect sizes and uncertainty.
• Treating a non-significant adjusted result as proof of no effect.
• Using correction to legitimize unrestricted model searching or selective reporting.

No p-value adjustment can repair poor study design, undisclosed outcome switching, invalid test assumptions, or a data-dependent family definition. Multiplicity control works best when hypotheses and decision rules are planned, transparent, and reproducible.

Frequently asked questions

When is multiple comparison adjustment necessary?

It is usually necessary when several related tests support one set of scientific or operational claims and the resulting false-positive risk would be unacceptable. The need depends on the defined family, purpose, and decision consequences, not on a fixed minimum number of tests.

Is Holm always better than Bonferroni?

Holm strongly controls FWER under general dependence and is uniformly at least as powerful as Bonferroni for hypothesis testing. Bonferroni may still be selected for simplicity, a prespecified protocol, or compatibility with a particular reporting framework.

What is the difference between an adjusted p value and an adjusted alpha?

An adjusted p value incorporates the multiplicity procedure and can usually be compared with the original alpha. An adjusted alpha changes the rejection threshold applied to raw p values. Both can encode the same decision for some procedures, but the report must state which quantity is shown.

Should I use Benjamini-Hochberg or Bonferroni?

Use the error criterion that matches the research goal. Bonferroni controls the probability of any false positive in the family and is cautious. Benjamini-Hochberg controls the expected false-discovery proportion and is generally more powerful for exploratory screening.

Do post-hoc tests and p-value adjustments mean the same thing?

No. A post-hoc procedure defines particular comparisons and may incorporate its own multiplicity control. A general p-value adjustment takes an already defined family of p values and adjusts their inferential thresholds. Avoid applying an extra adjustment without understanding what the post-hoc procedure already controls.

Can DataStatPro choose the correction automatically?

DataStatPro compares eight methods and explains their assumptions, but it cannot determine the scientific family, tolerance for false positives, or defensibility of dependence assumptions. Those decisions require the protocol, study design, and subject-matter context.