Method Comparison Tutorial
Method Comparison evaluates how closely two or more measurement methods agree when they are applied to the same specimens, participants, or items. This tutorial develops the statistical theory, shows how to run a study in DataStatPro, and works through continuous and cutoff-based examples by hand.
Table of Contents
- Study design and paired data
- Running Method Comparison in DataStatPro
- Descriptives and paired bootstrap intervals
- Bland–Altman agreement
- Passing–Bablok regression and CUSUM
- Deming regression and sensitivity analyses
- Binary and ordinal classification agreement
- Covariate and group analyses
- Worked examples
- Reporting and troubleshooting
- Quick reference
1. Study design and paired data
Consider a laboratory comparing a new instrument with its current instrument. Measuring each specimen on both instruments creates a pair , where identifies the specimen, is the first method, and is the second. One row represents one independent experimental unit. Both methods measure the same quantity on the same scale. For example, convert both columns to nmol/L before estimating differences or applying a common cutoff.
A method comparison study can answer several questions:
| Question | Output |
|---|---|
| How much higher or lower does Y read on average? | Bland–Altman mean bias |
| How large can individual paired differences be? | Bland–Altman limits of agreement |
| Does bias change across the measurement range? | Difference plot, Passing–Bablok slope, fitted differences |
| Do methods classify the same specimens similarly? | Matched classification counts and agreement measures |
| Does one method classify a larger share as positive? | Signed difference in positive rates and exact McNemar test |
| Do readings or differences vary with a covariate or across groups? | Subgroup summaries and adjusted models |
A large correlation is compatible with large measurement differences. If for every specimen and varies, Pearson’s equals 1, although . Agreement is evaluated against the identity line and against the magnitude of acceptable differences for the intended application. Establish that tolerance and any meaningful decision cutoffs before interpreting results. Include observations spanning the range where the methods will be used. Bland and Altman’s original agreement paper
The analyses assume independent rows. Several visits or replicate measurements from the same participant can produce correlated differences. Such a design calls for a repeated-measures agreement method and uncertainty estimates that account for clustering. Bland and Altman on agreement with multiple observations per individual
With selected methods, DataStatPro compares pairs. The earlier selected method is in each pair and differences are always . Three methods A, B and C therefore give B−A, C−A and C−B. The study uses the same set of rows for every pair by requiring a finite reading for every selected method. Compare included and excluded counts, especially when missingness may depend on the specimen or measurement level.
2. Running Method Comparison in DataStatPro
- Open Advanced Analysis → Method Comparison. Select an existing dataset and inspect the measurement columns in Dataset Preview & Editor.
- Use Two-method study to assign X and Y. Use Multi-method study to select and order several measurement columns.
- Optionally select a Numeric covariate such as temperature, age, or baseline concentration, and a Grouping variable such as site, instrument batch, or treatment arm. Choose the Reference group for adjusted group contrasts. Enter optional Covariate band boundaries in the covariate’s original units to summarize results by numeric range.
- Expand Cutoffs, sensitivity analyses and bootstrap. Use Include cutoff analyses for binary and ordinal comparisons across cutoffs. Use Include sensitivity analyses for additional analyses such as the inclusive band, the cutoff grid, Deming fits and alternative agreement limits.
- Review bootstrap resamples and the seed. The default is 5,000 resamples with seed 2026. Enter 0 to skip the bootstrap. The analytical plot and exact test outputs can still be available.
- Click Run study comparison. Start with Continuous agreement and Plots. Review Classification and Subgroups and models when relevant.
Data-derived suggestions
DataStatPro pools the readings from every selected method across the complete rows. It suggests the pooled tertiles for Classification cutoffs, the pooled quartiles for the Sensitivity band, and the pooled deciles for the Exploratory cutoff grid. Suggested concentrations for fitted differences use the suggested classification cutoffs. Suggestions are rounded to one decimal place. Repeated cutoff values are combined and rounded cutoffs at or outside the observed extremes are omitted. An interval requires distinct ordered bounds.
These are exploratory starting values. Edit them to use decision limits or ranges meaningful to your study. Restore data suggestions recalculates them for the current selection. A strict cutoff counts a reading as positive when it is below the threshold. The sensitivity band counts a reading as positive when it falls within the interval, including both endpoints.
3. Descriptives and paired bootstrap intervals
Summarizing the measurement range
Measurement descriptives reports the count, mean, sample standard deviation, median, quartiles, minimum and maximum. For any column ,
The median is and the quartiles are and . DataStatPro sorts values and uses linear interpolation. For probability , set , , and . The quantile is , with the maximum returned at . This same convention is used for the exploratory cutoffs and percentile intervals. The selected covariate is summarized over its available values in the retained measurement rows.
Resampling complete rows
For a statistic , draw complete rows with replacement and recalculate the statistic. Repeat this times. In each replicate the same sampled row indices apply to all methods and pairs, preserving the matching within each participant or specimen. The percentile interval is
Use the same seed and settings to reproduce results. DataStatPro requires at least two valid replicates and at least 90% of requested estimates to be valid for a bootstrap interval. Degenerate resamples can arise when readings or categories have little variation. Bootstrap intervals are marginal for individual estimates, and the fitted-line band is pointwise across the horizontal axis. Each point of that band uses jointly resampled intercepts and slopes.
4. Bland–Altman agreement
Mean bias and limits of agreement
For each specimen, calculate the difference and paired mean.
The mean difference estimates additive bias, with positive values indicating that Y reads higher on average. The sample SD of the differences describes their spread.
The conventional 95% limits of agreement use .
With approximately normal differences and a stable bias and spread across the range, the limits estimate an interval containing about 95% of individual paired differences in the study population. The 95% CI printed next to each endpoint measures sampling uncertainty about that endpoint. A mean bias close to zero can occur alongside wide limits. Compare both limits and their uncertainty with your tolerances. Bland and Altman’s derivation and examples
Difference plot and distribution checks
The Bland–Altman plot places on the horizontal axis and on the vertical axis. An upward or downward pattern suggests concentration-dependent differences. A funnel shape suggests changing scatter across the range. A point that lies far from the other pairs warrants a check of specimen matching, units and data quality.
Distribution of paired differences includes Shapiro–Wilk , skewness and excess kurtosis for , plus Pearson correlation for the paired raw readings. Pearson’s is the sample covariance divided by the product of sample SDs. Shapiro–Wilk uses the ordered differences and normal-order-statistic weights :
Small normality p-values suggest departures from a normal model. Skewness describes asymmetry and excess kurtosis describes tail weight relative to a normal distribution. Use these summaries alongside the plot. A high normality p-value in a small study can coexist with patterns that the test has little power to detect.
5. Passing–Bablok regression and CUSUM
Errors in the two measurements
Passing–Bablok fits for a positive, approximately linear relationship between two methods. The ideal agreement line has and . Passing–Bablok estimates the slope using ranks of pairwise slopes and the intercept using a median. This reduces the effect of some extreme values compared with a fit based on squared vertical residuals. Study design, the measurement range and the error structure still matter. The original model assumes independent errors between the methods, continuous error distributions of the same form and a constant ratio of measurement-error variances. Passing and Bablok’s original paper
Slope and intercept
For each pair of distinct observations calculate
DataStatPro skips identical coordinate pairs and slopes equal to . It includes vertical pairs at in the ordered calculation. Let be the number of retained slopes and the number below . Order the slopes from smallest to largest and use the shifted rank .
At a half-integer rank, the tangent convention averages the two adjacent slopes. A usable fit requires a finite positive slope and positive net concordance of the pairs. The plot shows the identity line, the fitted line and an inset with the fitted equation and model summary.
Confidence intervals and bias
The Continuous agreement table uses paired percentile bootstrap intervals. The plot inset uses analytical intervals where available. The analytical slope interval uses the ranks and , with
When all X values are on the same side of zero, the intercept interval transforms the slope endpoints through and orders the resulting endpoints. Analytical intervals can be unavailable with invalid ranks, nonfinite endpoints or X measurements spanning zero in the intercept calculation. Table and plot intervals can differ because their methods differ.
An intercept interval excluding 0 suggests a nonzero intercept. An intercept estimates bias at , which may be outside the study range. A slope interval excluding 1 suggests proportional bias. Intervals containing the identity values indicate that those values remain compatible with the data. For a practically meaningful X concentration use the fitted difference,
For example, with and , the estimated differences at and are and units. Review the fitted difference and its CI where decisions are made. DataStatPro labels concentrations outside the observed X range as extrapolations.
CUSUM test for linearity
Calculate the residuals . The test orders the pairs along the fitted direction using . Let and count positive and negative residuals. Residuals numerically equal to zero contribute zero. DataStatPro combines tied projections, computes the cumulative signed sum , and scales its largest absolute value.
Long runs of residuals on one side of the line produce large cumulative departures. DataStatPro uses an approximate two-sided Kolmogorov tail probability. A p-value at or below .05 flags evidence against linearity. A larger p-value indicates that the test did not detect a departure. If all nonzero residuals have the same sign, or all residuals are zero, the test is unavailable. R² is omitted from the Passing–Bablok summary because the agreement questions concern bias, linearity and the size of paired differences. Passing and Bablok’s discussion of the CUSUM procedure
6. Deming regression and sensitivity analyses
Deming regression
Deming regression allows measurement error in X and Y. It uses the assumed ratio . Estimate that ratio from precision data when available. The ratio concerns analytic measurement errors. The observed between-specimen variances also include real differences among specimens.
With sample variances and covariance , define . The fit is
A ratio of 1 gives equal weight to error variance on the two axes when they use the same units. Comparing estimates at different plausible ratios reveals how much the result depends on this assumption. Report the ratio with the fitted coefficients and their intervals. Linnet’s study of error-ratio misspecification
Empirical limits and ratios
Empirical limits of paired differences report and directly. These limits avoid a normal distribution assumption for , while sample extremes strongly influence the estimates in small studies.
Log-ratio means and limits of agreement are available for pairs with strictly positive readings. Calculate and then exponentiate the log-scale mean and limits.
A geometric mean ratio of 1.08 means Y is about 8% higher on average on the multiplicative scale. Ratio limits of 0.85 to 1.25 correspond to Y reading between roughly 15% lower and 25% higher for individual pairs under the log-scale assumptions. Ratio limits are useful when scatter grows roughly in proportion to measurement size. Bland and Altman’s discussion of transformations
Exploratory cutoff grid
At each grid cutoff , the app estimates the fraction for which . Review the curve of discordance across the measurement range using the ordered grid rows. Very low or very high cutoffs can produce high agreement because almost every observation is classified the same way. Report a cutoff selected after reviewing the grid as exploratory.
7. Binary and ordinal classification agreement
Read the matched 2×2 table
Under Classification, choose a positive rule and a pair. X defines the rows and Y defines the columns. Positive means below the selected cutoff or within both endpoints of the inclusive sensitivity band, depending on the chosen rule.
| Classification | Y positive | Y negative |
|---|---|---|
| X positive | ||
| X negative |
The total is . The diagonal entries agree and the off-diagonal entries show disagreements in each direction.
These two measures are symmetric in X and Y. If a denominator is zero, the corresponding estimate is undefined. The difference in positive proportions is
The app multiplies by 100 to report percentage points. Negative values indicate a lower positive classification rate for Y. Positive and negative agreement remain meaningful when both methods are candidates and there is no verified reference classification. Estimating diagnostic sensitivity and specificity requires defining an independent reference outcome.
Exact McNemar test and Holm adjustment
The null hypothesis states that the two directions of discordance are equally probable. Conditional on discordant pairs, the count in one direction follows under the null.
With , the p-value is 1. The test addresses a difference in marginal positive rates. For example, shows balanced positive rates despite 40 disagreements. Inspect the agreement statistics and the interval for alongside the p-value.
For each classification rule, DataStatPro applies Holm adjustment across the method pairs. For ordered raw p-values ,
The adjusted p-values account for the family of pair tests at that rule. Each cutoff and the inclusive band define their own family. R documentation for Holm’s method
Cohen’s kappa and ordinal weighted kappa
Cohen’s kappa compares observed agreement with the expected agreement from the marginal frequencies:
Kappa equals 1 for perfect agreement when defined. Zero corresponds to the agreement expected from those marginals. Kappa can be negative when observed agreement is lower than expected. It depends on the category frequencies, so include counts and observed agreement when reporting it. When , the denominator is zero.
With several cutoffs, the app assigns each reading to an ordered band and builds a matched table. For example, cutoffs 30 and 50 define , , and . For cell proportion and marginal proportions and , the linear weights are
The weighted kappa substitutes these weighted agreements into . Adjacent bands receive partial credit and the most distant bands receive none. The table also reports exact agreement, which counts only diagonal entries.
Wilson interval for a single proportion
Band membership and subgroup discordance use Wilson score intervals. For events among observations, put . The 95% interval is
It can express uncertainty even when the observed proportion is zero or one. The interval for a difference in paired positive rates comes from the paired bootstrap described above.
8. Covariate and group analyses
Subgroup tables summarize average readings by category and, when boundaries are supplied, by covariate band. These are separate summaries for each variable. With cutoff analyses enabled, they also report the fraction of matched pairs that disagree within each subgroup. Adjusted models use retained specimens with both covariate and group values available and at least two observed groups.
Let , where is the selected numeric covariate, and let indicate membership in group . The omitted group is the selected baseline. For each method reading the concentration model is
The covariate coefficient gives the expected change per one unit of the selected variable at a fixed group. Each group coefficient gives the expected difference from the reference group at a common covariate value. The fitted group lines have a common slope in this model.
For each method pair, the adjusted difference model also includes the centered pair mean.
Its covariate and group coefficients describe changes in at a common mean measurement level. The quadratic covariate sensitivity model adds . These estimates describe associations with readings or differences. Their interpretation depends on the observed covariate values, groups and measurement ranges.
HC3 inference
The coefficients are fitted by least squares. For a design matrix , residuals , and leverage from the diagonal of , the HC3 covariance estimate is
A coefficient CI uses , where is the number of fitted coefficients. Holm adjustment uses separate families for the concentration models, the covariate terms in paired differences, the group contrasts in paired differences, and the quadratic covariate terms. Review the model sample size when comparing these estimates with the wider study.
9. Worked examples
Example 1. Continuous agreement across the measurement range
Enter the following six synthetic pairs in a dataset. Assign the first method to X and the second to Y.
| Specimen | X | Y | Paired mean | Y − X |
|---|---|---|---|---|
| 1 | 10 | 11 | 10.5 | 1 |
| 2 | 20 | 23 | 21.5 | 3 |
| 3 | 30 | 32 | 31.0 | 2 |
| 4 | 40 | 43 | 41.5 | 3 |
| 5 | 50 | 56 | 53.0 | 6 |
| 6 | 60 | 65 | 62.5 | 5 |
- Mean difference. The differences sum to 20, giving units. The sample SD is units.
- Limits of agreement. Calculate to get approximately units.
- Passing–Bablok fit. All 15 pairwise slopes are eligible and none fall below . The eighth slope in sorted order is . The six values of are , giving median . The fitted line is .
- Interpretation. The estimated difference increases from units at to at . If the prespecified tolerance were units, the estimated upper agreement limit would exceed it. With six pairs, the endpoints are imprecise. Inspect the CIs and the difference plot before drawing a conclusion.
Example 2. Cutoff agreement and McNemar’s test
Suppose 30 specimens are classified positive below a cutoff of 23.3. X and Y classify 17 specimens positive together and 7 negative together. Six are positive only on X and zero only on Y.
| Classification | Y < 23.3 | Y ≥ 23.3 |
|---|---|---|
| X < 23.3 | 17 | 6 |
| X ≥ 23.3 | 0 | 7 |
- Overall agreement is .
- Positive agreement is . Negative agreement is .
- Positive rate difference is percentage points. Y classifies fewer specimens as positive.
- Cohen’s kappa is approximately . The expected agreement from the margins is .
- Exact McNemar p is . With one pair, the Holm-adjusted p-value is the same.
The table shows that all disagreements occur in one direction. The continuous analysis helps determine how large the underlying measurement differences are and where they arise.
10. Reporting and troubleshooting
Export APA Report creates the Word report after a successful analysis. In a report, state the specimen count, complete-case rule, units, direction of subtraction and acceptable measurement differences. Report mean bias and limits with their CIs, Passing–Bablok coefficients with interval methods and CUSUM p-value, and cutoff rules with their matched counts and signed differences when applicable. Record the resample count and seed for reproducibility.
| Situation | What to check |
|---|---|
| Too few complete rows | Check units, numeric column types, row matching, and missingness across all selected methods. At least three complete rows are required. |
| Regression coefficient or interval unavailable | Inspect ties, measurement range, the direction of association and the availability of valid bootstrap estimates. |
| CUSUM unavailable | Check whether the fit has residuals on both sides of the line. |
| Log-ratio summary unavailable | Check for zero or negative values in either method. |
| Kappa unavailable or small despite high agreement | Inspect category frequencies, off-diagonal counts and the expected agreement. |
| Adjusted model unavailable | Check the joint availability of covariate and group values, group levels, sample size and variation within the selected data. |
| Cutoff suggestions missing or collapsed | Check the distinct values and their range. Enter study-specific bounds when the pooled data provide insufficient distinct rounded values. |
| Need to start a new comparison | Use Reset to clear selections, settings and results while keeping the active dataset. |
11. Quick reference
| Output | What to read first |
|---|---|
| Bland–Altman | Direction of mean bias and whether both limits fit the tolerance |
| Passing–Bablok | Slope relative to 1, intercept relative to 0, and fitted differences at relevant concentrations |
| CUSUM | Evidence of nonlinear departures alongside the fitted plot |
| Deming sensitivity | Changes in the fit across plausible Y/X error variance ratios |
| Matched binary agreement | Counts , positive and negative agreement, and signed rate difference |
| McNemar test | Asymmetry of the discordant counts and Holm-adjusted p-value across pairs |
| Ordinal agreement | Matched bands, exact agreement and linear weighted kappa |
| Subgroups and models | Sample size, baseline group, effect units, HC3 CI and Holm-adjusted p-value |