Method Comparison Tutorial

Compare two or more methods on the same patients or specimens.

Quick answer

Compare two or more methods on the same patients or specimens. Apply this method only when the research question, target estimand, study design, outcome scale, and dependence structure are compatible with it. Before interpreting the result, verify method-specific assumptions, missing-data handling, diagnostics, uncertainty estimates, effect sizes where relevant, and sensitivity analyses or alternative procedures.

Method Comparison Tutorial: illustrated DataStatPro statistics tutorial
Method Comparison Tutorial statistics tutorial from DataStatPro.

Method Comparison Tutorial

Method Comparison evaluates how closely two or more measurement methods agree when they are applied to the same specimens, participants, or items. This tutorial develops the statistical theory, shows how to run a study in DataStatPro, and works through continuous and cutoff-based examples by hand.

Table of Contents

  1. Study design and paired data
  2. Running Method Comparison in DataStatPro
  3. Descriptives and paired bootstrap intervals
  4. Bland–Altman agreement
  5. Passing–Bablok regression and CUSUM
  6. Deming regression and sensitivity analyses
  7. Binary and ordinal classification agreement
  8. Covariate and group analyses
  9. Worked examples
  10. Reporting and troubleshooting
  11. Quick reference

1. Study design and paired data

Consider a laboratory comparing a new instrument with its current instrument. Measuring each specimen on both instruments creates a pair (xi,yi)(x_i,y_i), where ii identifies the specimen, XX is the first method, and YY is the second. One row represents one independent experimental unit. Both methods measure the same quantity on the same scale. For example, convert both columns to nmol/L before estimating differences or applying a common cutoff.

A method comparison study can answer several questions:

QuestionOutput
How much higher or lower does Y read on average?Bland–Altman mean bias
How large can individual paired differences be?Bland–Altman limits of agreement
Does bias change across the measurement range?Difference plot, Passing–Bablok slope, fitted differences
Do methods classify the same specimens similarly?Matched classification counts and agreement measures
Does one method classify a larger share as positive?Signed difference in positive rates and exact McNemar test
Do readings or differences vary with a covariate or across groups?Subgroup summaries and adjusted models

A large correlation is compatible with large measurement differences. If Y=2X+10Y=2X+10 for every specimen and XX varies, Pearson’s rr equals 1, although Y−X=X+10Y-X=X+10. Agreement is evaluated against the identity line Y=XY=X and against the magnitude of acceptable differences for the intended application. Establish that tolerance and any meaningful decision cutoffs before interpreting results. Include observations spanning the range where the methods will be used. Bland and Altman’s original agreement paper

The analyses assume independent rows. Several visits or replicate measurements from the same participant can produce correlated differences. Such a design calls for a repeated-measures agreement method and uncertainty estimates that account for clustering. Bland and Altman on agreement with multiple observations per individual

With kk selected methods, DataStatPro compares k(k−1)/2k(k-1)/2 pairs. The earlier selected method is XX in each pair and differences are always Y−XY-X. Three methods A, B and C therefore give B−A, C−A and C−B. The study uses the same set of rows for every pair by requiring a finite reading for every selected method. Compare included and excluded counts, especially when missingness may depend on the specimen or measurement level.

2. Running Method Comparison in DataStatPro

  1. Open Advanced Analysis → Method Comparison. Select an existing dataset and inspect the measurement columns in Dataset Preview & Editor.
  2. Use Two-method study to assign X and Y. Use Multi-method study to select and order several measurement columns.
  3. Optionally select a Numeric covariate such as temperature, age, or baseline concentration, and a Grouping variable such as site, instrument batch, or treatment arm. Choose the Reference group for adjusted group contrasts. Enter optional Covariate band boundaries in the covariate’s original units to summarize results by numeric range.
  4. Expand Cutoffs, sensitivity analyses and bootstrap. Use Include cutoff analyses for binary and ordinal comparisons across cutoffs. Use Include sensitivity analyses for additional analyses such as the inclusive band, the cutoff grid, Deming fits and alternative agreement limits.
  5. Review bootstrap resamples and the seed. The default is 5,000 resamples with seed 2026. Enter 0 to skip the bootstrap. The analytical plot and exact test outputs can still be available.
  6. Click Run study comparison. Start with Continuous agreement and Plots. Review Classification and Subgroups and models when relevant.

Data-derived suggestions

DataStatPro pools the readings from every selected method across the complete rows. It suggests the pooled tertiles for Classification cutoffs, the pooled quartiles for the Sensitivity band, and the pooled deciles for the Exploratory cutoff grid. Suggested concentrations for fitted differences use the suggested classification cutoffs. Suggestions are rounded to one decimal place. Repeated cutoff values are combined and rounded cutoffs at or outside the observed extremes are omitted. An interval requires distinct ordered bounds.

These are exploratory starting values. Edit them to use decision limits or ranges meaningful to your study. Restore data suggestions recalculates them for the current selection. A strict cutoff counts a reading as positive when it is below the threshold. The sensitivity band counts a reading as positive when it falls within the interval, including both endpoints.

3. Descriptives and paired bootstrap intervals

Summarizing the measurement range

Measurement descriptives reports the count, mean, sample standard deviation, median, quartiles, minimum and maximum. For any column v1,…,vnv_1,\ldots,v_n,

vˉ=1n∑i=1nvi,sv=∑i=1n(vi−vˉ)2n−1.\bar v=\frac{1}{n}\sum_{i=1}^{n}v_i,\qquad s_v=\sqrt{\frac{\sum_{i=1}^{n}(v_i-\bar v)^2}{n-1}}.

The median is Q0.50Q_{0.50} and the quartiles are Q0.25Q_{0.25} and Q0.75Q_{0.75}. DataStatPro sorts values and uses linear interpolation. For probability pp, set h=1+(n−1)ph=1+(n-1)p, j=⌊h⌋j=\lfloor h\rfloor, and g=h−jg=h-j. The quantile is v(j)+g(v(j+1)−v(j))v_{(j)}+g(v_{(j+1)}-v_{(j)}), with the maximum returned at p=1p=1. This same convention is used for the exploratory cutoffs and percentile intervals. The selected covariate is summarized over its available values in the retained measurement rows.

Resampling complete rows

For a statistic θ^\hat\theta, draw nn complete rows with replacement and recalculate the statistic. Repeat this BB times. In each replicate the same sampled row indices apply to all methods and pairs, preserving the matching within each participant or specimen. The percentile interval is

[Q0.025(θ^∗), Q0.975(θ^∗)].\left[Q_{0.025}(\hat\theta^{*}),\ Q_{0.975}(\hat\theta^{*})\right].

Use the same seed and settings to reproduce results. DataStatPro requires at least two valid replicates and at least 90% of requested estimates to be valid for a bootstrap interval. Degenerate resamples can arise when readings or categories have little variation. Bootstrap intervals are marginal for individual estimates, and the fitted-line band is pointwise across the horizontal axis. Each point of that band uses jointly resampled intercepts and slopes.

4. Bland–Altman agreement

Mean bias and limits of agreement

For each specimen, calculate the difference and paired mean.

di=yi−xi,mi=xi+yi2.d_i=y_i-x_i,\qquad m_i=\frac{x_i+y_i}{2}.

The mean difference estimates additive bias, with positive values indicating that Y reads higher on average. The sample SD of the differences describes their spread.

bias^=dˉ=1n∑idi,sd=∑i(di−dˉ)2n−1.\widehat{\mathrm{bias}}=\bar d=\frac{1}{n}\sum_i d_i,\qquad s_d=\sqrt{\frac{\sum_i(d_i-\bar d)^2}{n-1}}.

The conventional 95% limits of agreement use z=Φ−1(0.975)≈1.96z=\Phi^{-1}(0.975)\approx1.96.

LoA^lower=dˉ−zsd,LoA^upper=dˉ+zsd.\widehat{\mathrm{LoA}}_{\mathrm{lower}}=\bar d-zs_d,\qquad \widehat{\mathrm{LoA}}_{\mathrm{upper}}=\bar d+zs_d.

With approximately normal differences and a stable bias and spread across the range, the limits estimate an interval containing about 95% of individual paired differences in the study population. The 95% CI printed next to each endpoint measures sampling uncertainty about that endpoint. A mean bias close to zero can occur alongside wide limits. Compare both limits and their uncertainty with your tolerances. Bland and Altman’s derivation and examples

Difference plot and distribution checks

The Bland–Altman plot places mim_i on the horizontal axis and did_i on the vertical axis. An upward or downward pattern suggests concentration-dependent differences. A funnel shape suggests changing scatter across the range. A point that lies far from the other pairs warrants a check of specimen matching, units and data quality.

Distribution of paired differences includes Shapiro–Wilk WW, skewness and excess kurtosis for did_i, plus Pearson correlation for the paired raw readings. Pearson’s rr is the sample covariance divided by the product of sample SDs. Shapiro–Wilk uses the ordered differences d(i)d_{(i)} and normal-order-statistic weights aia_i:

r=sXYsXsY,W=(∑iaid(i))2∑i(di−dˉ)2.r=\frac{s_{XY}}{s_Xs_Y},\qquad W=\frac{\left(\sum_i a_i d_{(i)}\right)^2}{\sum_i(d_i-\bar d)^2}.

Small normality p-values suggest departures from a normal model. Skewness describes asymmetry and excess kurtosis describes tail weight relative to a normal distribution. Use these summaries alongside the plot. A high normality p-value in a small study can coexist with patterns that the test has little power to detect.

5. Passing–Bablok regression and CUSUM

Errors in the two measurements

Passing–Bablok fits y^=α^+β^x\hat y=\hat\alpha+\hat\beta x for a positive, approximately linear relationship between two methods. The ideal agreement line has α=0\alpha=0 and β=1\beta=1. Passing–Bablok estimates the slope using ranks of pairwise slopes and the intercept using a median. This reduces the effect of some extreme values compared with a fit based on squared vertical residuals. Study design, the measurement range and the error structure still matter. The original model assumes independent errors between the methods, continuous error distributions of the same form and a constant ratio of measurement-error variances. Passing and Bablok’s original paper

Slope and intercept

For each pair of distinct observations i<ji<j calculate

Sij=yj−yixj−xi.S_{ij}=\frac{y_j-y_i}{x_j-x_i}.

DataStatPro skips identical coordinate pairs and slopes equal to −1-1. It includes vertical pairs at +∞+\infty in the ordered calculation. Let MM be the number of retained slopes and KK the number below −1-1. Order the slopes from smallest to largest and use the shifted rank RR.

R=M+12+K,β^=S(R),α^=median⁡i(yi−β^xi).R=\frac{M+1}{2}+K,\qquad \hat\beta=S_{(R)},\qquad \hat\alpha=\operatorname{median}_i (y_i-\hat\beta x_i).

At a half-integer rank, the tangent convention averages the two adjacent slopes. A usable fit requires a finite positive slope and positive net concordance of the pairs. The plot shows the identity line, the fitted line and an inset with the fitted equation and model summary.

Confidence intervals and bias

The Continuous agreement table uses paired percentile bootstrap intervals. The plot inset uses analytical intervals where available. The analytical slope interval uses the ranks R−C/2R-C/2 and R+C/2R+C/2, with

C=round⁡ ⁣[zn(n−1)(2n+5)18].C=\operatorname{round}\!\left[z\sqrt{\frac{n(n-1)(2n+5)}{18}}\right].

When all X values are on the same side of zero, the intercept interval transforms the slope endpoints through median⁡(yi−bxi)\operatorname{median}(y_i-bx_i) and orders the resulting endpoints. Analytical intervals can be unavailable with invalid ranks, nonfinite endpoints or X measurements spanning zero in the intercept calculation. Table and plot intervals can differ because their methods differ.

An intercept interval excluding 0 suggests a nonzero intercept. An intercept estimates bias at X=0X=0, which may be outside the study range. A slope interval excluding 1 suggests proportional bias. Intervals containing the identity values indicate that those values remain compatible with the data. For a practically meaningful X concentration use the fitted difference,

Y−X^(x)=α^+(β^−1)x.\widehat{Y-X}(x)=\hat\alpha+(\hat\beta-1)x.

For example, with α^=−0.5\hat\alpha=-0.5 and β^=1.10\hat\beta=1.10, the estimated differences at x=20x=20 and x=50x=50 are 1.51.5 and 4.54.5 units. Review the fitted difference and its CI where decisions are made. DataStatPro labels concentrations outside the observed X range as extrapolations.

CUSUM test for linearity

Calculate the residuals ei=yi−α^−β^xie_i=y_i-\hat\alpha-\hat\beta x_i. The test orders the pairs along the fitted direction using ti=xi+β^yit_i=x_i+\hat\beta y_i. Let n+n_+ and n−n_- count positive and negative residuals. Residuals numerically equal to zero contribute zero. DataStatPro combines tied projections, computes the cumulative signed sum U(t)U(t), and scales its largest absolute value.

U(t)=∑ti≤twi,wi={n−/n+ei>0,−n+/n−ei<0,0ei=0,T=max⁡t∣U(t)∣n−+1.U(t)=\sum_{t_i\le t} w_i,\qquad w_i= \begin{cases} \sqrt{n_-/n_+} & e_i>0,\\ -\sqrt{n_+/n_-} & e_i<0,\\ 0 & e_i=0, \end{cases} \qquad T=\frac{\max_t|U(t)|}{\sqrt{n_-+1}}.

Long runs of residuals on one side of the line produce large cumulative departures. DataStatPro uses an approximate two-sided Kolmogorov tail probability. A p-value at or below .05 flags evidence against linearity. A larger p-value indicates that the test did not detect a departure. If all nonzero residuals have the same sign, or all residuals are zero, the test is unavailable. R² is omitted from the Passing–Bablok summary because the agreement questions concern bias, linearity and the size of paired differences. Passing and Bablok’s discussion of the CUSUM procedure

6. Deming regression and sensitivity analyses

Deming regression

Deming regression allows measurement error in X and Y. It uses the assumed ratio λ=σerror,Y2/σerror,X2\lambda=\sigma^2_{\mathrm{error},Y}/\sigma^2_{\mathrm{error},X}. Estimate that ratio from precision data when available. The ratio concerns analytic measurement errors. The observed between-specimen variances also include real differences among specimens.

With sample variances sXX,sYYs_{XX}, s_{YY} and covariance sXY≠0s_{XY}\ne0, define D=sYY−λsXXD=s_{YY}-\lambda s_{XX}. The fit is

β^D=D+D2+4λsXY22sXY,α^D=yˉ−β^Dxˉ.\hat\beta_D=\frac{D+\sqrt{D^2+4\lambda s_{XY}^2}}{2s_{XY}},\qquad \hat\alpha_D=\bar y-\hat\beta_D\bar x.

A ratio of 1 gives equal weight to error variance on the two axes when they use the same units. Comparing estimates at different plausible ratios reveals how much the result depends on this assumption. Report the ratio with the fitted coefficients and their intervals. Linnet’s study of error-ratio misspecification

Empirical limits and ratios

Empirical limits of paired differences report Q0.025(d)Q_{0.025}(d) and Q0.975(d)Q_{0.975}(d) directly. These limits avoid a normal distribution assumption for dd, while sample extremes strongly influence the estimates in small studies.

Log-ratio means and limits of agreement are available for pairs with strictly positive readings. Calculate ℓi=log⁡(yi/xi)\ell_i=\log(y_i/x_i) and then exponentiate the log-scale mean and limits.

ratio^=exp⁡(ℓˉ),ratio LoA^=[exp⁡(ℓˉ−zsℓ), exp⁡(ℓˉ+zsℓ)].\widehat{\mathrm{ratio}}=\exp(\bar\ell),\qquad \widehat{\mathrm{ratio\ LoA}}=\left[\exp(\bar\ell-zs_\ell),\ \exp(\bar\ell+zs_\ell)\right].

A geometric mean ratio of 1.08 means Y is about 8% higher on average on the multiplicative scale. Ratio limits of 0.85 to 1.25 correspond to Y reading between roughly 15% lower and 25% higher for individual pairs under the log-scale assumptions. Ratio limits are useful when scatter grows roughly in proportion to measurement size. Bland and Altman’s discussion of transformations

Exploratory cutoff grid

At each grid cutoff cc, the app estimates the fraction for which 1(xi<c)≠1(yi<c)\mathbf{1}(x_i<c)\ne\mathbf{1}(y_i<c). Review the curve of discordance across the measurement range using the ordered grid rows. Very low or very high cutoffs can produce high agreement because almost every observation is classified the same way. Report a cutoff selected after reviewing the grid as exploratory.

7. Binary and ordinal classification agreement

Read the matched 2×2 table

Under Classification, choose a positive rule and a pair. X defines the rows and Y defines the columns. Positive means below the selected cutoff or within both endpoints of the inclusive sensitivity band, depending on the chosen rule.

ClassificationY positiveY negative
X positiveaabb
X negativeccdd

The total is n=a+b+c+dn=a+b+c+d. The diagonal entries agree and the off-diagonal entries show disagreements in each direction.

Overall agreement=a+dn,Discordance=b+cn.\text{Overall agreement}=\frac{a+d}{n},\qquad \text{Discordance}=\frac{b+c}{n}. Positive agreement=2a2a+b+c,Negative agreement=2d2d+b+c.\text{Positive agreement}=\frac{2a}{2a+b+c},\qquad \text{Negative agreement}=\frac{2d}{2d+b+c}.

These two measures are symmetric in X and Y. If a denominator is zero, the corresponding estimate is undefined. The difference in positive proportions is

Δ=a+cn−a+bn=c−bn.\Delta=\frac{a+c}{n}-\frac{a+b}{n}=\frac{c-b}{n}.

The app multiplies Δ\Delta by 100 to report percentage points. Negative values indicate a lower positive classification rate for Y. Positive and negative agreement remain meaningful when both methods are candidates and there is no verified reference classification. Estimating diagnostic sensitivity and specificity requires defining an independent reference outcome.

Exact McNemar test and Holm adjustment

The null hypothesis states that the two directions of discordance are equally probable. Conditional on m=b+cm=b+c discordant pairs, the count in one direction follows Binomial⁡(m,1/2)\operatorname{Binomial}(m,1/2) under the null.

pexact=min⁡ ⁣(1, 2∑j=0min⁡(b,c)(mj)2−m).p_{\mathrm{exact}}=\min\!\left(1,\ 2\sum_{j=0}^{\min(b,c)}\binom{m}{j}2^{-m}\right).

With m=0m=0, the p-value is 1. The test addresses a difference in marginal positive rates. For example, b=c=20b=c=20 shows balanced positive rates despite 40 disagreements. Inspect the agreement statistics and the interval for Δ\Delta alongside the p-value.

For each classification rule, DataStatPro applies Holm adjustment across the method pairs. For MM ordered raw p-values p(1)≤⋯≤p(M)p_{(1)}\le\cdots\le p_{(M)},

p(i)Holm=min⁡ ⁣(1,max⁡1≤j≤i(M−j+1)p(j)).p^{\mathrm{Holm}}_{(i)}=\min\!\left(1, \max_{1\le j\le i}(M-j+1)p_{(j)}\right).

The adjusted p-values account for the family of pair tests at that rule. Each cutoff and the inclusive band define their own family. R documentation for Holm’s method

Cohen’s kappa and ordinal weighted kappa

Cohen’s kappa compares observed agreement Po=(a+d)/nP_o=(a+d)/n with the expected agreement from the marginal frequencies:

Pe=(a+b)(a+c)+(c+d)(b+d)n2,κ=Po−Pe1−Pe.P_e=\frac{(a+b)(a+c)+(c+d)(b+d)}{n^2},\qquad \kappa=\frac{P_o-P_e}{1-P_e}.

Kappa equals 1 for perfect agreement when defined. Zero corresponds to the agreement expected from those marginals. Kappa can be negative when observed agreement is lower than expected. It depends on the category frequencies, so include counts and observed agreement when reporting it. When Pe=1P_e=1, the denominator is zero.

With several cutoffs, the app assigns each reading to an ordered band and builds a matched k×kk\times k table. For example, cutoffs 30 and 50 define x<30x<30, 30≤x<5030\le x<50, and x≥50x\ge50. For cell proportion pijp_{ij} and marginal proportions pi+p_{i+} and p+jp_{+j}, the linear weights are

wij=1−∣i−j∣k−1,Po(w)=∑ijwijpij,Pe(w)=∑ijwijpi+p+j.w_{ij}=1-\frac{|i-j|}{k-1},\qquad P_o^{(w)}=\sum_{ij}w_{ij}p_{ij},\qquad P_e^{(w)}=\sum_{ij}w_{ij}p_{i+}p_{+j}.

The weighted kappa substitutes these weighted agreements into (Po(w)−Pe(w))/(1−Pe(w))(P_o^{(w)}-P_e^{(w)})/(1-P_e^{(w)}). Adjacent bands receive partial credit and the most distant bands receive none. The table also reports exact agreement, which counts only diagonal entries.

Wilson interval for a single proportion

Band membership and subgroup discordance use Wilson score intervals. For kk events among nn observations, put p^=k/n\hat p=k/n. The 95% interval is

p^+z22n ± zp^(1−p^)n+z24n21+z2n.\frac{\hat p+\frac{z^2}{2n}\ \pm\ z\sqrt{\frac{\hat p(1-\hat p)}{n}+\frac{z^2}{4n^2}}}{1+\frac{z^2}{n}}.

It can express uncertainty even when the observed proportion is zero or one. The interval for a difference in paired positive rates comes from the paired bootstrap described above.

8. Covariate and group analyses

Subgroup tables summarize average readings by category and, when boundaries are supplied, by covariate band. These are separate summaries for each variable. With cutoff analyses enabled, they also report the fraction of matched pairs that disagree within each subgroup. Adjusted models use retained specimens with both covariate and group values available and at least two observed groups.

Let Ci=ci−cˉC_i=c_i-\bar c, where cic_i is the selected numeric covariate, and let GigG_{ig} indicate membership in group gg. The omitted group is the selected baseline. For each method reading viv_i the concentration model is

vi=β0+βCCi+∑g≠baselineβgGig+εi.v_i=\beta_0+\beta_C C_i+\sum_{g\ne\mathrm{baseline}}\beta_g G_{ig}+\varepsilon_i.

The covariate coefficient gives the expected change per one unit of the selected variable at a fixed group. Each group coefficient gives the expected difference from the reference group at a common covariate value. The fitted group lines have a common slope in this model.

For each method pair, the adjusted difference model also includes the centered pair mean.

di=γ0+γCCi+∑g≠baselineγgGig+γM(mi−mˉ)+εi.d_i=\gamma_0+\gamma_C C_i+\sum_{g\ne\mathrm{baseline}}\gamma_g G_{ig} +\gamma_M(m_i-\bar m)+\varepsilon_i.

Its covariate and group coefficients describe changes in Y−XY-X at a common mean measurement level. The quadratic covariate sensitivity model adds γQCi2\gamma_Q C_i^2. These estimates describe associations with readings or differences. Their interpretation depends on the observed covariate values, groups and measurement ranges.

HC3 inference

The coefficients are fitted by least squares. For a design matrix ZZ, residuals eie_i, and leverage hiih_{ii} from the diagonal of H=Z(ZTZ)−1ZTH=Z(Z^\mathsf{T}Z)^{-1}Z^\mathsf{T}, the HC3 covariance estimate is

Var⁡^HC3(β^)=(ZTZ)−1ZTdiag⁡ ⁣(ei2(1−hii)2)Z(ZTZ)−1.\widehat{\operatorname{Var}}_{\mathrm{HC3}}(\hat\beta)= (Z^\mathsf{T}Z)^{-1}Z^\mathsf{T} \operatorname{diag}\!\left(\frac{e_i^2}{(1-h_{ii})^2}\right) Z(Z^\mathsf{T}Z)^{-1}.

A coefficient CI uses β^j±t0.975,n−pSEHC3(β^j)\hat\beta_j\pm t_{0.975,n-p}\mathrm{SE}_{\mathrm{HC3}}(\hat\beta_j), where pp is the number of fitted coefficients. Holm adjustment uses separate families for the concentration models, the covariate terms in paired differences, the group contrasts in paired differences, and the quadratic covariate terms. Review the model sample size when comparing these estimates with the wider study.

9. Worked examples

Example 1. Continuous agreement across the measurement range

Enter the following six synthetic pairs in a dataset. Assign the first method to X and the second to Y.

SpecimenXYPaired meanY − X
1101110.51
2202321.53
3303231.02
4404341.53
5505653.06
6606562.55
  1. Mean difference. The differences sum to 20, giving dˉ=20/6=3.333\bar d=20/6=3.333 units. The sample SD is 1.8621.862 units.
  2. Limits of agreement. Calculate 3.333±1.96(1.862)3.333\pm1.96(1.862) to get approximately [−0.316, 6.983][-0.316,\ 6.983] units.
  3. Passing–Bablok fit. All 15 pairwise slopes are eligible and none fall below −1-1. The eighth slope in sorted order is 1.101.10. The six values of yi−1.10xiy_i-1.10x_i are 0,1,−1,−1,1,−10,1,-1,-1,1,-1, giving median −0.50-0.50. The fitted line is y^=−0.50+1.10x\hat y=-0.50+1.10x.
  4. Interpretation. The estimated difference increases from 1.51.5 units at X=20X=20 to 4.54.5 at X=50X=50. If the prespecified tolerance were ±5\pm 5 units, the estimated upper agreement limit would exceed it. With six pairs, the endpoints are imprecise. Inspect the CIs and the difference plot before drawing a conclusion.

Example 2. Cutoff agreement and McNemar’s test

Suppose 30 specimens are classified positive below a cutoff of 23.3. X and Y classify 17 specimens positive together and 7 negative together. Six are positive only on X and zero only on Y.

ClassificationY < 23.3Y ≥ 23.3
X < 23.3176
X ≥ 23.307
  • Overall agreement is (17+7)/30=80%(17+7)/30=80\%.
  • Positive agreement is 34/(34+6)=85%34/(34+6)=85\%. Negative agreement is 14/(14+6)=70%14/(14+6)=70\%.
  • Positive rate difference is (0−6)/30=−20(0-6)/30=-20 percentage points. Y classifies fewer specimens as positive.
  • Cohen’s kappa is approximately 0.5690.569. The expected agreement from the margins is (23⋅17+7⋅13)/302≈0.536(23\cdot17+7\cdot13)/30^2\approx0.536.
  • Exact McNemar p is 2(1/2)6=0.031252(1/2)^6=0.03125. With one pair, the Holm-adjusted p-value is the same.

The table shows that all disagreements occur in one direction. The continuous analysis helps determine how large the underlying measurement differences are and where they arise.

10. Reporting and troubleshooting

Export APA Report creates the Word report after a successful analysis. In a report, state the specimen count, complete-case rule, units, direction of subtraction and acceptable measurement differences. Report mean bias and limits with their CIs, Passing–Bablok coefficients with interval methods and CUSUM p-value, and cutoff rules with their matched counts and signed differences when applicable. Record the resample count and seed for reproducibility.

SituationWhat to check
Too few complete rowsCheck units, numeric column types, row matching, and missingness across all selected methods. At least three complete rows are required.
Regression coefficient or interval unavailableInspect ties, measurement range, the direction of association and the availability of valid bootstrap estimates.
CUSUM unavailableCheck whether the fit has residuals on both sides of the line.
Log-ratio summary unavailableCheck for zero or negative values in either method.
Kappa unavailable or small despite high agreementInspect category frequencies, off-diagonal counts and the expected agreement.
Adjusted model unavailableCheck the joint availability of covariate and group values, group levels, sample size and variation within the selected data.
Cutoff suggestions missing or collapsedCheck the distinct values and their range. Enter study-specific bounds when the pooled data provide insufficient distinct rounded values.
Need to start a new comparisonUse Reset to clear selections, settings and results while keeping the active dataset.

11. Quick reference

OutputWhat to read first
Bland–AltmanDirection of mean bias and whether both limits fit the tolerance
Passing–BablokSlope relative to 1, intercept relative to 0, and fitted differences at relevant concentrations
CUSUMEvidence of nonlinear departures alongside the fitted plot
Deming sensitivityChanges in the fit across plausible Y/X error variance ratios
Matched binary agreementCounts a,b,c,da,b,c,d, positive and negative agreement, and signed rate difference
McNemar testAsymmetry of the discordant counts and Holm-adjusted p-value across pairs
Ordinal agreementMatched bands, exact agreement and linear weighted kappa
Subgroups and modelsSample size, baseline group, effect units, HC3 CI and Holm-adjusted p-value