Skip to contents

Validating whole-slide imaging means showing that pathologists reach the same scores on digital images as on glass slides. This article shows how to analyse such a study with the Interrater Reliability analysis (agreement(), meddecide > Agreement > Interrater Reliability), using HER2 immunohistochemistry scored on a five-level ordinal scale (Null, Ultralow, 1+, 2+, 3+), the setting studied by Xiao et al. (2025).

The example file her2_agreement_data.csv (100 cases; columns pathologist_a, pathologist_b, pathologist_c, glass_slides, digital_images) is in the data-raw/non-rda folder of the ClinicoPath repository. The code is shown without output.

her2 <- read.csv("her2_agreement_data.csv")
her2_levels <- c("Null", "Ultralow", "1+", "2+", "3+")
for (v in c("pathologist_a", "pathologist_b", "pathologist_c",
            "glass_slides", "digital_images")) {
  her2[[v]] <- factor(her2[[v]], levels = her2_levels, ordered = TRUE)
}

Declaring the factor levels in clinical order matters: weighted kappa and the ordinal statistics use that order.

1. Inter-observer agreement on one modality

library(meddecide)

res <- agreement(
  data = her2,
  vars = c("pathologist_a", "pathologist_b", "pathologist_c"),
  kripp = TRUE,
  krippMethod = "ordinal",
  allPairsKappa = TRUE,
  wght = "squared",          # applied to the all-pairs Cohen's kappa
  agreementHeatmap = TRUE
)

res$krippTableGroup$krippTable$asDF   # ordinal Krippendorff's alpha, 3 raters
res$allPairsKappaTable$asDF           # quadratic-weighted kappa per pathologist pair

With three raters the headline Fleiss’ kappa is unweighted; when weights are requested, the analysis reports them per pair (allPairsKappa) instead.

2. Glass versus digital

Treat the two modalities as two “raters” of the same cases.

res_mod <- agreement(
  data = her2,
  vars = c("glass_slides", "digital_images"),
  wght = "squared",
  sft = TRUE,                 # frequency (contingency) table
  specificAgreement = TRUE,   # agreement within each HER2 category
  stuartMaxwell = TRUE,       # marginal homogeneity: does one modality score higher?
  bhapkar = TRUE
)

res_mod$irrtable$asDF
res_mod$specificAgreementGroup$specificAgreementTable$asDF
res_mod$raterBiasGroup$stuartMaxwellTable$asDF
  • Weighted kappa measures overall concordance, giving partial credit to one-step differences.
  • Specific agreement shows where the modalities disagree; for HER2 the Null/Ultralow boundary matters most, because it decides eligibility for HER2-directed antibody-drug conjugates.
  • Stuart-Maxwell / Bhapkar test whether the two modalities have the same category distribution. A significant result means one modality systematically scores higher or lower; inspect the contingency table to see which way.

3. Comparing agreement under two conditions

If every pathologist scored all cases on glass and again on digital images, the paired agreement test compares the panel’s kappa between the two conditions using a bootstrap over cases. Put the condition-A rater columns in vars and the condition-B columns, in the same rater order, in conditionBVars.

agreement(
  data = study,                                     # one row per case
  vars = c("glass_rater1", "glass_rater2", "glass_rater3"),
  conditionBVars = c("digital_rater1", "digital_rater2", "digital_rater3"),
  pairedAgreementTest = TRUE,
  pairedBootN = 2000
)$pairedAgreementGroup$pairedAgreementTable$asDF

4. Consensus reference

In jamovi, Computed Variables > Create Consensus Variable adds a consensus column to the data set (rule: consensusRule = "majority", "supermajority" or "unanimous"; name: consensusName). Cases that do not meet the rule get no consensus label. This column is a jamovi computed variable; it is not returned when agreement() is called from R.

Interpreting kappa

Landis and Koch (1977) bands, a descriptive convention rather than a validated clinical threshold:

Kappa Label
< 0 Poor
0.00 - 0.20 Slight
0.21 - 0.40 Fair
0.41 - 0.60 Moderate
0.61 - 0.80 Substantial
0.81 - 1.00 Almost perfect

Whether a given level is good enough for clinical use is a decision for the validation protocol (for whole-slide imaging, see the CAP guideline, Evans et al. 2022), judged on the lower confidence limit rather than the point estimate.

References

  • Xiao A, Vohra P, Chen YY, Ung L, Kim MO, Geradts J. Comparative study of intra- and inter-observer variability in manual scoring of HER2 immunohistochemical stains on glass slides versus paired digital images with emphasis on the low end of the expression spectrum. Hum Pathol. 2025;161:105860.
  • Evans AJ, Brown RW, Bui MM, et al. Validating whole slide imaging systems for diagnostic purposes in pathology: guideline update from the College of American Pathologists in collaboration with the American Society for Clinical Pathology and the Association for Pathology Informatics. Arch Pathol Lab Med. 2022;146(4):440-450.
  • Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.