Digital Pathology Validation: Agreement Between Glass and Digital Reads
Source:vignettes/meddecide-09-digital-pathology-validation.Rmd
meddecide-09-digital-pathology-validation.RmdValidating whole-slide imaging means showing that pathologists reach
the same scores on digital images as on glass slides. This article shows
how to analyse such a study with the Interrater
Reliability analysis (agreement(),
meddecide > Agreement > Interrater Reliability),
using HER2 immunohistochemistry scored on a five-level ordinal scale
(Null, Ultralow, 1+, 2+, 3+), the setting studied by Xiao et
al. (2025).
The example file her2_agreement_data.csv (100 cases;
columns pathologist_a, pathologist_b,
pathologist_c, glass_slides,
digital_images) is in the data-raw/non-rda
folder of the ClinicoPath
repository. The code is shown without output.
her2 <- read.csv("her2_agreement_data.csv")
her2_levels <- c("Null", "Ultralow", "1+", "2+", "3+")
for (v in c("pathologist_a", "pathologist_b", "pathologist_c",
"glass_slides", "digital_images")) {
her2[[v]] <- factor(her2[[v]], levels = her2_levels, ordered = TRUE)
}Declaring the factor levels in clinical order matters: weighted kappa and the ordinal statistics use that order.
1. Inter-observer agreement on one modality
library(meddecide)
res <- agreement(
data = her2,
vars = c("pathologist_a", "pathologist_b", "pathologist_c"),
kripp = TRUE,
krippMethod = "ordinal",
allPairsKappa = TRUE,
wght = "squared", # applied to the all-pairs Cohen's kappa
agreementHeatmap = TRUE
)
res$krippTableGroup$krippTable$asDF # ordinal Krippendorff's alpha, 3 raters
res$allPairsKappaTable$asDF # quadratic-weighted kappa per pathologist pairWith three raters the headline Fleiss’ kappa is unweighted; when
weights are requested, the analysis reports them per pair
(allPairsKappa) instead.
2. Glass versus digital
Treat the two modalities as two “raters” of the same cases.
res_mod <- agreement(
data = her2,
vars = c("glass_slides", "digital_images"),
wght = "squared",
sft = TRUE, # frequency (contingency) table
specificAgreement = TRUE, # agreement within each HER2 category
stuartMaxwell = TRUE, # marginal homogeneity: does one modality score higher?
bhapkar = TRUE
)
res_mod$irrtable$asDF
res_mod$specificAgreementGroup$specificAgreementTable$asDF
res_mod$raterBiasGroup$stuartMaxwellTable$asDF- Weighted kappa measures overall concordance, giving partial credit to one-step differences.
- Specific agreement shows where the modalities disagree; for HER2 the Null/Ultralow boundary matters most, because it decides eligibility for HER2-directed antibody-drug conjugates.
- Stuart-Maxwell / Bhapkar test whether the two modalities have the same category distribution. A significant result means one modality systematically scores higher or lower; inspect the contingency table to see which way.
3. Comparing agreement under two conditions
If every pathologist scored all cases on glass and again on digital
images, the paired agreement test compares the panel’s kappa between the
two conditions using a bootstrap over cases. Put the condition-A rater
columns in vars and the condition-B columns, in the same
rater order, in conditionBVars.
4. Consensus reference
In jamovi, Computed Variables > Create Consensus
Variable adds a consensus column to the data set (rule:
consensusRule = "majority",
"supermajority" or "unanimous"; name:
consensusName). Cases that do not meet the rule get no
consensus label. This column is a jamovi computed variable; it is not
returned when agreement() is called from R.
Interpreting kappa
Landis and Koch (1977) bands, a descriptive convention rather than a validated clinical threshold:
| Kappa | Label |
|---|---|
| < 0 | Poor |
| 0.00 - 0.20 | Slight |
| 0.21 - 0.40 | Fair |
| 0.41 - 0.60 | Moderate |
| 0.61 - 0.80 | Substantial |
| 0.81 - 1.00 | Almost perfect |
Whether a given level is good enough for clinical use is a decision for the validation protocol (for whole-slide imaging, see the CAP guideline, Evans et al. 2022), judged on the lower confidence limit rather than the point estimate.
References
- Xiao A, Vohra P, Chen YY, Ung L, Kim MO, Geradts J. Comparative study of intra- and inter-observer variability in manual scoring of HER2 immunohistochemical stains on glass slides versus paired digital images with emphasis on the low end of the expression spectrum. Hum Pathol. 2025;161:105860.
- Evans AJ, Brown RW, Bui MM, et al. Validating whole slide imaging systems for diagnostic purposes in pathology: guideline update from the College of American Pathologists in collaboration with the American Society for Clinical Pathology and the Association for Pathology Informatics. Arch Pathol Lab Med. 2022;146(4):440-450.
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.