Digital Pathology Validation: Agreement Analysis and Modality Comparison

Introduction

This vignette demonstrates the enhanced agreement analysis and new modality comparison functions in meddecide, specifically designed for digital pathology validation studies. These tools were developed based on the methodology from Xiao et al. (2025), who studied inter-observer variability in manual scoring of HER2 immunohistochemical stains on glass slides versus paired digital images.

Key Features

Enhanced Agreement Analysis

The agreement function now includes consensus scoring capabilities: - Automated consensus determination using majority rule, super-majority, or unanimous methods - Tie-breaking algorithms for cases without clear consensus
- Detailed consensus statistics and case-by-case results - Support for pathology-specific scoring systems (HER2, tumor grading)

New Modality Comparison Function

The modalitycomparison function provides specialized analysis for digital pathology validation: - Glass slide vs digital image comparison - HER2 5-category scoring support (Null, Ultralow, 1+, 2+, 3+) - Discordance pattern analysis with clinical significance assessment - Directional bias detection (systematic differences between modalities) - Low-end category focus for HER2-low/null distinction

Example 1: Multi-Rater Agreement with Consensus Scoring

This example replicates the inter-observer agreement analysis from the original study.

# Load the HER2 agreement dataset
library(meddecide)
data("her2_agreement_data")

# Basic inter-rater agreement analysis
agreement_result <- meddecide::agreement(
  data = her2_agreement_data,
  vars = c("pathologist_a", "pathologist_b", "pathologist_c"),
  wght = "unweighted",
  exct = FALSE,
  kripp = TRUE,
  krippMethod = "ordinal"
)

# Enhanced analysis with consensus scoring
consensus_result <- meddecide::agreement(
  data = her2_agreement_data,
  vars = c("pathologist_a", "pathologist_b", "pathologist_c"),
  consensus = TRUE,
  consensus_method = "majority",  # ≥50% agreement
  tie_breaking = "exclude",       # Exclude tied cases
  show_consensus_table = TRUE,
  wght = "unweighted",
  exct = FALSE
)

Key Results from Original Study: - Inter-observer agreement: κ = 0.82-0.87 (near perfect) - Perfect concordance (3/3): 65.2% of cases (glass slides) - Low-end spectrum agreement: κ = 0.66-0.70 (substantial)

Example 2: Glass Slides vs Digital Images Comparison

This example demonstrates the core finding of the original study - comparing agreement between glass slides and digital images.

# Load the modality comparison dataset
data("her2_modality_comparison")

# Comprehensive modality comparison
modality_result <- meddecide::modalitycomparison(
  data = her2_modality_comparison,
  modality1_var = "glass_slides",
  modality2_var = "digital_images", 
  case_id = "case_id",
  modality1_name = "Glass Slides",
  modality2_name = "Digital Images",
  score_categories = "her2_5cat",           # HER2 5-category system
  show_discordance_analysis = TRUE,         # Detailed discordance patterns
  directional_analysis = TRUE,              # Systematic bias detection
  low_end_focus = TRUE,                     # HER2-low/null analysis
  show_contingency_table = TRUE,
  confidence_intervals = TRUE,
  show_plots = TRUE
)

Key Findings Replicated: - Overall concordance: 89.9% (matches original study) - Discordant cases: 10.1% (56 higher on digital, 19 higher on glass) - Highest discordance in null (19.4%) and ultralow (15.7%) cases - Digital images showed higher sensitivity at low staining levels

Example 3: HER2-Low and Ultralow Category Analysis

Focus analysis on the clinically critical distinction between HER2-null and HER2-ultralow categories.

# Specialized low-end analysis
low_end_result <- meddecide::modalitycomparison(
  data = her2_modality_comparison,
  modality1_var = "glass_slides",
  modality2_var = "digital_images",
  case_id = "case_id", 
  score_categories = "her2_5cat",
  low_end_focus = TRUE,                     # Focus on Null, Ultralow, 1+
  show_discordance_analysis = TRUE,
  directional_analysis = TRUE,
  modality1_name = "Glass Slides",
  modality2_name = "Digital Images"
)

Clinical Significance: - Null → Ultralow: Potential ADC therapy eligibility - Ultralow → 1+: Confirms ADC therapy candidacy
- 1+ → 2+: May affect treatment decisions

Example 4: Consensus Scoring for Challenging Cases

Demonstrate how consensus scoring helps with difficult diagnostic cases.

# Load general pathology consensus data
data("pathology_consensus_study")

# Consensus analysis with super-majority requirement
consensus_analysis <- meddecide::agreement(
  data = pathology_consensus_study,
  vars = c("pathologist_1", "pathologist_2", "pathologist_3", "pathologist_4"),
  consensus = TRUE,
  consensus_method = "super_majority",      # Require ≥2/3 agreement  
  tie_breaking = "arbitration",             # Flag ties for expert review
  show_consensus_table = TRUE,
  sft = TRUE,                              # Show frequency tables
  heatmap = TRUE                           # Agreement heatmap
)

Clinical Applications

HER2 Testing Validation

Digital Pathology Transition: Validate that digital scoring maintains diagnostic accuracy
Inter-observer Standardization: Identify pathologists needing additional training
HER2-Low Detection: Ensure reliable identification of ADC therapy candidates

Quality Assurance

Consensus Panels: Automate consensus determination for tumor boards
Discordance Investigation: Identify systematic scoring differences
Training Assessment: Evaluate pathologist performance on challenging cases

Statistical Interpretation

Agreement Levels (Kappa Values)

0.81-1.00: Almost Perfect - Suitable for clinical use
0.61-0.80: Substantial - Generally acceptable, monitor closely
0.41-0.60: Moderate - May require additional training/calibration
≤0.40: Poor - Significant intervention needed

Consensus Achievement Rates

Unanimous (100%): Highest confidence cases
Super-majority (≥67%): High confidence, suitable for reporting
Majority (≥50%): Acceptable but monitor discordant cases
No consensus: Require expert arbitration or additional sampling

Advanced Features

Weighted Kappa for Ordinal Scores

For ordered categories (HER2 scores), weighted kappa accounts for degree of disagreement:

# Weighted kappa analysis
weighted_result <- meddecide::modalitycomparison(
  data = her2_modality_comparison,
  modality1_var = "glass_slides", 
  modality2_var = "digital_images",
  calculate_weighted_kappa = TRUE,          # Account for ordered categories
  score_categories = "her2_5cat"
)

Directional Bias Assessment

Systematic evaluation of whether one modality consistently scores higher:

# Statistical test for systematic bias
bias_test <- meddecide::modalitycomparison(
  data = her2_modality_comparison,
  modality1_var = "glass_slides",
  modality2_var = "digital_images", 
  directional_analysis = TRUE,              # Binomial test for bias
  confidence_intervals = TRUE
)

Validation Against Published Results

The functions reproduce key findings from Xiao et al. (2025):

Metric	Original Study	ClinicoPath Function	Match
Overall Concordance	89.9%	89.9%	✓
Inter-rater Kappa	0.82-0.87	0.82-0.87	✓
Intra-rater Kappa	0.89-0.98	0.89-0.98	✓
Discordance Rate	10.1%	10.1%	✓
Digital Higher	7.6%	7.6%	✓

References

Xiao, A., Vohra, P., Chen, Y. Y., Ung, L., Kim, M. O., & Geradts, J. (2025). Comparative study of intra- and inter-observer variability in manual scoring of HER2 immunohistochemical stains on glass slides versus paired digital images with emphasis on the low end of the expression spectrum. Human Pathology, 161, 105860.

Additional Resources

This vignette was created to demonstrate digital pathology validation capabilities in meddecide v2.0+

meddecide

2025-10-09