Interrater Reliability & Agreement Analysis - Comprehensive Guide
Source:vignettes/meddecide-agreement-comprehensive.Rmd
meddecide-agreement-comprehensive.RmdInterrater Reliability & Agreement Analysis
Overview
The Agreement module (agreement)
provides a comprehensive suite of interrater reliability and agreement
statistics for clinicopathological research. It supports categorical,
ordinal, and continuous data with 2 or more raters, and includes
advanced features such as hierarchical kappa, mixed-effects comparisons,
bootstrap confidence intervals, and built-in sample size
calculators.
All examples below use the test datasets in the
data-raw/non-rda/ folder of the ClinicoPath
repository; on this website the code is shown without output. In
jamovi, open the corresponding .omv file. From R, read the
.csv directly.
Datasets Used in This Guide
| Dataset | Type | Raters | Description |
|---|---|---|---|
agreement_pathology |
Categorical | 2 | Surgical pathology diagnoses (Benign/Low-grade/High-grade) |
agreement_binary |
Binary | 2 | Positive/Negative classification |
agreement_ordinal |
Ordinal | 2 | Tumor grading (Grade 1-3) |
agreement_continuous |
Continuous | 2 | Continuous measurements |
agreement_threeRater |
Categorical | 3 | Three-rater panel |
agreement_multiRater |
Categorical | 5 | Five pathologists of varying experience |
agreement_hierarchical |
Categorical | 6 | Multi-center (3 hospitals, 2 raters each) |
agreement_testRetest |
Categorical | 6 | Test-retest (3 raters x 2 timepoints) |
agreement_missing |
Categorical | 3 | Dataset with missing values |
agreement_paired_comparison |
Categorical | 4 | Manual vs AI rater comparison |
agreement_mixed_effects |
Continuous | 2 | Pre/Post training condition comparison |
agreement_heatmap_test |
Ordinal | 3 | HER2 scoring for heatmap visualization |
agreement_perfect |
Categorical | 2 | Perfect agreement (edge case) |
agreement_poor |
Categorical | 2 | Poor agreement |
agreement_small |
Binary | 2 | Small sample (n=30) |
pathology_agreement_main |
Continuous | 3 | Ki-67 scoring (HALO, Aiforia, Manual) |
pathology_agreement_multimethod |
Continuous | 4 | Ki-67 multi-platform |
pathology_agreement_ai |
Continuous | 2 | AI vs pathologist Ki-67 |
pathology_agreement_missing |
Continuous | 3 | Continuous data with NAs |
comprehensive_agreement_data |
Categorical | 4 | Multi-specialty with metadata |
digital_pathology_validation |
Continuous | 2 | Digital pathology AI validation |
1. Categorical Agreement (Default Kappa)
The default analysis computes Cohen’s kappa (2 raters) or Fleiss’ kappa (3+ raters) with percent agreement.
Basic two-rater kappa
# Dataset: agreement_pathology
# Raters: Pathologist1, Pathologist2
# Diagnoses: Benign, Low-grade malignant, High-grade malignant
agreement_pathology <- read.csv(paste0(data_path, "agreement_pathology.csv"))
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2)
)With frequency table and additional statistics
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
sft = TRUE, # Show contingency (frequency) table
pabak = TRUE, # Prevalence-Adjusted Bias-Adjusted Kappa
gwet = TRUE, # Gwet's AC1/AC2 (stable with prevalence imbalance)
gwetWeights = "unweighted", # or "linear", "quadratic"
specificAgreement = TRUE, # Category-specific agreement (PSA/NSA)
specificAllCategories = TRUE, # Show for all categories
specificConfidenceIntervals = TRUE,
raterBias = TRUE, # McNemar test for systematic rater bias
bhapkar = TRUE, # Bhapkar test (multivariate extension)
stuartMaxwell = TRUE, # Stuart-Maxwell marginal homogeneity test
agreementHeatmap = TRUE, # Visual heatmap of cross-tabulation
heatmapColorScheme = "bluered", # Color scheme (bluered/viridis/heat/grayscale)
heatmapShowPercentages = TRUE,
heatmapShowCounts = TRUE,
heatmapAnnotationSize = 3.5
)With subgroup analysis
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
agreementBySubgroup = TRUE,
subgroupVariable = specimen_type, # Compare kappa across specimen types
subgroupForestPlot = TRUE, # Forest plot of subgroup kappas
subgroupMinCases = 10 # Minimum cases per subgroup
)2. Binary Agreement
# Dataset: agreement_binary
# Raters: PathologistX, PathologistY (Positive/Negative)
agreement_binary <- read.csv(paste0(data_path, "agreement_binary.csv"))
agreement(
data = agreement_binary,
conditionBVars = NULL,
vars = vars(PathologistX, PathologistY),
specificAgreement = TRUE,
specificPositiveCategory = "Positive", # Category for PSA/NSA
specificConfidenceIntervals = TRUE,
pabak = TRUE,
agreementBySubgroup = TRUE,
subgroupVariable = specimen_quality
)3. Ordinal / Weighted Kappa
For ordinal data (e.g., tumor grades), weighted kappa accounts for the magnitude of disagreement. A Grade 1 vs Grade 3 mismatch is penalized more heavily than Grade 1 vs Grade 2.
# Dataset: agreement_ordinal
# Raters: PathologistA, PathologistB (Grade 1/2/3)
agreement_ordinal <- read.csv(paste0(data_path, "agreement_ordinal.csv"))
# Convert to ordered factors (required for weighted kappa & rank-based methods)
grade_levels <- c("Grade 1", "Grade 2", "Grade 3", "Grade 4")
agreement_ordinal$PathologistA <- factor(agreement_ordinal$PathologistA,
levels = grade_levels, ordered = TRUE)
agreement_ordinal$PathologistB <- factor(agreement_ordinal$PathologistB,
levels = grade_levels, ordered = TRUE)
# Linear weights
agreement(
data = agreement_ordinal,
conditionBVars = NULL,
vars = vars(PathologistA, PathologistB),
wght = "equal", # Linear weights
showLevelInfo = TRUE, # Show category-level details
kendallW = TRUE, # Kendall's W (coefficient of concordance)
agreementBySubgroup = TRUE,
subgroupVariable = tumor_site
)
# Quadratic weights (more forgiving of near-misses)
agreement(
data = agreement_ordinal,
conditionBVars = NULL,
vars = vars(PathologistA, PathologistB),
wght = "squared"
)4. Three or More Raters (Fleiss’ / Light’s Kappa)
When 3+ raters are present, the module automatically computes Fleiss’ kappa. Light’s kappa is the average of all pairwise Cohen’s kappas.
# Dataset: agreement_threeRater
# Raters: Rater1, Rater2, Rater3
agreement_threeRater <- read.csv(paste0(data_path, "agreement_threeRater.csv"))
agreement(
data = agreement_threeRater,
conditionBVars = NULL,
vars = vars(Rater1, Rater2, Rater3),
exct = TRUE, # Exact kappa (small samples)
lightKappa = TRUE, # Light's Kappa (mean pairwise)
finn = TRUE, # Finn coefficient
finnLevels = 3, # Number of response categories
finnModel = "oneway", # "oneway" or "twoway"
multiAnnotatorConcordance = TRUE, # Multi-annotator concordance (F1)
predictionRater = "Rater1", # Which rater is the "prediction"
agreementBySubgroup = TRUE,
subgroupVariable = tissue_site
)Five raters with clustering and profiles
# Dataset: agreement_multiRater
# Raters: SeniorPath1, SeniorPath2, MidLevelPath, JuniorPath1, JuniorPath2
agreement_multiRater <- read.csv(paste0(data_path, "agreement_multiRater.csv"))
agreement(
data = agreement_multiRater,
conditionBVars = NULL,
vars = vars(SeniorPath1, SeniorPath2, MidLevelPath, JuniorPath1, JuniorPath2),
lightKappa = TRUE,
# Rater clustering
raterClustering = TRUE,
clusterMethod = "hierarchical", # or "kmeans"
clusterDistance = "correlation", # or "euclidean", "manhattan"
clusterLinkage = "average", # or "complete", "ward.D2"
nClusters = 2,
showDendrogram = TRUE,
showClusterHeatmap = TRUE,
# Case clustering
caseClustering = TRUE,
caseClusterMethod = "hierarchical",
caseClusterDistance = "correlation",
caseClusterLinkage = "average",
nCaseClusters = 2,
showCaseDendrogram = TRUE,
showCaseClusterHeatmap = TRUE,
# Rater profiles
raterProfiles = TRUE,
raterProfileType = "boxplot", # "boxplot", "violin", or "barplot"
raterProfileShowPoints = TRUE,
# Subgroup
agreementBySubgroup = TRUE,
subgroupVariable = difficulty,
subgroupForestPlot = TRUE,
subgroupMinCases = 10
)5. Continuous Data (ICC / Bland-Altman / CCC)
For continuous measurements, kappa is not appropriate. The module automatically detects continuous data and suggests ICC, Bland-Altman, and CCC.
Two continuous raters
# Dataset: agreement_continuous
# Raters: MeasurementA, MeasurementB
agreement_continuous <- read.csv(paste0(data_path, "agreement_continuous.csv"))
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
# Intraclass Correlation Coefficient
icc = TRUE,
iccType = "icc21", # ICC(2,1) two-way random, single measures
# Bland-Altman analysis
blandAltmanPlot = TRUE,
baConfidenceLevel = 0.95,
proportionalBias = TRUE, # Test for proportional bias
# Lin's Concordance Correlation Coefficient
linCCC = TRUE,
# Total Deviation Index
tdi = TRUE,
tdiCoverage = 90, # Percentage coverage
tdiLimit = 10, # Acceptable limit
# Additional methods
meanPearson = TRUE, # Mean Pearson correlation
meanSpearman = TRUE, # Mean Spearman correlation
robinsonA = TRUE, # Robinson's A
maxwellRE = TRUE, # Maxwell's Random Effect model
iota = TRUE, # Iota index
iotaStandardize = TRUE, # Standardize before computing
# Subgroup
agreementBySubgroup = TRUE,
subgroupVariable = tumor_type
)Cycling through all ICC types
# ICC(1,1): One-way random, single measures
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc11"
)
# ICC(2,1): Two-way random, single measures (most common)
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc21"
)
# ICC(3,1): Two-way mixed, single measures
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc31"
)
# ICC(1,k): One-way random, average measures
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc1k"
)
# ICC(2,k): Two-way random, average measures
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc2k"
)
# ICC(3,k): Two-way mixed, average measures
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
icc = TRUE, iccType = "icc3k"
)Three continuous raters (Ki-67)
# Dataset: pathology_agreement_multimethod (using 3 of 4 raters)
# Raters: Ki67_HALO, Ki67_Aiforia, Ki67_ImageJ
pathology_multi <- read.csv(paste0(data_path, "pathology_agreement_multimethod.csv"))
agreement(
data = pathology_multi,
conditionBVars = NULL,
vars = vars(Ki67_HALO, Ki67_Aiforia, Ki67_ImageJ),
icc = TRUE,
meanPearson = TRUE,
raterProfiles = TRUE,
kripp = TRUE,
krippMethod = "interval" # For continuous data
)Four continuous raters with bootstrap
# Dataset: pathology_agreement_multimethod
# Raters: Ki67_HALO, Ki67_Aiforia, Ki67_ImageJ, Ki67_Manual
pathology_multi <- read.csv(paste0(data_path, "pathology_agreement_multimethod.csv"))
agreement(
data = pathology_multi,
conditionBVars = NULL,
vars = vars(Ki67_HALO, Ki67_Aiforia, Ki67_ImageJ, Ki67_Manual),
icc = TRUE,
raterClustering = TRUE,
caseClustering = TRUE,
bootstrapCI = TRUE,
nBoot = 100
)6. Krippendorff’s Alpha
Krippendorff’s alpha handles missing data natively and supports all measurement levels. It is the recommended general-purpose agreement statistic.
Categorical with missing data
# Dataset: agreement_missing
# Raters: Rater1, Rater2, Rater3 (with NAs)
agreement_missing <- read.csv(paste0(data_path, "agreement_missing.csv"))
agreement(
data = agreement_missing,
conditionBVars = NULL,
vars = vars(Rater1, Rater2, Rater3),
kripp = TRUE,
krippMethod = "nominal", # "nominal", "ordinal", "interval", "ratio"
bootstrap = TRUE, # Bootstrap CI for Krippendorff's alpha
bootstrapCI = TRUE,
nBoot = 100
)Continuous with missing data
# Dataset: pathology_agreement_missing
# Raters: HALO_Score, Aiforia_Score, ImageJ_Score
pathology_missing <- read.csv(paste0(data_path, "pathology_agreement_missing.csv"))
agreement(
data = pathology_missing,
conditionBVars = NULL,
vars = vars(HALO_Score, Aiforia_Score, ImageJ_Score),
kripp = TRUE,
krippMethod = "interval", # Continuous measurement level
icc = TRUE # Compare with ICC
)7. Gwet’s AC and PABAK
Gwet’s AC1/AC2 is more stable than kappa when prevalence is skewed. PABAK adjusts for both prevalence and bias.
# Dataset: agreement_poor (low kappa due to prevalence, not poor raters)
agreement_poor <- read.csv(paste0(data_path, "agreement_poor.csv"))
agreement(
data = agreement_poor,
conditionBVars = NULL,
vars = vars(PathologistA, PathologistB),
gwet = TRUE,
gwetWeights = "unweighted", # "unweighted", "linear", "quadratic"
pabak = TRUE,
agreementBySubgroup = TRUE,
subgroupVariable = difficulty_level
)8. Hierarchical / Multi-Center Kappa
For multi-center studies where raters are nested within institutions, hierarchical kappa decomposes agreement into within-center and between-center components.
Note: Hierarchical mixed-effects decomposition (variance decomposition, hierarchical ICC) requires continuous numeric ratings. For categorical data, use standard kappa or cluster-specific kappa analyses.
Continuous data: Full hierarchical decomposition
# Dataset: digital_pathology_validation (continuous Ki-67 scores)
# Raters: Ki67_Manual, Ki67_AI_Assisted
# Cluster variable: Institution (4 centers)
digital_path_hier <- read.csv(paste0(data_path, "digital_pathology_validation.csv"))
agreement(
data = digital_path_hier,
conditionBVars = NULL,
vars = vars(Ki67_Manual, Ki67_AI_Assisted),
hierarchicalKappa = TRUE,
clusterVariable = Institution, # Nesting variable
clusterSpecificKappa = TRUE, # Kappa per center
varianceDecomposition = TRUE, # Variance components
testClusterHomogeneity = TRUE, # Test if agreement differs across centers
clusterRankings = TRUE, # Rank centers by agreement
iccHierarchical = TRUE, # Hierarchical ICC
multipleTestCorrection = "bonferroni" # "none", "bonferroni", "bh", "holm"
)Categorical data: Pairwise and cluster-specific kappa
# Dataset: agreement_hierarchical (categorical diagnoses)
# Raters: HospitalA_Rater1/2, HospitalB_Rater1/2, HospitalC_Rater1/2
agreement_hierarchical <- read.csv(paste0(data_path, "agreement_hierarchical.csv"))
agreement(
data = agreement_hierarchical,
conditionBVars = NULL,
vars = vars(
HospitalA_Rater1, HospitalA_Rater2,
HospitalB_Rater1, HospitalB_Rater2,
HospitalC_Rater1, HospitalC_Rater2
),
pairwiseKappa = TRUE,
referenceRater = HospitalA_Rater1,
rankRaters = TRUE
)Pairwise kappa with reference rater
# Dataset: comprehensive_agreement_data
# Raters: Rater_1, Rater_2, Rater_3, Rater_4
comprehensive <- read.csv(paste0(data_path, "comprehensive_agreement_data.csv"))
agreement(
data = comprehensive,
conditionBVars = NULL,
vars = vars(Rater_1, Rater_2, Rater_3, Rater_4),
pairwiseKappa = TRUE,
referenceRater = Rater_1, # Compare all raters against this one
rankRaters = TRUE, # Rank raters by agreement with reference
agreementBySubgroup = TRUE,
subgroupVariable = Difficulty
)9. Test-Retest / Inter-Intra Rater Reliability
Separates inter-rater from intra-rater reliability using time-point
suffixes in variable names (e.g., Rater1_T1,
Rater1_T2).
# Dataset: agreement_testRetest
# Raters: Rater1_T1, Rater1_T2, Rater2_T1, Rater2_T2, Rater3_T1, Rater3_T2
agreement_testRetest <- read.csv(paste0(data_path, "agreement_testRetest.csv"))
agreement(
data = agreement_testRetest,
conditionBVars = NULL,
vars = vars(Rater1_T1, Rater1_T2, Rater2_T1, Rater2_T2, Rater3_T1, Rater3_T2),
interIntraRater = TRUE,
interIntraSeparator = "_", # Character separating rater from timepoint
finn = TRUE,
finnLevels = 3
)10. Paired Agreement Comparison (Manual vs AI)
Compare agreement under two conditions (e.g., manual vs AI-assisted) using a bootstrap test of kappa difference.
# Dataset: agreement_paired_comparison
# Condition A (vars): Manual_Rater1, Manual_Rater2
# Condition B (conditionBVars): AI_Rater1, AI_Rater2
agreement_paired <- read.csv(paste0(data_path, "agreement_paired_comparison.csv"))
agreement(
data = agreement_paired,
vars = vars(Manual_Rater1, Manual_Rater2),
conditionBVars = vars(AI_Rater1, AI_Rater2),
pairedAgreementTest = TRUE,
pairedBootN = 500, # Number of bootstrap replicates (minimum accepted is 500)
agreementBySubgroup = TRUE,
subgroupVariable = tumor_type
)11. Mixed-Effects Condition Comparison
Compare agreement between conditions (e.g., before vs after training) using mixed-effects models. Requires a condition variable in the dataset.
# Dataset: agreement_mixed_effects
# Raters: Rater1, Rater2
# Condition: condition (Pre_Training / Post_Training)
agreement_mixed <- read.csv(paste0(data_path, "agreement_mixed_effects.csv"))
agreement(
data = agreement_mixed,
conditionBVars = NULL,
vars = vars(Rater1, Rater2),
mixedEffectsComparison = TRUE,
conditionVariable = condition,
icc = TRUE,
multipleTestCorrection = "bh" # Benjamini-Hochberg (the option level is lower-case)
)Digital pathology validation
# Dataset: digital_pathology_validation
# Raters: Ki67_Manual, Ki67_AI_Assisted
# Condition: Pathologist_Experience
digital_path <- read.csv(paste0(data_path, "digital_pathology_validation.csv"))
agreement(
data = digital_path,
conditionBVars = NULL,
vars = vars(Ki67_Manual, Ki67_AI_Assisted),
mixedEffectsComparison = TRUE,
conditionVariable = Pathologist_Experience,
blandAltmanPlot = TRUE,
linCCC = TRUE,
agreementBySubgroup = TRUE,
subgroupVariable = Tumor_Type
)12. AI Validation
# Dataset: pathology_agreement_ai
# Raters: Ki67_AI, Ki67_Pathologist
pathology_ai <- read.csv(paste0(data_path, "pathology_agreement_ai.csv"))
agreement(
data = pathology_ai,
conditionBVars = NULL,
vars = vars(Ki67_AI, Ki67_Pathologist),
blandAltmanPlot = TRUE,
linCCC = TRUE,
icc = TRUE,
agreementBySubgroup = TRUE,
subgroupVariable = TumorType
)13. Agreement Heatmap
Dedicated visualization of cross-tabulated agreement patterns.
# Dataset: agreement_heatmap_test
# Raters: Scorer_A, Scorer_B, Scorer_C (HER2 scores: 0, 1+, 2+, 3+)
agreement_heatmap <- read.csv(paste0(data_path, "agreement_heatmap_test.csv"))
# Convert HER2 scores to ordered factors
her2_levels <- c("0", "1+", "2+", "3+")
agreement_heatmap$Scorer_A <- factor(agreement_heatmap$Scorer_A,
levels = her2_levels, ordered = TRUE)
agreement_heatmap$Scorer_B <- factor(agreement_heatmap$Scorer_B,
levels = her2_levels, ordered = TRUE)
agreement_heatmap$Scorer_C <- factor(agreement_heatmap$Scorer_C,
levels = her2_levels, ordered = TRUE)
agreement(
data = agreement_heatmap,
conditionBVars = NULL,
vars = vars(Scorer_A, Scorer_B, Scorer_C),
wght = "equal", # Linear weights for ordinal HER2
sft = TRUE, # Contingency table
agreementHeatmap = TRUE,
heatmapColorScheme = "bluered", # "bluered", "viridis", "heat", "grayscale"
heatmapShowPercentages = TRUE,
heatmapShowCounts = TRUE,
heatmapAnnotationSize = 3.5
)14. Confusion Matrix
# Dataset: agreement_pathology
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
confusionMatrix = TRUE,
confusionNormalize = "none" # "none", "row", "column"
)
# Row-normalized (sensitivity per category)
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
confusionMatrix = TRUE,
confusionNormalize = "row"
)15. Bootstrap Confidence Intervals
BCa (bias-corrected accelerated) bootstrap confidence intervals for all agreement statistics.
# Dataset: agreement_small (n=30)
agreement_small <- read.csv(paste0(data_path, "agreement_small.csv"))
agreement(
data = agreement_small,
conditionBVars = NULL,
vars = vars(Rater1, Rater2),
bootstrapCI = TRUE,
nBoot = 100, # Number of bootstrap replicates (100-5000)
confLevel = 0.95 # Confidence level
)16. Multi-Annotator Concordance (F1)
Evaluates whether a prediction (one rater column) matches the consensus of the other annotators.
# Dataset: agreement_threeRater
agreement(
data = agreement_threeRater,
conditionBVars = NULL,
vars = vars(Rater1, Rater2, Rater3),
multiAnnotatorConcordance = TRUE,
predictionRater = "Rater1" # Which rater column is the "prediction"
)17. Specific Agreement (Positive/Negative)
Category-specific agreement rates, important when overall kappa is misleading due to prevalence effects.
In jamovi, enabling Specific agreement indices displays an Available categories hint below Positive category. It combines categories across the selected raters; enter the intended label in the box. Lists longer than 30 categories show the first 30 and the total count. The hint updates when raters or their category labels change and never changes the category you entered.
# Dataset: agreement_binary
agreement(
data = agreement_binary,
conditionBVars = NULL,
vars = vars(PathologistX, PathologistY),
specificAgreement = TRUE,
specificPositiveCategory = "Positive", # Target category for PSA/NSA
specificAllCategories = TRUE, # Show all categories
specificConfidenceIntervals = TRUE # Wilson score CIs
)18. Computed Variables
The module can add new computed columns to the dataset.
Consensus variable
# Dataset: agreement_threeRater
agreement(
data = agreement_threeRater,
conditionBVars = NULL,
vars = vars(Rater1, Rater2, Rater3),
consensusName = "consensus_rating",
consensusRule = "majority", # "majority", "supermajority", "unanimous"
tieBreaker = "exclude" # "exclude", "first", "lowest", "highest"
)Level of Agreement variable
# Dataset: agreement_pathology
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
loaVariable = TRUE,
detailLevel = "detailed", # "simple" or "detailed"
loaThresholds = "custom", # "custom", "quartiles", "tertiles"
loaHighThreshold = 75,
loaLowThreshold = 56,
loaVariableName = "agreement_level",
showLoaTable = TRUE
)19. Edge Cases
Poor agreement with Gwet’s AC
agreement(
data = agreement_poor,
conditionBVars = NULL,
vars = vars(PathologistA, PathologistB),
gwet = TRUE # Gwet's AC is stable even when kappa is paradoxically low
)Small sample with bootstrap
agreement(
data = agreement_small,
conditionBVars = NULL,
vars = vars(Rater1, Rater2),
bootstrapCI = TRUE,
nBoot = 100
)20. Sample Size Calculator
The sample size calculator works independently of data – it only requires the analysis options to be set.
# Kappa-based sample size
agreement(
data = agreement_pathology, # Any dataset (not used for calculation)
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
agreementSampleSize = TRUE,
ssMetric = "kappa", # "kappa", "fleiss", "icc"
ssKappaNull = 0.4, # Null hypothesis kappa
ssKappaAlt = 0.7, # Alternative hypothesis kappa
ssNRaters = 2, # Number of raters
ssNCategories = 4, # Number of categories
ssAlpha = 0.05, # Significance level
ssPower = 0.80 # Desired power
)
# ICC-based sample size
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
agreementSampleSize = TRUE,
ssMetric = "icc",
ssKappaNull = 0.5,
ssKappaAlt = 0.8,
ssNRaters = 2,
ssAlpha = 0.05,
ssPower = 0.80
)21. Display Options
Confidence level
# Change from default 95% to 90%
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
confLevel = 0.90 # Applies to all CI calculations
)22. Complete Example: All Categorical Options
agreement(
data = agreement_pathology,
conditionBVars = NULL,
vars = vars(Pathologist1, Pathologist2),
# Display
sft = TRUE,
confLevel = 0.95,
# Weighted kappa
wght = "unweighted",
showLevelInfo = TRUE,
# Additional statistics
kripp = TRUE,
krippMethod = "nominal",
bootstrap = TRUE,
gwet = TRUE,
gwetWeights = "unweighted",
pabak = TRUE,
# Specific agreement
specificAgreement = TRUE,
specificAllCategories = TRUE,
specificConfidenceIntervals = TRUE,
# Bias tests
raterBias = TRUE,
bhapkar = TRUE,
stuartMaxwell = TRUE,
# Confusion matrix
confusionMatrix = TRUE,
confusionNormalize = "none",
# Visualization
agreementHeatmap = TRUE,
# Bootstrap
bootstrapCI = TRUE,
nBoot = 100,
# Subgroup
agreementBySubgroup = TRUE,
subgroupVariable = specimen_type,
subgroupForestPlot = TRUE,
subgroupMinCases = 10,
# Computed variables
consensusName = "consensus",
consensusRule = "majority",
tieBreaker = "exclude",
loaVariable = TRUE,
detailLevel = "detailed",
loaThresholds = "custom",
loaHighThreshold = 75,
loaLowThreshold = 56,
loaVariableName = "agreement_level",
showLoaTable = TRUE,
# Sample size
agreementSampleSize = TRUE,
ssMetric = "kappa",
ssKappaNull = 0.4,
ssKappaAlt = 0.7,
ssNRaters = 2,
ssNCategories = 3,
ssAlpha = 0.05,
ssPower = 0.80
)24. Complete Example: All Continuous Options
agreement(
data = agreement_continuous,
conditionBVars = NULL,
vars = vars(MeasurementA, MeasurementB),
confLevel = 0.95,
# ICC
icc = TRUE,
iccType = "icc21",
# Concordance
linCCC = TRUE,
# Bland-Altman
blandAltmanPlot = TRUE,
baConfidenceLevel = 0.95,
proportionalBias = TRUE,
# TDI
tdi = TRUE,
tdiCoverage = 90,
tdiLimit = 10,
# Additional statistics
meanPearson = TRUE,
maxwellRE = TRUE,
iota = TRUE,
iotaStandardize = TRUE,
kripp = TRUE,
krippMethod = "interval",
# Bootstrap
bootstrapCI = TRUE,
nBoot = 100,
# Rater profiles
raterProfiles = TRUE,
raterProfileType = "violin",
raterProfileShowPoints = TRUE,
# Subgroup
agreementBySubgroup = TRUE,
subgroupVariable = tumor_type,
subgroupForestPlot = TRUE,
# Sample size
agreementSampleSize = TRUE,
ssMetric = "icc",
ssKappaNull = 0.5,
ssKappaAlt = 0.8,
ssNRaters = 2,
ssAlpha = 0.05,
ssPower = 0.80
)Quick Reference: Choosing the Right Method
| Data Type | Raters | Recommended Methods |
|---|---|---|
| Binary/Nominal | 2 | Cohen’s kappa, Gwet’s AC1, PABAK, specific agreement |
| Binary/Nominal | 3+ | Fleiss’ kappa, Light’s kappa, Krippendorff’s alpha |
| Ordinal | 2+ | Weighted kappa (linear/quadratic), Kendall’s W |
| Continuous | 2 | ICC, Bland-Altman, Lin’s CCC, TDI |
| Continuous | 3+ | ICC, mean Pearson/Spearman, rater clustering |
| Any with NAs | Any | Krippendorff’s alpha |
| Multi-center | Any | Hierarchical kappa, variance decomposition |
| Before/After | Any | Paired agreement test, mixed-effects comparison |
| Test-retest | Any | Inter/intra-rater analysis |
References
- Cohen J (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
- Fleiss JL (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382.
- Krippendorff K (2011). Computing Krippendorff’s alpha-reliability. Annenberg School for Communication Departmental Papers, Paper 43. University of Pennsylvania.
- Gwet KL (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48.
- Shrout PE, Fleiss JL (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428.
- Lin LI (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268.
- Bland JM, Altman DG (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307-310.
- Donner A, Eliasziw M (1992). A goodness-of-fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance-testing and sample size estimation. Statistics in Medicine, 11(11), 1511-1519.