Inter-Rater Agreement and Reliability in Pathology
Source:vignettes/03-agreement-analysis.Rmd
03-agreement-analysis.RmdThis article is a short, practical guide to the Interrater
Reliability analysis (agreement(), jamovi menu
meddecide > Agreement > Interrater Reliability).
It shows which statistic fits which kind of rating, and how to read the
result tables from R. For every option, see ?agreement or
the comprehensive
guide.
Each rater is one column of data; each row is one case.
Pass the rater columns to vars. With two raters the
analysis reports Cohen’s kappa; with three or more, Fleiss’ kappa.
Choosing a statistic
| Ratings | Primary statistic | Options |
|---|---|---|
| Nominal (e.g. diagnosis) | Cohen’s / Fleiss’ kappa | default; kripp = TRUE, gwet = TRUE
|
| Ordinal (e.g. grade 1-3) | Weighted kappa (2 raters); ordinal alpha or Gwet’s AC2 (3+ raters) |
wght = "equal" (linear) or "squared"
(quadratic); with 3+ raters allPairsKappa = TRUE gives
weighted kappa per pair, or krippMethod = "ordinal",
gwetWeights
|
| Rare category, skewed prevalence | Gwet’s AC1, PABAK |
gwet = TRUE, pabak = TRUE
|
| Continuous (e.g. Ki-67 %) | ICC, Lin’s CCC, Bland-Altman |
icc = TRUE, linCCC = TRUE,
blandAltmanPlot = TRUE
|
| Missing ratings, any scale | Krippendorff’s alpha |
kripp = TRUE,
krippMethod = "nominal"/"ordinal"/"interval"/"ratio"
|
| Which pairs disagree | All-pairs kappa | allPairsKappa = TRUE |
Nominal ratings: three raters
The bundled histopathology data has three binary rater
columns.
data(histopathology)
res <- agreement(
data = histopathology,
vars = c("Rater 1", "Rater 2", "Rater 3"),
kripp = TRUE,
gwet = TRUE,
allPairsKappa = TRUE
)
res$irrtable$asDF # Fleiss' kappa
#> method subjects raters peragree kappa ci_lower ci_upper z
#> "1" Fleiss' κ 250 3 76.30522 0.6512801 0.572085 0.7304752 16.19697
#> p
#> "1" 8.688896e-41
res$krippTableGroup$krippTable$asDF # Krippendorff's alpha
#> method subjects raters alpha
#> "1" Krippendorff's Alpha (nominal) 250 3 0.6540881
#> interpretation
#> "1" Insufficient agreement - discard variable
res$gwetGroup$gwetTable$asDF # Gwet's AC1
#> method subjects raters coefficient se ci_lower
#> "1" Gwet's AC1 (unweighted) 250 3 0.69976 0.03545 0.63
#> ci_upper p
#> "1" 0.77 7.61843e-53
res$allPairsKappaTable$asDF # Cohen's kappa for every pair
#> rater_a rater_b n peragree kappa se ci_lower ci_upper
#> "1_2" Rater 1 Rater 2 249 0.9196787 0.8205535 0.03832152 0.7454447 0.8956623
#> "1_3" Rater 1 Rater 3 249 0.8433735 0.6661395 0.04800118 0.5720589 0.7602201
#> "2_3" Rater 2 Rater 3 250 0.7600000 0.4943707 0.05604538 0.3845237 0.6042176
#> z p
#> "1_2" 21.412343 1.025272e-101
#> "1_3" 13.877566 8.663573e-44
#> "2_3" 8.820899 1.135447e-18Several tables live inside result groups, so the path has two parts
(res$krippTableGroup$krippTable,
res$gwetGroup$gwetTable,
res$iccGroup$iccTable).
Ordinal ratings: weighted kappa
For two raters scoring an ordered scale, weighted kappa gives partial
credit for near-misses. wght = "squared" (quadratic
weights) is the usual choice for grades.
Continuous measurements: ICC and Lin’s CCC
res_c <- agreement(
data = histopathology,
vars = c("Measurement1", "Measurement2"),
icc = TRUE,
iccType = "icc21",
linCCC = TRUE,
blandAltmanPlot = TRUE
)
res_c$iccGroup$iccTable$asDF
#> model subjects raters icc_value ci_lower
#> "1" ICC(2,1) - Two-way random, single 250 2 0.1718531 0.04903747
#> ci_upper f_value df1 df2 p
#> "1" 0.2895482 1.414754 249 249 0.003187137
res_c$iccGroup$linCCCTable$asDF
#> comparison subjects ccc ci_lower ci_upper
#> "1_2" Measurement1 vs Measurement2 250 0.1712834 0.06751243 0.2713883
#> pearson_r bias_factor interpretation
#> "1_2" 0.2011813 0.8513884 Poor concordancePick the ICC model from the study design (Shrout & Fleiss 1979; McGraw & Wong 1996):
iccType |
Model | Use when |
|---|---|---|
icc11 / icc1k
|
One-way random effects | each case is rated by a different set of raters |
icc21 / icc2k
|
Two-way random effects, absolute agreement | the same raters rate every case and are a sample of possible raters |
icc31 / icc3k
|
Two-way mixed effects, consistency | the same fixed raters rate every case and only their ranking matters |
The 1 forms give the reliability of a single rater; the
k forms give the reliability of the mean of the k raters,
which is always higher.
Rater clustering
raterClustering = TRUE groups raters by how alike they
rate. See Diagnostic Style
Clustering.
Interpreting kappa
Landis and Koch (1977) proposed these descriptive bands. They are a convention, not a validated clinical threshold; always report the confidence interval.
| Kappa | Landis & Koch label |
|---|---|
| < 0 | Poor |
| 0.00 - 0.20 | Slight |
| 0.21 - 0.40 | Fair |
| 0.41 - 0.60 | Moderate |
| 0.61 - 0.80 | Substantial |
| 0.81 - 1.00 | Almost perfect |
For the ICC, Koo and Li (2016) suggest < 0.50 poor, 0.50 - 0.75 moderate, 0.75 - 0.90 good and > 0.90 excellent, judged on the 95% confidence interval.
Kappa depends on prevalence: with one very common category, kappa can be low while percent agreement is high (the “kappa paradox”). Report percent agreement alongside kappa, and consider Gwet’s AC1 or PABAK.
Sample size
To plan an agreement study, use the kappa sample-size tools under
jamovi’s Power menu: kappaSizeCI(),
kappaSizePower() and kappaSizeFixedN().
References
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.
- Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-428.
- McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1:30-46.
- Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15:155-163.
- Gwet KL. Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol. 2008;61:29-48.
- Krippendorff K. Content Analysis: An Introduction to Its Methodology. 4th ed. Sage; 2018.