Inter-rater reliability: kappa and alternatives

Inter-rater reliability asks one question: when two people rate the same items independently, how often do they give the same rating, once you account for the agreement you would get by luck? For categories the standard answer is Cohen’s kappa.

This post computes kappa by hand and in base R from a two-rater table, then covers three things that trip people up: chi-square as an agreement test, weighted kappa for ordered grades, and equal raw agreement giving very different kappas. The worked example uses a deterministic engine run on seeded synthetic data, so every number quoted from the engine is reproducible. The kappa figures are computed from that run’s table with the code shown on this page.

What kappa measures

Raw agreement is the share of items on which the two raters gave the same rating. It is easy to understand, and it flatters the raters: if one category is common, they will agree often with no skill at all. Kappa removes that part.

Write po for the observed agreement and pe for the agreement expected by chance, calculated from how often each rater used each category. Then:

kappa = (po - pe) / (1 - pe)

The numerator is the agreement beyond chance, and the denominator is the most agreement beyond chance that was available. Kappa is 1 for perfect agreement, 0 when agreement is exactly what the category frequencies alone would produce, and negative when raters agree less often than that. Cohen introduced it in 1960 for nominal categories and two raters who each rate every item.

For the chance term, take each category, multiply the share of items rater A put in it by the share rater B put in it, then add up across categories. That is the agreement you would see if each rater assigned categories at random according to their own habits, independently of the other.

Kappa does not say whether either rater is right: two raters can agree perfectly on the wrong answer. It measures consistency, not validity, which needs a reference standard and different measures.

The design kappa assumes

Before the arithmetic, check that the data fit the design.

• The same items are rated by both raters. Each item gives one pair of ratings, so the unit of analysis is the pair, as in a paired test.
• The ratings are made independently. If one rater sees the other’s rating, or they discuss cases, agreement is inflated and kappa stops measuring reliability.
• The categories are fixed in advance and the same for both raters.
• The items are independent of each other. Several items taken from the same subject or slide are not independent of one another, and the simple standard error below is then too small. Use methods that account for the grouping, or one item per subject.
• There are two raters. For three or more, see the alternatives section below.

If the ratings are measurements on a numeric scale rather than categories, kappa is the wrong measure. The last section says what to use instead.

Worked example: two raters, 120 images

The data are seeded synthetic data, built so that the truth is known. Each of 120 images has a hidden true status, present with probability 0.35. Each of two raters reports the true status correctly with probability 0.85 and the opposite status otherwise, independently of the other rater. The numbers describe no real images, raters or people.

The 2 x 2 table of the two raters’ ratings, from the engine run’s data:

	Rater B: absent	Rater B: present	Row total
Rater A: absent	64	12	76
Rater A: present	16	28	44
Column total	80	40	120

The two raters agree on 64 + 28 = 92 images, so the raw agreement is 92 / 120, or 0.7667. Rater A said “absent” for 76 of 120 images and rater B for 80, so by chance alone they would both say “absent” on (76 / 120) x (80 / 120) = 0.4222 of the images. For “present” the same calculation gives (44 / 120) x (40 / 120) = 0.1222. Adding the two gives pe = 0.5444, so more than half the raw agreement is what the raters’ habits would produce anyway.

kappa = (0.7667 - 0.5444) / (1 - 0.5444) = 0.4878

The same calculation in base R, with a function that also handles the ordered categories used below:

kappa_from_table <- function(tab, weights = c("none", "linear", "quadratic")) {
  weights <- match.arg(weights)
  p <- tab / sum(tab)
  k <- nrow(tab)
  dist <- abs(outer(seq_len(k), seq_len(k), "-")) / (k - 1)
  w <- switch(weights, none = 1 - diag(k), linear = dist, quadratic = dist^2)
  expected <- outer(rowSums(p), colSums(p))
  1 - sum(w * p) / sum(w * expected)
}

ratings <- matrix(c(64, 16, 12, 28), nrow = 2)
round(kappa_from_table(ratings), 4)             # 0.4878

With no weights, a disagreement costs 1 and an agreement costs 0, which reduces to the formula above.

An interval for kappa

A single kappa without an interval hides how little 120 images can tell you. A simple large-sample approximation to its standard error is the square root of po (1 - po) / (n (1 - pe)^2). With these counts it gives about 0.085, and an approximate 95% interval of 0.32 to 0.65. Fuller variance formulas exist and give slightly different intervals, and for small samples a bootstrap over the items is a reasonable check. Either way the lesson is the same: with 120 images the data are compatible with anything from fair to substantial agreement on the usual benchmarks.

The same formula helps plan a study. The half-width of the interval shrinks with the square root of the number of items, so halving it takes about four times as many: with this agreement, 480 images would give about 0.08 instead of 0.17. Sim and Wright (2005) tabulate the items needed to show that kappa exceeds a minimum acceptable value, a more useful target than zero.

Reading the number

Landis and Koch (1977) proposed labels that are quoted everywhere: below 0 poor, 0.00 to 0.20 slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect. They described the benchmarks as arbitrary, and kappa depends on how common the categories are, so the labels are a convention for talking, not a standard of adequacy. A better practice is to decide before the study what level of agreement the downstream use needs and to report the table, the raw agreement, kappa and its interval together.

Why a chi-square test is not an agreement test

Because the data are a 2 x 2 table, many people reach for the chi-square test of association. The engine ran it to show what it tells you.

Quantity	Engine output
Images analysed	120
Chi-square statistic	26.6 on 1 degree of freedom with Yates’ continuity correction (28.7 without), p < 0.001
Cramer’s V	0.4891, 95% CI 0.339 to 1 (one-sided interval)
Odds ratio (present vs absent)	9.333, 95% CI 3.909 to 22.28
Expected counts at least 5	passed

The test is highly significant, and the effect size of 0.4891 looks close to the kappa of 0.4878. That closeness is partly luck. For a 2 x 2 table, Cramer’s V equals kappa in magnitude when both raters have the same category frequencies, and the two part company as soon as their margins differ. Here the margins are slightly different (76 and 44 against 80 and 40), so the figures are near but not equal.

More importantly, the chi-square test asks whether the ratings are associated, not whether they match. A rater who always gives the opposite of the other rater produces a perfect association and zero agreement: the chi-square statistic would be at its largest for that table, and kappa would be negative. The test also answers a question nobody asked, because two raters looking at the same images are expected to be associated.

So the chi-square run is shown as a warning, not as the analysis. Celeus does not compute kappa or an intraclass correlation: the agreement coefficients on this page are calculated outside the engine, from the table counts, with the R code shown or the published formulas. What the run does contribute is a check on the table: the counts are large enough that no expected cell count falls below 5, so the chi-square approximation itself is sound. That check does not cover the kappa interval, which is why a bootstrap is suggested above.

Ordered categories: weighted kappa

Many rating scales are ordered, such as grades or severity bands. Unweighted kappa treats every disagreement as equally bad, so a rater who calls an item “middle” when the other calls it “low” is penalised exactly as much as one who calls it “high”. Weighted kappa (Cohen, 1968) gives partial credit to near misses.

Each cell of the table gets a disagreement weight between 0 and 1. With k categories, linear weights are the distance between the two categories divided by k - 1, and quadratic weights are that figure squared. Kappa is then one minus the weighted observed disagreement divided by the weighted disagreement expected by chance, which is what the function above computes.

For a worked illustration, take 100 items graded 1 to 3 by two raters. These counts are typed in by hand for the illustration and are not part of the engine run:

	Rater B: 1	Rater B: 2	Rater B: 3
Rater A: 1	30	6	1
Rater A: 2	5	25	6
Rater A: 3	0	7	20

grades <- matrix(c(30, 6, 1,  5, 25, 6,  0, 7, 20), nrow = 3, byrow = TRUE)
round(sapply(c("none", "linear", "quadratic"), function(w) kappa_from_table(grades, w)), 4)
#      none    linear quadratic
#    0.6217    0.6960    0.7749

Raw agreement here is 75 of 100. Kappa is 0.6217 unweighted, 0.6960 with linear weights and 0.7749 with quadratic weights. It rises with the weights because most of the disagreements are by one grade, and only one pair of ratings is two grades apart. The weighting scheme is a choice about how much a near miss matters, and it changes the answer. Decide the weights before you see the data, say which you used every time you quote the number, and do not compare a quadratic-weighted kappa with a linear or unweighted one from another study.

Fleiss and Cohen (1973) showed that quadratic-weighted kappa is closely related to the intraclass correlation coefficient, which is the more direct tool if your ratings are really a numeric scale.

The prevalence problem, in numbers

The most common complaint about kappa is that it can be low when raw agreement is high. Feinstein and Cicchetti (1990) described two paradoxes: high agreement with low kappa when one category dominates, and a higher kappa when the two raters’ margins are unbalanced in opposite directions than when they match. Byrt, Bishop and Carlin (1993) tied them to prevalence and rater bias. The mechanism is the chance term: when nearly every item falls in one category, pe is large, and little room is left above it.

Two hand-typed illustration tables of 100 items each show it. Both have raw agreement of 0.90 and the same number of disagreements, 10:

	Balanced: B absent	Balanced: B present	Skewed: B absent	Skewed: B present
A absent	45	5	88	5
A present	5	45	5	2

balanced <- matrix(c(45, 5, 5, 45), nrow = 2)
skewed   <- matrix(c(88, 5, 5, 2), nrow = 2)
round(sapply(list(balanced = balanced, skewed = skewed), kappa_from_table), 4)
# balanced   skewed
#    0.800    0.232

The balanced table gives kappa 0.80. The skewed table gives 0.2320, although the raters agree just as often. In the skewed table, “absent” is so common that chance agreement is already about 0.87, so 0.90 is only a small step above it. Where one category dominates, agreement on the common category says little about whether the raters can tell the categories apart. If the rare category is what matters, a low kappa is a fair warning. If both matter equally, it may understate how consistent the raters are.

The practical consequence is to report more than kappa: the full table, raw agreement, and the category frequencies. A kappa quoted alone cannot be compared across studies with different prevalence.

Alternatives when kappa misleads

None of the following is a drop-in replacement, and each has its own caveats.

Prevalence-adjusted, bias-adjusted kappa (PABAK). For two categories, PABAK equals 2 x po - 1, so both tables above give 0.8. The cost is that it replaces the chance term with the value expected if both categories were equally common, which discards information about the real frequencies. It is best used as a companion to kappa, to show how much prevalence is moving the number, never as a replacement that hides it.

Gwet’s AC1. Gwet (2008) proposed a different chance-agreement model that is much less sensitive to a dominant category. On the same data the AC1 for the skewed table is about 0.885 against a kappa of 0.2320, and for the 120-image example it is about 0.5719 against 0.4878. Whether its chance model is more believable than kappa’s depends on how you think raters behave when unsure, so state which coefficient you used and why.

More than two raters, or incomplete ratings. Fleiss’ kappa (Fleiss, 1971) extends the chance-corrected idea to a fixed number of raters per item, who need not be the same people for every item. Krippendorff’s alpha (Hayes and Krippendorff, 2007) allows missing ratings and treats nominal, ordinal, interval and ratio data under one definition. Whichever coefficient you choose, name it, give the software and its settings, and give an interval.

When the ratings are numbers

Kappa is for categories. If two raters or instruments each give a numeric measurement of the same quantity, the question changes from “do they pick the same category” to “how far apart are their values”. The intraclass correlation coefficient measures reliability for numeric ratings, and Koo and Li (2016) describe how to select and report the right form for your design. Shrout and Fleiss (1979) distinguished one-way, two-way random and two-way mixed models, each for a single or an averaged rating, and McGraw and Wong (1996) separated absolute agreement from consistency. Name the form you used, because the values can differ. For method comparison, where the size of the typical difference in the original units matters, the Bland-Altman guide shows how to compute bias and limits of agreement. If you dichotomise a numeric rating to use kappa, you throw away information, and the result depends on where you cut.

For choosing between agreement measures and tests of a difference, see choosing a statistical test.

Reporting a reliability study

A complete report gives:

1. the number of items and raters, and how items were chosen;
2. how ratings were collected, and the confirmation that raters worked independently;
3. the full table of counts, or the margins at least;
4. raw agreement and the chosen coefficient, with its name and any weights;
5. an interval for the coefficient, and the method used for it;
6. the category frequencies, so a reader can judge prevalence effects.

A sentence in that spirit for the example: “Two raters independently classified 120 images as present or absent. They agreed on 92 (76.7%). Cohen’s kappa was 0.49 (approximate 95% interval 0.32 to 0.65).”

This page describes statistical methods for research data. It makes no claim about any individual case, and a reliability coefficient says nothing on its own about whether a rating is correct for a particular item.

For the engine’s own method reference for the chi-square test, see the chi-square test reference. A fuller treatment of the test itself is in chi-square test explained.

# Results

- Test: Chi-squared test of association
- n: 120
- statistic: 26.6
- p-value: < 0.001
- effect size: Cramer's V = 0.4891 (95% CI 0.339 to 1)
  (one-sided CI (upper bound fixed at 1))
- seed: 20261022

## Assumption checks
- expected cell counts >= 5: passed=TRUE (all expected counts >= 5)

## Effect measures (present vs absent; event = present)
| Measure | Estimate | 95% CI | Method |
|---|---|---|---|
| Risk ratio | 4.03 | 2.29 to 7.093 | log-scale Wald |
| Risk difference | 0.4785 | 0.3144 to 0.6426 | Wald |
| Odds ratio | 9.333 | 3.909 to 22.28 | log-scale Wald |

- RR and RD assume the groups were sampled by exposure (cohort, trial, cross-sectional); if participants were sampled by outcome (case-control), only the odds ratio is interpretable, and it approximates the risk ratio only when the event is rare.

## Power (sensitivity, not observed power)
- minimum detectable effect at n=120: Cohen's w = 0.2557
- to detect a conventional medium effect: 88 total to detect w=0.30
- alpha = 0.05, target power = 0.8

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Chi-squared test of association  |  engine 0.31.1  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261022)
.data_path <- "data/inter-rater-reliability.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("rater_a", "rater_b")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
result <- chisq.test(table(data[["rater_a"]], data[["rater_b"]]))
print(result)

# Sensitivity: minimum detectable effect at n=120 (power=0.8, alpha=0.05). NOT observed power.
power <- pwr::pwr.chisq.test(N = 120, df = 1, sig.level = 0.05, power = 0.8)
print(power)