Analysing Likert scale data: items and scales

If your thesis collects answers on a five-point agree-disagree scale, the first decision is not which test to run. It is what you are analysing: one item on its own, or a score built from several items that measure the same thing. A single item is ordinal and is best analysed with an ordinal method. A summed or averaged scale of several items usually behaves well enough for a t-test, ANOVA or regression, provided you have checked that the items belong together.

Items, scales and “Likert-type” questions

Three terms get used loosely, and committees notice.

A Likert item is one statement with ordered answer options, for example “I feel confident choosing a statistical test”, answered from 1 (strongly disagree) to 5 (strongly agree).

A Likert scale, in the original sense, is a set of several items written to measure one underlying construct, such as statistics anxiety, combined into one score by summing or averaging. Sullivan and Artino describe this practice and note that it is often recommended for abstract constructs that no single question can capture.

A Likert-type item is a single rating question that uses the same answer format but is not part of a designed scale.

The distinction matters because the two produce different kinds of numbers. A single item takes only five values, and nothing guarantees that the step from “disagree” to “neutral” equals the step from “agree” to “strongly agree”. A score averaged over eight such items takes many more values, tends to be less lumpy, and is what the scale was designed to measure.

The debate, honestly

There are two camps, and both have a point.

The ordinal camp argues that because the answer options are ordered categories with unknown spacing, means are not meaningful. Sullivan and Artino summarise this traditional position: frequencies, contingency tables, chi-square tests, Spearman’s rho or the Mann-Whitney test instead of parametric tests.

The robustness camp argues that parametric tests give the right answer on this kind of data in practice. In his 2010 review, Norman dissects the objections that samples are too small, that data are not normal and that Likert data are ordinal, and argues from studies going back to the 1930s that methods such as ANOVA, regression and correlation are robust to these violations. Sullivan and Artino come down on the same side for Likert scale responses, citing Norman.

The counterpoint that newer work adds is sharper. Liddell and Kruschke surveyed the articles in three major psychology journals that mentioned the term “Likert”, and found that every one of them that analysed ordinal data did so with a metric model. They then showed that treating ordinal responses as metric can produce false positives, missed effects and even reversals of the direction of an effect, and that averaging several ordinal items did not remove these problems. They recommend ordinal models that describe the data directly.

What the evidence supports is a defensible default:

• A single item: use an ordinal method. It answers the question without pretending the five labels are equally spaced.
• A multi-item scale score: a t-test, ANOVA or regression on the score is common, usually reasonable and backed by the robustness literature, especially with many items and a score not piled up at one end. If a committee member raises the ordinal objection, rerunning the main comparison with an ordinal model is a cheap answer.

Do not choose the approach after seeing which gives the smaller p-value. Decide in your analysis plan, before the data are in.

Choosing the test by design

The design questions are the usual ones: how many groups, independent or repeated, and whether to adjust for anything.

Design	Single Likert item (ordinal)	Multi-item scale score
Two independent groups	Ordinal logistic regression, or Mann-Whitney	Welch’s t-test
Three or more independent groups	Ordinal logistic regression, or Kruskal-Wallis	One-way ANOVA (Welch’s version if spreads differ)
Same people, two time points	Wilcoxon signed-rank test	Paired t-test
Adjusting for covariates	Ordinal logistic regression	Linear regression or ANCOVA
Association between two measures	Spearman’s rho	Pearson’s r, or Spearman’s rho if the relationship is monotonic but not linear

Mann-Whitney and Kruskal-Wallis compare groups through ranks, which suits an item, but they cannot adjust for anything. Ordinal logistic regression can, and it gives an effect size, the odds ratio, with a confidence interval, which is why the worked example uses it. For a scale score, the engine’s Welch’s t-test reference page shows the two-group default. The ordinal or multinomial logistic regression guide explains the model in more depth, and the t-test or Mann-Whitney guide covers the two-group choice for a scale score. If your design is not in the table, the guide to choosing a statistical test starts from the outcome type.

Before you analyse a scale score

A scale score is only as good as the items in it. These points belong in your methods section before any test runs.

Reverse-keyed items. Many scales include statements worded in the opposite direction (“I find statistics confusing” alongside “I feel confident with statistics”). These must be reverse-scored (6 minus the answer on a five-point item) before summing, or they cancel the items they should reinforce.

Internal consistency. Report a reliability coefficient for your sample, usually Cronbach’s alpha, which Cronbach introduced in 1951. Sullivan and Artino note that alpha, or factor analysis, provides evidence that the items are intercorrelated enough to be treated as measuring one thing. If you use a published scale, follow its scoring manual and report alpha in your own data anyway.

Missing items. Decide in advance how a respondent who skipped an item is scored. A sum over fewer items is not comparable with a full sum, which is one reason to use the mean of the answered items, which also stays on the 1 to 5 answer scale.

Design choices that shape the analysis. A neutral midpoint lets a respondent with no view answer honestly, but those answers then need a stated place in the analysis; removing it forces a lean. More options give a finer ranking, though respondents may not discriminate between very many labels. Whichever you choose, keep the format identical across the items of one scale and word statements neutrally. A small pilot is the cheapest way to find an ambiguous item.

Worked example: one five-point item by group

The data are synthetic, generated with a fixed seed, and describe no real study or person. They mimic a common thesis design: 120 students in two independent groups of 60, a workshop group and a comparison group, each answering once. The outcome is the single item “I feel confident choosing a statistical test”, scored 1 to 5. A second variable records whether each student had taken a statistics course before the study. It is in the model on design grounds: it was recorded before the workshop, so the workshop cannot have changed it; it plausibly relates to confidence; and where students choose their group, it may differ between groups. Name a covariate like this in the analysis plan before the data are in.

The research question: do workshop students give higher ratings than comparison students, after allowing for a prior course?

One caution: the responses were generated from the very model fitted below, so its main assumption holds by construction. Real survey items come with no such guarantee, which is why the assumption check matters on your own data.

Seeded and run in R; the method was chosen before the run. The full output and the R script appear below the article.

What the model estimates

Ordinal logistic regression, also called the proportional-odds model, treats the five answers as ordered without assigning them distances. McCullagh set out the model in 1980 as a way to use the ordering without having to score the categories. It looks at every split of the scale into “this answer or below” against “above it”: 1 against 2 to 5, 1 to 2 against 3 to 5, and so on. For each predictor it estimates one odds ratio: how much higher the odds of answering above any given point are in one group than in the other. The proportional-odds assumption is that this odds ratio is the same at every split.

The result

Quantity	Value
Students analysed	120
Workshop vs comparison, log odds ratio	0.947 (standard error 0.347)
95% confidence interval, log odds ratio	0.2746 to 1.637
Workshop vs comparison, odds ratio	2.579
Wald statistic, p-value (two-sided)	2.731, p = 0.00632
Prior course vs none, odds ratio	2.663 (log odds ratio interval 0.3054 to 1.671, p = 0.00485)
Proportional-odds check	statistic 4.037, p = 0.672

Reading it

Start with the effect. The odds ratio of 2.579 means that, among students with the same prior-course status, a workshop student had about two and a half times the odds of a comparison student of answering above any given point on the scale.

Next, the interval. The engine reports it on the log odds scale, the scale the model is fitted on: 0.2746 to 1.637. Both ends are above zero, so on the odds ratio scale both ends are above 1. To report odds ratios, exponentiate the two ends, which exp(confint(fit)) does in R. The interval is wide: the data are compatible with a modest advantage and with a large one. The interval comes from profiling the likelihood, the p-value from the Wald statistic (the estimate divided by its standard error); they can disagree near p = 0.05, but this result is not near it.

Then the p-value. The Wald statistic of 2.731 gives p = 0.00632, so at the 0.05 level the data are not compatible with no difference between the groups. That is a statement about compatibility, not proof that the workshop caused the difference: in a thesis where students chose whether to attend, other differences between the groups could produce the same pattern. Random allocation is what would support a causal reading.

The prior-course coefficient is an adjustment, not a second research question. Its odds ratio of 2.663 says that, within each group, students with a prior course also tended to rate themselves higher.

The assumption checks

Proportional odds: no departure detected. The engine runs Brant’s test from 1990, which compares the separate binary fits at each split with the common odds ratio. The statistic was 4.037 with p = 0.672. That does not show the odds ratios are equal; with small groups or sparse categories such tests have little power. On your own data, also look at the proportion of each group at or above each answer; if one group differs only at the top of the scale, a single odds ratio is the wrong summary.

Sample size: passed. With 120 students, the data clear the engine’s minimum-size floor, a sanity check that says nothing about model fit or power. The engine does not compute power for this model; plan it by simulation, sized for the smallest effect that would matter.

Answer order. The engine noted that it treated the outcome as ordered 1 < 2 < 3 < 4 < 5. That is correct here, but it is the first thing to check on real data: answer labels imported as text sort alphabetically, which puts “agree” before “disagree”, and a reverse-keyed item left unreversed puts the order backwards.

The scale-score side, in R

For a multi-item scale, the steps below assume a data frame d with eight items q1 to q8 on a 1 to 5 scale, one of them reverse-keyed, and a group column. This is code only: the worked example above is the one engine-run analysis in this guide.

items <- paste0("q", 1:8)
d$q3 <- 6 - d$q3                      # reverse-key the negatively worded item first

# Internal consistency of the eight items in your sample
alpha_items <- d[, items]
k <- ncol(alpha_items)
alpha <- (k / (k - 1)) * (1 - sum(apply(alpha_items, 2, var)) / var(rowSums(alpha_items)))

# The scale score: mean of the items, on the 1 to 5 answer scale
d$scale_score <- rowMeans(d[, items])

# Two independent groups: Welch's t-test on the scale score
t.test(scale_score ~ group, data = d)

# The ordinal check a committee might ask for: the same comparison on one item
MASS::polr(factor(q1, ordered = TRUE) ~ group, data = d, Hess = TRUE)

Report the scale’s mean and standard deviation by group, the difference with its confidence interval, an effect size such as Cohen’s d with its interval, and alpha. Welch’s version is the default for two groups because it does not assume equal variances; the t-tests explained guide sets out why. For where Likert outcomes sit among the other designs a thesis uses, see the guide to choosing statistics for a dissertation.

How to report it

For the worked example, a results sentence would read:

Ratings of confidence in choosing a statistical test (1 = strongly disagree, 5 = strongly agree) were analysed with a proportional-odds ordinal logistic regression, with group and prior statistics course as predictors (n = 120). Workshop students had higher odds of a higher rating than comparison students, log OR = 0.95, 95% CI 0.27 to 1.64 (OR = 2.58), Wald z = 2.73, p = 0.006. Brant’s test did not detect a departure from proportional odds (p = 0.672).

That interval is on the log odds scale. To report it as odds ratios instead, use exp(confint(fit)).

For a single item, add a table or stacked bar chart of the percentage of each group choosing each answer. It shows the reader what the odds ratio summarises. Sullivan and Artino note that a frequency distribution usually describes Likert responses better than a mean, and that the long-standing advice for a single central value is the median. A diverging stacked bar, with disagree answers extending left of a centre line and agree answers right, makes the balance of opinion readable at a glance. A “top-box” figure, the share choosing the top one or two answers, is a common headline; define it in the methods and report it beside the full distribution, because collapsing categories discards information. If you used a rank test, report an effect size with it, such as the rank-biserial correlation or the probability that a random workshop student outranks a random comparison student (counting ties as half), rather than the p-value alone.

Common mistakes

Averaging a single item and running a t-test without saying why. It may give a similar p-value, but the mean of five ordered labels has no agreed meaning, and Liddell and Kruschke show that metric analysis of ordinal responses can go wrong in either direction. Use an ordinal method, or justify the choice in your plan.

Running a test on every item separately. Ten items and ten tests give many chances of a false positive. Analyse a scale as a score; for separate questions, name the primary ones in advance and correct for the rest.

Summing items without checking them. An unreversed negative item weakens the score, and without alpha the reader has no evidence the items measure one thing.

Rerunning the example

Below the article you will find the R script exactly as the engine generated it, and the engine’s recorded output. The script fits MASS::polr(confidence_item ~ group + prior_course, data = data, Hess = TRUE), the same model the engine reports. On the same package versions it reproduces the coefficient table, including the Wald statistic. The script does not print the p-value; the engine computes it from that statistic as 2 * pnorm(-abs(t)). To rerun the interval and the proportional-odds check, add:

fit <- MASS::polr(confidence_item ~ group + prior_course, data = data, Hess = TRUE)
confint(fit)        # profile-likelihood 95% interval, log odds scale (the engine's numbers)
exp(confint(fit))   # the same interval on the odds-ratio scale
car::poTest(fit)    # Brant-type proportional-odds tests

The engine’s proportional-odds check is the Overall row that car::poTest() prints.

# Results

- Test: Ordinal logistic regression
- n: 120
- statistic: 2.731
- p-value: 0.00632
- effect size: odds ratio = 2.579
- 95% CI: [0.2746, 1.637]
- seed: 20261021

## Assumption checks
- proportional odds (car::poTest): passed=TRUE (no evidence against the proportional-odds assumption (the predictors' effects look consistent across cut points))
- sample size: passed=TRUE (adequate n)

## Power
- Power is not calculated for this test.

# Celeus - reproduce.R  (run from this package folder; bundled data in data/)
# Test: Ordinal logistic regression  |  engine 0.30.0  |  catalog c39-d3cd
# Research context: not provided for this run.
# Reproduces the recorded result deterministically.

set.seed(20261021)
.data_path <- "data/likert-scale-data-analysis.csv"
if (!file.exists(.data_path)) stop("Run reproduce.R from the package folder (the bundled data is in data/).")
data <- readr::read_csv(.data_path, show_col_types = FALSE)  # needs the readr package
clean_cells <- function (df) 
{
    ctrl_collapse <- "[\r\n\t\v\f]+"
    zero_width_chars <- intToUtf8(c(8203L, 65279L), multiple = TRUE)
    uni_space_chars <- intToUtf8(c(160L, 8239L, 8199L), multiple = TRUE)
    clean_text <- function(x) {
        out <- x
        for (ch in zero_width_chars) out <- gsub(ch, "", out, 
            fixed = TRUE)
        out <- gsub(ctrl_collapse, " ", out)
        for (ch in uni_space_chars) out <- gsub(ch, " ", out, 
            fixed = TRUE)
        trimws(gsub(" +", " ", out))
    }
    changed_cols <- character(0)
    cells_cleaned <- 0L
    retyped <- character(0)
    for (j in seq_along(df)) {
        col <- df[[j]]
        if (!is.character(col)) 
            next
        cleaned <- clean_text(col)
        n_changed <- sum(!is.na(col) & !is.na(cleaned) & col != 
            cleaned, na.rm = TRUE)
        if (n_changed > 0L) {
            changed_cols <- c(changed_cols, names(df)[j])
            cells_cleaned <- cells_cleaned + n_changed
        }
        df[[j]] <- cleaned
        non_blank <- cleaned[!is.na(cleaned) & nzchar(cleaned)]
        if (length(non_blank) && all(grepl("^[+-]?(0|[1-9][0-9]*)([.][0-9]+)?$", 
            non_blank))) {
            parsed <- suppressWarnings(as.numeric(cleaned))
            was_blank <- is.na(cleaned) | !nzchar(cleaned)
            lost <- (is.na(parsed) | is.infinite(parsed)) & !was_blank
            if (!any(lost)) {
                finite <- parsed[!is.na(parsed)]
                all_whole <- all(finite == floor(finite))
                in_range <- all(abs(finite) <= .Machine$integer.max)
                df[[j]] <- if (all_whole && in_range) 
                  as.integer(parsed)
                else parsed
                retyped <- c(retyped, names(df)[j])
            }
        }
    }
    names(df) <- clean_text(names(df))
    if (length(changed_cols)) {
        attr(df, "ingest_clean_columns") <- changed_cols
        attr(df, "ingest_clean_cells_cleaned") <- cells_cleaned
    }
    if (length(retyped)) 
        attr(df, "ingest_clean_retyped") <- retyped
    df
}
data <- clean_cells(data)
# Complete-case analysis: drop rows missing any analysis value.
.acols <- c("confidence_item", "group", "prior_course")
data <- data[stats::complete.cases(data[, .acols, drop = FALSE]), , drop = FALSE]
data[["confidence_item"]] <- factor(data[["confidence_item"]], levels = c("1", "2", "3", "4", "5"), ordered = TRUE)
result <- summary(MASS::polr(confidence_item ~ group + prior_course, data = data, Hess = TRUE))
print(result)