130  Friedman Test

The Friedman Test (Friedman 1937) is a non-parametric alternative to the Repeated Measures ANOVA (Chapter 129). It is used when the same subjects are measured under three or more conditions, but the assumptions of normality or sphericity are not satisfied. The test extends the Wilcoxon Signed-Rank Test (Chapter 117) to three or more related groups and is based on ranks within each subject (block).

130.1 Hypotheses

The Friedman Test evaluates whether the distributions across conditions are identical:

\[ \begin{cases}\text{H}_0: F_1 = F_2 = \ldots = F_k \\\text{H}_A: \exists\; i \neq j: F_i \neq F_j\end{cases} \]

where \(F_i\) denotes the distribution function of condition \(i\) and \(k\) is the number of conditions or time points.

The test works by:

  1. Ranking the observations within each subject (block) from 1 to \(k\).
  2. Summing the ranks for each condition across all subjects.
  3. Testing whether the rank sums differ more than would be expected by chance.

The Friedman test statistic \(Q\) approximately follows a Chi-squared distribution with \(k - 1\) degrees of freedom.

130.2 Analysis based on p-values and confidence intervals

130.2.1 Software

The Friedman Test can be computed in RFC under the “Hypotheses / Empirical Tests” menu item (select “Friedman Test” from the ANOVA type dropdown), or by using the R code shown below.

130.2.2 Data & Parameters

The data should contain:

  • A quantitative (or ordinal) response variable
  • A categorical variable identifying the condition or time point
  • A subject/block identifier

The data must represent a balanced design: each subject must have exactly one observation per condition. Missing values will cause the test to fail.

130.2.3 Output

Consider the problem of measuring the reaction time (in milliseconds) of 12 subjects under three different conditions: no caffeine, moderate caffeine, and high caffeine. Each subject is tested under all three conditions. The results from the Friedman analysis are shown below.

Interactive Shiny app (click to load).
Open in new tab

The output reports the Friedman chi-squared statistic, condition descriptive statistics, and (if the overall test is significant) post-hoc pairwise Wilcoxon signed-rank tests with Bonferroni correction. The plots show box plots per condition and subject profile (spaghetti) plots.

The same analysis can also be replicated with R code:

# Simulated reaction time data
set.seed(42)
n_subjects <- 10
no_caffeine <- rnorm(n_subjects, mean = 350, sd = 30)
moderate_caffeine <- rnorm(n_subjects, mean = 320, sd = 30)
high_caffeine <- rnorm(n_subjects, mean = 300, sd = 30)

# Create data in matrix format (subjects x conditions)
reaction_matrix <- cbind(no_caffeine, moderate_caffeine, high_caffeine)

# Friedman test
friedman.test(reaction_matrix)

    Friedman rank sum test

data:  reaction_matrix
Friedman chi-squared = 9.8, df = 2, p-value = 0.007447

The output reports the Friedman chi-squared statistic, the degrees of freedom (\(k - 1\)), and the p-value. If the p-value is smaller than the chosen type I error \(\alpha\), we reject the Null Hypothesis and conclude that at least two conditions differ.

130.2.3.1 Post-hoc pairwise comparisons

When the overall Friedman test is significant, post-hoc comparisons identify which specific conditions differ. The Nemenyi test or pairwise Wilcoxon signed-rank tests with Bonferroni correction can be used:

# Create long-format data for pairwise comparisons
reaction_data <- data.frame(
  subject = factor(rep(1:n_subjects, 3)),
  condition = factor(rep(c("None", "Moderate", "High"), each = n_subjects),
                     levels = c("None", "Moderate", "High")),
  reaction_time = c(no_caffeine, moderate_caffeine, high_caffeine)
)

# Pairwise Wilcoxon signed-rank tests with Bonferroni correction
pairwise.wilcox.test(reaction_data$reaction_time, reaction_data$condition,
                     paired = TRUE, p.adjust.method = "bonferroni")

    Pairwise comparisons using Wilcoxon signed rank exact test 

data:  reaction_data$reaction_time and reaction_data$condition 

         None   Moderate
Moderate 0.1113 -       
High     0.0059 1.0000  

P value adjustment method: bonferroni

Alternatively, if the PMCMRplus package is available, the Nemenyi test (Nemenyi 1963) provides a more appropriate post-hoc procedure:

if (requireNamespace("PMCMRplus", quietly = TRUE)) {
  library(PMCMRplus)
  frdAllPairsNemenyiTest(reaction_data$reaction_time,
                         reaction_data$condition,
                         reaction_data$subject)
}
         None  Moderate
Moderate 0.037 -       
High     0.010 0.896

130.3 R code

To compute the Friedman Test on your local machine, the following script can be used in the R console:

# Example: Compare the effectiveness of three teaching methods
# 12 students are tested under all three methods (within-subjects design)
set.seed(123)
n <- 12
method_A <- round(rnorm(n, mean = 70, sd = 10))
method_B <- round(rnorm(n, mean = 75, sd = 10))
method_C <- round(rnorm(n, mean = 80, sd = 10))

scores_matrix <- cbind(method_A, method_B, method_C)
colnames(scores_matrix) <- c("Method A", "Method B", "Method C")

# Friedman test
friedman.test(scores_matrix)

# Post-hoc: pairwise Wilcoxon signed-rank tests
scores_long <- data.frame(
  student = factor(rep(1:n, 3)),
  method = factor(rep(c("A", "B", "C"), each = n)),
  score = c(method_A, method_B, method_C)
)
pairwise.wilcox.test(scores_long$score, scores_long$method,
                     paired = TRUE, p.adjust.method = "bonferroni")
Warning in wilcox.test.default(xi, xj, paired = paired, ...): cannot compute
exact p-value with ties
Warning in wilcox.test.default(xi, xj, paired = paired, ...): cannot compute
exact p-value with ties
Warning in wilcox.test.default(xi, xj, paired = paired, ...): cannot compute
exact p-value with ties

    Friedman rank sum test

data:  scores_matrix
Friedman chi-squared = 6, df = 2, p-value = 0.04979


    Pairwise comparisons using Wilcoxon signed rank test with continuity correction 

data:  scores_long$score and scores_long$method 

  A     B    
B 1.000 -    
C 0.023 0.274

P value adjustment method: bonferroni

Note that friedman.test() expects either a matrix (subjects in rows, conditions in columns) or a formula with a grouping variable and a block variable.

130.4 Assumptions

The Friedman Test makes the following assumptions:

  • Dependent/repeated measures: The data must come from a within-subjects design where the same subjects (or matched blocks) are measured under each condition.
  • Ordinal or continuous outcome: The response variable must be measured on at least an ordinal scale.
  • Balanced design: Each subject must have exactly one observation per condition (no missing values).

There is no need to assume normality or sphericity, which makes the Friedman Test more robust than the Repeated Measures ANOVA.

130.5 Alternatives

  • Repeated Measures ANOVA (Chapter 129): When the assumptions of normality and sphericity are satisfied, the parametric Repeated Measures ANOVA has greater statistical power.
  • Linear mixed-effects models: A more flexible approach that can handle unbalanced designs and missing data. Available through the lme4 package in R.