DRAFT This draft is under development — DO NOT CITE OR SHARE.

84  Survey Scores Rank Order Comparison

84.1 Definition

The Survey Scores Rank Order Comparison (SSROC) attempts to assess whether or not a series of survey scores for similar questions¹ (which are based on a Likert scale²) can be treated as quasi-interval variables (i.e. quantitative variables for which it is possible to compute quantitative statistics, such as the Arithmetic Mean).

In a first step, the SSROC computes the following statistics:

• The Arithmetic Average \(\bar{X}\) of the Likert Scores for each item (i.e. question).
• The sum of positive scores \(P_s\) for the Likert scores, after subtracting the midpoint of the Likert scale.
• The absolute sum of negative scores \(N_s\) for the Likert scores, after subtracting the midpoint of the Likert scale.
• The count of positive scores \(P_c\) for the Likert scores, after subtracting the midpoint of the Likert scale.
• The count of negative scores \(N_c\) for the Likert scores, after subtracting the midpoint of the Likert scale.
• The sum-based statistic \(A_s = \frac{P_s - N_s}{P_s + N_s}\).
• The count-based statistic \(A_c = \frac{P_c - N_c}{P_c + N_c}\).

For instance, if the 5-point Likert scale \(X = [4, 3, 1]\) then the following values are obtained:

• \(\bar{X} = \frac{4 + 3 + 1}{3} \simeq 2.667\).
• \(P_s = 1\) because the sum of positive values in \(X - \text{midpoint of Likert scale} = [4 - 3, 3 - 3, 1 - 3] = [1, 0, -2]\) is 1.
• \(N_s = 2\) because the absolute sum of negative values in \(X - \text{midpoint of Likert scale} = [4 - 3, 3 - 3, 1 - 3] = [1, 0, -2]\) is 2.
• \(P_c = 1\) because the count of positive values in \([1, 0, -2]\) is 1.
• \(N_c = 1\) because the count of negative values in \([1, 0, -2]\) is 1.
• The sum-based average \(A_s = \frac{1 - 2}{1 + 2} = \frac{-1}{3}\).
• The count-based average \(A_c = \frac{1 - 1}{1 + 1} = 0\).

The Arithmetic Mean should only be used if the underlying Likert scores \(X\) can be interpreted as a (truly) quantitative variable (rather than an ordinal variable). The count-based average \(A_c\), however, can always be used without problems (even if the Likert scores can not be interpreted as a quantitative variable). The sum-based average \(A_s\) is somewhere in-between.

Hence, if all these statistics are computed for similar questions it is possible to assess whether or not the three statistics (i.e. \(\bar{X}\), \(A_s\), and \(A_c\)) preserve their rank orders. If the rank order is preserved (i.e. the Rank Order Correlation is close to +1) then the Likert scores can be interpreted as quantitative variables (because the result of ranking similar questions does not depend on which statistic is used). However, if the rank orders are not strongly correlated then it is certainly not wise to treat the Likert scores as a truly quantitative variable. As a practical guideline, Rank Order Correlations above 0.9 suggest that the three statistics agree and the Arithmetic Mean can be used with reasonable confidence; lower correlations indicate that the ordinal nature of the scale should be respected.

84.2 R Module

84.2.1 Public website

The SSROC is available on the public website:

• https://compute.wessa.net/rwasp_surveyscores.wasp

84.2.2 RFC

The SSROC is also available in RFC under the “Descriptive / Multivariate Descriptive Statistics” menu item.

To compute the SSROC on your local machine, the following script can be used in the R console:

x <- array(round(runif(241*20, 1, 5)), dim=c(241,20), 
           dimnames=list(1:241, c('A1','A2','A3','A4','A5','A6','A7','A8','A9','A10',
                                  'A11','A12','A13','A14','A15','A16','A17','A18','A19','A20')))
par1 = '1 2 3 4 5'
docor <- function(x,y,method) {
  r <- cor.test(x,y,method=method)
  paste(round(r$estimate,3),' (',round(r$p.value,3),')',sep='')
}
nx <- length(x[,1])
cx <- length(x[1,])
mymedian <- median(as.numeric(strsplit(par1,' ')[[1]]))
myresult <- array(NA, dim = c(cx,7))
rownames(myresult) <- paste('Q',1:cx,sep='')
colnames(myresult) <- c('mean',
                        'Sum of pos (Ps)',
                        'Sum of neg (Ns)', 
                        '(Ps-Ns)/(Ps+Ns)', 
                        'Count of pos (Pc)', 
                        'Count of neg (Nc)', 
                        '(Pc-Nc)/(Pc+Nc)')
for (i in 1:cx) {
  spos <- 0
  sneg <- 0
  cpos <- 0
  cneg <- 0
  for (j in 1:nx) {
    if (!is.na(x[j,i])) {
      myx <- as.numeric(x[j,i]) - mymedian
      if (myx > 0) {
        spos = spos + myx
        cpos = cpos + 1
      }
      if (myx < 0) {
        sneg = sneg + abs(myx)
        cneg = cneg + 1
      }
    }
  }
  myresult[i,1] <- round(mean(as.numeric(x[,i]),na.rm=T)-mymedian,2)
  myresult[i,2] <- spos
  myresult[i,3] <- sneg
  myresult[i,4] <- round((spos - sneg) / (spos + sneg),2)
  myresult[i,5] <- cpos
  myresult[i,6] <- cneg
  myresult[i,7] <- round((cpos - cneg) / (cpos + cneg),2)
}
print(myresult)

cat("\nPearson correlations of survey scores\n")
cor(myresult[,c(1,4,7)], method = "pearson")
cat("\nKendall rank correlations of survey scores\n")
cor(myresult[,c(1,4,7)], method = "kendall")

     mean Sum of pos (Ps) Sum of neg (Ns) (Ps-Ns)/(Ps+Ns) Count of pos (Pc)
Q1   0.03             124             116            0.03                94
Q2  -0.01             116             118           -0.01                86
Q3  -0.02             121             126           -0.02                86
Q4   0.06             129             114            0.06                95
Q5   0.04             122             113            0.04                87
Q6   0.02             116             110            0.03                86
Q7   0.03             123             116            0.03                95
Q8  -0.05             107             119           -0.05                81
Q9  -0.08             115             135           -0.08                91
Q10  0.03             122             115            0.03                95
Q11 -0.01             116             118           -0.01                88
Q12 -0.05             109             122           -0.06                87
Q13 -0.06             114             129           -0.06                87
Q14  0.02             120             114            0.03                89
Q15 -0.12             109             137           -0.11                83
Q16 -0.05             117             129           -0.05                86
Q17  0.09             139             118            0.08                99
Q18 -0.07             114             130           -0.07                87
Q19  0.21             157             106            0.19               114
Q20 -0.06             117             132           -0.06                90
    Count of neg (Nc) (Pc-Nc)/(Pc+Nc)
Q1                 89            0.03
Q2                 91           -0.03
Q3                 89           -0.02
Q4                 91            0.02
Q5                 83            0.02
Q6                 83            0.02
Q7                 87            0.04
Q8                 97           -0.09
Q9                102           -0.06
Q10                81            0.08
Q11                89           -0.01
Q12                92           -0.03
Q13                96           -0.05
Q14                83            0.03
Q15               101           -0.10
Q16                95           -0.05
Q17                87            0.06
Q18               100           -0.07
Q19                80            0.18
Q20                95           -0.03

Pearson correlations of survey scores
                     mean (Ps-Ns)/(Ps+Ns) (Pc-Nc)/(Pc+Nc)
mean            1.0000000       0.9966956       0.9474221
(Ps-Ns)/(Ps+Ns) 0.9966956       1.0000000       0.9462873
(Pc-Nc)/(Pc+Nc) 0.9474221       0.9462873       1.0000000

Kendall rank correlations of survey scores
                     mean (Ps-Ns)/(Ps+Ns) (Pc-Nc)/(Pc+Nc)
mean            1.0000000       0.9720482       0.7768625
(Ps-Ns)/(Ps+Ns) 0.9720482       1.0000000       0.7452413
(Pc-Nc)/(Pc+Nc) 0.7768625       0.7452413       1.0000000

To compute the SSROC, the R code iterates over all columns of the multivariate dataset and computes the various statistics (see column names of myresult). In addition, the Pearson and Kendall correlation matrices are also computed.

84.3 Purpose

The purpose of the SSROC is to determine whether Likert scores of similar items in a survey can be treated as quantitative variables (which allows one to compute the Arithmetic Mean and other quantitative statistics). If this is not the case, the data should be properly categorized and/or treated as purely qualitative data.

84.4 Pros & Cons

84.4.1 Pros

The SSROC has the following advantages:

• It allows one to assess the appropriateness of quantitative statistics (such as the Arithmetic Mean) for Likert scores of similar items in a survey.
• It is easy to compute the alternative averages \(A_s\) and \(A_c\).
• The interpretation of \(A_c\) is very easy and has the potential to be very informative (it can always be applied and it does not depend on the neutral scores).

84.4.2 Cons

The SSROC has the following disadvantages:

• Most readers do not know about the SSROC.
• Most researchers simply compute quantitative statistics for Likert scores and do not want to be bothered about the validity of doing so.
• It can only be used if there are sufficient items that can be compared in the rank order comparison.

84.5 Example

The following analysis shows the three statistics (\(\bar{X}\), \(A_s\), and \(A_c\)) for 10 similar items based on 7-point Likert scores. The statistics \(\bar{X}\) and \(\frac{P_s-N_s}{P_s+N_s}\) are scaled between -3 and 3 (because we use a 7-point Likert score). The statistic \(\frac{P_c-N_c}{P_c+N_c}\) always has a minimum of -1 and a maximum of 1.

Interactive Shiny app (click to load).

Open in new tab

Observe how, for instance, the first item has a count-based average score of 0.59 (this is 59% of the maximum) while the Arithmetic Mean is only 0.86 (for a maximum of +3). The count-based score does not only apply to ordinal as well as quantitative variables, it also ignores the neutral scores of the survey!

84.6 Task

In the previous example, change the scale and examine what happens with the scores. Do you see why it is important to set the correct Likert scale?

---

1. In this context, similar questions are defined as questions which attempt to measure the same underlying opinion but with different phrases. Surveys often contain similar questions in order to improve the validity of the survey.

2. A Likert scale represents an ordinal measurement. For instance, a 5-point Likert scale assigns the values 1, 2, 3, 4, and 5 to represent the degree to which the respondent agrees with a specified statement (i.e. 1 = totally disagree and 5 = totally agree).