Tuesday 6 October 2015

Making the Grade

Rensis Likert, inventor of the Likert scale
I’ve been looking at the National Student Survey (NSS) rankings of Students’ Unions recently (and more of that another time) but one thing which can’t wait is the question of how you actually rank the outcomes.

The NSS includes a question asking respondents to rate their response to the statement “I am satisfied with the Students' Union at my institution”, where a 1 corresponds to Strongly Disagree, a 5 to Strongly Agree, and 2-4 are the points in between, with 3 being neutral. It’s a five-item Likert scale of the sort very commonly used in surveys.

The NSS data is used in the Key Information Set, with the score for a Students’ Union being calculated as the sum of the Agree and Strongly Agree ratings.  For example, a students’ union with the following outcomes in NSS Q24:


1
2
3
4
5
Poppleton University
3%
15%
10%
45%
27%

would be ranked as 72% on KIS (45% + 27% = 72%).

But there are different combinations of results which would sum to 72% positive: to take a more extreme case:


1
2
3
4
5
Poppleton Metropolitan University
23%
5%
0%
55%
17%

This also has a KIS rating of 72%, but it doesn’t look like the same sort of response at all: almost a quarter of respondents are really unsatisfied with Poppleton Metropolitan SU, as opposed to the handful at Poppleton SU.

A way round this problem is to use a Grade-Point Average (GPA) measure. The combines in one number the proportion of respondents at each level – so, for instance, Poppleton University SU’s GPA would be 3% of 1 plus 15% of 2 plus 10% of 3 plus 45% of 4 plus 27% of 5, which works out at a GPA of 3.78.  And this GPA of 3.78 means that the average respondent fell between ‘Neither agree nor disagree’ (3) and ‘Agree’ (4), and was nearer to agree than to the neutral position.

Compare this with Poppleton Metropolitan SU, which has a GPA of 3.38. Again, this falls between ‘Neither agree nor disagree’ and ‘Agree’, but is closer to neutral than to a positive endorsement. So both SU’s have a positive GPA, but it’s clearer that there’s also a difference in how students perceive each of them. 

“OK Hugh,” I hear the more polite ones amongst you say, “thanks for the statistics lesson. But so what?” Always a good question. Let’s have a look at the difference it makes in practice.

Here’s the top ten Students’ Unions, in NSS 2015, using the KIS measure and the GPA:


KIS Measure
GPA Measure
1
The University of Sheffield
The University of Sheffield
2
St Mary's University College
Loughborough University
3
Loughborough University
St Mary's University College
4
The University of Leeds
The University of Leeds
5
University of Dundee
University of Dundee
6
Royal Northern College of Music
Cardiff University
7
Cardiff University
Royal Welsh College of Music and Drama
8
Teesside University
Royal Northern College of Music
9
Royal Welsh College of Music and Drama
The University of Keele
10
The University of Keele
Teesside University

These are the same ten universities, but in a different order, as Eric Morecambe might have said. So perhaps it isn’t the end the world what measure we use. What about the bottom ten?


KIS Measure
GPA Measure
150
The University of Westminster
University for the Creative Arts
151
Oxford Brookes University
Ravensbourne
152
Queen Margaret University, Edinburgh
Oxford Brookes University
153
Liverpool Institute for Performing Arts
Queen Margaret University, Edinburgh
154
University of the Highlands and Islands
Liverpool Institute for Performing Arts
155
Royal Conservatoire of Scotland
Royal Conservatoire of Scotland
156
University of Durham
University of Durham
157
University of Bristol
University of Oxford
158
University of Oxford
University of Cambridge
159
University of Cambridge
University of Bristol

It isn’t the same ten universities on both sides here: the measure makes a difference for the University of the Highlands and Islands, Ravensbourne, the University of Westminster, and the University of the Creative Arts.

And there are some very stark differences if you look within the rankings. Let’s have a look at the top ten Students’ Unions for differential GPA-versus-KIS-ranking-performance (take a deep breath and say it slowly - it’s only a matter of time before this becomes a standard measure, you know …):


GPA Rank
KIS Rank
Difference
The University of Law
66
111
45
The Open University
69
112
43
Heythrop College
119
86
33
Hull and York Medical School
38
66
28
Birkbeck College
113
137
24
University of Sussex
71
49
22
Bath Spa University
110
89
21
University of the West of Scotland
98
118
20
Royal Holloway, University of London
111
91
20
The University of Liverpool
115
95
20

In five of these cases the KIS ranking is better, in five the GPA ranking.  Let’s have a look at the numbers in detail:


1
2
3
4
5
GPA
KIS
The University of Law
4%
4%
29%
20%
42%
3.89
62%
The Open University
2%
2%
35%
32%
30%
3.89
62%
Heythrop College
8%
13%
13%
44%
23%
3.64
67%
Hull and York Medical School
1%
2%
27%
33%
37%
4.03
70%
Birkbeck College
5%
4%
34%
31%
26%
3.69
57%
University of Sussex
4%
6%
16%
41%
32%
3.88
73%
Bath Spa University
6%
9%
18%
42%
25%
3.71
67%
University of the West of Scotland
5%
8%
27%
30%
31%
3.77
61%
Royal Holloway, University of London
6%
9%
20%
43%
23%
3.71
66%
The University of Liverpool
5%
8%
21%
42%
23%
3.67
65%

Broadly speaking, if the GPA rank is higher than the KIS rank (Law, Open, Hull-York, Birkbeck, West of Scotland), it points to a more even distribution of scores across the five categories, if KIS is higher than GPA (Heythrop, Sussex, Bath Spa, Royal Holloway, Liverpool), then it points to more extreme scores in 1 and 2. Broadly speaking.

And that’s the key to it. The KIS ranking answers a simple question: how many like it.  The GPA ranking is more nuanced, but also requires more interpretation. 

That gets to the heart of the ranking and league table problem. Rankings and league tables are a way of simplifying what is complex. There’s often a direct trade-off between simple and realistic. And that’s why it matters what methodology a league table uses, but why it is also inevitable that there’s game playing within the rankings. 

So be careful always to read a ranking or a league table having taken a suitably sized pinch of salt.

No comments:

Post a Comment