 |
| Rensis Likert, inventor of the Likert scale |
The NSS includes a question asking respondents to rate their response to the statement “I am satisfied with the Students' Union at my institution”, where a 1 corresponds to Strongly Disagree, a 5 to Strongly Agree, and 2-4 are the points in between, with 3 being neutral. It’s a five-item Likert scale of the sort very commonly used in surveys.
The NSS data is used in the Key Information Set, with the score for a Students’ Union being calculated as the sum of the Agree and Strongly Agree ratings. For example, a students’ union with the following outcomes in NSS Q24:
|
|
1
|
2
|
3
|
4
|
5
|
|
Poppleton
University
|
3%
|
15%
|
10%
|
45%
|
27%
|
would be ranked as 72% on KIS (45% + 27% = 72%).
But there are different combinations of results which would sum to 72% positive: to take a more extreme case:
|
|
1
|
2
|
3
|
4
|
5
|
|
Poppleton Metropolitan
University
|
23%
|
5%
|
0%
|
55%
|
17%
|
This also has a KIS rating of 72%, but it doesn’t look like the same sort of response at all: almost a quarter of respondents are really unsatisfied with Poppleton Metropolitan SU, as opposed to the handful at Poppleton SU.
A way round this problem is to use a Grade-Point Average (GPA) measure. The combines in one number the proportion of respondents at each level – so, for instance, Poppleton University SU’s GPA would be 3% of 1 plus 15% of 2 plus 10% of 3 plus 45% of 4 plus 27% of 5, which works out at a GPA of 3.78. And this GPA of 3.78 means that the average respondent fell between ‘Neither agree nor disagree’ (3) and ‘Agree’ (4), and was nearer to agree than to the neutral position.
Compare this with Poppleton Metropolitan SU, which has a GPA of 3.38. Again, this falls between ‘Neither agree nor disagree’ and ‘Agree’, but is closer to neutral than to a positive endorsement. So both SU’s have a positive GPA, but it’s clearer that there’s also a difference in how students perceive each of them.
“OK Hugh,” I hear the more polite ones amongst you say, “thanks for the statistics lesson. But so what?” Always a good question. Let’s have a look at the difference it makes in practice.
Here’s the top ten Students’ Unions, in NSS 2015, using the KIS measure and the GPA:
|
|
KIS Measure
|
GPA Measure
|
|
1
|
The
University of Sheffield
|
The
University of Sheffield
|
|
2
|
St Mary's
University College
|
Loughborough
University
|
|
3
|
Loughborough
University
|
St Mary's
University College
|
|
4
|
The
University of Leeds
|
The
University of Leeds
|
|
5
|
University
of Dundee
|
University
of Dundee
|
|
6
|
Royal
Northern College of Music
|
Cardiff
University
|
|
7
|
Cardiff
University
|
Royal
Welsh College of Music and Drama
|
|
8
|
Teesside
University
|
Royal
Northern College of Music
|
|
9
|
Royal
Welsh College of Music and Drama
|
The
University of Keele
|
|
10
|
The
University of Keele
|
Teesside
University
|
These are the same ten universities, but in a different order, as Eric Morecambe might have said. So perhaps it isn’t the end the world what measure we use. What about the bottom ten?
|
|
KIS Measure
|
GPA Measure
|
|
150
|
The
University of Westminster
|
University
for the Creative Arts
|
|
151
|
Oxford
Brookes University
|
Ravensbourne
|
|
152
|
Queen
Margaret University, Edinburgh
|
Oxford
Brookes University
|
|
153
|
Liverpool
Institute for Performing Arts
|
Queen
Margaret University, Edinburgh
|
|
154
|
University
of the Highlands and Islands
|
Liverpool
Institute for Performing Arts
|
|
155
|
Royal
Conservatoire of Scotland
|
Royal
Conservatoire of Scotland
|
|
156
|
University
of Durham
|
University
of Durham
|
|
157
|
University
of Bristol
|
University
of Oxford
|
|
158
|
University
of Oxford
|
University
of Cambridge
|
|
159
|
University
of Cambridge
|
University
of Bristol
|
It isn’t the same ten universities on both sides here: the measure makes a difference for the University of the Highlands and Islands, Ravensbourne, the University of Westminster, and the University of the Creative Arts.
And there are some very stark differences if you look within the rankings. Let’s have a look at the top ten Students’ Unions for differential GPA-versus-KIS-ranking-performance (take a deep breath and say it slowly - it’s only a matter of time before this becomes a standard measure, you know …):
|
|
GPA Rank
|
KIS Rank
|
Difference
|
|
The
University of Law
|
66
|
111
|
45
|
|
The Open
University
|
69
|
112
|
43
|
|
Heythrop
College
|
119
|
86
|
33
|
|
Hull and
York Medical School
|
38
|
66
|
28
|
|
Birkbeck
College
|
113
|
137
|
24
|
|
University
of Sussex
|
71
|
49
|
22
|
|
Bath Spa
University
|
110
|
89
|
21
|
|
University
of the West of Scotland
|
98
|
118
|
20
|
|
Royal
Holloway, University of London
|
111
|
91
|
20
|
|
The
University of Liverpool
|
115
|
95
|
20
|
In five of these cases the KIS ranking is better, in five the GPA ranking. Let’s have a look at the numbers in detail:
|
|
1
|
2
|
3
|
4
|
5
|
GPA
|
KIS
|
|
The
University of Law
|
4%
|
4%
|
29%
|
20%
|
42%
|
3.89
|
62%
|
|
The Open
University
|
2%
|
2%
|
35%
|
32%
|
30%
|
3.89
|
62%
|
|
Heythrop
College
|
8%
|
13%
|
13%
|
44%
|
23%
|
3.64
|
67%
|
|
Hull and
York Medical School
|
1%
|
2%
|
27%
|
33%
|
37%
|
4.03
|
70%
|
|
Birkbeck
College
|
5%
|
4%
|
34%
|
31%
|
26%
|
3.69
|
57%
|
|
University
of Sussex
|
4%
|
6%
|
16%
|
41%
|
32%
|
3.88
|
73%
|
|
Bath Spa
University
|
6%
|
9%
|
18%
|
42%
|
25%
|
3.71
|
67%
|
|
University
of the West of Scotland
|
5%
|
8%
|
27%
|
30%
|
31%
|
3.77
|
61%
|
|
Royal
Holloway, University of London
|
6%
|
9%
|
20%
|
43%
|
23%
|
3.71
|
66%
|
|
The
University of Liverpool
|
5%
|
8%
|
21%
|
42%
|
23%
|
3.67
|
65%
|
Broadly speaking, if the GPA rank is higher than the KIS rank (Law, Open, Hull-York, Birkbeck, West of Scotland), it points to a more even distribution of scores across the five categories, if KIS is higher than GPA (Heythrop, Sussex, Bath Spa, Royal Holloway, Liverpool), then it points to more extreme scores in 1 and 2. Broadly speaking.
And that’s the key to it. The KIS ranking answers a simple question: how many like it. The GPA ranking is more nuanced, but also requires more interpretation.
That gets to the heart of the ranking and league table problem. Rankings and league tables are a way of simplifying what is complex. There’s often a direct trade-off between simple and realistic. And that’s why it matters what methodology a league table uses, but why it is also inevitable that there’s game playing within the rankings.
So be careful always to read a ranking or a league table having taken a suitably sized pinch of salt.
No comments:
Post a Comment