Making the Grade

Rensis Likert, inventor of the Likert scale

I’ve been looking at the National Student Survey (NSS) rankings of Students’ Unions recently (and more of that another time) but one thing which can’t wait is the question of how you actually rank the outcomes.

The NSS includes a question asking respondents to rate their response to the statement “I am satisfied with the Students' Union at my institution”, where a 1 corresponds to Strongly Disagree, a 5 to Strongly Agree, and 2-4 are the points in between, with 3 being neutral. It’s a five-item Likert scale of the sort very commonly used in surveys.

The NSS data is used in the Key Information Set, with the score for a Students’ Union being calculated as the sum of the Agree and Strongly Agree ratings.  For example, a students’ union with the following outcomes in NSS Q24:

	1	2	3	4	5
Poppleton University	3%	15%	10%	45%	27%

would be ranked as 72% on KIS (45% + 27% = 72%).

But there are different combinations of results which would sum to 72% positive: to take a more extreme case:

	1	2	3	4	5
Poppleton Metropolitan University	23%	5%	0%	55%	17%

This also has a KIS rating of 72%, but it doesn’t look like the same sort of response at all: almost a quarter of respondents are really unsatisfied with Poppleton Metropolitan SU, as opposed to the handful at Poppleton SU.

A way round this problem is to use a Grade-Point Average (GPA) measure. The combines in one number the proportion of respondents at each level – so, for instance, Poppleton University SU’s GPA would be 3% of 1 plus 15% of 2 plus 10% of 3 plus 45% of 4 plus 27% of 5, which works out at a GPA of 3.78.  And this GPA of 3.78 means that the average respondent fell between ‘Neither agree nor disagree’ (3) and ‘Agree’ (4), and was nearer to agree than to the neutral position.

Compare this with Poppleton Metropolitan SU, which has a GPA of 3.38. Again, this falls between ‘Neither agree nor disagree’ and ‘Agree’, but is closer to neutral than to a positive endorsement. So both SU’s have a positive GPA, but it’s clearer that there’s also a difference in how students perceive each of them. 

“OK Hugh,” I hear the more polite ones amongst you say, “thanks for the statistics lesson. But so what?” Always a good question. Let’s have a look at the difference it makes in practice.

Here’s the top ten Students’ Unions, in NSS 2015, using the KIS measure and the GPA:

	KIS Measure	GPA Measure
1	The University of Sheffield	The University of Sheffield
2	St Mary's University College	Loughborough University
3	Loughborough University	St Mary's University College
4	The University of Leeds	The University of Leeds
5	University of Dundee	University of Dundee
6	Royal Northern College of Music	Cardiff University
7	Cardiff University	Royal Welsh College of Music and Drama
8	Teesside University	Royal Northern College of Music
9	Royal Welsh College of Music and Drama	The University of Keele
10	The University of Keele	Teesside University

These are the same ten universities, but in a different order, as Eric Morecambe might have said. So perhaps it isn’t the end the world what measure we use. What about the bottom ten?

	KIS Measure	GPA Measure
150	The University of Westminster	University for the Creative Arts
151	Oxford Brookes University	Ravensbourne
152	Queen Margaret University, Edinburgh	Oxford Brookes University
153	Liverpool Institute for Performing Arts	Queen Margaret University, Edinburgh
154	University of the Highlands and Islands	Liverpool Institute for Performing Arts
155	Royal Conservatoire of Scotland	Royal Conservatoire of Scotland
156	University of Durham	University of Durham
157	University of Bristol	University of Oxford
158	University of Oxford	University of Cambridge
159	University of Cambridge	University of Bristol

It isn’t the same ten universities on both sides here: the measure makes a difference for the University of the Highlands and Islands, Ravensbourne, the University of Westminster, and the University of the Creative Arts.

And there are some very stark differences if you look within the rankings. Let’s have a look at the top ten Students’ Unions for differential GPA-versus-KIS-ranking-performance (take a deep breath and say it slowly - it’s only a matter of time before this becomes a standard measure, you know …):

	GPA Rank	KIS Rank	Difference
The University of Law	66	111	45
The Open University	69	112	43
Heythrop College	119	86	33
Hull and York Medical School	38	66	28
Birkbeck College	113	137	24
University of Sussex	71	49	22
Bath Spa University	110	89	21
University of the West of Scotland	98	118	20
Royal Holloway, University of London	111	91	20
The University of Liverpool	115	95	20

In five of these cases the KIS ranking is better, in five the GPA ranking.  Let’s have a look at the numbers in detail:

	1	2	3	4	5	GPA	KIS
The University of Law	4%	4%	29%	20%	42%	3.89	62%
The Open University	2%	2%	35%	32%	30%	3.89	62%
Heythrop College	8%	13%	13%	44%	23%	3.64	67%
Hull and York Medical School	1%	2%	27%	33%	37%	4.03	70%
Birkbeck College	5%	4%	34%	31%	26%	3.69	57%
University of Sussex	4%	6%	16%	41%	32%	3.88	73%
Bath Spa University	6%	9%	18%	42%	25%	3.71	67%
University of the West of Scotland	5%	8%	27%	30%	31%	3.77	61%
Royal Holloway, University of London	6%	9%	20%	43%	23%	3.71	66%
The University of Liverpool	5%	8%	21%	42%	23%	3.67	65%

Broadly speaking, if the GPA rank is higher than the KIS rank (Law, Open, Hull-York, Birkbeck, West of Scotland), it points to a more even distribution of scores across the five categories, if KIS is higher than GPA (Heythrop, Sussex, Bath Spa, Royal Holloway, Liverpool), then it points to more extreme scores in 1 and 2. Broadly speaking.

And that’s the key to it. The KIS ranking answers a simple question: how many like it.  The GPA ranking is more nuanced, but also requires more interpretation. 

That gets to the heart of the ranking and league table problem. Rankings and league tables are a way of simplifying what is complex. There’s often a direct trade-off between simple and realistic. And that’s why it matters what methodology a league table uses, but why it is also inevitable that there’s game playing within the rankings. 

So be careful always to read a ranking or a league table having taken a suitably sized pinch of salt.