Making the Grade

Rensis Likert, inventor of the Likert scale

I’ve been looking at the National Student Survey (NSS) rankings of Students’ Unions recently (and more of that another time) but one thing which can’t wait is the question of how you actually rank the outcomes.

The NSS includes a question asking respondents to rate their response to the statement “I am satisfied with the Students' Union at my institution”, where a 1 corresponds to Strongly Disagree, a 5 to Strongly Agree, and 2-4 are the points in between, with 3 being neutral. It’s a five-item Likert scale of the sort very commonly used in surveys.

The NSS data is used in the Key Information Set, with the score for a Students’ Union being calculated as the sum of the Agree and Strongly Agree ratings.  For example, a students’ union with the following outcomes in NSS Q24:

	1	2	3	4	5
Poppleton University	3%	15%	10%	45%	27%

would be ranked as 72% on KIS (45% + 27% = 72%).

But there are different combinations of results which would sum to 72% positive: to take a more extreme case:

	1	2	3	4	5
Poppleton Metropolitan University	23%	5%	0%	55%	17%

This also has a KIS rating of 72%, but it doesn’t look like the same sort of response at all: almost a quarter of respondents are really unsatisfied with Poppleton Metropolitan SU, as opposed to the handful at Poppleton SU.

A way round this problem is to use a Grade-Point Average (GPA) measure. The combines in one number the proportion of respondents at each level – so, for instance, Poppleton University SU’s GPA would be 3% of 1 plus 15% of 2 plus 10% of 3 plus 45% of 4 plus 27% of 5, which works out at a GPA of 3.78.  And this GPA of 3.78 means that the average respondent fell between ‘Neither agree nor disagree’ (3) and ‘Agree’ (4), and was nearer to agree than to the neutral position.

Compare this with Poppleton Metropolitan SU, which has a GPA of 3.38. Again, this falls between ‘Neither agree nor disagree’ and ‘Agree’, but is closer to neutral than to a positive endorsement. So both SU’s have a positive GPA, but it’s clearer that there’s also a difference in how students perceive each of them. 

“OK Hugh,” I hear the more polite ones amongst you say, “thanks for the statistics lesson. But so what?” Always a good question. Let’s have a look at the difference it makes in practice.

Here’s the top ten Students’ Unions, in NSS 2015, using the KIS measure and the GPA:

	KIS Measure	GPA Measure
1	The University of Sheffield	The University of Sheffield
2	St Mary's University College	Loughborough University
3	Loughborough University	St Mary's University College
4	The University of Leeds	The University of Leeds
5	University of Dundee	University of Dundee
6	Royal Northern College of Music	Cardiff University
7	Cardiff University	Royal Welsh College of Music and Drama
8	Teesside University	Royal Northern College of Music
9	Royal Welsh College of Music and Drama	The University of Keele
10	The University of Keele	Teesside University

These are the same ten universities, but in a different order, as Eric Morecambe might have said. So perhaps it isn’t the end the world what measure we use. What about the bottom ten?

	KIS Measure	GPA Measure
150	The University of Westminster	University for the Creative Arts
151	Oxford Brookes University	Ravensbourne
152	Queen Margaret University, Edinburgh	Oxford Brookes University
153	Liverpool Institute for Performing Arts	Queen Margaret University, Edinburgh
154	University of the Highlands and Islands	Liverpool Institute for Performing Arts
155	Royal Conservatoire of Scotland	Royal Conservatoire of Scotland
156	University of Durham	University of Durham
157	University of Bristol	University of Oxford
158	University of Oxford	University of Cambridge
159	University of Cambridge	University of Bristol

It isn’t the same ten universities on both sides here: the measure makes a difference for the University of the Highlands and Islands, Ravensbourne, the University of Westminster, and the University of the Creative Arts.

And there are some very stark differences if you look within the rankings. Let’s have a look at the top ten Students’ Unions for differential GPA-versus-KIS-ranking-performance (take a deep breath and say it slowly - it’s only a matter of time before this becomes a standard measure, you know …):

	GPA Rank	KIS Rank	Difference
The University of Law	66	111	45
The Open University	69	112	43
Heythrop College	119	86	33
Hull and York Medical School	38	66	28
Birkbeck College	113	137	24
University of Sussex	71	49	22
Bath Spa University	110	89	21
University of the West of Scotland	98	118	20
Royal Holloway, University of London	111	91	20
The University of Liverpool	115	95	20

In five of these cases the KIS ranking is better, in five the GPA ranking.  Let’s have a look at the numbers in detail:

	1	2	3	4	5	GPA	KIS
The University of Law	4%	4%	29%	20%	42%	3.89	62%
The Open University	2%	2%	35%	32%	30%	3.89	62%
Heythrop College	8%	13%	13%	44%	23%	3.64	67%
Hull and York Medical School	1%	2%	27%	33%	37%	4.03	70%
Birkbeck College	5%	4%	34%	31%	26%	3.69	57%
University of Sussex	4%	6%	16%	41%	32%	3.88	73%
Bath Spa University	6%	9%	18%	42%	25%	3.71	67%
University of the West of Scotland	5%	8%	27%	30%	31%	3.77	61%
Royal Holloway, University of London	6%	9%	20%	43%	23%	3.71	66%
The University of Liverpool	5%	8%	21%	42%	23%	3.67	65%

Broadly speaking, if the GPA rank is higher than the KIS rank (Law, Open, Hull-York, Birkbeck, West of Scotland), it points to a more even distribution of scores across the five categories, if KIS is higher than GPA (Heythrop, Sussex, Bath Spa, Royal Holloway, Liverpool), then it points to more extreme scores in 1 and 2. Broadly speaking.

And that’s the key to it. The KIS ranking answers a simple question: how many like it.  The GPA ranking is more nuanced, but also requires more interpretation. 

That gets to the heart of the ranking and league table problem. Rankings and league tables are a way of simplifying what is complex. There’s often a direct trade-off between simple and realistic. And that’s why it matters what methodology a league table uses, but why it is also inevitable that there’s game playing within the rankings. 

So be careful always to read a ranking or a league table having taken a suitably sized pinch of salt.

An engagement challenge

Here’s a challenge to our universities – can you match the academic outreach work of one of the Ivy League giants?

A big hat-tip to AnneMarie Cunningham (@amcunningham) for tweeting an article from Quartz about the Stanford Encyclopedia [sic] of Philosophy, which I hadn’t come across before. In short, the SEP is an online encyclopaedia of philosophy (the name gives that bit away) which is academically edited, rigorous, up-to-date, and free to the world.

I’m not going to be able to improve on Nikhil Sonnad’s article about the SEP, and he also has a good line or two about the inevitable shortcomings of Wikipedia. Instead I want to pick up on the question of why there aren’t more initiatives like this.

Denis Diderot - an earlier encyclopaedist

If universities have any single purpose, it is surely about the creation, preservation and dissemination of knowledge. This is operationalised through teaching and research, but at its heart the university is a means of ensuring that we learn as a species and that we remember the lessons learnt before us. This is a grand and noble challenge (and also why I think that helping universities be better is a good thing for me to do.)

Stanford had to seed fund the SEP, and despite the various grants from foundations and trusts to fund its development, still bears some of the recurrent costs. But the amount must be negligible, compared to Stanford’s overall funds. And the reputational gains for a UK university to do something similar (in a different discipline, obviously) would be large. And think of the clear contribution to meeting the charitable objects of the university.

Would some disciplines be more amenable to the encyclopaedic treatment than others? Perhaps so – the speed with which some scientific disciplines develop, and the very minor developments which can count as progress, mean that an encyclopaedia of physics, say, may well be less useful than the latest journals in the field. But I don’t expect the SEP will replace the journals in philosophy: maybe I’m overplaying this.

Here, then, is the challenge – will a UK university step up to the plate and fund a similar development? And if so, can we spell encyclopaedia properly, please?