League tables

Used a lot in sport! Basically ranking things, typically in our field, services, by some performance metric as a way of deciding which are not performing well (far more often than with the itention of finding out how it is that the services ranked best are doing that).

Details #

The idea is simple, easy to understand and can be useful but there are six main weaknesses.

The first has really been introduced above: they tend to be used with the assumption that the ranking criterion is uniquivocal and hence can be used to make relative judgements. Sometimes simply using them as a basis for exploration of differences might be much more appropriate particularly if any of #4, #5 or #6 apply. (And they often do!)
This is very simple. If you have purely random continuous variables on which you rank things then one thing (if there is sufficient range of values in the variable) will be the “best” and one will be the “worst”. However as the variable is random then there is no true difference between any of the ranked things. A lesser but still crucial aspect of this problem is that sometimes the differences between many of the things ranked may be purely random despite some of them coming above the middle rank and some below. Depending what is being measured there are statistical methods that can help say whether differences in ranks are robust or likely to be random.
An exacerbation of that issue for may things that may be used for league tables in our fields is that the variable being used may be measured unreliable so you will have randomness in the ranking on the league table down to sampling fluctations and down to measurement unreliability.
Choice of ranking variable. It is very tempting, whether considering the issue above or not, to rank on very easily, often also on very cheap to measure, variables. These may not be the most important ones when we think seriously about what we want from good services.
Relating to #4 often what we want from good services is complex and not at all reducible to a single variable. We will want economic efficiency: the best of everything else and at the lowest cost. We will want brief waits for therapy and equity in service delivery by gender, ethnicity, age, severity of problems. We will want good changes to be achieved by clients but this may require longer therapies, clashing with waiting times and economic costs minimisation. We will want to have a low suicide rate but suicides are rare (we hope) so how do we weight them against less severe measures of change? There are statistical methods that in principle can rank, i.e. create league tables, even with multivariate measures but the reality is that the criteria for those methods to apply simply aren’t present in real life for services.
Differences between the challenges faced by services. Services in areas of great social deprivation, perhaps services in areas with huge ethnic diversity and mistrust of professional services probably make it very hard for these services to achieve the rankings that services in largely socially well off and homogenously fairly priviledged areas will achieve. Again, in priniciple there are statistical methods to take into account such “predictor” or contextual variables but the realities of modelling them robustly are almost never present.