Futility audit

Theresa May’s “racial disparity audit” announced on 27 August, is really just a political gesture that works best if it never delivers findings. I’m reminded of the scene in Yes, Minister (or is it The Thick Of It? Or both?) where the protagonists are all in trouble for something and when the prime minister announces that there will be a public inquiry to find out what went wrong, they are delighted. They know that inquiries are the political equivalent of long grass, with the intention being that everybody involved has retired by the time it reports*.

Larry knows better than to look for mice in 300,000 different places.

It’s not entirely clear what is meant by audit here. Not in the accountants’ sense, surely. Something more like clinical audit? Audit, done properly, is pretty cool. Timely information on performance can get fed back to professionals who run public services, and they can use those data to examine potential problems and improve what they do. But when central agencies examine the data and make the call, it is not the same thing. The trouble is that, whatever indicators you measure, indicators can only indicate; it takes understanding of the local context to see whether it really is a problem.

But there’s another, more statistical problem in this plan: it is impossible to deliver all those goals in the announcement from the prime ministers office:

  • audit to shine a light on how our public services treat people from different backgrounds
  • public will be able to check how their race affects how they are treated on key issues such as health, education and employment, broken down by geographic location, income and gender
  • the audit will show disadvantages suffered by white working class people as well as ethnic minorities
  • the findings from this audit will influence government policy to solve these problems

So that pulls together data across the country from all providers of health services, all schools and colleges, all employers. There needs to be sufficient numbers to break them down into categories by ethnicity (18 categories are used by the Census in England), location at sufficient scale to influence policy (152 local authorities, presumably), income (maybe deciles?) and gender (in this context, they probably need more than two, let’s allow four). Also, social class has been dropped into the objectives, so they will need to collect at least three categories there.

This gives about 300,000 combinations. Inside each of these, sufficient data are needed in order to give precise estimates of fairly rare (one hopes) adverse outcomes. Let’s say maybe 200 people’s data. On total, data from 60,000,000 people, which is just short of the entire UK population, but that includes babies etc, who are not relevant to some of the indicators above. Oh dear. Now, those data need to be collected in a consistent and comparable way, analysed and fed back, including a public-friendly league table from the sounds of it, in timely fashion, say within six months of starting.

I’m being fast and loose with the required sample size, because there are some efficiency savings through excluding irrelevant combinations, multilevel modeling, assumptions of linearities or conditional independence etc, but it is still hopeless. I suspect then that this was never intended actually to happen, but just to be a sop to critics who regard our current government as representing the interests of white UK citizens only, while throwing some scraps to disenchanted white working class voters who chose Brexit and might now be disappointed that police are not going door to door rounding up Johnny Foreign.

One more concern and then I’ll be done: when politicians ask experts to do something, and everybody says no, they sometimes like to look for trimmed down versions such as a simpler analysis based on previously collected data. After all, it would be embarrassing to admit that you couldn’t do a project. However, that would be a serious mistake because of the inconsistencies and problems in making the extant sources commensurate. I hope any agency or academic department approached says no to this foolish quest.

* – you might like to compare with Nick Bostrom’s criticism of the great number of twenty-year predictions for technology: close enough to be exciting, but still after the predictor’s retirement.


1 Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s