Tag Archives: quality indicators

Performance indicators and routine data on child protection services

The parts of social services that do child protection in England get inspected by Ofsted on behalf of the Department for Education (DfE). The process is analogous to the Care Quality Commission inspections of healthcare and adult social care providers, and they both give out ratings of ‘Inadequate’, ‘Requires Improvement’, ‘Good’ or ‘Outstanding’. In the health setting, there’s many years’ experience of quantitative quality (or performance) indicators, often through a local process called clinical audit and sometimes nationally. I’ve been involved with clinical audit for many years. One general trend over that time has been away from de novo data collection and towards recycling routinely collected data. Especially in the era of big data, lots of organisations are very excited about Leveraging Big Data Analytics to discover who’s outstanding, who sucks, and how to save lives all over the place. Now, it may not be that simple, but there is definitely merit in using existing data.

This trend is just appearing on the horizon for social care though, because records are less organised and electronic, and because there just hasn’t been that culture of profession-led audit. Into this scene came my colleagues Rick Hood (complex systems thinker) and Ray Jones (now retired professor and general Colossus of UK social care). They wanted to investigate recently open-sourced data on child protection services and asked if I would be interested to join in. I was – and I wanted to consider this question: could routine data replace Ofsted inspections? I suspected not! But I also suspected that question would soon be asked on the cash-strapped corridors of the DfE, and I wanted to head it off with some facts and some proper analysis.

We hired master data wrangler Allie Goldacre, who combed through, tested and verified and combined together the various sources:

  • Children in Need census, and its predecessor the Child Protection and Referrals returns
  • Children and Family Court Advisory and Support Service records of care proceedings
  • DfE’s Children’s Social Work Workforce statistics
  • SSDA903 records of looked-after children
  • Spending statements from local authorities
  • Local authority statistics on child population, deprivation and urban/rural locations.

Just because the data were ‘open’ didn’t mean they were useable. Each set had its own quirks and each local authority had its own problems and definitions in some cases. The data wrangling was painstaking and painful! As it’s all in the public domain, I’m going to add the data and code to my website here, very soon.

Then, we wrote this paper investigating the system and this paper trying to predict ‘Inadequate’ ratings. The second of these took all the predictors in 2012 (the most complete year for data) and tried to predict Inadequates in 2012 or 2013. We used the marvellous glmnet package in R and got down to three predictors:

  • Initial assessments within the target of 10 days
  • Re-referrals to the service
  • The use of agency workers

Together they get 68% of teams right, and that could not be improved on. We concluded that 68% was not good enough to replace inspection, and called it a day.

But lo! Soon afterwards, the DfE announced that they had devised a new Big Data approach to predict Inadequate Ofsted scores, and that (what a coincidence!) it used the same three indicators. Well I never. We were not credited for this, nor indeed had our conclusion (that it’s a stupid idea) sunk in. Could they have just followed a parallel route to ours? Highly unlikely, unless they had an Allie at work on it, and I get no impression of the nuanced understanding of the data that would result from that.

Ray noticed that the magazine Children and Young People Now were running an article on the DfE prediction, and I got in touch. They asked for a comment and we stuck it in here.

A salutary lesson that cash-strapped Gradgrinds, starry eyed with the promises of big data after reading some half-cocked article in Forbes, will clutch at any positive message that suits them and ignore the rest. This is why careful curation of predictive models matters. The consumer is generally not equipped to make the judgements about using them.

A closing aside: Thomas Dinsmore wrote a while back that a fitted model is intellectual property. I think it would be hard to argue that coefficients from an elastic-net regression are mine and mine only, although the distinction may well be in how they are used, and this will appear in courts around the world now that they are viewed as commercially advantageous.

1 Comment

Filed under research

How to assess quality in primary care

Jim Parle, of the University of Birmingham, and I have an editorial just out in the BMJ responding to the recent Health Foundation report on quality indicators in primary care. There’s a lot one could say about this subject but we had to be brief and engaging. Hopefully the references serve as a springboard for readers who want to dig in more. In brief:

  • We think it’s great that composite indicators received a strongly worded ‘no’; remember that Jeremy Hunt (and probably Jennifer Dixon too) started this process quite keen on global measures of quality reducing all the complexity of primary care organisation and care to a traffic light.
  • We agree that a single port of call would be invaluable. Too much of this information is scattered about online
    • but along with that, there’s a need for standardised methods of analysis and presentation; this is not talked about much but it causes a lot of confusion. At NAGCAE, my role is to keep banging on about this to make analysts learn from the best in the business and to stop spending taxpayers’ money reinventing wheels via expensive private-sector agencies
    • and interactive online content is ideally suited to this, viz ‘State of Obesity’
  • We think they should have talked about the value of accurate communication of uncertainty, arising from many different sources. Consider Elizabeth Ford’s work on GP coding, or George Leckie and Harvey Goldstein on school league tables (googlez-vous).
  • We also think they should have talked about perverse incentives and gaming. It never hurts to remind politicians of Mr Blair’s uncomfortable appearance on Question Time

Leave a comment

Filed under healthcare

Visualising data for clinical audit – videos now online

HQIP held a most enjoyable seminar recently on presenting healthcare quality data to clinicians and the public. It’s all now online on YouTube and I recommend it to anyone working in this field.

It took them hours to do my make-up but it was worth it.

Really, I’d now like to reverse the order of topics in my talk (and a nother version of it popped up at Hertfordshire University Business School), starting with interactivity and trends, then going into chart design and perception a bit more. Stats could appear at the end if there’s time, along with software, if there’s time. I’ve also decided to ditch the silly pictures and have concrete examples, good and bad, at every stage. We live and learn (mostly).

Leave a comment

Filed under Visualization

Healthcare quality is trendy again

Yesterday I received a promotional email from a publishing company telling me that next week is “Healthcare Quality Week” (but perhaps only in the USA). Who’da thought it? Apparently a great way to celebrate this would be to subscribe to one of their journals. Well, I managed to say no to a danish with my coffee at East Croydon station this morning, so I think I can resist this lesser temptation too.

But the quality of care in the UK is definitely getting a lot of attention. It sort of was trendy for a while in the noughties, discredited through Star Ratings and Sunday newspaper league tables, went away and is now coming back into vogue. It’s worth doing, but it’s jolly hard to look across indicators, specialisms, care settings and disease topics and find an underlying pattern identifying the “bad apples”. (When I say jolly hard, I actually mean impossible. It’s British understatement.)


Yet finding the wrong uns is what people generally expect will happen. People like our Secretary of State for Health, Jeremy Hunt, who said:

As an MP I know how well each school in my constituency is doing thanks to independent and thorough Ofsted inspections. But because the Care Quality Commission only measures whether minimum standards have been reached, I do not know the same about hospitals and care homes. I am not advocating a return to the old ‘star ratings’ but the principle that there should be an easy-to-understand, independent and expert assessment of how well somewhere is doing relative to its peers must be right.

“Right” as in good, I suppose, not “right” as in correct. Well, as the song goes, we are where we are, let’s all get on with it.

The Nuffield Trust and Health Foundation are prominent in this, launching this week something called QualityWatch, joining the field as another publisher of league tables and interactive graphics using existing official data. There is no government endorsement, although that may come. Nuffield Trust ran Mr Hunt’s consultation on rating healthcare providers earlier this year, giving it the thumbs-up, so you can imagine they are in favour.

There is a long history of statisticians versus league tables, which is why I am now writing a retrospective with this title, perhaps for Significance magazine if they like it, perhaps for a medical / health services research audience. I’m going to focus on health because it’s my field but you should look up Harvey Goldstein’s equally critical tone in education, particularly given that “I know how well each school is doing” comment.

I am a member of NHS England’s National Advisory Group on Clinical Audit and Confidential Enquiries. This is entirely my own opinion and not that of NAGCAE, NHS England or HM Government.

Leave a comment

Filed under Uncategorized

Hospital mortality league tables: another layer of uncertainty

There’s a nice paper just out in BMC Health Services Research by Kristofferson and colleagues where they looked at hospital mortality stats in Norway and counted deaths in three different ways:

  • exclude patients transferred between hospitals and count deaths in hospital
  • exclude patients transferred between hospitals and count deaths within 30 days wherever they happened
  • count patients weighted by the proportion of time in each hospital, and count deaths within 30 days wherever they happened

OK, that’s not every possibility but the point is to test how sensitive a league table would be to changing this definition. The assumption is often made that mortality is the best statistic to fall back on when all else fails, but the notion that a patient is either dead or alive is all very well until you get down to the fine details of how you count these deaths… and then it gets complicated.

They found a considerable number of hospitals moving in and out of being “outliers” when the definition of mortality changed. This is no great surprise to anyone who has analysed comparative hospital stats, or has looked ito the methodological literature on it. But it remains the case that league tables get a lot of attention both from journalists and bureaucrats.

As further reading I cannot recommend highly enough the book “Performance measurement for health system improvement” and the landmark JRSS paper.

PS: the graphs in the Kristofferson paper are bad: inadequately lablled, ugly and confusing

Leave a comment

Filed under Uncategorized