In England, and to some extent other parts of the UK (it’s confusing over here), clinical audits with a national scope are funded by HM Government via the Healthcare Quality Improvement Partnership (HQIP). Today, they have released a report from ongoing work to find out how these different audits operate. You can download it here. I am co-opted onto one of the sub-groups of the NHS England committee that decides which projects to fund, and as a statistician I always look for methodological rigour in these applications. The sort of thing that catches my eye, or more often worries me by its absence: plans for sampling, plans for data linkage, plans for imputing missing data, plans for risk adjustment and how these will be updated as the project accumulates data. Also, it’s important that the data collected is available to researchers, in a responsible way, and that requires good record-keeping, archiving and planning ahead.
I’ve just looked through the audit-of-audits report for statistical topics (which are not its main focus) and want to pick up a couple of points. In Table 3, we see that the statistical analysis plan is the area most likely to be missed out of an audit’s protocol. It’s amazing really, considering how central that is to their function. 24/28 work streams provide a user manual including data dictionary to the poor devils who have to type in their patients’ details late at night when they were supposed to have been at their anniversary party long ago (that’s how I picture it anyway); this really matters because the results are only as good what got typed in at 1 am. 4 of them take a sample of patients, rather than aiming for everyone, and although they can all say how many they are aiming for, only one could explain how they check for external validity and none could say what potential biases existed in their process. 20/28 use risk-adjustment, 16 of whom had done some form of validation.
Clearly there is some way to go, although a few audits achieve excellent standards. The problem is in getting those good practices passed along. Hopefully this piece of work will continue to get support and to feed into steady improvements in the audits.