Firstly, let’s namecheck the wittiest punner in stats for that title, Stephen Senn.
This recent post on Andrew Gelman’s blog is essential reading. I suspect my readers are all over there too, but I’ll mention it here because of this point in wrapping up:
Peer review can serve some useful purposes. But to the extent the reviewers are actually peers of the authors, they can easily have the same blind spots. I think outside review can serve a useful purpose as well.
I’ve seen this a lot in my life as a medical (read ‘health and social care more broadly, with a dash of education’) statistician. There are distinct tribes of healthcare professionals and they do things, including research designs, analytical methods and communicating findings, in their own sweet way. There’s generally no reason, it’s just custom and ritual. If you don’t fit that mould to some extent, you get rejected. Often, I find myself consciously peppering the paper / slides with some shibboleths that will ease my journey to REFland. (Of which, sample size calculations for anything that isn’t a randomised controlled trial is the most common, although I am no stranger to the Totally Unnecessary Reporting Diagram (TURD). I draw the line at Cohen’s D though; D stands for d’oh.)
‘Outside review’ reminds me of the idea of ‘strong inference’, or having your worst enemy analyse your data too and see if they can destroy your conclusions. You don’t have to go that far though, you could just make sure that reviewers extend beyond the specialism and profession of the authors to break that parochialism and question the unquestionable.
Essentially, if they can’t understand it, then it’s not written well. I don’t accept any argument that the subject is just too complex for outsiders – because the authors’ interests were once upon a time confined to Lego or the Smurfs, so it must be possible – nor do I claim to have got it right myself – it’s a constant challenge to pitch Bayesian latent variable models just so for a subject-expert audience.
I don’t know about you, but I enjoy reading academic papers outside my own field (OK, no critical theory please, but I’ll consider pretty much anything else). Maybe I should start an occasional series of randomly selected academic papers here, or maybe I just don’t have time for that.