On methodological terrorism

Elevator pitch

A psychologist and journal editor has written that outspoken statistical critics on blogs and social media are “methodological terrorists”. Some people are upset by this. I think it is pretty accurate, in that incumbent statistical cultures will not change from their current data-torturing ways until they are scared out of it. In this post, I take a level-headed view of terrorism and revolution and see what we can learn from the early Soviet Union. Disclaimer: don’t hurt anyone, readers.

tl;dr

Some methodologists are increasingly critical in blogs and social media of bad analyses, typically carried out by people who are not statistically trained, and typically arriving at overly definitive headline-grabbling messages. Now, there is discussion about whether we should confine critique to polite scientific journals and conferences, continue with the tweeting and blogging but be nicer, or continue the attack. Most recently, the focus has turned more specifically to scientists who refuse to accept that their work could be wrong after criticism, or even respond that it is irrelevant whether they are wrong or not. I have been mulling this over in light of the term “methodological terrorists” deployed in protest against unpleasant criticism (the irony seems not to have been detected by the author, one Susan Fiske). I got Trotsky off the shelf, then Azrael, and had a little bit of early-USSR reading. I thought about whether this is some kind of revolution, and whether terrorism is in fact an accurate term and maybe even not to be rejected but mined for ideas about forcing change in the system. I also thought back on things I saw while growing up in Apartheid South Africa. I came to these conclusions:

  • Dilettante analyses kill people, and methodologists who care are duty bound to take strong measures to change the damaging norms in scientific culture. There is no logical reason to reject unpleasantness, and in fact it is likely given the imbalance of power in certain scientific fields that intimidation is necessary. This is not something to be scared of; methodological intimidation gave us randomised clinical trials with allocation concealment, and to some extent pre-specified protocols and analysis plans.
  • To avoid any misunderstanding, bodily threats and violence should not be employed; hard-nosed critics who are not convinced by an appeal to their better natures should note that posting mystery powders to social science departments will simply be counter-productive
  • The current system of a small number of the same people approving funding for studies, doing them, and editing the journals where they are published is arguably corrupt. The subject experts who run it benefit so much from it that they certainly don’t allow dissenting voices on their patch, and, unable to control self-publication on blogs and social media, react forcefully. Journals and conferences are used as an organ of repression, and we should focus on influencing them and not allowing them to be a refuge for irresponsible conduct.
  • We need to stop appearing as venomous cranks with axes to grind. Criticise someone’s work once, and then move on. Don’t keep picking at them. The next generation of experts needs to fear us, not because we are unpleasant, but because we are right.
  • We need every practitioner of dilettante analysis to feel the gaze of potentially career-limiting critics. Criticise more people, more studies and put less venom into each one. (I like the idea of explicitly stating an intention to critique one paper chosen at random from each issue of a chosen journal, and intend to announce my own chosen journal soon.)
  • We need to use the media to further these arguments, or we will never get off the ground. There are just not enough of us. Also, the thrill of appearing in the newspapers or whatever is one of the big drivers of silly interpretations of studies. Journalists love balance, which most scientists shudder at, and let’s not go there now, other than to say that we can use balance to our advantage, by becoming the dissenters. So, when you write your critique in your blog, also send it to the science editor at various news outlets. Make an e-mail list of these poor saps so you can make the whole process as quick as possible for yourselves. And do it fast! They want to report the study today and move on to something else tomorrow.
  • We need to acknowledge and indeed reward efforts to fix the problem. Critiques online should be amended to reflect any effort by the target to make good (although I feel uneasy about deleting things). Everyone makes mistaks. In fact that is the only way science can proceed. It’s what they do next that matters.

I also suggest some classic tactics that the dilettanti may take to suppress further criticism in scientific fields where they hold the balance of power. We need to be alert to these.

Terrorism and positivism

To recap for those who have had better things to focus on recently: there is a so-called crisis of replication. Starting in psychology and social sciences and then spreading elsewhere, big names are getting criticised for their statistical practices, which allegedly lead to headline-grabbing conclusions that evaporate when someone attempts to re-run the study. A more general tendency to blog outspoken criticisms has gained momentum too. Even your mild-mannered author has had a pop at fanciful concoctions of pseudo-science cooked up by politicians, politicians appropriating stats for potentially damaging tautologies, influential scientists who line up a series of logical fallacies to support a political cause, powerful people who wish for the moon with stats, and famous studies with screwed up analyses. I have many more that I don’t blog about but keep for teaching examples.

For a while, the scientists in the firing line kept quiet but now have started fighting back. One publication that is getting a lot of attention is an article in a magazine of the Association for Psychological Science by Susan Fiske. I picture old Susan* rattling this editorial off on her porch in the Hamptons with a Tom Collins or two on the go; the language is emotional and gets more florid as it goes along. She coins a few neat terms: destructo-critics, data police, vigilante critique, but it is “methodological terrorists” that has irked many. The destructo-critics with the best sense of humour, like Eric-Jan Wagenmakers, have enjoyed adopting this new job title. (The closest thing to destruction I’ve seen EJ do is to consume an intemperate amount of sushi. I mean, when Abu Bakr al Baghdadi gives one of his talks to stir up the troops or whatever, does he use a slide with Spongebob Squarepants to illustrate a point? No.) I am too small a fish to be honoured by anyone with such a label, although, unlike Pablo Picasso, I do — fairly — get called an asshole for being outspoken. But I too will add it to my name badge, with all the unseemly bandwagon haste of Dee Dee Ramone putting out a rap album.

* – I don’t know how old Susan is or isn’t. For some reason this scene came to mind and the sheer Salinger-ness of it all changed it into his voice in some terribly mystical way and all, just for your information.

Vis a vis assholes, this article by Jesse Singal (who, I’m sure, is a nice guy) is a great overview, summarising the counter-argument by Tal Yarkoni (likewise) that the fact someone is unpleasant in pointing out that you screwed up doesn’t mean that you, in fact, didn’t screw up. If there’s one really surprising turn in this story, it’s that the methodo-destructos have solidified the logical basis of their actions, and done so quickly, from several independent writers, while the complaining ex-spurts have resorted to emotion and rhetoric.

Now, some people get upset about the terrorism tag, but I’d like to take it seriously and explore it — or explode it, if you will. (Fiske, in the cold light of day, has decided to delete it from her article, in case you’re wondering where it went.) Some terrorism aims to provoke larger scale insurrection or revolution (the good old Cold War variety: PLO, Angry Brigade, Unabomber, N17, etc etc, possibly Isis, but it’s not relevant here anyway), some to consolidate a prior revolution (the Bolshevik variety, which we will explore further), and some is just a chance to lash out or pretend you’re in a real war (the IRA or ETA variety, oddballs claimed by Isis post hoc). We don’t fit any of those, but scientific revolution may be at hand, in the Kuhn sense. I am considering to what extent this recent development constitutes a revolution, which is to say that the previously meek statisticians rise up and impose a new order on their former masters, the subject experts.

Fiske is using the term terrorism here in the fashion of Karl Kautsky (who was, like, y’know, surely there’s no need to be so unpleasant, our ideas are great so let’s win them over with words instead). So, I took a look at Trotsky’s Terrorism And Communism, which was written as a riposte to Kautsky and a defence of Bolshevik terror, (and he was, like, don’t be naive, they are perfectly comfortable as they are, and will destructo us if we don’t destructo them first) to consider what we can learn about the term. It certainly seems to be true that statisticians are more critical than we used to be, or at least some of them are more outspoken and abrasive in doing so. Andrew Gelman (who doesn’t like being called a terrorist) has recently pointed out in his essential-reading blog post which offers a short history of events leading up to this crisis of replication that the Neuroskeptic blog was prominent in establishing the destructo genre — principally because the crisis took root firstly in the field of psychology. I don’t believe that the problems are unique to psychology or even that they are at their worst there; consider the sisyphean labour of medico-destructo Ben Goldacre, or the universal condemnation of John Ioannides.

I also am aware of a number of older statisticians who feel uneasy about the abrasive confrontational approach (not to mention PR skills) of the younger generation. It does feel as though they regard a statistician’s proper place as a discreet consultant who does their thing and steps back into the shadows afterwards (and I understand that for many, their (self-)employment requires this, so destructo-mania is a luxury for the academic, for the most part). David Colquhoun wrote recently:

“I recall talking to a statistician at a recent meeting of the Royal Statistical Society. He was involved in analysis of clinical trials. I asked why he allowed the paper to make claims of discoveries based on P values close to 0.05. His answer was that if he didn’t allow that he’d lose his job and the clinician would hire a more compliant statistician. That lies at the heart of the problem.”

So, this is not just a methodologist vs subject expert thing. What is it?

Is it that we methodologo-destructos are no longer seen, or see ourselves, as servants of a larger scientific technical process? We surely have insights of a methodological nature to contribute. Perhaps it is because our numbers are greater than before. The advent of Data Science, bringing a probabilistic statistics background into contact with a computer science background has clearly been a major driving force here. Also, the mass of critical writing about statistical practices has increased (viz Ziliak and McCloskey), along with popular writing on cognitive biases (Gigerenzer vs that other more famous guy whose name eludes me). And finally, there is the internet, so we can self-publish even if we are still just cogs in a bigger science factory. If you’ve been in the same stats job for years, you might not have noticed any of this. Tenure and job longevity correlates with age, so the idea that this is a generation effect might be a red herring.

So, in what sense could we call this revolutionary? It has the characteristic of a lesser power base (the method-people) challenging the norms of a greater power base (the ex-spurts). The meth-heads have pointed out that they are unhappy, and been ignored (or so they feel), and now in some quarters resort to force. The X-Men must feel somewhat threatened because upon finding the statisticians are getting too big for their boots, they too have reacted with force. What does Trotsky have to say about the role terror plays in subduing incumbent powers, and what parallels might there be between bloody armed conflict and online arguments over methodology?

Is there a cause worth fighting for?

Firstly, for me, it is worth noting that bad publically-funded research takes tax money away from other causes and wastes them on vanity projects — and that’s the best-case scenario. The retired professor may console themselves about their life’s work of fanciful claims built by p-hacking from the safety of their little cottage near Aix-en-Provence, but for the most part, their research projects achieved precisely nothing. That’s the good news. The bad news is that sometimes they waste money and also do harm, when they deal with the health and wellbeing of the nation. I have said before: dilettante analyses kill people. In that context, it is not unreasonable to conceive of this as class struggle. If we view it as our civic duty to promote good research, it is also our civic duty not to tolerate bad research. If you saw someone in a lab coat and goggles hurting an old person to get their hands on some money, would you intervene? I hope so, and this situation is no different. There is a corrupt system which you are obliged to end, and you will have to act outside the system to do so. Not by blowing up their offices, as I explain below, but by confronting their work when it is wrong, in the best scientific tradition, and refusing to go away until it is fixed. Fiske said “it’s careers that are getting broken”; yes, that is precisely the objective. Acting out of ignorance, then seeing the light and fixing the problem is one thing, fighting not to change is another, and someone who refuses to learn and improve is not a scientist but a dancer. No offense to actual dancers. The difference between us and Trotsky is that we are not fighting for survival of one group or the other, we want the great majority of subject experts to continue their work, but to be aware of methoological problems and to behave better in future.

“‘But in that case, in what do your tactics differ from the tactics of tsarism?’ we are asked by the high priests of liberalism and Kautskianism. You do not understand this, holy men? We shall explain to you. The terror of tsarism was directed against the proletariat. The gendarmerie of tsarism throttled the workers who were fighting for the socialist order. Our Extraordinary Commissions shoot landlords, capitalists and generals who are striving to restore the capitalist order. Do you grasp this… distinction? Yes? For us Communists it is quite sufficient.”

What is to be done?

Now we must diverge from Trotsky. We are not rebuilding society after civil war, which is his general excuse for all manner of terrorist acts. We want to reform it without breaking it, and even more delicately without breaking the trust or conversation between the subject experts and the methodologists. But we cannot be nice about it or we will just be ignored. Let’s consider our one real success, perhaps the only one.

Getting ethical review and having randomised, blinded allocation is still grumbled about in my field (healthcare), where doctor still knows best, but they get on and do it. Why? Because they have seen the light? Sometimes, yes, but sometimes because they know it would be career-ending to breach the protocols. If that isn’t terror, I don’t know what is. But not all terror will work. We have to change the culture, probably without removing the experts (see Expert Power… below) (although a few will doubtless jump). Let’s be clear and avoid any misunderstandings: actual acts of violence, harrassment or physical threat are simply wrong and, anyway, will serve only to make your argument easily dismissed. Who doesn’t want to be the noble scientist persecuted by the barbarians? They already have role models attacked by animal rights people.

We should scare them all right, but in a thoroughly scientific way. It needs to be clear that nobody’s blunders are safe from being called out. We need to go after anyone and everyone, not just the big names. Perhaps a good way to do this is to allocate a methodological destructo-critic to each scientific journal of interest. Then, with each issue, you simply pick a paper at random and blog your critique, without pulling any punches. But crucially, we need to incentivise good behaviour too: congratulate authors of well-conducted studies, and especially importantly, go back and acknowledge when someone you criticised has taken steps to fix the problem.

The role of the ANC in South Africa was pre-revolutionary, in contrast to the Bolsheviks. Notably, Mandela distanced himself and uMkhonto we Sizwe from the word terrorism, promoting “armed struggle” at all times. From his self-defence at the Rivonia trial:

Firstly, we believed that as a result of Government policy, violence by the African people had become inevitable, and that unless responsible leadership was given to canalize and control the feelings of our people, there would be outbreaks of terrorism which would produce an intensity of bitterness and hostility between the various races of this country which is not produced even by war. Secondly, we felt that without violence there would be no way open to the African people to succeed in their struggle against the principle of white supremacy. All lawful modes of expressing opposition to this principle had been closed by legislation, and we were placed in a position in which we had either to accept a permanent state of inferiority, or to defy the Government.

But the violence which we chose to adopt was not terrorism.

The notion of canalizing is perhaps relevant to us today.

Trotsky was keen to defend the censorship of newspapers, again appealing to the war footing which he claimed to be indefinitely extended while the Bolsheviks felt the pressure of their enemies. We have the opposite problem: our business is largely conducted through publications, but these are somewhat incontinent of studies, releasing the good and bad all together (why not just offer to peer-review for them? see below under Peace And Love). How then can we change the behaviour of the publications too? Well, they are commercial enterprises to some extent, and bad publicity about methods hurts them. Publicity is also one of the motivators for scientists who seek an exciting story out of their data by any means. We need to turn publicity against them. We need to give fresh, exciting (whiff of scandal) material to journalists. They are obsessed with balance: if you have someone on the news representing political party A, you must must must get someone from party B to speak against them. They then rather simplistically extend this to science, bringing on all manner of tinfoil-hat cranks and giving them disproportionate coverage. Whatever. Let’s give them something better. Every time they have a story about some professor who found that wearing socks with sandals is an early sign of dementia, we rapidly wheel out someone who can eloquently and scientifically shoot it down. I guarantee you those silly claims will dry up soon. The problem is that journalists want to cover the story today and then forget it, so rapid response is essential — but that’s what the blogging and tweeting destructo-critics are good at!

One thing that worries me is the Dunning-Kruger effect. When you are new to a subject (a recent stats graduate, for example), your new-found confidence is as yet untempered by harsh reality. It would be a pity if inexperienced destructo-critics (trying, perhaps, to build an online presence and land a great job leveraging big data analytics) made asses of themselves and undermined the whole movement. But what can you do?

Expert power and methodological politics

The relationship between experts and methodologists may not be very well represented by the USSR in, say, 1922, but it could more closely follow the 1930s tension between industrial managers (that’s them) and the party in its various guises (that’s us). So I leafed through Jeremy Azrael’s book Managerial Power and Soviet Politics. At the end of the 1920s, managers (who were largely still in post since before the revolution) were routinely accused of sabotage if production did not meet targets. Sabotage could take any form, including just not working hard enough. You can see how scary and frustrating that would be, and doubtless this is how many subject experts feel now. But at the same time, their expertise was needed, and experiments to replace them with soviets were not entirely successful. Indeed, many of them had strong ambitions to up the scale of their work (like subject experts talking a good game on replication but not really knowing how to get there), which made them valuable assets, known as “Americans” for their industrialising zeal. There were many layers of regulation and inspection that had sprung up organically around them, but the most relevant was the presence of Red Directors, who were intended to impose Party priorities but could also come under accusations of sabotage. To be a Red Director might sound like a good career move, but it was not a comfortable situation. Oppressive regimes are most scared by their ‘own’ people.

Broadly, both camps were won over to support Stalin in the years immediately after Lenin’s death with promises of immunity from the sabotage accusations, and recognition of a policy of “one-man management”. But this was not to be. Their position was undermined finally at the 16th Party Congress in 1929, where “one-man management” was rejected. In the years that followed, five-year plans, league tables, performance indicators and purges served to entrench a system of corruption and perverse incentives. The opportunity to build on the skill of the “Americans” had been lost, and was never regained. I think we have to be careful not to go down that route. The crucial difference is that we do not have a shared system for critique and control of scientific analysis (the Party), but rather disparate individuals (guerrillas, if you will), so there is no risk at present of destroying science in quite the apocalyptic way that Fiske envisages. However, I call here for more action, and quite possibly for more organisation of the destructo-critics, and so we need to be careful that any organisation that does emerge doesn’t turn into an ossified set of rules that will stifle quantitative science.

Peace and love, man

Not everyone writing about this has suggested getting tougher. Jeff Leek, pitching himself as the Desmond Tutu of replication, blogged in a more conciliatory tone recently and proposed six steps to improve matters while taking the heat out of the argument. Deborah Mayo picked this up and contrasted it with Gelman’s rejoinder, which has more of a this-is-war attitude. Mayo comes down on Gelman’s side. I agree — who couldn’t — with Leek’s sixth point that statistical analyses should involve statistical experts, but the rest seems naive to me. Journals want meth-heads but can’t find them. X-men who want meth-fuelled collaborators also can’t find them. Other X-men have been told after a six-lecture course in t-tests and SPSS that they now know everything there is to know about stats, so they’re never going to start looking. In fact, they start doing the analyses for all their X-buddies and the whole thing becomes a love-in (you may choose to insert an earthier epithet here). If medical students had a spare day in their timetable, they would spend it memorising more metabolic pathways, not learning stats (and so they should). None of Leek’s apple pie stuff is going to come to pass, in large part because of simple supply and demand because, as you all know by now, statistics is the sexy job of the 21st century. They want to hire one of us, but they can’t find one. We are few, and many of us have already been ruled, found, brought and bound in Santa Clara county or The City, and are saying nothing to anyone, ever.

Professional identity

Another angle I started thinking along was how other professions/jobs would react, or do react, in comparable situations. Let’s consider accountants, stock market traders and dentists. Doubtless there are crooked accountants who help clients evade tax and launder criminal proceeds, just as there are statisticians who comply with requests for repeated hypothesis tests until p<0.05, but for the most part they act as police, simply because the repercussions of default for them as individuals is too great, and the rewards not sufficient to warrant the risk. In contrast, the rogue trader may well accept the risk thanks to the enormous reward. Perhaps it also helps that they don’t have a shared and valued professional identity. Now, statisticians have a professional identity, but it’s not strong. If it was bolstered with greater promotion of chartered / PStat status in the public eye, and a willingness on the part of stats societies to strike members off, then we might get tougher at this. At present, most young statisticians feel too anxious to attempt anything like this. Now, consider the dentist. You pay them to tell it like it is. If your molar is rotten and has to come out, you want to hear it and have some straight-talking advice on what to do about it. You don’t enjoy hearing the news, but better now than later in agony. That is the service they provide — to tell you the facts, not to be your friend. We need to stop being friends of subject experts and start being their dentists instead. (The dentist analogy deserves a big hairy hat-tip to Mike Monteiro and his excellent keynote talk from interaction15.)

Counter-revolutionary tactics

It will be interesting to see how this power struggle plays out in the next few years. Will the critics grow in numbers? Will it mature from sniping to more concerted and funded campaigns (like AllTrials)? Is it a generational thing, in which case there really will be a shift across all sciences? Will the dilettanti adopt some classic counter-revolutionary measures to divide-and-conquer? Here are some forms they might take; we need to be vigilant to them.

  • Critical statisticians could be blacklisted in some way so that pliable Quislings appear as collaborators on all manner or studies, giving that gloss of methodological respectability.
  • They could launch a Pledge To Be Nice and pressure all graduates and early career researchers to sign it or be blacklisted. Will it come under the disguise of Safe Places and Trigger Warnings? It would certainly be easy to tag onto that in the USA at present.
  • They could push for collective punishment, identifying a whole institution as having inappropriate ‘checks and balances’ or ‘governance’ if someone speaks out, effectively blacklisting the whole place.
  • They could promote peer punishment, where it is your stated duty to act against methodological terrorism, perhaps by refusing to collaborate ever again with your outspoken colleague. Failure to do so makes you suspect too.
  • Informants can be recruited too by the same logic, and one can compile a database of methodologists. The trouble with this is the lack of external governance: your dad calls his friend who calls his friend and suddenly the guy who upset you is getting a 4am raid by the B.O.S.S.

If you combine these things into a database and an institutional pledge to good conduct, which is cascaded down into people’s job descriptions and objectives, you have a deadly combination. I say it with some trepidation, but it’s better to be forewarned. Don’t think this stuff doesn’t happen.

The good news for us bad guys is that today’s terrorist is tomorrow’s statesman, so I look forward to Andrew Gelman’s keynote talk at the 2036 APA Convention. Our goal in the future is analogous to the Bolsheviks’ (although Trotsky fumbles the distinction (see above)): once everyone understands philosophy of science and statistics, there will be no more interdisciplinary conflict. That’s if we “win”. If the status quo continues and we go back to being meek, there is perpetual oppression of the sort that Colquhoun highlighted, and waste of public resources, and sometimes harm to vulnerable people.

One day I’ll revisit this from the perspective of the Puritans before and after the First English Civil War, but probably not until I’ve retired.

Advertisements

2 Comments

Filed under Uncategorized

2 responses to “On methodological terrorism

  1. Robert:

    You write, “Criticise someone’s work once, and then move on. Don’t keep picking at them.” I disagree. The point of my criticism is not the particular people being criticized: I don’t care, one way or another, what happens to the ovulation-and-voting researchers or the beauty-and-sex-ratio researchers or the air-rage researchers or etc etc etc. The reason to keep reconsidering (“picking at”) these examples is that we can learn from considering specific examples carefully. It’s the same reason that in statistics I keep reanalyzing the 8-schools data: I don’t care about those particular 8 schools in New Jersey from the 1970s; I just learn from continuing to explore one example in depth. It takes an investment to get into an example, and that investment can pay off from repeated takes on that example, from many directions. To put it another way: my goal in criticizing published research is not to find the maximum number of different flawed papers in the literature, but rather to better understand the problems in statistics, scientific practice, and scientific folkways that lead to these problems. And “keep picking at them” can be a good way for me to move forward in this way.

    • Yes, I can see where you’re coming from, and I’ve certainly learnt a lot from your blog in that regard. I’m also thinking of influencing the perpetrators to get better and I worry that they can write us off like Fiske tried to do. But ultimately there’s so many bad analyses that there’s room in this game for all of us.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s