Check out this article in online magazine Slate by Andrew Gelman. He tackles a recently published paper with the above title. The issue is one of how there are many ways in which you could take a hypothesis and actually test it. It’s an enjoyable read and thought-provoking too. The authors came back with a response and then more debate continued on Gelman’s blog. Among the ideas in there, I like:
- It’s easy to eat up a lot of degrees of freedom without even realising you’re doing it. In just the same way running lots of alternative tests on the same data drains its power to discriminate fact from fiction, if you are capable of and willing to explain several alternative patterns as supporting your theory in different ways, alarm bells should be ringin’
- Here’s a dilemma: if you really do have a binary exposure and a binary outcome, and you are absolutely certain how to measure them, why collect more data? Just do a 2-by-2 table. On the other hand, sometimes seeing more detailed data makes you change your mind about how certain you are. And how can you know until you see the data? Really, the problem here is not the amount of data but what you do with it, because…
- …a single p-value is pretty useless. This is not an exam with a pass/fail result, it’s an ongoing quest to search out and Studdy the secrett of Nature by way of Experiment.
- Context matters: you will interpret the results differently depending on where they fit into a larger framework of theory and evidence, but nobody really knows how to capture this so that we can be more open about it
- I’ve said it before and I’ll say it again. Once the PR people get a hold of your study (“What, this research is actually about sex? and clothes? Cool!”) and the soundbite starts rolling, there’s nothing you can do to bring it back. It’s all very well for people like me to pick over the facts. We are paid to do this kind of thing, and we work in places that spend a lot of money getting us access to any scientific literature we want, when we want it. It’s totally unapproachable to the rest of the world, and all they are left with is the soundbite.