Felix Schönbrodt has blogged recently about how a statistic (correlation, in his case) wiggles around and gradually stabilises as a sample accumulates. And he draws what I call a cumulative funnel plot. Schönbrodt seems to have basically reinvented elementary statistical inference, and I would suggest that if you read his blog, you not get excited and start referring to POSs and COSs, lest a statistician take a dim view of your new clothes. However, I think the cumulative funnel plot is a great way of conveying the notion of uncertainty from sampling error. A couple of years ago I commended it here, and although Spiegelhalter and colleagues made a valid rebuttal about false alarms, we’re aiming for different goals. I’m thinking about the longer-term goal of improving public understanding of uncertainty. A few false alarms is part of life, and I think people can handle them and don’t have to be shielded by well-meaning statisticians. As soon as the line wiggles over outside the funnel, you get interested, but you don’t swing into action and close the hospital down. I would have thought that obvious… ah well maybe not. You wait until the pattern keeps happening. Best of all, you have some prespecified endpoint and you do one significance test then, but sometimes when lives are at stake, we have to compromise on the statistical purity in order to get early warnings. But to come back to the notion of communicating inferential principles, the other thing I like about the plot is the transparent, superimposed bootstrapped trajectories. I have no doubt it is easier for the newcomer to understand this sort of depiction of uncertainty than the theoretical stuff (like the funnel). At ICOTS this summer I’ll be attending the workshop on simulation in introductory stats teaching and I hope to report back soon afterwards with some new ideas.
Around that time, I was making a Stata command for drawing these plots, and it somehow ended up on my pile of dormant projects. Maybe I’ll get it back up and running some day.