Last year, Li and colleagues at Harvard brought out a paper looking into restless legs syndrome and the risk of depression (AmJEpidemiol (2012);176(4):279–288). This paper really caught my eye because it is one of those associations that we see in medical research but where it is incredibly hard to demonstrate causality. It turned out to be a really nice example of the many hurdles one has to overcome in retrospective analysis of observational data.
They used data from the Nurses’ Health Study to follow an impressive 56,399 women over 6 years. There had been a number of cross-sectional and case-control studies before but nothing prospective.
One major challenge in looking at retrospective data like this is defining when someone becomes at risk and when an endpoint of interest really occurs. It is quite straightforward in a hospital setting but in the community you are reliant on the participants’ own perceptions of their health, and their attitudes to seeking help. The first task was to exclude women with pre-existing (prevalent) depression, which took out 11% of the total 2002 study participants. A small number reported a diagnosis of depression at the next round of questionnaires but weren’t sure when the symptoms had begun, so they got excluded too. So far, so good. The only exclusion for prevalence that worries me is that they took out 9% of the 2002 total participants because they had incomplete data on depression and medication, and that seems a high proportion to me. They question that comes to mind immediately is whether these 7000+ women are different in any way from those who did provide complete data. If they are, this could be a problem of non-ignorable missingness. We are also told that participants were regarded as having depression if they had a clinical diagnosis and were regular users of antidepressants. That seems quite stringent to me and I would have liked to have seen sensitivity analysis allowing any one of those through and reporting the results as an upper bound on the incidence of depression. (You have to remember that not everyone will seek or get a diagnosis, yet not everyone taking antidepressants is doing so for depression; some are used for neuropathic pain etc)
Next, the most powerful and widely used tool we have to remove confounding is the multiple regression model. Remember that the aim is to obtain an estimate of a causal effect between restless legs and depression, so we have to control for other factors which we know or believe can cause depression, and which are correlated (causation not required) with restless legs. This is the classic epidemiological definition of a confounder and it is frequently forgotten by analysts who stick anything and everything into the regression model. This paper is one of those kitchen sink models, unfortunately. Most of the factors that go into the model are sensible but I don’t see how smoking in itself can cause depression or how serum cholesterol is correlated with restless legs.
There are some more problems with the predictors in the models. Iron deficiency was not measured in the study (we shouldn’t blame the authors for this) and is replaced with the proxy measure of taking iron supplements (but this is questionable). As surely as we need to identify confounders by the classic definition, we must also bear in mind that adjusting for things that don’t fit that definition can make matters worse and not better (the key text here is Greenland et al 1991 Am J Epidemiol).
Another problem is that sleep problems were only captured once in the study, in 2002, and by adding this into the model as a single predictor the authors were essentially assuming that it never changes. This is arguably acceptable as there is no other good alternative with the data, but sleep problems are so central to the relationship between restless legs and depression that some people might argue it’s best to pack up and go home. I would suggest that doing some analysis with this imperfect but large dataset is laudable and potentially informative, although it would be nice to have more sensitivity analysis.
One of the measures used to capture depression (CESD-10) contained a question about restless sleep, which clearly is going to pick up restless legs with or without depression and might inflate the relationship between these conditions. The authors have taken a pragmatic and sensible approach to this by removing that question and otherwise recalculating the CESD in the same way as before. This is reasonable to my mind because any scale is just a rough guide anyway, but I’m sure there will be plenty of scale fiends out there who get upset whenever anyone tinkers with a scale which has undergone the ritualistic initiation of Cronbach’s alpha, factor analysis &c &c.
The sensitivity analyses that do appear in the results include a cautious exclusion of any women who were diagnosed with depression in the first three years of the study period. If they already had some symptoms at the time of the 2002 sleep questions, then including them will inflate the relationship between restless legs and depression. It will be as though none of them had any symptoms and then they all acquired them quite quickly. Whether this really is a problem depends on your interpretation of the study. If you want to get causal effects of restless legs on depression symptoms, exclusion is probably wise, but if you are interested in the diagnosis, it doesn’t make sense. This cohort has both forms of outcome data on depression but often you don’t have the choice, especially for less common conditions. Context always determines the appropriateness of the analytical choices, although most people are taught that there is one correct method for every problem and it leads to the unequivocal truth. Real-life research is messier and more complex than that, although from my point of view it is also more interesting