Homeopathy for Insomnia
Another interesting article on homeopathy, but with tantalising holes in the protocol. (Now updated twice with response from the authors.)
Everyone's favourite homeopathic MP, David Tredinnick, has today submitted four Early Day Motions to the House of Commons, each of which expresses support for the homeopathic principle and specific papers in particular.
The last of these, EDM 287, refers to a paper from Durban Univerity, South Africa, which concludes that it has shown clinically significant results (above placebo) for homeopathic insomnia remedies.
The full paper is by David Francis Naudé, Ingrid Marcelline Stephanie Couchman and Ashnie Maharaj, and is entitled Chronic primary insomnia: Efﬁcacy of homeopathic simillimum. It was published in Homeopathy (2010), 99, pages 63-68. The link will get you a PDF of the paper.
The paper itself makes an interesting read. The protocol seems reasonable, although the sample size is low - we will come back to that later. Each of 33 patients was assigned to either a treatment or a placebo group at random, and the identity of the groups kept concealed from patient and physician until the end of the experiment.
The patients were required to keep a diary of their sleep patterns, which were then assessed on a standardised scale to give a numerical value known as the Sleep Impairment Index.
Over the course of the trial, each patient was seen by a qualified homeopath at regular intervals and prescribed an individualized treatment. They were then dispensed either the treatment or a placebo, according to their assigned group. Treatment consisted of sugar pills treated in the homeopathic tradition with a very dilute solution of the treatment agent, and the placebo was the same pills with same solvent applied, but no agent.
Since homeopathic preparations are usually diluted way past any plausible dilution limit, and therefore have no active ingredient anyway, conventional medicine says that the two sets of pills should be identical.
The two groups, however, had slight but significant differences in outcome. The homeopathic group were getting more sleep than the control group.
This looks good for homeopathy, but hold on a minute.
The weak link in the blinding chain was the dispensary. In order to substitute the placebo for the patients in that group, the dispensing practitioner must have known which group was the control, and which the experimental group.
It is very easy for unconscious body language to betray this kind of knowledge - this is why double blinding is so important - and it may be that there was just enough of a hint to tip off the patients and modify their expectations of treatment.
The randomization process was basically just pulling names out of a hat. This is reasonable as a first step, but given the small sample size, the makeup of each of the groups becomes important.
The age range of the participants was stated as from 20 to 58 years of age. No information was given as to the distribution of those ages other than the average (37). It is known that the amount of sleep we need changes as we age. If there were only 2 people over 55, and they were both in one group, then this could easily have skewed the results.
This would be a significant confounding factor, yet the age makeup of the two groups is not stated in the paper, even as limits and average.
As previous studies have shown, it is possible for unconscious bias to manifest in the analysis stage of the results, especially when subjective judgments have to be made. The highest profile example of this is the investigation of Jacques Benveniste by the Horizon program, where subjective judgments on cell health were affected by the observer's belief that they were in the control group.
The blinding on this test was said to have been broken after the end of the test period, but it is not stated whether this is before or after analysis of the sleep diaries to obtain the SII scores.
If broken before analysis, bias can again creep in.
In all of the above, I am not suggesting in any way that deliberate bias has been introduced. Far from it. But the holes in the protocol and the lack of documentation of potential confounding factors make some kind of chance effect a plausible explanation for the stated p-values of the results.
I will be sending an email to Mr. Naudé asking for clarification of these points, and will update this article if I get a reply.
I received a response from Mr. Naudé, addressing each of the points I outlined above, for which I would like to publicly thank him.
Here are the original questions from my email, and his responses:
Question: First, I would like to understand in greater detail what happens at the dispensary. What measures were taken to isolate the person selecting the remedy or placebo (who is not blinded), from the person dispensing it to the test subjects?
- I think there is a misunderstanding here with regard to the words 'dispensing/dispenser' and he/she who 'selects' the medicine. When I use the word 'dispense' I mean the actual act of preparing the medicine from the dispensary, i.e. taking the remedy and medicating the powder sachets according to the prescription or using placebo, the person who does this is called the dispenser, (he was not blinded) the person who dispensed the medicine was independent of the study (a laboratory/dispensary technician) who is employed by the Department to run the dispensary.
- The researcher/prescriber (who was blind to what was ultimately dispensed) who you call the one who 'selected' the medicine did not communicate directly with the laboratory/dispensary technician, once the prescription was made the file is submitted via reception staff to the dispensary technician who then dispensed the active medicine or placebo according to the randomisation list (thus the dispensary technician was not blinded). The dispensary technincian is bound by a code of conduct with regard to confidentiality in this regard and cannot reveal what was prescribed to the researcher, we can confidently assume he upheld this code of conduct.
Question: Second, the age range of the participants was stated as from 20 to 58 years of age, with an average of 37. No further information was given as to the distribution of those ages in the group as a whole, nor in the placebo and experimental groups. What was the age distribution in the two groups? A list of the ages of the subjects in each group would be ideal, and should not pose any problems with confidentiality.
- Average age of participants in placebo group = 33.35yrs range was 20-56 years. Average age of participants in the treatment group = 40.5 range was 22-58 years.
Question: Finally, the blinding on this test is described in the paper as having been broken after the end of the test period, but it is not stated whether this is before or after analysis of the sleep diaries to obtain the SII scores. Could you let me know whether the analysis was performed before or after the blinding was removed?
- The researcher was given the list of names and to which group they were allocated i.e. Group 1 or Group 2 but not the identity each respective group, after analysis the identity of Group 1 and Group 2 were revealed.
Update to Conclusion
I think that Mr. Naudé has closed the loopholes in blinding that I was worried about. I agree with him that there is no reason to doubt the conduct of the dispensary staff, and that blinding there seems adequate to remove cues to the participants in the study. Also, by breaking the blinding only after analysis, this removes the possibility of subconscious bias in that phase of proceedings.
However, the difference in average age between the two groups (33.35 v 40.5) still appears to me to be a plausible confounding factor.
I will take a look at some other sleep studies, to see what kinds of correlation between age and sleep exist, and report back here with more information if and when I find it.
I asked Mr Naudé if he thought that the difference in age distributions could affect the result. His response was:
- Statistically the difference in age was not significant when comparing all the members of each group although the mean age differed somewhat, we did do some testing on this. The randomization process we undertook just happened to make the age distribution slightly different i.e. there were also a few outliers which skewed the average age.
- I can only speculate on this issue, Clinically we usually see that older patients are typically slower to respond on average to homoeopathic medicine with younger patients responding much quicker, with the treatment group being the slightly 'older' one could argue from a clinical point of view that the odds were probably against this group responding as quickly, ultimately this group outperformed the placebo (younger group) though which further is suggestive of the clinical effectiveness of this intervention over placebo.
- We should probably have limited the age range in the inclusion criteria, I would recommend this in future studies or possibly stratify the sample according to age.
Second Update to Conclusions
I found this reply unsatisfying. He is right that the age controls could have been better, and he acknowledges that there may be an age-related effect. Together, these two observations introduce enough uncertainty to bias the test.
However, he discounts this as an explanation due to his expectations being the other way. The reason he gives for the difference - that homeopathic medicines act at different speeds in different ages - does not pass scrutiny because it anticipates the conclusion that homeopathy is effective.
The results of the two groups are different, but the two groups are different. This, coupled with the small size of the study, makes it very difficult to rule out the null hypothesis, especially when coupled with the very low a priori plausibility of the treatment.
For comparison, I am still looking for studies or systematic reviews that show the normal frequency of primary insomnia in "normal" people in the age ranges reported in this study. Most of the results I have found either concentrate on specific groups ("Randomized controlled trial of brief cognitive–behavioural interventions for insomnia in recovering alcoholics") or for people over 60 ("Predicting treatment response in older adults with insomnia").