Science and how we know we are wrong 3

This study made it into several newspapers when it was published last week. Science-blogger Orac gives a detailed look at the study, and points out the flaws in the arguments in the accompanying editorial, which tried to defend alternative medicine in the face of evidence of absence of efficacy.

In the meantime, angry doc finds the study interesting for a different reason.

The study looked at the subjects' performance on the ADHD Rating Scale-IV and Clinical Global Impression Improvement Scale (which are assessed and scored by clinicians) as primary end points, and also analysed the Parent-Reported Child Behavior Checklist or Youth Self Report Form (for subjects 11 years and older), Conners Parent Rating Scale, and parent-report and child-report forms of the Pediatric Quality of Life Inventory (which are scored by the subjects or their parents), and found that there were no statistically-significant differences in the scores before and after treatment between the group receiving St. John's Wort and the group receiving placebo.

But that's not all.

If you look at the scores (you'll need to register to read the full paper on JAMA), you will find that except for the Youth Self Report Form, scores for both groups improved at follow-up.

In other words, the clinicians involved in the trial and the parents of the subjects all *thought* that the children were improving.

What's more, children in the placebo group 'improved' more than children in the treatment group, even if the differences in the scores were not statistically-significant.

To angry doc, this trial demonstrates the power of the placebo effect, where just the fact that the children are being given a pill three times a day, regardless of whether it contains "a mixture of rice protein powder and a small amount of activated charcoal" or "0.3% hypericin... free of heavy metals, pesticides, and adulterants" can affect a clinician or parent's assessement of their behaviour.

So the next time you are tempted to say "I know it works because he got better after taking it", stop and think of this study, won't you?