When research isn’t science

Some time ago, a link to this quasi-review article crossed my desk. Briefly, it explains why, in the author’s opinion, the subject articles on genetic research in autism are somehow untrustworthy or flawed.

In some ways, it’s well-written. I like the bibliography. It’s all Nature, Science, NEJM, and Nature Genetics.

But to me, the real story is what’s not on the page. It’s too bad the author didn’t cite leaky gut papers from those journals or anything related to his “environmental” arguments/theories/guesses. That’s the real story – he can complain about these genetic studies because they’re published and methods are revealed and all the rest is an open book — including funding sources, conflicts of interest, and even access to many of the underlying samples. In some cases, the journals themselves require researchers to share all their data even if the lab wants to keep some of it private so they can look for more publishable findings.

Anecdotal evidence isn’t. Much of what is passed off to parents as “research” or “science” or “medicine” is so opaque or poorly structured that no one writes about it critically. It’s not deserving of criticism, and the practitioners who operate in this way don’t really complain about being shut out of mainstream scientific discourse. They haven’t even ante’d up the price of admission to the discussion.

I asked a research scientist once about whether there was any point to telling these other doctors and “researchers” (and I feel compelled to put it in quotes) about what constitutes a reviewable study. The response: all they have to do is read the journals and they can see what types of studies get published in terms of study design, replication, confirmatory evidence, and disclosures about ancillary matters (conflicts of interest and funding).

When you think about it like that, you see that the real reason the “alternative” and “biomedical” research is ignored or rejected is that it just doesn’t get presented in a way that looks like what society has spent hundreds of years refining as science and the scientific method.

Yes, it’s true that this post is aimed at discussions about the science of autism. But the same issues are there when it comes time to select specific interventions for remediating reading problems for children with learning disabilities. Some things work, and some don’t. If there isn’t research supporting the use of a particular intervention in the circumstance that a specific child presents (not just children in general or dyslexics or “most” autistic children) and if another reasonable intervention isn’t identified, then good advocacy demands that an IEP include a process for determining whether progress is being made and whether that progress is satisfactory.