Pseudoscientific Research -- In Nutrition & Education

The May-June issue of the Skeptical Inquirer prints an important article on the science of nutrition by Dr. Reynold Spector, who is currently a clinical professor at the Robert Wood Johnson Medical School in New Jersy. (He formerly taught at Iowa , Stanford, and Harvard-MIT and was formerly executive VP at Merck in charge of drug development.)

Spector begins by pointing out that “in recent years” progress in research on nutrition has “lagged behind” many other medical fields.

Some of this lag is the result of pseudoscientific beliefs and practices that funders and some researchers, but certainly many science journalists and much of the public erroneously believe is based on scientific methods.

This, my readers from the world of educational policy research will recognize, is all too reminiscent of the mantra in our field: “Studies show….” A phrase that -- in the field of education -- should always cause the waving of a warning flag.

Spector notes that critics of the scientific field of nutrition quite understandably say that “much nutritional research and practice” is, to paraphrase Thomas Hardy, “science’s laughingstock.”

Spector contends that the field of nutrition and the people who rely on its findings are harmed by the absence of “rigorous scientific principles and methods.” He uses data to show that much conventional wisdom about dietary pyramids, food supplements, megavitamins, and weight-loss diets is “unproven, erroneous, or even harmful and is often based on pseudoscience” resting on shoddy research methods.

In nutrition research, there is too much reliance on epidemiology/observation studies for causal conclusions about nutritional facts and practices. Instead, Spector says, such conclusions should be based on sound, rigorous scientific studies (randomized, single variable, hypothesis-driven, with validated instruments and proper statistical analyses) .

[Reading this in Spector’s article immediately reminded me of the fact that when we were both in the U.S. Department of Education, IES Director Russ Whitehurst and I were delighted by Gary Taubes's pro-rigorous-research article on health (“Do We Really Know What Makes Us Healthy?”) in the Sept. 16, 2007 New York Times Sunday Magazine. Taubes had earlier authored the path-breaking 2002 article “ What if It's All Been a Big Fat Lie? ” ]

Here (from Spector’s article) is a chart showing how the public has been led astray by reliance on false-positive results from epidemiology/observation studies:



Spector points out that the appropriate methodology for finding the truth in nutrition research “has been known for decades.” Nonetheless, Spector says, it is often “either not followed or scientific data are resisted.”

Spector suggests that to understand why this happens, we take a sociological approach. He says, “A useful part of such an analysis is the question: who benefits from a particular event or behavior?” He proposes that the following groups benefit from shoddy nutrition research:

• Journals and their editors (a constant flow of content);
• Academics (“It’s easy to publish almost anything; certain types of studies (e.g., case…studies) require much less effort and resources than controlled trials to yield a publication.”);
• Business interests (who market products and services based on the shoddy research); and
• The news media (controversy and novelty sell newspapers and bring in TV viewers).

Many might say that like beneficiaries are present in the education industry as well. Some might add as potential beneficiaries in the education industry: the education Establishment and the politicians allied with them. Spector also notes that “the unwillingness of investigators who perform pseudoscientific studies” to acknowledge error “cannot be underestimated.”

Spector insists that “unless proper studies are done,” the research literature is “doomed to potential, often-unknown bias and confounding.” He adds that although “it is difficult and expensive” to do “long-term adequate…studies,” they can be done and have been done.

Who, Spector asks, is harmed by shoddy nutritional research? Spector provides a list of those harmed. If we modified his list to apply instead to the education industry, it would read as follows:

• Students and their parents;
• Effective teachers and other efficacy-minded instructional specialists (textbook writers, in-service trainers of teachers, etc.);
• Scientifically-minded education researchers and ed-school professors;
• Resources (which are wasted);
• The news media (“Reporting advice that ultimately requires revision or repudiation makes them look foolish."); and
• Professional standards (undermined by lack of rigor in pursuit of the truth).

Spector concludes by making recommendations for readers of future research in the field of nutrition:

1. "Readers of medical reports and journals should focus on studies that employ methods that test a hypothesis definitively.
2. "Readers should be skeptical of the results of [epidemiology/observation studies] that test a contributory causal hypothesis and draw causal conclusions…. Such studies must be considered at best hypothesis-generating. Moreover, unless such studies have a clear “upfront” hypothesis and prespecified data analysis plan and are not the result of “data-dredging,” they merit even less credence....
3. "Readers should encourage journal editors, academicians, and funding agencies to support quality studies (e.g., randomized controlled studies) rather than those unlikely to answer questions definitively (e.g., [epidemiology/observation studies ], case-control studies, or cohort studies)."