Bill Gardner is cautiously—very cautiously—hopeful that big data from electronic health records (EHRs) will enable unbiased estimates of the effects of medical treatment, without any randomized controlled trials. Gardner’s hope, his caution and his description of the data needed—“all the factors that determine who gets what treatment”—are all right on the mark. And EHRs will massively increase data on detailed clinical factors that drive clinical decisions.
But I fear researchers will focus only on the data laying around in EHRs. To cure confounding, you need to go out and measure all the confounders—everything related to both treatment and outcome. Continue reading →
That lack of support was a problem for me. I did not doubt the basic truth that many academic papers are uncited. But to be sure 90% was not urban legend and to learn the context and caveats, I needed to find the original research paper. I was not the only one who wanted the supporting evidence. So, I dove into Google scholar, searching the disparaged academic literature for articles on academic citation rates.
When the mayor of Fort Lee asked why those—now infamous—toll lanes to the George Washington Bridge were closed, he was told, “a study.” Yet when his desperate pleas to Port Authority officials were ignored, he did not turn to the independent official body in charge of making sure that studies are conducted ethically. Such a body must ensure studies don’t harm people—don’t, say, trap ambulances in traffic.
Why didn’t the Fort Lee mayor turn to that body? Don’t research studies that affect humans have to show that they don’t harm those humans? Or at least that benefits exceed harms? Decades ago, after scandals like the Tuskegee study which kept poor sharecroppers ignorant of their syphilis, and therefore untreated, we created rules and bureaucracies to protect human research subjects. Currently, regulation 45 CFR 46 ensures this, mandating the creation of Institutional Research Boards (IRBs) who are charged with ensuring that research on humans is ethical.
The mayor of Fort Lee couldn’t turn to an IRB because there wasn’t one. IRBs don’t apply to a government agency—or school or business—trying to improve operations. Studies that are for “internal management” purposes, don’t count as research, which is defined as producing “generalizable knowledge.” (Generalizable means providing information beyond just the specific setting, place and time of the study.)
Generalizability, however, is an arbitrary, potentially dangerous and just plain stupid way to decide which studies are subject to ethical review. Continue reading →