Editor’s note: A version of this article originally appeared on the Frederick National Laboratory for Cancer Research website.
Scientific leaders from government, industry, academia, and publishing identified and proposed solutions to one of the most complex challenges facing scientists today—reproducibility in research—at the recent Hood College – Frederick National Laboratory Life Sciences Symposium.
“Research reproducibility has been in the spotlight for many years, and while some progress has been made, it’s a very complex issue with multiple layers that require many solutions, rather than a one-size-fits-all fix,” said Leonard Freedman, Ph.D., chief science officer at FNLCR, who led the symposium and the development of the FNLCR Scientific Standards Hub.
“It’s a very important topic, … and it really matched to the highest expectations,” said attendee Manpreet Singh, Ph.D., a computational research scientist at The Henry M. Jackson Foundation for the Advancement of Military Medicine, in support of the Biotechnology High Performance Computing Software Applications Institute.
Several presentations showcased tools and plans for improving reproducibility, with topics ranging from data repositories to assay manuals to genomic and proteomic techniques. Audience members took notes and posed thoughtful questions. There was palpable interest.
“I’m going to take … and try to build upon” the information once back in the lab, said Gabriel Benton, Ph.D., a quality control scientist in FNLCR’s Applied and Developmental Research Directorate.
A First Step: What is Reproducibility?
Keynote speaker Harvey Fineberg, M.D., Ph.D., professor emeritus of Health Policy and Management at the Harvard T. H. Chan School of Public Health, said just defining reproducibility is challenging, as disciplines view it differently.
The National Science Foundation says reproducibility is “the ability of a researcher to duplicate the results of a prior study using the same materials and procedures used by the original investigator.”
Fineberg proposed a blanket definition, that “reproducibility” refers to all scientific efforts to reproduce, duplicate, and replicate, regardless of the question asked or approach used. He then outlined distinct subcategories and classifications. The structure establishes a shared vocabulary and divides the overarching issue into clearer portions that scientists can more easily address.
Some attendees cautioned their peers against equating reproducibility with scientific validity, urging them to monitor their work for biases. Remaining clear-eyed would foster reproduction of sound data, they said.
Others reminded attendees that irreproducibility is more often due to extenuating factors or misinterpreted data than fraud. Multiple attendees advocated for better outreach to the public to build confidence in science.
“The scientific community needs to really listen to what the general public thinks about the need for data reproducibility,” said attendee Thomas “TC” D.Y. Chung, Ph.D., director of Translational Programs Outreach at the Conrad Prebys Center for Chemical Genomics.
Document the Work in Detail and Publish Negative Data
Given their diverse scientific fields, attendees later broke into smaller groups to allow for in-depth discussions among leading stakeholders.
The proteins group came up with a three-phase plan to establish common standards and a protein repository. The committee on AI outlined best practices for wider adoption. Others made comparable progress.
In the preclinical drug studies group, moderator Nathan Coussens, Ph.D., director of FNLCR’s Molecular Pharmacology Laboratory, asked the participants to identify areas for improvement. Nearly half indicated laboratory protocols, data analysis, and data reporting.
“What are the minimum details someone needs to reproduce [protocols]?” one panelist asked, adding scientists should record small steps they might normally omit.
One panelist likened published protocols to “a cooking recipe.” They skip some details, like an assay’s pH conditions, under the assumption the reader knows or will work in similar conditions.
Another panelist said his well-documented “bulletproof” assay failed when executed by his postdoctoral fellow. He discovered an undocumented, seemingly inconsequential factor—the amount of time the samples remained at room temperature—was the culprit.
Around the room, ideas sprouted from the exchange.
One participant suggested curating and promoting repositories where scientists can publish their full protocols, which are often too extensive for scientific journals.
Incentivize scientists to publish negative data, their experiments that don’t work, another participant said. This would tell others not to waste their time on the failed hypothesis. It also reduces data biases caused by the abundance of published successful studies.
“Maybe bioRxiv would be a way for people to not only publish their negative data but get credit for it,” he said.
In all, the symposium focused on finding solutions to bolster reproducibility in research rather than pointing blame or criticism at the many ways it fails to occur, Freedman said, and noted that the topic resonated with the nearly 200 highly engaged participants.
Samuel Lopez leads the editorial team in Scientific Publications, Graphics & Media (SPGM). He writes for newsletters; informally serves as an institutional historian; and edits scientific manuscripts, corporate documents, and sundry other written media. SPGM is the creative services department and hub for editing, illustration, graphic design, formatting, and multimedia.
