VCU study finds that many published psychology experiments lack evidence of validity

An examination of nearly 350 published psychological experiments found that nearly half failed to show that they were based on a valid foundation of empirical evidence, suggesting that a wide swath of psychological science is based on an “untested foundation.”

The study



—



conducted by



David Chester

, Ph.D., a psychology professor at Virginia Commonwealth University, and Emily Lasko, a psychology doctoral student at VCU



—



focuses on the practice of experimental manipulations, in which psychologists induce specific mental states, such as giving research participants insulting or complimentary feedback to manipulate how angry they feel.

To conduct these experimental manipulations in a scientifically valid way, researchers must first establish whether their manipulations actually affect the intended psychological variable (for example: make people feel angry) and not other closely related variables (for example: make people feel sad). However, the extent to which psychologists actually examine the validity of their manipulations remains unknown.

Chester and Lasko investigated 348 psychological manipulations included in peer-reviewed studies. They found that roughly 42% of the experiments were paired with no validity evidence, and that the remaining psychological manipulations were validated in ways that were extremely limited.

“These findings call into question the accuracy of one of psychology’s most common practices and suggest that the field needs to strongly improve its practices in this methodological domain,” said Chester, an assistant professor in the



Department of Psychology



in the



College of Humanities and Sciences

.

The forthcoming study, “

Construct Validation of Experimental Manipulations in Social Psychology: Current Practices and Recommendations for the Future

,” will be published in the journal

Perspectives on Psychological Science

.

The study is the first of its kind to systematically document the extent to which psychology’s experiments are based on a valid foundation of empirical evidence.

Importantly, Chester said, the study’s findings do not suggest that the experimental psychologists’ findings were necessarily wrong or invalid.

“We do not find such experiments are invalid, instead we simply don’t have the evidence to know one way or another how valid they are,” he said. “Almost all of the manipulations we examined failed to provide the necessary evidence that they were valid, which does not mean they are invalid



—



their validity is just unknown.”

As a result, he said, the study suggests that “the findings of experimental psychology likely rest on an untested foundation.”

“This framework might be weak, it might be strong, it is more likely both of these things depending on many factors,” he said. “We have outlined a prescribed series of recommendations for experimenters to ensure that this is not the case going forward



—



that the validity of each experimental manipulation is tested in a systematic and accurate way.”

Chester added that he and Lasko hope their findings encourage experimental psychologists to include validity evidence in future research.

“We hope our paper makes experimenters aware of this untested aspect of their research, motivates them to change their practices, and provides a road map of precisely what to do to make such changes,” he said.

About VCU and VCU Health

Virginia Commonwealth University is a major, urban public research university with national and international rankings in sponsored research. Located in downtown Richmond, VCU enrolls more than 30,000 students in 233 degree and certificate programs in the arts, sciences and humanities. Twenty-two of the programs are unique in Virginia, many of them crossing the disciplines of VCU’s 11 schools and three colleges. The VCU Health brand represents the VCU health sciences academic programs, the VCU Massey Cancer Center and the VCU Health System, which comprises VCU Medical Center (the only academic medical center in the region), Community Memorial Hospital, Children’s Hospital of Richmond at VCU, MCV Physicians and Virginia Premier Health Plan. For more, please visit

www.vcu.edu

and

vcuhealth.org

.