Measuring the world of social phenomena

Economists working with Professor Marko Sarstedt from Otto von Guericke University Magdeburg are demanding that the same scientific standards be applied to economics and the behavioral sciences in general as are used in the natural sciences. They believe that the inherent uncertainties in measured values must be described and quantified in order to enhance the reproducibility of measurement series. Only in this way can the sources of errors be identified and eliminated.

In a recently published study, the economists investigated why so much research work in the behavioral sciences is not reproducible – or only to a limited extent. They argue that established checklists that are intended to make important aspects of the studies transparent are inadequate. According to the economist and co-author, Professor Marko Sarstedt, in physics no study would pass muster without stating the accuracy of the measurements used. “Although providing corresponding calibration data for the measurement of social phenomena is difficult to conceive of, it is not impossible.” In the study the economists advocate transferring concepts from the natural sciences to the behavioral sciences and show, using a sample calculation, what major consequences the allowance for measurement uncertainty can have for a seemingly statistically significant result.

Researchers in the behavioral sciences need to try to incorporate the impact of interference factors directly and fully into their results. “Physicists, engineers, physicians and biologists do this by stating the measurement inaccuracy of their instruments, for example of a thermometer or scale,” explains Sarstedt. “Behavioral scientists such as economists and psychologists must do the same when measuring unobservable concepts such as satisfaction and happiness – even if it is difficult.”

Whilst physics generally deal with precisely defined and measurable quantities such as lengths, masses and temperatures, concepts such as satisfaction and happiness are not clearly defined and are measured and interpreted differently depending on the team of researchers. But, according to Sarstedt, despite the disparity between the subjects being studied, reproducibility is simply an essential element of science. Only through the repeated confirmation of research results can they be accepted as fact.

“Protocols in physics or biology specify clear guidelines, which even in terms of the rather imprecise subjects being studied in our discipline must be adapted,” explains the economist. “As in physics, we need institutions to establish standards for measurements in the behavioral sciences and to quantify uncertainties in standard measurements.”

The background to the study is that according to the authors, many areas of social science research find themselves in a replication crisis. In recent years it has not been possible to verify fundamental effects in psychology, management and marketing research in repeated studies under almost identical conditions. Since then, various groups of researchers around the Center for Open Science have made efforts to define standards for empirical studies in order to enhance their reproducibility. From this a variety of checklists have resulted, that are intended to assist researchers in documenting important aspects of their studies and making them transparent.

The study by the authors, Edward E. Rigdon (Georgia State University), Marko Sarstedt (University of Magdeburg) and Jan-Michael Becker (University of Cologne) recently appeared in the renowned journal,

Nature Human Behaviour

under the title “Quantify uncertainty in behavioral research”.

###