Study examines state of social, personality psychology research

Is the quality and overall state of social and personality research “rotten to the core,” as has been debated by psychologists in recent years?

The answer is no, according to University of Illinois at Chicago researchers who conducted two studies to examine how practices have changed, if at all.

In one study, the UIC researchers surveyed over 1,100 social and personality psychologists from the three largest professional organizations — the Society for Personality and Social Psychology, the European Society for Social Psychology, and the Society for Australasian Social Psychologists — about how the current debate has affected their perceptions of their own research practices and the field’s.

Matt Motyl, UIC assistant professor of psychology and lead author of the study. Photo: Jenny Fontaine

“Scientists said they are less likely to use questionable research practices, and more likely to use best research practices, following the ongoing discussion of scientific reliability,” said Matt Motyl, UIC assistant professor of psychology and lead author of the study. “Upon examining justifications for using questionable research practices, we found they tended to be quite justifiable and reasonable.”

Journal editors requesting that scientists omit some analyses, or even whole experiments, and researchers remedying statistical problems, such as having too-small sample sizes to yield reliable results, are examples of the proactive field improvements noted in the report.

Despite the progress, the report indicates perceptions of the current state of the field are more pessimistic than optimistic. Most respondents think that improvement is needed in replicability, which is the ability to repeat an experiment and find the same result.

A second study involved a random selection of 30 percent of all articles published in the four leading journals — the Journal of Personality and Social Psychology, the Personality and Social Psychology Bulletin, the Journal of Experimental Social Psychology, and Psychological Science – in the 2003-2004 and 2013-2014 periods. The UIC investigators manually coded the research practices and important statistics contained in the studies.

“Journals did show a bias to publish significant results, but we did not find widespread evidence of researcher misconduct,” Motyl said. “While there is always room for improvement, none of the findings suggest science was getting worse over this 10-year period.”

Overall, the report suggests the field is evolving in a positive direction.

“Social and personality science seems to be improving, and we should have increased confidence in findings moving forward,” he said. “Although, there is still a long way to go.”

The researchers advise that more research should be pre-registered; data should be shared more widely; and data analysis should be more transparent.

“By increasing transparency, it will be easier to evaluate the truth-value of studies and estimate what findings are real and what findings are more likely to be flukes,” Motyl said.

“The State of Social and Personality Science: Rotten to the Core, Not so Bad, Getting Better, or Getting Worse?” is published online in the Journal of Personality and Social Psychology.

Co-authors are UIC graduate students Alexander Demos, Timothy Carsel, Brittany Hanson, Zachary Melton, Allison Mueller, JP Prims, Jiaqing Sun, Anthony Washburn, Kendal Wong and Caitlyn A. Yantis; and Linda Skitka, UIC professor of psychology.