New, massive study finds only half of social science findings replicate

Research & Discovery

New, massive study finds only half of social science findings replicate

In 2012, in the Journal of Organizational Behavior found extroverts experience a closer bond to their workplace, team or school.

It was one of many research findings at the time showing feelings affect performance. Companies like Google began leaning into employee engagement, with themed workspaces, free snacks and meals, recreation areas and game rooms, .

The trend popularized open offices, focus rooms, and collaboration and wellness spaces. Soon came the backlash. The Harvard Business Review found open offices meant . people used email and Slack more than they had direct conversations.

The trouble with that 2012 study was that it may have been wrong. University of Virginia researchers were unable to reproduce the findings.

“The original study found that extroverts don’t just think of themselves as part of their organization – they feel pride, happiness and emotional attachment,” said Ryan Wright, an artificial intelligence expert in �鶹�ƽ�� McIntire School of Commerce.

“Our team tried to replicate it. We measured students’ personalities and then asked how they connect to their university – six different ones in total – both emotionally and cognitively. The link between extroversion and that emotional connection didn’t show up in our data. Being an extrovert didn’t predict whether someone felt a warm bond with the university.”

Checking the science

Nosek says replicability is essential for establishing reliable, generalizable knowledge. (Photo by Lathan Goumas, University Communications)

Wright was one of hundreds of researchers who worked on a new, huge replication project that UVA psychology professor Brian Nosek led. Nosek first broke news in 2015, when his team tried to replicate 100 psychology studies. In the largest replication study to that point, they could .

In 2019, he expanded his study to 11 areas of social psychology, including health sciences, leadership and organizational behavior.

��ǲ��’s from around the world to investigate nearly 3,900 scientific studies published across 62 journals from 2009 to 2018. Of the studies, researchers sought to replicate 164 papers. With funding from the U.S. Defense Advanced Research Projects Agency, they successfully replicated 49% of them.

“These numbers are unsurprising, in the sense that they are consistent with what we’ve seen in prior investigations,” Nosek said. “Nevertheless, they still are surprising in that our natural assumption as a reader of the published literature is that most of the findings would be replicable if we tried to do it again.”

He said the findings are important because replicability is a key component of establishing generalizable knowledge. “Usually in research, we’re making a more general claim. ‘When you eat this kind of food, you have this kind of health outcome.’ And that should apply if you eat that food tomorrow,” he explained.

As part of the work, documented , ��ǲ��’s team to help reproduce some of the studies. They found there is “room for improvement,” noting AI is not yet reliable enough to replace human judgment or actual replication.