A new study poses a bit of a riddle in its first few paragraphs. He notes that scientific progress depends on the ability to update ideas that are deemed acceptable in the face of new knowledge. But science itself has produced no shortage of evidence that people are bad at updating their beliefs and suffer from problems such as confirmation bias and motivated thinking. Since scientists are actually humans, the problems with updating beliefs should severely limit the ability of science to progress.
And there are some signs of it. Max Planck wrote, for example, that “a new scientific truth does not triumph by convincing its opponents and bringing them to light, but because their opponents eventually die and a new generation grows up.”
However, a new study suggests that this may not be a huge problem. Some scientists took advantage of a scheduled replication study and interviewed their colleagues before and after the results of the replication study were published. And most scientists seemed to update their beliefs with little effort.
Before and after
The design of the new study is simple. The researchers behind it used a planned replication study – one that would repeat some prominent experiments to see if they got the same results. Before the replication results were announced, the researchers contacted approximately 1,100 people involved in psychological research. These participants were asked what they thought of the original results.
When the replication work was complete, some of the earlier experiments were replicated, which gave greater confidence in the original results. Others failed, which raised the question of whether the original results were reliable. This should give the research community an opportunity to update their beliefs. To see if this was the case, the researchers went back behind the new paper and found what the same 1,100 people thought of the experiments, in terms of whether the experiments were repeated.
In a practical sense, the research team’s subjects were asked to read the results of the replicated studies and then assess whether the results were likely to represent a “non-trivial” effect. Participants were also asked whether they believed in these previous results or whether they personally invested in them (as might be the case if they based their own research on the results). Half of the participants were asked about the quality of the replication experiments and whether they succeeded in reproducing the conditions of the original experiments.
After the replication was complete, all participants were asked again to assess whether the effect tested in the replication is likely not trivial and their confidence in the effect. They also rated the quality of the replication experiments.
This setup allowed the researchers behind the new study to assess whether participants updated their thinking in response to the new data. It also provides an opportunity for researchers to examine some of the factors that affect motivated thinking, such as: B. a personal interest in the result. And a participant who argues with motivation might dismiss replication as inferior, which the researchers also asked. Overall, this appeared to be a thorough study.
Apply the update
Overall, the participants see the study quite well. If a replication was successful, they were more confident that the replicated experiment showed a significant effect. When replication failed, they adjusted their trust in the opposite direction. In fact, the participants updated their beliefs more than they themselves expected.
They also showed little evidence of motivated thinking. There was little evidence that the researchers changed their minds about the quality of replication, even if the data challenged their earlier thoughts. Nor did they focus on differences between the original experiments and replication. Personal interest in the results also made no difference.
Knowing about possible sources of bias might protect people from motivated thinking, but again there was no evidence of it. The only thing that seemed to correlate with appropriate belief updates was a self-reported sense of intellectual humility.
Overall, psychologists do not seem to suffer from cognitive biases that prevent people from correctly integrating new information. At least when it comes to science – it is very likely that they will do so in other areas of their lives.
There are two major caveats. For one thing, participants knew that their responses would be kept confidential so they could afford to express opinions that could cause problems if published. Thus, there can still be a gap between what each participant thinks privately and how the field as a whole reacts to differences in replication status.
The other caveat is that participants knew they were doing a reproducibility study. So they might be expected to shade their answers to make them look good to their fellow researchers. The main thing that speaks against it is that the participants haven’t changed their minds as much as you’d expect given the size of the difference between original and replication results. In other words, participants were cautious about a failed replication – not something you would expect from someone who does reputation management.
Even with these caveats, these results are probably worth investigating. The kind of behaviors that allow people to maintain their beliefs despite evidence to the contrary is a major societal problem. If scientists can break away from these beliefs in some contexts, it would be useful to understand how they do so.
Nature human behavior, 2021. DOI: 10.1038 / s41562-021-01220-7 (Via DOIs).
This article was previously published on Source link