The Perceived Purpose and Limitations of Peer Review
Several commenters discuss the true purpose and limitations of peer review, especially in the context of public perception and trust in science. Some argue that the public overestimates the rigor and reliability ensured by peer review.
- "Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?" - jxjnskkzxxhx
- "I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?" - yupitsme123
- "The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers." - jostmey
- "One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing." - yummypaint
Erosion of Public Trust in Science and the Role of Peer Review
A recurring theme is the declining public trust in science, often attributed to the perceived misuse of research for political agendas and general scientific illiteracy. Some suggest that changes like transparent peer review can help address this issue, while others are skeptical that these changes will have a meaningful impact on public opinion.
- "The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus." - yupitsme123
- "The public has lost trust in science because 10 to 30% of it is scientifically illiterate." - JumpCrisscross
- "Transparent peer review raises the bar for both academics and enhances the potential trust in the process." - aDyslecticCrow
- "I highly doubt it's a meaningful factor in public trust." - thfuran
- "Of the people who distrust science, how many of them have ever read a scientific paper?I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either." - jfengel
- "Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail." - NoMoreNicksLeft
- "When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it." - JumpCrisscross
- "So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews." - aDyslecticCrow
The Prevalence of Low-Quality vs. Dishonest Research
Several comments discuss the sources of problems in scientific publications. A distinction is drawn between deliberate fraud and issues stemming from pressures to publish, which can lead to lower-quality work.
- "Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science." - aDyslecticCrow
- "There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important." - aDyslecticCrow
- "Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes." - yummypaint
Critique of a "Scientific Society" Model
The discussion touches upon the potential benefits and drawbacks of reverting to a model resembling older scientific societies to improve peer review, highlighting concerns about bias and stifled innovation.
- "We need more scientific societies... it was all about a real community—setting high standards and having just a few people decide what to publish...“Should we publish this?” asked the society." - dr_dshiv
- "That model had some major issues. Too many opposing theories to the 'established norm' were dismissed, among things we now know were wrong all along...it restricted innovative ideas or unpopular outcomes." - aDyslecticCrow
- "Sounds like a recipe for serious inequity to me." - striking
- "The problem with the society method is it didn't work better than a decentralised scientific system." - JumpCrisscross
- We want to believe, which leads to farces like Piltdown man" - aspenmayer
Potential Risks Associated with Transparent Peer Review
Some participants express concern about the potential downsides of increased transparency in peer review, particularly regarding power dynamics, potential harassment of reviewers, and the discouragement of critical feedback.
- "I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor." - eig
- "Imagine saying 'no' to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you... a measure that women and inexperienced researchers do not support is a measure that favors only those who are already part of the club." - probably_wrong
- ""The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."" - JumpCrisscross (countering the previous point)