Open science was one of the key topics at the Science, Technology and Innovation Indicators (STI) conference that CWTS organised in September 2023 in Leiden, the Netherlands. Open science was not only discussed at the conference but was also put into practice in the publication and peer review process of the conference. By way of experiment, all papers submitted to the conference were published as a preprint before they were peer reviewed. Moreover, the peer review process was open: Review reports were published alongside the preprinted papers, and some reviewers also chose to sign their reports.

What did we learn from this experiment? To evaluate the experiment, we invited all 186 authors of conference submissions to complete a brief anonymous survey (survey form; survey responses). Survey invitations were sent after the conference. We received 93 responses, corresponding to a response rate of 50%. Below we discuss the outcomes of the survey.

Preprinting

We first asked survey respondents to share their opinion about preprinting of conference submissions. Preprinting of conference submissions is strongly supported. 51% of the respondents selected the highest score (5), indicating they "like it a lot", and an additional 37% of the respondents selected the second-highest score (4). Only a small minority of 6% of the respondents do not support preprinting and selected the lowest (1; "I don't like it at all") or second-lowest (2) score.

When asked what they see as advantages and disadvantages of preprinting of conference submissions, respondents indicated they appreciate the early access and visibility of research through preprinting. The ability to receive feedback and engage with peers is also seen as an advantage. In addition, preprinting enables researchers to establish priority on emerging topics. However, concerns revolve around the quality assurance of preprints, with worries that some submissions represent ongoing work that still needs further development, in which case preprinting may undermine the credibility of the research. There is also a risk that early-career researchers may feel pressured by the open nature of preprints.

Open peer review

Support for open peer review is almost as strong as support for preprinting. 73% of the survey respondents expressed support for open peer review (score of 4 or 5). Only 14% of the respondents were opposed to open peer review (score of 1 or 2).

Respondents recognized several advantages of open peer review, including enhanced transparency, constructive feedback, and accountability of reviewers. However, respondents expressed concerns about variations in review quality and about the potential impact of negative reviews, especially on early-career researchers. Some respondents indicated a preference for fully anonymous peer review, to make sure author identities do not influence reviewers.

Future editions of the STI conference

Survey respondents are very positive about the idea of adopting preprinting and open peer review at future editions of the STI conference. 76% of the respondents are in favor of this idea (score of 4 or 5), while only 7% oppose the idea (score of 1 or 2).

Respondents also offered suggestions for improvement. They would value more thorough reviews, increased transparency in the decision-making process, and well-defined evaluation criteria. Some respondents recommended improving the selection and assignment of reviewers. Several respondents felt there is a need to have a more user-friendly platform for publishing and reviewing conference submissions.

Strengths and weaknesses of the Orvium platform

The opinions of survey respondents about the Orvium platform, which was used to publish and review conference submissions, are a bit more mixed. When asked about their satisfaction with the platform, 66% of the respondents gave a positive response (score 4 or 5). Only 8% gave a negative response (score 1 or 2), but there is also a significant share of the respondents (25%) that gave a neutral response (score of 3).

Respondents mentioned several strengths of the Orvium platform, including the platform's ability to issue DOIs, support for having multiple versions of a paper, and the generation of ‘previews’ of papers for convenient reading in a web browser. However, a number of weaknesses were also mentioned. While some respondents considered the platform easy to use, others found that navigation was challenging and the interface confusing. Respondents also reported problems with the platform's search function.

Impact of the STI 2023 conference on open science perspectives

We also asked survey respondents whether the STI 2023 conference changed their ideas about open science. Most respondents reported that their views on open science remained largely unchanged, as they were already supportive of open science. However, a significant share of the respondents found that the conference reinforced their positive views of open science. These respondents for instance mentioned that they appreciate the practical implementation of open science at the conference. At the same time, some respondents reported that the conference made them more aware of potential drawbacks of open science, such as information overload and problems in quality assurance processes.

Conclusion

Based on the outcomes of the survey, we consider the experiment with preprinting and open peer review at the STI 2023 conference to be successful. While some survey respondents are critical, a large majority of the respondents are positive about the experiment. Moreover, over three-quarters of the respondents recommend that future editions of the STI conference adopt preprinting and open peer review. The feedback provided by respondents offers excellent guidance for making further improvements to publication and peer review processes at the STI conference, hopefully enabling the full potential of open science to be realized.

We dedicate this blog post to Leo Waaijers. Leo was a passionate open science activist and a driving force behind the experiment discussed in this post. He passed away in August 2023, while the preparations for the STI 2023 conference were ongoing.