Setting the course: our first two years in the focal area Evaluation & Culture
The Evaluation and Culture focal area at CWTS is dedicated to studying, discussing, and advocating for renewed forms of scholarly communication, fair research evaluation, and inclusive research cultures. This blog post offers a reflection on our activities and progress over the past two years.
In early 2023, CWTS introduced a new organisational structure of so-called focal areas to address the challenges outlined in its knowledge agenda. As discussed in a previous post, the knowledge agenda is not another research agenda, but rather combines research and intervention around the mission of CWTS to better understand how research is practiced and governed, and how it serves society.
The focal area Evaluation & Culture started with a collective effort to translate the overarching ambitions of the CWTS knowledge agenda into more specific goals. This led us to two main foci: scholarly communication on one hand, and research evaluation and research culture on the other. Our activities around these topics involve work on three levels: understanding, intervening, and practising what we preach.
Scholarly communication
One of our defining achievements has been the launch of MetaROR, a platform for open peer review of research articles in the field of metaresearch. Launching MetaROR is one of our contributions to the Research on Research Institute (RoRI), an alliance of research funders and science studies researchers at several universities in which CWTS is one of the participating research organisations. The launch of the platform has entailed infrastructural and development work, but also community-building activities to introduce and discuss the concept of publish-review-curate among academic audiences, some of which don’t have a tradition of experimenting with publishing models.
Another concrete example of our approach to both understanding and intervening was our involvement in a major consultation to elicit feedback from relevant stakeholder groups in scholarly communication on the reform proposal ‘Towards Responsible Publishing’, published by cOAlition S in 2024. This gave us the opportunity to play a role in shaping discussions around prominent ideas for reforming scholarly communication around principles of open access, transparency, and equity.
Research evaluation & research culture
In regard to our second area of activities, we aimed to develop a more systematic understanding of attempts to diversify notions of research quality that are used to inform evaluative processes in research across different organizational levels over the last decades. We have produced conceptually informed research on science reform movements internationally and in the Netherlands.
We have also sought to follow a ‘practice what we preach’ approach in our projects. In our service projects, evidence and its interpretation are provided for SEP-related (Strategy Evaluation Protocol) self-evaluations and contributions, following the insights of our research. Through these projects we have also strengthened relations and exchanges with policy-support officers, scientific directors, and researchers in general.
Our work has allowed us to contribute to the knowledge base for new interventions and innovations in research assessment reform. As part of our involvement in RoRI, we have carried out research on the evaluative use of narrative CVs. So far, we have published preliminary results from a comparative study involving three different funding bodies as well as a set of practical recommendations for funders, and we are currently exploring possibilities to develop new services aimed to support organisations interested in introducing narrative CVs.
We have also played a major role in mapping changes in national research assessment and funding systems. As part of RoRI, we will in May launch an online observatory to summarise and compare national assessment systems around the world. A major survey of research funding agencies’ responses to the Responsible Research Assessment (RRA) agenda is forthcoming.
A selection of focal area contributions in this area:
|
Next steps: improve inclusivity and keep asking questions
These achievements notwithstanding, the transition from a more disciplinary organisational structure to a mission-driven format hasn’t been without difficulties. These include for example the challenge of creating openness for different interests and disciplinary backgrounds and making sure to embed especially junior colleagues in the projects and discussions of the focal area. Connecting our research projects to the service projects we carry out for universities, research funders, and other actors in the research system also needs further attention.
Furthermore, it is important to avoid making (the wrong) concessions to the technocratic nature of much of the debate around research evaluation and continue asking questions of a fundamental nature - e.g., is (more) evaluation always a good idea? What are the costs of evaluation? What are the strengths and limits of voluntary reform initiatives in coordinating and catalysing cultural transitions in science?
The members of the focal area Evaluation & Culture: Louise Bezuidenhout, Clara Calero Medina, Soohong Eum, Thomas Franssen, Nina Gogadze, Kathleen Gregory, Tjitske Holtrop, Tanya Lee, Thed van Leeuwen, Ed Noijons, Ana Parrón Cabañero, Alex Rushforth, Clifford Tatum, Judit Varga, Ludo Waltman, Eleonora Dagiene, Dmitrii Kotchetkov, Paul Wouters, Marta Sienkiewicz, Sarah de Rijcke, Kwun Hang Lai, Helen Clare, Clifford Tatum, Rinze Benedictus, Carole de Bordes, Rose Bieszczad, Ismael Rafols, Andrew Hoffman, Margaret Gold, Juan Bascur Cifuentes, Anestis Amantidis, André Brasil, Laurens Hessels together with the coordinators: Wolfgang Kaltenbrunner, Andrea Reyes Elizondo, Inge van der Weijden look forward to the next two years.
Header image by Markus Spiske on Unsplash.
DOI: 10.59350/cr9ah-sxj73 (export/download/cite this blog post)
0 Comments
Add a comment