Determining the quality, value, impact, or merit of research is no easy task. Results can vary considerably depending on how these criteria are defined, what elements are assessed and how the assessments are used. Research continues to be evaluated around the world at individual, institutional and national levels, yet considerable discussion remains as to what strategies effectively address the complexity of the task.  

A Dutch approach to research evaluation

The Strategy Evaluation Protocol (SEP) is a central element of research evaluation in the Netherlands. Focused at a research unit level, the SEP moves away from the exclusive use of quantitative indicators, emphasising instead "informed peer review" and qualitative assessment as key elements of evaluation. This protocol is updated every six years, and since 2021, it has focused on the research unit's own strategy as well as on formative and development-oriented evaluation. 

In March 2026, an update to the protocol was released for the SEP 2027-2033. In addition to its primary focus on quality, relevance and viability, the updated SEP also considers four key elements, namely PhD policy and training, Open Science, responsible research practices, and academic culture.

The focus on Open Science in the SEP offers an exciting opportunity to enrich the impact of the Dutch National Open Science Programme 2030 (NPOS2030) by embedding Open Science more firmly within research unit strategic planning and action. Nonetheless, it must be recognised that realising this ambition is not without challenges. 

The first reflects a capacity issue. The roll-out of Open Science in the Netherlands is still a work-in-progress and many research units are still at the beginning stages of their journey towards openness. This can mean that research units may struggle to articulate Open Science in their strategic planning and research trajectories. 

The second is an issue of scoping. The NPOS2030 was intentionally aligned to the 2021 UNESCO Recommendation on Open Science that outlines a very ambitious framing of Open Science. This has moved conversations beyond the more “traditional” framings of Open Science (Open Access and Open Data) to include a wide range of additional areas, such as Citizen Science, Open Software, and public engagement. While the broad remit of Open Science offers considerable opportunity, it can be overwhelming for research units to identify where and how to direct their efforts.

The third issue relates to evidence. Within the SEP protocol, the qualitative narrative is accompanied by a variety of indicators. These indicators support the narrative and enable reviewers to delve deeply into the outputs and outcomes of the research unit. It is well-recognised that monitoring and evaluation of Open Science is in its infancy, and many of the elements outlined in the UNESCO Recommendation do not have established approaches for monitoring. This can cause considerable confusion within research units, particularly if approaches to record evidence are not put in place at the beginning of the SEP cycle. 

Research Assessment Frameworks and Open Science monitoring

In recognition of these challenges, at Leiden University we are exploring an Open Science monitoring strategy designed to assist research units in developing their SEP self-assessment. This approach takes inspiration from the Finnish approach to Open Science monitoring, namely the Readiness Assessment Framework (RAF) tool. A RAF is a self-evaluation tool for services, policies and practices (see below). Taking a sequential approach to Open Science, a RAF guides research units through the steps needed to embed Open Science as a research culture while aligning with national policies on Open Science. The purpose of the tool is to assist research units in identifying gaps and opportunities and to monitor their progress.

Representation of the Finnish RAF
The Finnish RAF focuses on five areas that reflect the Finnish national Open Science approach, namely culture of Open Scholarship, evaluation, education, research data, and publications. Within each area a number of subcategories offer clear guidance as to what would constitute a minimal and optimal level of openness. The Finnish RAF also offers a heterogeneous list of examples of evidence that could be used to further illustrate each category. An example of this is shown below.

Exploring a SEP-aligned Readiness Assessment Framework

The wholesale adoption of the Finnish RAF in the Netherlands is, of course, not practical due to differences in the research ecosystems and approaches to Open Science. Nonetheless, the adaption of the methodology and approach for use in a Dutch context offers an exciting opportunity for the Netherlands and the evolution of the SEP. In response to the first concern outlined above, RAFs offer structured roadmapping that supports the SEP approach of strategic planning. This can help research units starting out in their journey toward openness.

The modular nature of the RAF enables new indicators to be added and the model to be scaled. This addresses the second concern above relating to the broad scope of the NPOS2030 and the UNESCO Recommendation. A RAF can evolve with Open Science in the Netherlands, rather than acting as a straightjacket.

Third, and perhaps most exciting, the use of a RAF in the SEP will support the structuring of the narrative self-assessments relating to Open Science. This offers an opportunity to better utilise SEP self-assessments into national strategies to monitor Open Science. Most pertinently, it enables Dutch Open Science monitoring to move beyond traditional approaches that simply count open research products, to better visualise and reward practices and processes that open up and strengthen the entire research lifecycle.

Header picture by Patrick Perkins on Unsplash.
DOI: 10.59350/9hnhe-jy476 (export/download/cite this blog post)