Research evaluation in context 3: The board, the research unit and the assessment committee
What is the procedure for the evaluation of academic research? What are the roles and responsibilities of those involved? This is the 3rd post in a series on research evaluation in the Netherlands and is dedicated to the Strategy Evaluation Protocol 2021-2027.
In a previous post the SEP 2021-2027 was introduced. As a reminder: the Association of Universities in the Netherlands (VSNU), the Netherlands Organisation for Scientific Research (NWO) and the Royal Netherlands Academy of Arts and Sciences (KNAW) share the responsibility for the evaluation of all academic research units. According to the SEP 2021-2027, research units are evaluated in the light of their context, aims and strategy. Also, an evaluation should be understood as integral part of ongoing quality assurance. Evaluation is intended to reflect and learn, it is of formative nature.
The board is responsible and commissions the evaluation,…
The board of an academic research performing organisation is responsible for the evaluation. This includes ensuring that all its research units are evaluated once every six years. It also includes commissioning the evaluation, determining the Terms of Reference (ToR) for the evaluation, appointing an assessment committee and ensuring follow-up.
The board develops the ToR for each evaluation, specifying the aims, criteria, procedure and schedule of the evaluation. Most of it is laid down in the protocol, but the board can add extra aspects or evaluation questions. These might relate to everything and anything: a recent reorganisation of the unit; the upcoming retirement of key staff; novel, maybe potentially disruptive, research opportunities; external funding, etc. The board also appoints the assessment committee, after consultation with the unit.
… the research unit provides evidence in the form of a self-evaluation report…
Given the attention for context, aims and strategy, the research unit has considerable influence in the evaluation process. And that is intentionally. After all, the evaluation takes place in light of the aims, strategy and context of the unit.
The unit is asked to write a so-called self-evaluation report. It is, in fact, more a concise report in which the unit presents itself. The aims and strategy of the unit are central. With the aims and strategy as a reference, the unit describes the achievements of the past six years and it presents future plans. All this in 20 pages or less, and written as a coherent and narrative argument.
… - with indicators of choice - ….
That’s not all. The unit is asked to identify and present forms of evidence that underpin its argument in a robust way. In other words: the unit itself chooses indicators that best fit its context, aims and strategy. It should present the indicators in the report, explain the choice for the indicators and explain how these indicators relate to the unit’s aims and strategy.
The protocol doesn’t prescribe the use of certain indicators, on the contrary; it leaves room for plurality. It merely presents potential indicators. However, the protocol mentions six categories for which the unit is expected to present evidence: Research products for peers (1) and for societal target groups (2); Demonstrable use of products by peers (3) and by societal target groups (4); Marks of recognition from peers (5) and by societal target groups (6).
There is but one restriction: it is not allowed to use the Journal Impact Factor. And there is a firm warning against the use of the H-index. The protocol refers to the DORA declaration and presents arguments for these choice.
… with a number of annexes…..
The annexes to the self-evaluation report contains basic information, such as number of staff, funding and PhD candidates. It also contains the evidence to the self-chosen indicators.
Moreover, the annex should contain a number of case studies. These cases can relate to anything specific that the unit considers important, such as particular projects or programmes, how the unit has organized its interaction between research activities and society or between research and the PhD programmes. Note that these cases don’t need to relate to societal impact per se. Yet the protocol mentions that the case studies are pre-eminently suited to indicate the connection between the academic and the societal. A connection, the protocol explains, that is seen as essential in many academic domains and disciplines.
A final annex, that comes closest to a self-evaluation, is a “SWOT” analysis. The unit is asked to analyze its own strengths and weaknesses, and to identify external opportunities and threats. The SWOT analysis should be used to inform the strategy for the years to come.
… and an independent assessment committee,…
For each evaluation, a dedicated and independent assessment committee is appointed. The unit is expected to propose members. After all, the unit is evaluated in light of its own context, aims and strategy. However the board decides, can remove an add members, and formally appoints the committee.
The committee as a whole (not every member separately) should be able to assess the unit, all criteria and every aspect. It should be diverse, in terms of gender and cultural, national and disciplinary background. It is advised to include a non-academic expert, but this is not required. There are only two strict requirements: at least one PhD candidate and one early-/mid-career researcher have to be part of the assessment committee.
The committee is supported by an independent secretary. The secretary should have experience with assessment processes within the context of scientific research in the Netherlands. Moreover, the secretary should be independent of the board and the research unit.
… based on the self-evaluation and a site-visit,…
The committee is expected to come to its assessment based on the documentation and a site-visit. The documentation includes the self-evaluation report with annexes, as well as the previous assessment report.
During the site-visit the committee members meet with representatives of the research unit (management and researchers, including research leaders; tenured and temp staff; PhD candidates), as well as with representatives of the research organisation and external partners. This allows to verify and supplement the information provided in the self-evaluation.
In some cases the visit takes place on neutral grounds, for instance when a number of universities collaborate in a nationwide evaluation. And in recent months, because of the restrictions due to the pandemic, the site-visits took place virtually, online. However, the idea is that a site-visit allows the committee to take the situation on site, including the infrastructure, into account.
… formulates its assessment…
The committee presents its assessment in a written report. It should be a sharp, discerning and clear assessment. It should describe positive issues and – very distinctly, but constructively – weaknesses and suggest improvements. The text should give a clear assessment regarding all criteria and aspects. In case extra aspects were addressed in the self-evaluation report or the ToR, the committee is expected to address these in the assessment report as well.
… upon which the board formulates a position and ensures follow-up.
The committee submits the report to the board, upon which the board formulates a position. The board is expected to reflect on the assessment and to state how it will follow up on the assessment. The SEP 2021-2027 positions evaluation as part of ongoing quality assurance in the research performing organisation, even more so than before. It urges the board to discuss the assessment outcome and potential actions with the unit, and to use the report as a reference in the years to come.
The board needs to follow-up on the evaluation in a different way as well. For public accountability reasons, the board is required to publish the evaluation portfolio on in its website. This includes a summary of the self-evaluation report, the case studies, the assessment report and the position document of the board.
And what about the follow-up of this series?
This and the previous post describe the formal procedure, goals and criteria of an evaluation, as described in the current Strategy Evaluation Protocol 2021-2027. The text of the SEP 2021-2027 has been a major source for both blog posts. But there is more to understanding research evaluation in context / in the Netherlands. So a future blog post will be dedicated to the practice - or should one say the challenges? - of research evaluation in context. And another one to the practice of designing a protocol, that is responsive to developments and concerns in research (policy).