The changing meaning of bibliometric indicators

For many years, publication and citation indicators, such as those used in the CWTS Leiden Ranking, could be interpreted in relatively straightforward terms. Publication counts were treated as a proxy for research activity and volume. Citation impact was generally understood as a marker of scientific influence and visibility. Debates focused primarily on technical questions, including normalization procedures, counting methods, and field delineation. The integrity of the publication system itself was rarely a central concern. That relative interpretive stability, however, is now more difficult to maintain.

What has changed is not the presence of distortion, but its scale and organization. The rise of paper mills, coordinated citation practices, editorial failures, and industrialized publication models has reshaped the environment in which bibliometric indicators operate. As a result, publication and citation metrics increasingly function as mixed signals, reflecting both genuine research strength and systemic noise within the contemporary publishing ecosystem.

These developments do not invalidate publication- and citation-based rankings. However, they complicate their interpretation. High output and strong field-normalized citation impact may no longer reflect scientific performance alone. In some institutional contexts, these indicators may also signal elevated exposure to high-risk publishing environments within the global research system.

These challenges have prompted growing interest in incorporating integrity-related signals into institutional assessment. One recent example is the Research Integrity Risk Index (RI²), developed as a diagnostic framework to identify institutional exposure to systemic research-integrity risks using transparent bibliometric indicators. RI² is not yet embedded in formal ranking systems.

From performance measurement to risk diagnostics

RI² does not measure misconduct, nor does it replace traditional performance metrics. Instead, it focuses on institutional-level indicators, currently including:

• D-Rate (Delisted Journal Rate): The share of an institution’s articles published in journals dropped from coverage in Scopus or Web of Science following formal review of editorial and publication standards. D-Rate reflects exposure to journals that no longer meet indexing criteria.
• R-Rate (Retraction Rate): Retractions normalized by publication output, reflecting exposure to research formally withdrawn due to author error, author misconduct, or editorial failure.
• S-Rate (Self-Citation Rate): Institutional self-citation intensity relative to global disciplinary baselines. Self-citation is a normal feature of scholarly work. S-Rate identifies statistical outliers where institutional self-referencing substantially exceeds field expectations.

Work is also underway to extend the framework with an additional indicator capturing post-retraction citation exposure (C-Rate), introduced in a beta version of RI² released in February 2026.

These signals are not judgments of misconduct. They are diagnostic signals. They serve as early-warning indicators of environments in which conventional output and impact measures may be partially inflated or structurally distorted.

Importantly, RI² is not a ranking. It is a diagnostic overlay that can sit alongside existing rankings and help interpret performance metrics more cautiously. It is intended to encourage methodological evolution in ranking systems as the publishing environment changes.

Integrity as institutional responsibility

Research integrity failures ultimately manifest in individual decisions. Authors submit manuscripts. Reviewers evaluate them. Editors accept or reject them. Yet these decisions do not occur in a vacuum. They are shaped by institutional incentives, evaluation systems, promotion criteria, and the signals universities send about which forms of productivity are rewarded.

In high-pressure publication environments, career advancement, funding, and reputation are closely tied to output and citation thresholds. Under sustained pressure, quantity-based metrics can weaken safeguards designed to protect research quality.

Engagement with paper mills, predatory journals, or systematic citation manipulation rarely emerges spontaneously. Such patterns tend to flourish where oversight is limited, accountability mechanisms are weak, or evaluation systems disproportionately reward scale over scrutiny.

Research integrity is therefore not only an individual duty but also an institutional responsibility. Persistent exposure to elevated structural indicators constitutes a governance concern, even without direct evidence of misconduct. This perspective aligns with the Hong Kong Principles, which emphasize assessment systems that reward responsible research practices rather than volume alone.

Framing these dynamics as structural risks shifts attention away from individual failings toward the conditions that make questionable behavior more likely. Evidence from several national research evaluation systems suggests that incentive structures tied primarily to publication counts or impact thresholds can unintentionally legitimize risky publication strategies and gradually erode trust in scholarly communication.

Why this matters for the Leiden Ranking

The Leiden Ranking is known for methodological transparency. It excludes author self-citations and removes retracted publications from its calculations. These choices reflect careful treatment of citation signals. However, broader publishing dynamics introduce additional interpretive challenges:

• Production ambiguity: High publication volume may reflect organized publication dynamics rather than organic research capacity.
• Impact ambiguity: Field-normalized citation indicators remain vulnerable to coordinated citation patterns, even when self-citations and retractions are excluded.
• Field concentration effects: Disciplines such as biomedical sciences, materials science, and computer science have experienced disproportionate exposure to editorial failures and large-scale commercial publishing practices. Institutions concentrated in these areas may therefore display amplified distortion signals.

These challenges highlight the importance of Leiden’s ’s commitment to transparent methodology and suggest opportunities for complementary overlays such as RI².

A possible path forward: A dual-lens approach

The response is not to abandon performance metrics, but to pair them with integrity-risk diagnostics. Under such a dual-lens approach:

• Traditional indicators continue to measure output and impact.
• Complementary risk indicators provide context when strong performance coincides with elevated integrity signals.

This does not change how metrics are calculated; it changes how they are interpreted. This approach preserves the strengths of rankings while acknowledging shifts in the publishing environment. It reflects the spirit of the Leiden Manifesto, which calls for contextualized and responsible use of metrics rather than mechanical reliance on indicator outputs.

Avoiding extremes

The current moment presents two unhelpful reactions:

• Treating bibliometric indicators as if the integrity crisis does not affect them
• Concluding that all quantitative evaluation is irreparably compromised.

Both positions oversimplify a more complex reality. Bibliometric systems evolve as the environments in which they operate change. As distortions emerge in the scholarly record, measurement frameworks must adapt accordingly. RI² represents one such adaptation.

For example, a university may display strong publication output alongside an elevated D-Rate. This does not imply misconduct. Rather, it suggests that publication venues merit closer scrutiny before interpreting growth as unambiguous evidence of research strength. Conversely, institutions with low RI² exposure can demonstrate resilience in their publishing practices, strengthening confidence in their performance metrics.

Concluding reflection

The Leiden Ranking has long supported transparent and nuanced evaluation of research performance. Current integrity challenges do not weaken that contribution; they create an opportunity to deepen it.

The key question is no longer only how productive or influential a university appears. It is also under what structural conditions that productivity is generated and how resilient its impact is to distortion or artificial amplification.

Reframing the discussion in this way does not weaken bibliometric evaluation. On the contrary, it reinforces its credibility. By pairing performance indicators with integrity-sensitive diagnostics, the Leiden Ranking can continue to lead global efforts in responsible metrics, ensuring that signals of impact remain interpretable, credible, and governance-relevant.