Peer review versus the h-index for evaluation of individual researchers in the biological sciences

Abstract

Past performance is a key consideration when rationalising the allocation of grants and other opportunities to individual researchers. The National Research Foundation of South Africa (NRF) has long used a highly structured system of ‘rating’ the past performance of individual researchers. This system relies heavily on peer review, and has seldom been benchmarked against bibliometric measures of research performance such as Hirsch’s h-index. Here I use data for about 600 rated researchers in the biological sciences to evaluate the extent to which outcomes of peer review correspond to bibliometric measures of research performance. The analysis revealed that values of the h-index based on the Scopus database are typically 5–20 for researchers placed in the NRF’s C rating category (‘established’), 20–40 for those in the B rating category (‘considerable international recognition’) and >40 for those in the A rating category (‘leading international scholars’). Despite concerns that citation patterns differ among disciplines, the mean h-index per rating category was remarkably consistent across five different disciplines in the biological sciences, namely animal sciences, plant sciences, ecology, microbiology and biochemistry/genetics. This observation suggests that the NRF rating system is equitable in the sense that the outcomes of peer review are generally consistent with bibliometric measures of research performance across different disciplines in the biological sciences. However, the study did reveal some notable discrepancies which could reflect either bias in the peer-review process or shortcomings in the bibliometric measures, or both.

Significance:

• NRF rating categorisations (estimates of standing in a research field based on peer review) are a reasonably good predictor of the h-index of individual researchers in the biological sciences.
• The relationships between rating categorisations and the h-index are remarkably consistent across five sub-disciplines in the biological sciences.
• Peer review and the h-index were compared in terms of their relative advantages and disadvantages and the combined use of both approaches is advocated for measurement of research performance.