NRF ratings and h-index for engineers : Are we missing the point ?

Introduction Polanyi1 showed how scientific progress relies on a system of ‘mutual control’, or simply the means by which ‘scientists keep watch over each other’ to prevent resources being grossly wasted. In our present context, two ways in which this ‘mutual control’ is exercised is through peer-reviewed ratings from the South African National Research Foundation (NRF) and Hirsch’s bibliometric h-index.2


Introduction
showed how scientific progress relies on a system of 'mutual control', or simply the means by which 'scientists keep watch over each other' to prevent resources being grossly wasted. In our present context, two ways in which this 'mutual control' is exercised is through peer-reviewed ratings from the South African National Research Foundation (NRF) and Hirsch's bibliometric h-index. 2 Application for an NRF rating requires individuals to prepare portfolios. These portfolios are peer reviewed and individuals are given an A-rating (leading international researchers), B-rating (internationally acclaimed researchers) or C-rating (established researchers). The subsidiary ratings for young researchers are not considered here.
Whilst guidelines for what can be included in these portfolios have been tailored to specific disciplines, emphasis across all disciplines is placed on peer-reviewed journal papers. 3 Other outputs (e.g. confidential reports of applied research for industry in engineering) require considerably more justification and are therefore assessed with greater subjectivity. Nevertheless, this subjectivity is perhaps the main advantage of the NRF-rating system as reviewers can assess individuals holistically and in context. 4 Hirsch's h-index (where h is the number of publications cited h times) is significantly easier to obtain as it is based purely on bibliometrics. The h-index is advantageous as it measures the impact of a body of work better than relying on total publications or total citations. Although h-indices are based on a formula, they can differ based on the database used. For instance, Google Scholar h-indices tend to be higher than Scopus h-indices, as the former's database is generated by an algorithm scouring the web, whereas the latter is based on a smaller curated database. The main disadvantage of the h-index is that it is a metric that does not consider the context of a researcher, which can both unduly inflate an individual's h-index or fail to capture an individual's impact. 4 Despite differences, it is unsurprising that a relationship exists between the two metrics. Johnson 4 compared 614 NRF-rated biological scientists to their respective h-indices in 2020 and found fairly distinct ranges of h-index associated with each NRF rating. In 2013, Fedderke 5 also found a significant statistical relationship between NRF ratings and h-index for 1932 scholars across various disciplines, including engineering. Nevertheless, it is considered worthwhile reassessing this relationship for engineers as citation practices are rapidly changing 4 and engineering faculties increasingly emphasise research, particularly in promotion criteria.

Method
Following the same methodology as Johnson 4 , the latest NRF ratings (23 March 2021) were downloaded from the NRF website (www.nrf.ac.za) and Scopus h-indices were obtained for individuals listing 'Engineering sciences' as one of their primary disciplines (accessed over 16/17 August 2021). Sub-disciplines were not considered, as it was questionable whether this would add significantly to the debate; however, future work may want to consider this aspect. Unlike in Johnson 4 , differences between h-index and NRF rating were tested using a single factor ANOVA test. Similarities between engineers and biologists were tested using two-sample t-tests assuming equal variance. Pearson correlation was used to test whether h-index increased with time from date of rating.

Results
The latest NRF ratings (23 March 2021) included 385 rated individuals with 'Engineering sciences' as one of their primary disciplines. In the 2020 study 4 , a total of 644 rated scientists listed 'Biological sciences' as a primary discipline. In the latest ratings, 'Biological sciences' researchers are the most rated (18%), followed closely by 'Social sciences' also at 18%; the rest of the top five are 'Humanities' at 16%, 'Health sciences' at 12% and Engineering at 9%. Figure 1 depicts ranges of h-index for each of the three main NRF ratings for both biologists and engineers. It is clear that, similarly to biologists, h-indices for engineers fall into distinct ranges for each rating (F(2, 301) = 81, p<0.001). Johnson suggested h-index norms for biologists were about 5-20 for C-ratings, 20-40 for B-ratings and >40 for A-ratings. 4 These h-index norms are shown in Figure 1 by green shading and the percentage of individuals falling within these norms is indicated above these bars. While the majority of engineers also fell within these norms, a number of A-and B-rated engineers fell below the suggested norms. Consequently, in all cases, the mean h-index, per rating, was statistically lower for engineers than for biologists (p<0.05 in all three cases). This difference was 19 for A-ratings, 5 for B-ratings and 4 for C-ratings.
Johnson 4 reported significant differences in h-index, per rating, with time since rating. However, our analysis of biologists showed a weak relationship between h-index and time since rating, with r = 0.2 for A-ratings, r = 0.04 for B-ratings and r = 0.2 for C-ratings. For engineers, the relationship between h-index, per rating, and time since rating was not significantly better, with r = 0.4 for A-ratings, r = 0.2 for B-ratings and r = 0.2 for C-ratings.

Discussion
A striking result was how few engineers were rated. Ascertaining whether this result reflects vastly different numbers of individuals in these fields is difficult. Estimating this from the number of graduates in each field is problematic. Firstly, disciplines are not similarly captured, and, secondly, scientists, whether studying humans or nature, are far more likely to do research compared to engineers who typically enter applied engineering fields. Considering the number of academics in each field is also problematic as scientific research is often not university-bound. Engineering faculties also focus on producing engineers and academics may not feel an equal need to create knowledge compared to those in more pure fields. While a more detailed study could be undertaken, we hope this discrepancy is not due to higher student to academic ratios in engineering, or fewer research-active engineers, or engineering research being inferior, but that it reflects fewer engineers applying for rating. This may change particularly as university promotion criteria increasingly prioritise research and academics seek to demonstrate their worth.
The low number of rated engineers raises the question, Are engineers missing out? NRF ratings are a means of allocating public resources to research and engineers may well be missing out. However, engineering research being more applied, is more likely to obtain private funding as the commercial utility is clearer. Consequently, the question we rather focus on is: Are we missing the point? Returning to Polanyi, scientific worth is a function of three factors, which vary with domain: exactitude, systematic importance and intrinsic interest of the subject matter. 1 While engineering may be deficient in the first two factors, it adequately makes up with 'intrinsic interest', especially for the layperson. Nevertheless, engineering researchers need to be kept in check, so we now consider whether an NRF rating or an h-index is the correct measure.
A key driver of engineering research is developing methods for practising engineers to solve problems. 6 While dissemination of these methods is done in journals, dissemination is more often via conferences, standards, guidelines and trade magazines (so-called grey literature). While tracking grey literature has been attempted 7-9 , doing so is largely impractical. Tracking engineering research within real-world engineering is almost impossible as outputs (i.e. designs, reports and drawings) are 'invisible' to the public and rarely use standardised referencing conventions. Research-active engineers may therefore feel it is impossible to show research impact within the guidelines suggested for rating. The strong correlation between NRF ratings and h-indices will likely reinforce this perception.
As alluded to in the previous paragraph, the users of engineering research ought to be practising engineers. There is a growing body of work that shows that journal papers are a poor means of knowledge transfer between academia and the applied world, particularly within engineering. 10,11 Practising engineers are also contributing less to journals -either by way of original research or by discussioncompared to decades past. 12 Consulting co-workers and supervisors is favoured 13 , with some suggesting that it is only as problem complexity increases that journal articles are considered 14 . However, it is more likely that textbooks are referred to than original research papers. Increasingly, engineers are also turning to social media for information. 15 Fraser et al. 10 contend, and show within Australian engineers, that this lack of trust in journals is due to a perceived research-practice gap. That is, research by academics is perceived to have little or no practical relevance. Fraser et al. 10 argue that the measure of engineering research impact should be the practical use of research. Interestingly, medical practitioners are shown to have a much higher view of journal publications 10 , suggesting medical researchers are much better at articulating the practical significance of their work.
Research-active engineers seemingly face a dilemma of demonstrating the worth of their research. Much engineering research ends up in grey literature, which is hard to index and track. It is also questionable whether high h-indices reflect practising engineers using cited research, given evidence that practising engineers rarely consider journals and the 'invisible' nature of engineering outputs. NRF ratings, in principle, seem the better metric as they take a more holistic view of engineers, but guidance documents emphasise peer-reviewed journal articles and in practice show close agreement to bibliometric patterns present in other disciplines.
Engineers strive to be pragmatic and propose solutions to all problems. In our view, research-active engineers should be able to demonstrate a close relationship with practising engineers. They should show how they are actively approached by industry to solve complex problems and show how they have solved these problems. Solving problems is fundamentally what research is all about. 16 Complex problems are rarely tackled alone and teams of peers contribute and critique each other along the way. Engineers should demonstrate how they are striving to bring about knowledge transfer, not simply through traditional media (e.g. trade magazines, courses and discussion) but also through digital platforms. Altmetrics are increasingly able to track research impact across digital platforms. 17 Peer-reviewed journal articles remain a key way to formalise research, but impact needs to be measured through more than just citations. Citations in textbooks, guidelines and standards should receive emphasis. Industry recognition through invited lectures, prizes and fellowship to voluntary associations should also be heavily weighted as these provide evidence of peer review and standing. In our view, basing an engineer's research impact on a few well-cited papers is a simplification of what is entailed in engineering research.