Assessing the scientific relevance of a single publication over time

Quantitatively assessing the scientific relevance of a research paper is challenging for two reasons. Firstly, scientific relevance may change over time, and secondly, it is unclear how to evaluate a recently published paper. The temporally averaged paper-specific impact factor is defined as the yearly average of citations to the paper until now including bonus citations equal to the journal impact factor in the publication year. This new measure subsequently allows relevance rankings and annual updates of all (i.e. both recent and older) scientific papers of a department, or even a whole scientific field, on a more objective basis. It can also be used to assess both the average and overall time-dependent scientific relevance of researchers in a specific department or scientific field.


Introduction
Identifying the most relevant scientific papers within a certain department or field may seem to be a simple task at first sight, but the resultant need to rank the papers makes it a serious challenge.A selection compiled by superiors or (alleged) experts may be immediately challenged by those whose papers do not feature.Accusations of bias and general discontent may follow.In order to avoid such discord, the selection could alternatively be based on a compromise between all the stakeholders.For instance, each researcher could be allowed to select a certain number of their papers depending on years already spent in research, although such an approach may lead to suboptimal selections from the viewpoint of scientific relevance.
Usually, quantitative measures of the scientific relevance of the publishing journal are considered, with the journal impact factor being the most common choice. 1 The journal impact factor can be regarded as an estimate of the average number of citations a journal paper will receive in the first 2 years following the publication year.Most notable is the poor correlation of a journal's impact factor and the actual number of citations of the individual papers within the journal. 2 Therefore, measuring the scientific relevance of single papers by the impact factor of the journal in which they appear seems reasonable for papers recently published.However, for all other papers, the use of the journal impact factor is limited by a levelling effect as both highly cited and infrequently cited papers are given the same impact factor.There is therefore a need for a quantitative paper-specific evaluation criterion.
The basic idea behind paper-specific scientific relevance is that highly relevant papers will be cited more often than papers which are less relevant.A simple and straightforward first approach is therefore to sum the number of citations a paper has received to date.The citation h-index for single publications is another approach adapted from the author h-index. 3,4A publication has citation h-index H, if H of its citing papers have been cited at least H times each, and the remainder of the citing papers have been cited no more than H times each.Both the number of citations received and the citation h-index can never decrease over time.The former can be considered a measure of the first-generation spread of a publication and the latter a measure of its second-generation spread.Neither can be used to compare papers with different publication dates, as older papers are obviously favoured over newer ones.
Dividing the number of citations received and the citation h-index by the number of years since the respective publication appeared yields the average number of citations per year and the citation m-quotient, respectively.The latter is analogous with the m-quotient for authors. 3The average number of citations per year is a disadvantage only for recently published papers whose first citing papers have yet to be written, submitted and published.This disadvantage is even more prolonged for the citation m-quotient, for which a second generation of citing papers is required.However, all these approaches ignore a psychological influence, in that they do not appraise the reputation an author acquires when publishing in a journal with a high impact factor.Such journals are usually flooded with submissions and simply having a paper accepted for publication is considered an achievement sui generis, even if the paper is rarely cited afterwards.Keeping all these factors in mind, we aimed to derive a measure which 1. considers citations of a paper as crucial 2. avoids disadvantages for a paper just published 3. recognises the reputation of publishing in a journal with a high impact factor The temporally averaged paper-specific impact factor (TAPS IF ) fulfills these requirements.TAPS IF is the paper's average number of citations per year (including the publication year) combined with bonus cites for the publishing journal's prestige -which is taken as the journal impact factor from the publication year.Furthermore, the annual TAPS IF values of all the papers of an author can be appropriately combined to measure the overall scientific relevance of the author over time.

Methods
The temporally averaged paper-specific impact factor (TAPS IF ) of a scientific paper for a given year is defined as where pubyear denotes the publication year of the paper, IF(pubyear) denotes the journal impact factor in the publication year, and citations(i) denotes the number of citations the paper has received in the i-th year.
The basic idea is that in the publication year the paper's scientific value is derived from the impact factor -the scientific relevance or prestige of the journal.With the passing of time, the journal's relevance becomes less important and the paper's own performance -determined by the number of citations received -comes to the fore.Note that alternatively the journal impact factor of the paper submission year or of the year for which TAPS IF is computed could be considered.The former would account for the relevance of the journal when it was chosen by the authors, the latter would account for temporal changes in relevance (for details see Bloching 5 ).
Assume that an author has published n(year) papers up to a specific year.The TAPS IF can now be used to define a measure for the scientific relevance of the author for this year -the temporally summarised authorspecific impact factor (TSAS IF ): where TAPS IF,j (year) denotes the TAPS IF of the j-th paper in the specific year.An annual mean TAPS IF for the papers of the author can also be defined -the temporally averaged author-specific impact factor (TAAS IF ): n (year) Equation 3

Results and discussion
For demonstration purposes assume that researchers in a scientific department have published three papers -A, B and C -in three different scientific journals in 2004 (and none before or afterwards).Furthermore assume that the journal impact factors of the journals in which Papers A and B were published were 2.5 and 10 in 2004, respectively.Assume Paper C was published in a local scientific journal or conference proceedings for which no impact factor was available, so a value of zero is used.The assumed citation counts and the resultant TAPS IF values for the three papers for the years 2004 to 2011 are shown in Table 1.

Table 1:
An illustrative example showing the computation of the temporally averaged paper-specific impact factor (TAPS IF ) of three hypothetical papers  The TAPS IF can also provide a reasonable basis for researcher evaluation beyond the popular h-index.Crucial features of the h-index are its monotonic increase over time and its favouring of researchers who steadily publish medium-quality papers over those who rarely publish highly cited papers.
The TAAS IF enables a yearly comparison of the individual average scientific relevance across various researchers, irrespective of their seniority and productivity.The TSAS IF measures a researcher's overall scientific relevance in a specific year.A researcher's seniority and productivity are considered in the TSAS IF .If a researcher's scientific relevance begins to decline, then both TAAS IF and TSAS IF will decrease over time.As for TAPS IF , the use of TAAS IF and TSAS IF should be restricted to single departments or single scientific fields for the purposes of comparison.
Papers B and C start with a TAPS IF value equal to their respective journal impact factor in the assumed publication year 2004.Paper A shows a TAPS IF value larger than its journal impact factor as it has already received a citation in the publication year.Paper A regularly receives a small number of citations over the next years.The resulting TAPS IF values vary slightly, but tend to stabilise in the long run.The high impact factor of the journal in which Paper B was published leads to a high TAPS IF value for 2004.Because the paper is not cited thereafter, the TAPS IF constantly declines.Paper B would be considered less relevant than Paper A from 2009 onwards.Paper C exemplifies the case of an originally unappreciated publication which suddenly attracts attention.The TAPS IF values reflect this pattern over time by grading Paper C as most relevant from 2008 onwards.A paper published in conference proceedings or in a journal without an impact factor is often considered a wasted effort.However, through citations it can gain in value over time as TAPS IF is recomputed each year.With regard to our initial question, such a paper could even, at some point, score highly in a selected ranking.As citation habits and frequencies are much too different between scientific subject areas, TAPS IF values, like journal impact factors, should not be compared between them.In other words, the TAPS IF is ideally suited for ranking scientific papers of a single department (or, more generally, a single scientific field) and not for ranking those of a whole university (or, more generally, across various scientific fields).
Hypothetical Papers A, B and C were published in journals with assumed impact factors in 2004 of 2.5, 10 and 0, respectively.