Impact factors and their significance; overrated or misused?

KEY POINTS * Impact factor (IF) is a useful tool to evaluate a scientific publication. * The impact of a publication on the community can bear little relationship to the impact factor. *

This paper describes how IF is calculated and some of its uses, with misuses highlighted. * The IFs of the dental journals are presented. ABSTRACT The journal _impact factor_ (IF) is in

widespread use for the evaluation of research and researchers, and considerable controversy surrounds it. The concept behind the IF is _citations_, and the number of them. The IF is a useful

tool for the evaluation of journals, but it must be used carefully. Considerations include the number of review papers, letters or other types of material published in a journal, variations

between disciplines, and item-by-item impact. Perhaps the most important use of the IF is in the process of academic evaluation. The extent to which the IF is appropriate for the evaluation

of the _quality_ of a specific article or journal and particularly for the evaluation of individual and collective research achievements is highly debatable. INTRODUCTION It has always been

difficult to judge the importance of scientific papers or journals, and the impact of these on users and customers is a very fickle concept. Impact factor (IF), a term familiar to most

people in the dental, medical, and scientific academic communities, appeared about 30 or more years ago (http://www.isinet.com/essays/journalcitationreports/7html/),1,2,3 and has come to be

regarded by some as of increasing importance, yet few fully understand its meaning or implications. Perhaps the most important use of the IF is as a useful tool for the evaluation of

journals but, since it is also employed in some situations in the process of academic evaluation, considerable controversy surrounds it. The 'impact' of a publication on the

community can bear little relationship to the 'impact factor'. DEFINITIONS The journal _impact factor_ (IF) is published annually by the ISI (a private, profit-oriented commercial

Philadelphia-based organisation which was formerly termed the Institute for Scientific Information; http://www.isinet.com). Access to the ISI databank is not free of charge. The IF is a

measure applied to a journal and is a measure of the frequency with which the 'average article' has been cited in a particular year or period. The key idea behind the IF is

citations, and the number of them. Citations are the papers and book chapters listed in the references at the end of a scientific paper. Journal citation patterns (ie which author was being

cited and where) have been studied since the 1920s and, since the 1960s, the Science Citation Index (SCI) and the Journal Citation Reports (JCR) have been produced based upon

computer-compiled statistical reports. The citation index makes it feasible to produce computer-compiled statistical reports not only on the output of journals but also in terms of their

citation frequency. The JCR provides quantitative tools for ranking, evaluating, categorising, and comparing journals. The JCR uses the IF, along with other criteria, to compare, evaluate

and rank journals; it can also indicate the largest journals, the 'hottest' journals, what publications a journal cites and which publications cite the journal itself. The IF is

calculated by dividing the number of current citations to articles published in a specific journal in the previous two-year period by the total number of articles published in the same

journal in the corresponding two-year period (Table 1). The annual JCR IF is a ratio between citations and recent citable items published. The impact factors are published every September.

So, in September 2003, the IF for a journal for 2002 would have been calculated from the number of citations in the year 2002 to articles published in the journal in 2000 and 2001, divided

by the number of articles published in the journal in 2000 and 2001. The IF is thus calculated from the number of times the articles _are_ cited divided by the number of articles in that

journal that _could_ be cited. It should be noted that, when the IF is calculated, the numerator is the total number of citations _to virtually any item_ in the journal, all types of

articles, such as editorials, letters and abstracts from meetings are included in the numerator — the cited items. However, the denominator is the number of _articles_ only and includes only

normal articles, notes and reviews — the citable items. The IF can be affected, for example, by a large correspondence section or by publishing controversial editorials. If a journal

publishes more additional items, they become included in the numerator and thus the impact factor may well increase significantly. In contrast, an increase in citable items can have the

opposite effect. For example when, in 1997, the _Lancet_ divided its 'Letters' section into 'Correspondence' and 'Research Letters' — the latter being

peer-reviewed and hence 'citable' for the denominator, the increase in the denominator led to a fall in IF from about 17 to about 12. Only journals covered by the SCI database are

scanned for their reference lists, only a very small proportion (about 5,000 science and technology titles) of the world total of over 100,000 English language journals and chiefly those

from the USA are favoured, and tend to enjoy higher IF than non-English language journals. Books and book chapters are not scanned for their bibliographies, or included in any IF

calculation. WHERE CAN IMPACT FACTORS BE FOUND? Journals often advertise their IF on their websites and publicity fliers (eg http://www.elsevier.com/locate/oraloncology). Otherwise, a

university library or a library of one of the larger institutions can access them either on CD or on the Internet (http://www.isinet.com/isi or http://wos.mimas.ac.uk). IMPACT FACTORS OF

DENTALLY-RELATED JOURNALS The impact factors for 2002 for the first 20 of the 49 dental journals evaluated in JCR are shown in Table 2. The IFs ranged from 1.047 to 2.956, and no other

journal in this field had an IF ≤1 Interestingly, two of the three journals with the highest IFs are review journals ie they contain essentially no original research. USES OF IMPACT FACTOR

The IF can be helpful to evaluate a journal's relative 'importance', especially when compared to others in the same field but, as noted above, this can be comparable to

comparing apples with pears. The IF is useful in clarifying the significance of absolute (or total) citation frequencies. It eliminates some of the bias of such counts which favour larger

journals over small ones, of older journals over newer ones or of frequently issued journals over those less frequently issued. However, larger journals have a larger citable body of

literature than smaller or younger journals and, generally speaking, the larger the number of previously published articles, the more often a journal will be cited and the higher the IF will

be. Publishers often use IF for marketing (many fliers give the journal IF) or in identifying opportunities for new journal launches or taking decisions on whether to expand, merge or

discontinue existing titles. Authors sometimes use IF to decide where to publish and to discover other journals in their specialty. Other uses of IF are discussed below. CARE IN THE USE OF

IMPACT FACTORS Informed and careful use of IF data is essential. Ill-formed conclusions based on IF statistics may arise unless several caveats are considered. Criticisms levelled against IF

have been well documented4 and the main ones are summarised here. _The scientific field to which the journal belongs influences IF_. The scientific field to which the journal belongs

influences the IF greatly; ISI recognises this and warns against making comparisons between fields. For example, the highest impact factor in the ISI subject category 'Dentistry'

is 2.956, whereas that in 'Oncology' is _CA Cancer J Clin_, with a massive IF of 32.886. But does that really mean that Oncology is 15 times as good or important as Dentistry?

Different disciplines have widely differing citation practices, and general journals are at a particular advantage over more specialist journals. _Scientific journals generally rank higher

than clinical journals_. Scientific journals generally rank higher than clinical journals in the IF league, partly due to the fact that scientific papers tend to cite only scientific and not

clinical articles, whereas clinical papers tend to cite both scientific and clinical articles. _Self-citation is also possible, increasing the IF_. Self-citation is also possible, so

authors happily cite previous papers in the same journal and editors may cite editorials. Some journals have been known to try to manipulate the IF by writing to authors asking them to add

references to articles published in that journal. _Errors, misprints and inconsistencies in citations can distort the IF_. Errors, misprints and inconsistencies in citations can distort the

IF, and cause damage. Seglen4 suggests that misprints in lists of references may affect up to one-quarter of references. Errors in the published IF have caused considerable damage to at

least two journals in the dental field. _IFs are biased toward journals that mainly publish review articles_. IFs are biased toward journals that are review journals or mainly publish review

articles, since those tend to be cited more frequently — often in authors' introductions.5 Amongst the dental journals with the highest IFs (Table 2) are the review journals _Critical

Reviews in Oral Biology and Medicine_, and _Periodontology 2000_. _Multi-author and consortia articles sometimes pose a problem_. Multi-author and consortia articles sometimes pose a problem

regarding who — or what — should be cited, and how. For example, the International Human Genome Sequencing Consortium described the sequencing of the human genome in a paper in Nature in

2001 — probably the most important scientific advance ever and of phenomenal impact, but the IF was low.6 _Greater availability tends to raise the IF_. Free electronic access, or the

inclusion of a journal as part of the membership to a society and therefore greater availability, tends to raise the IF of a journal. Accessibility may mean that the source is cited but it

does not necessarily follow that the most appropriate or 'best' reference has been chosen to be cited. For example, in the SCI subject category 'Dentistry', the journal

with the highest impact factor (Table 2) is the _Journal of Dental Research_. Apart from the high quality, the fact that _J Dent Res_ is taken by most dental researchers, has surely helped

it become the most highly cited journal. Controversial or poor papers may increase the IF.7 It is worth remembering that a paper may be cited as an example of poor research or may be highly

cited if it covers a controversial topic eg suggesting that HIV is not the cause of AIDS. _Citation counts in JCR do not distinguish between letters, reviews, or original research_. So, if a

journal publishes a large number of letters, there will usually be a temporary increase in citations of those letters. Letters to the _Lancet_ may indeed be cited more often that letters to

_JAMA_ or vice versa, but the overall citation count and IF recorded would not take into account this artefact. _A change in journal title may adversely affect the IF_. In the first year

after a change, the IF is not available for the new title unless the data for old and new can be unified, and in the second year, the IF is split. The new title may rank lower than expected

and the old title may rank higher than expected because only one year of source data are included in the calculation. For example, the journal _Oral Oncology_, once a daughter journal of the

_European Journal of Cancer_, was undercited for several subsequent years, since many authors and journals were mistakenly still quoting it as _Eur J Cancer_ rather than as _Oral Oncology_.

Thus the IF for _Eur J Cancer_ was artificially inflated and that for _Oral Oncology_ was falsely low. CONTENTIOUS USES OF IMPACT FACTORS Citation analysis, in the hands of non-experts, can

be an extremely blunt instrument.8 Probably the most serious criticism of IFs relates to their erroneous and potentially dangerous use to determine 'author impact'. In some

countries and institutions, academic administrators significantly (and controversially) use IF as a convenient tool in the process of deciding on promotion and tenure, ie the IFs of journals

in which candidates have published are used to help determine the impact or importance of their research. IFs can represent an all too convenient shortcut to bypass proper appraisal, and

the work involved in obtaining the more meaningful information of citation counts for individual articles and authors. By extension, and perhaps erroneously, IFs may be taken as an

indication of a person's scientific worth. By further extension, there is a possibility of IFs being used to compare institutions. However, it cannot be over-emphasised that the IF was

created with the intent of comparing _journals_, not authors or individual articles. The IF can be used to provide a gross approximation of the prestige of journals in which individuals have

published, but this is best done in conjunction with other considerations such as peer review, productivity, and subject specialty citation rates. Nevertheless, despite these problems, many

bodies responsible for hiring, promotion, tenure, or the assessment of research groups and grant proposals for funding and resource allocation continue to evaluate the quality of an

individual's research and publications by looking at the IF of the journals in which they have published. The widespread belief that the IF is representative of an individual author (or

article) has been countered by the finding that citations of individual articles in a journal show a very skewed distribution.4 Authors consequently feel significant pressure to submit

papers to a journal with a high IF, whether or not that journal is the most appropriate platform for their work. Many believe that the higher the journal IF that published their paper, the

more their paper will be cited, but this is a myth.4 CONCLUSIONS The impact factor is a useful tool for evaluation of journals, but it must be used very carefully. Considerations include the

amount of review articles, letters or other types of material published in the journal, variations between disciplines, and item-by-item impact. The IF is of questionable value in certain

circumstances and the extent to which the IF of a journal is appropriate for the evaluation of the _quality_ of a specific article or journal and particularly of individual and collective

research achievements is undoubtedly highly questionable.9,10,11,12,13,14 Certainly, the actual impact on the community of an article is not necessarily related to the IF. REFERENCES *

Garfield E . Citation analysis as a tool in journal evaluation. _Science_ 1972; 178: 471– 479. Article  Google Scholar  * Garfield E . 'Citations to' divided by 'items

published' gives journal impact factor. Essays of an Information Scientist. _Current Contents_ 1972; 1: 270– 273. Google Scholar  * Garfield E . Which medical journals have the greatest

impact? _Ann Intern Med_ 1986; 105: 313– 320. Article  Google Scholar  * Seglen P O . Why the impact factor of journals should not be used for evaluating research. _BMJ_ 1997; 314: 498–

502. Article  Google Scholar  * Krauze T K, Hillinger C . Citations, references and the growth of scientific literature: a model of dynamic interaction. _J Am Soc Info Sci._ 1971; 22: 333–

336. Article  Google Scholar  * Human genomes: public and private. _Nature_ 2001; 409: 745 (editorial). * Moed H F, Van Leeuwen T N V . Improving the accuracy of Institute for Scientific

Information's journal impact factors. _J Am Soc Info Sci_ 1995; 46: 461– 467. Article  Google Scholar  * Adam D . The counting house. _Nature_ 2002; 415: 726– 729. Article  Google

Scholar  * Saper C B . What's in a citation impact factor? A journal by any other measure. _J Comp Neurol_ 1999; 411: 1– 2. Article  Google Scholar  * Bloch S, Walter C . The Impact

Factor: time for change. _Aust N Z J Psychiatry_ 2001; 35: 563– 568. Article  Google Scholar  * Chew F S, Relyea-Chew A . How research becomes knowledge in radiology: an analysis of

citations to published papers. _Am J Roentgenol_ 1988; 150: 31– 37. Article  Google Scholar  * Garfield E . Citation indexing for studying science. _Nature_ 1970; 227: 669– 671. Article 

Google Scholar  * Kaltenborn K F, Kuhn K . [The journal impact factor as a parameter for the evaluation of researchers and research] _Med Klin_ (Munich) 2003; 98: 153– 169. Article  Google

Scholar  * Talamanca A . The impact factor in the evaluation of research. _Bull Group Int Rech Sci Stomatol Odontol_ 2002; 44: 2– 9. Google Scholar  Download references AUTHOR INFORMATION

AUTHORS AND AFFILIATIONS * Eastman Dental Institute for Oral Health Care Sciences, 256 Gray's Inn Road C Scully * UCL, University of London, WC1X 8LD C Scully * Eastman Dental Institute

for Oral Health Care Sciences, 256 Gray's Inn Road H Lodge * UCL, University of London, WC1X 8LD H Lodge Authors * C Scully View author publications You can also search for this author

inPubMed Google Scholar * H Lodge View author publications You can also search for this author inPubMed Google Scholar CORRESPONDING AUTHOR Correspondence to C Scully. ADDITIONAL

INFORMATION REFEREED PAPER RIGHTS AND PERMISSIONS Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE Scully, C., Lodge, H. Impact factors and their significance; overrated or

misused?. _Br Dent J_ 198, 391–393 (2005). https://doi.org/10.1038/sj.bdj.4812185 Download citation * Received: 04 May 2004 * Accepted: 02 June 2004 * Published: 09 April 2005 * Issue Date:

09 April 2005 * DOI: https://doi.org/10.1038/sj.bdj.4812185 SHARE THIS ARTICLE Anyone you share the following link with will be able to read this content: Get shareable link Sorry, a

shareable link is not currently available for this article. Copy to clipboard Provided by the Springer Nature SharedIt content-sharing initiative