Bibliometrics, a term coined by Pritchard in 1969, is a measure used to understand the output and impact of scientific communication. Publications and Citations are the two important variables normally used in bibliometrics. Bibliometrics has arguably provided ways and means of benchmarking and evaluation of scholarly work. In recent years bibliometrics has been a growing field of interest in Library and Information Science, yielding to various resultant factors in rankings and decision making process in library management and information services to the users. For example the citation analysis is also used as one of the methods employed in user studies.

Bibliometrics – as a method and as a discipline – has received a greater deal of significance since its germination or genesis. One of the important aspects of the increasing interest in bibliometrics, in the libraries as well as in academia in general is the increasing use of bibliometric indicators to evaluate research performance of faculty and researchers, “especially in the university and government laboratories, and also by policymakers, research directors and administrators, information specialists and librarians and researchers by themselves” (Pendlebury, 2009).The objectivity with which the assessments can be made and repeatability of the analyses are basic reasons for its popularity. Other Source: European Physical Society, EDP reasons, for accepting bibliometrics as a measurement

Sciences, 2009

tool, is its being relatively inexpensive in terms of time, money and effort in its study, and provided a good data source is made available. Scalability is one of the main advantages of bibliometrics as a tool. In other words, it can be applied from a micro level, i.e., an individual researcher or an institute, to a macro level, i.e., country or global level. Ability for comparative analyses – temporal, geographic, linguistic, biographic, etc. – in bibliometrics has drawn the attention of many scholars. It has been universally accepted as an ideal method for assessing the research productivity.

The use in library administration and management is one of the early applications of bibliometrics. Use of bibliometrics in collection development and management in libraries is a well known practice, not the least in relation to digital library management. As a student of Library and Information Science (LIS), you would be interested to know that bibliometrics is a well-established part of LIS research. There has been an increase in the number of research activities by the LIS profession in recent years (Naseer and Mahmood, 2009).

Not everything is green for bibliometrics. It has received some criticisms as well. While bibliometric data bring useful information, the implementation often seems to arise from a loss of critical and rational mind and application. In this Unit an attempt is made to identify and to understand the limitations of bibliometrics studies.

2.0 Limitations of Bibliometric Analysis

Bibliometrics undoubtedly has its distinct strengths. However, it is also subjected to several valid criticisms. Using bibliometrics to measure the quality in scholarly work has attracted attention of the critiques. As Laloë and Mosser (2009) put it, the use of indices such as the H-index at the level of individuals is easy and therefore attractive, but mostly unscientific. Publication counts, one of the widely used indicators, is criticised for this measure of only the quantity and not the quality. Publication counts have also been subject to other criticisms, such as problems associated with gratuitous co-authoring of articles; different publication practices across fields; and difficulties of defining fields of research especially given strong trends towards collaborative research (Lundberg, 2006). These kinds of criticisms can be seen for other measures also.

3.0 Limitations of Citation Analysis

Citation analysis, one of the most applied bibliometrics techniques, is also not far from receiving the criticisms. Let us try to understand the limitations of citation analysis in more detail in the subsequent section.

There exists a relationship between cited and citing documents; and a citation represents the relationship between them. This is the fundamental principle on which citation analysis is based upon. The citation analysis is the area of bibliometrics which deals with the study of this relationship between cited and citing references. Undoubtedly, citation analysis is the most used and useful technique of all. As a concept, it has been responsible for the development of most of the citation databases such Thomson Reuters’s Citation Index, Elsiever’s Scopus, Google’s Google Scholar and so on. Citation analysis is the basis for the development of ‘Citation Indexing’ by Eugene Garfield. Citation indexing is a kind of indexing in which the bibliographic references of documents are made searchable and refer to the documents being indexed. The cited references in a document are made to be a part of the subject access points available for information retrieval.

Why do researchers cite? What is the citers’ motivation for referencing earlier work? These questions have been comprehensively answered by Garfield (1979), the person who originated the citation analysis. He gives a list of 15 reasons for authors to cite earlier works. On examination of these reasons, one may find some of them really show the scholarly impact; and some of them indicate that they have less-than-noble intention behind them. It is the second category of reasons which attract the attention of the critics; and hampers the very purpose of citation analysis.

Citation analysis is based on a few assumptions: 1) Citation of a document implies use of that document by the citing author; 2) Citation of document (author, journal, etc.) reflects the merit (quality, significance, impact) of that document (author, journal, etc.); 3) Citations are made to best possible works; 4) A cited document is semantically related in content to the citing document (if two documents are bibliographically coupled, they are related in content; and if two documents are co-cited, they are related in content); and 5) All citations are equal. All these assumptions have inherent flaws in them which are succinctly described below:

Criticism about the first assumption: The first assumption implies that the citing author has either partly or fully used/influenced by the ideas in the cited work; and secondly that all the cited documents were indeed used by the citing author. Failure to meet any one of these conditions amounts to sins of omission and commission. These ‘sins’ will have a negative impact on the fundamental principle on which citation analysis is based upon.
Criticism about the second assumption: The citations received by a document show its quality is the meaning of this assumption. On the face value, one may say that there is a positive correlation between the number of citations received and the quality of the article. But it is not always true. Sometimes, citations are received for wrong reasons also. It has been shown by many studies that the authors have the habit of giving citations to the spurious works also. This also shakes the very fundamentals of citation analysis.
Criticism about the third assumption: Studies often show that not all good works are cited. Number of Citations, among other reasons, also depend upon the availability (accessibility) of the documents to the authors. Accessibility of a document may be a function of its form, place of origin, age, and language. Hence, non-availability of documents may have a negative impact on the citation analysis.
Criticism about the fourth assumption: This assumption has a bearing on the information retrieval function of the citation databases. Martyn (1964) contends that a bibliographic coupling is not a valid unit of measurement because one does not know that two documents citing a third are citing the identical unit of information in it. The same applies to co-citation as well; the fact that two papers are co-cited does not guarantee a common relation- ship between their contents.
Criticism about the fifth assumption: It is not difficult to see that all the papers cited in an article are not of equal importance in the context of the citing work. However, this varied importance of the cited articles are neither reflected in the reference list nor used for citation analysis. This might have a negative impact on the overall results.

Apart from the above, as Smith (1981) lists, the other parameters which may induce errors in the citation analysis are: multiple authorship, self-citations, homographs, synonyms, types of sources, implicit citations, fluctuations with time, field variations, and errors. The limitations of citation analysis do not negate its value as a research method when used with care. There are, in fact, several application areas where citation analysis has been used successfully.

4.0 Characteristics and Limitations of Bibliometric Indicators

It is not uncommon to use bibliometrics as the tool for research assessment as it is considered as an objective, quantitative and un-obstructive method. These bibliometric indicators have become important for individual researchers and organizations. For researchers they are important because they provide objective mode of assessment of diffusion and impact of articles on their works. For organizations, the bibliometric indicators are significant because they are the means of assessing the quality of particular work, person, or group. However, as said in the last segment of the previous section, the usability of the indicators has to be properly assessed before applying them for quantification. When applied judiciously with care and caution, the bibliometric indicators provide valid results useful for decision making.

The indicators are normally used as unit of analysis in various bibliometric studies. The indicators can be categorised, for the sake of explanation here, into three groups, viz., Publication Indicators; Citation Indicators; and Journal Indicators. The following section provides a brief explanations of indicators used in bibliometrics under each of these categories along with their advantages and limitations, if any. The list is compiled from the list given by Rehn (2007).

4.1 Publication Indicators

Number of Publications: It is the number of publications published either by an author, institute, country or so on. The time span is also taken in many situations to suit the temporal scope. The data is collected either directly from the original publications from databases. It is relatively easy to collect data. Although this count is a very straightforward indicator that can be easily calculated by the authors themselves, one must be very careful when using it to compare authors or research groups. The disadvantages of this indicator are: when used does not take the size of the analysed unit; and does not speak of the impact of the publications counted.
Number of ISI Publications: It is the number of publications indexed by Thomson ISI indices. Temporal and geographic filters are applied many a times in many studies. Quite easy to collect data as it can be directly collected from the databases. The disadvantages are: when used does not take the size of the analysed unit; has inherent problem of scope and coverage as that of ISI indices; and does not count non-ISI publications.
Number of Publications in Top Journals: It is the number of publications the analyzed unit has published in a selected number of journals during the analyzed time span. The selection of journals is usually made on some criteria. The advantage is that as the data is collected from top journals (which show their relative importance among others in the group); it is a better count than a mere publication count. The disadvantages are: does not take the size of the analyzed unit into account; and has the limitations of the selection criteria. Although this approach may look like a performance indicator, it was designed to address the shortcoming of the above-mentioned quantity indicator.

4.2 Citation Indicators

• Number of citations: It is the total number of references received from other works,

i.e., number of citations to articles published by an analyzed unit during the analyzed

time span. The citation of one article by another is characteristic of scientific publications, and it is generally accepted that the number of citations of a particular article receives is a reflection of its impact in the scientific community (Rhen, 2006). The data has to be collected from the citation databases such as Thomson Reuters Web of Knowledge, Scopus, Google Scholar, CiteseerX and so on. As collected from databases, data and results are verifiable. Limitations of this indicator include: it does not take into account older articles as usually are more cited and that citation rates vary between document types and subject areas; and does not compensate for size of the unit.

Citations per publication: It is the average number of citations to articles published by an analyzed unit during the analyzed time span. It is calculated by the ratio of total publications and dividing by the total number of publications considered. The limitations are: Does not take into account that older articles usually are more cited if a variable, cumulative citation time window is used, and that citation rates vary between document types and subject areas.
Field Normalized Score: This indicator corresponds to the relative number of citations to publications from a specific unit, compared to the world average of citations to publications of the same document type, age and subject area. It is calculated as follows: The number of citations to each of the unit’s publications is normalized by dividing it with the world average of citations to publications of the same document type, publication year and subject area, which is called the field reference value (µf). If an article is classified as belonging to several subject areas, a mean value of the areas is used. The limitations include if the normalization is done on an article level, a few highly cited articles in a moderately cited research area may contribute un-proportionately to the value of the field normalized citation score.
Total field normalized citation score: This indicator gives an indication of both the impact and the production volume of the analyzed unit. The score is got by adding together the item oriented field normalized citation scores for all the publications of the analyzed unit. The disadvantage is that it does not compensate for the size of the analyzed unit.
Journal normalized citation score: This indicator corresponds to the number of citations to publications from a specific unit during an analyzed time span, compared to the world average of citations to publications of the same document types, ages and in the same journals. The calculation is as follows: the number of citations to each of the unit’s publications is normalized by dividing it with the world average of citations to publications of the same document type, published in the same year in the same journal. The indicator is the mean value of all the normalized citation counts for the unit’s publications.
Crown indicator: It is developed by Center for Science and Technologies Studies at Leiden University. It intends to measure the scientific impact of a researcher or a research group. This indicator is calculated by dividing the average number of received citations (from a researcher or a research group) by the average number that could be expected for publications of the same type, during the same year, and published in journals within the same field (Lundberg, 2007). It has a few flaws also. First, its dependence on categories published by Thomoson Reuters leads to a problem that it does not take into account that publications from a particular field are often published in journals categorized in another field. Second, the size of a research group influences its productivity — quite simply, the more researchers in a group, the larger the number of published articles. It is therefore recommended to compare research groups with the crown indicator only if the groups are of similar sizes (Lundberg, 2007).
h-index: The h-index is an index that attempts to measure both the scientific productivity and the apparent scientific impact of a scientist. J.E. Hirsch introduced it

in 2005 and defined it in the following way: “A scientist has index h if h of [his/her] Np papers have at least h citations each, and the other (Np -h)papers have at most h citations each”. The ‘Web of Science’ now gives direct access to the H index in a few mouse clicks. It is calculated as follows: find the unit’s published articles in a citation index and sort them in descending order by number of citations. Count articles from the top of the list and downwards, and when the number of an article rises above the citation count for that very article, the number of the preceding article is to be counted as the h-index. H-index is criticized as it gives positive bias to senior researchers with older articles, since these have had more time to be cited, though the demand that new articles with comparable citation levels has to be added has a certain damping effect on that bias.

Uncitedness: It is the share of a unit’s publications that that remain uncited after a certain time period. Self-citations should be removed from the citation count. It requires data from a comprehensive citation database such as the Thomson citation indices and validation of the unit’s publications.
Self-citation: It is the share of a unit’s received citations where authors refer to their own papers. The calculation of the self-citation is as follows: count the total number of citations to the unit’s publications during the analyzed time span. Check where citations are coming from and count the number coming from the unit itself. Divide the second number with the first to get share of self citedness. The requirement for getting self-citation is that it requires data from a comprehensive citation database such as the Thomson citation indices, validation of publications and analysis of citing articles, which can be done in the ISI Web of Science.

4.3 Journal Indicators

Impact factor: The impact factor (IF) of an academic journal is a measure reflecting the average number of citations to recent articles published in the journal and intended to gauge the importance of a journal in its given field. It is perceived that higher impact factor of a journal more important it is in that field than those with lower ones. The impact factor was devised by Eugene Garfield, the founder of the Institute for Scientific Information. Impact factor is calculated yearly for those journals that are indexed in the Journal Citation Reports. IFs are available in the SCI (Science Citation Index) Journal Citation Reports and on the Web of Knowledge for more than 8000 selected scientific journals. The IF does have several limitations (Durieux & Gevenois, 2010). First, although a higher IF can suggest a greater impact of a journal, it does not reflect the quality of each particular article published by that journal. Consequently, it is not clear whether a high IF is due to a moderate degree of citation of all of the articles published or to a high degree of citation of only some articles. Second, multidisciplinary journals usually have a higher IF than specialized journals. Third, there are differences between research fields, including in research intensity. The highest ranking journal in each specialized field may have a very different IF from specialty to specialty. Fourth, the types of articles published by a journal also influence its IF. Review articles and technical reports are more frequently quoted than are original research articles, case reports, and pictorial essays.
Normalized Journal Impact Factor: This indicator corresponds to the relative number of citations to publications in one specific journal, compared to the world average of citations to publications of the same document type, age and subject area. The indicator is stated as a decimal number that shows the relation of the number of citations to the world average. As an example, 0.9 means that publications in this journal are cited 10% below average and 1.2 that they are cited 20% above average.
Immediacy Index: It is an indicator which measures the current importance of the work published by a journal by calculating the average number of times articles published during a particular year by a specific journal is cited over the course of that same year. The immediacy index is useful for identifying the journals publishing the articles in the emerging areas. It is said that immediacy index has an unintended bias towards articles published in the earlier part of the year as they would have better and more chance to get cited than those articles published later in the year.
Journal-to-field impact score: The journal-to-field impact score has been proposed by the Center for Science and Technologies Studies of Leiden University (Leiden, the Netherlands) as an alternative to the IF. It measures the average number of cited articles in a specific journal and compares this number with that of other journals in the same research field category. The field categorization of journals is based on the journal subject categories, which are defined by Thomson Reuters. By ranking journals in a given subject category, this score overcomes the limitations of IF related to research field characteristics such as productivity, citation habits, and citation dynamics

5.0 Conclusion

Bibliometric and Scientometric analysis are based on many indicators. These indicators seek to measure the quantity and impact of research publications. The indicators are used increasingly in evaluation processes in universities, and public and private research institutes. Each of these indicators has its own merits and limitations. Interpreted as exhaustive measures of scientific/research output, the indicators would present a biased story. However, when used with caution, they can reveal some insights through trends regarding aspects of scientific/research production at global level.

As a conclusion, let us bear in mind the following words-of-caution given by Durieux and Gevenois (2010) while using the bibliometric and scientometric indicators for analysis:

Performance indicators are based on the assumption that the quality of a particular article is reflected by the frequency of its citations in other articles.
Given differences between fields of research in terms of productivity, citation habits, and citation dynamics, bibliometric indicators should not be used for comparing researchers, research groups, or journals from different fields.
It is recommended to measure the quality and the impact of scientific journals, research groups, or particular researchers through several indicators rather than only one.

6.0 Summary

Bibliometrics is a set of mathematical and statistical tool used to measure quantity and quality of books, articles, and other genera of publications. Studies indicate that bibliometrics, as a metric, is objective, quantitative and un-obstructive method used for. However, it is not without any limitations. The limitations in citation analysis is discussed in the Unit Critics have also shown the flaws in most of the indicators used in bibliometric studies. Each of these indicators along with their advantages and limitations has been explained. The bibliometric analysis and the use of indicators would serve the purpose when used with care. This unit provides an overview of the currently used bibliometric indicators and summarizes the limitations and characteristics one should be aware of when evaluating the quantity and quality of scientific output.

you can view video on Challenges of Bibliometric and Scientometric Studies

References

Durieux, V., & Gevenois, P. A. (2010). Bibliometric Indicators: Quality Measurements of Scientific Publication 1. Radiology, 255(2), 342-351.
Garfield, E. (1979). Citation indexing: Its theory and application in science, technology, and humanities (Vol. 8). New York: Wiley.
Laloë, F., & Mosseri, R. (2009). Bibliometric evaluation of individual researchers: not even right… not even wrong!. Europhysics News, 40(5), 26-29.
Lundberg, J. (2006). Bibliometrics as a research assessment tool: impact beyond the impact factor. Doctoral Thesis. Sweden:Korolinska Instituet.
Lundberg, J. (2007). Lifting the crown—citation z-score. Journal of informetrics, 1(2), 145-154.
Martyn, J. (1964). Bibliographic coupling. Journal of documentation, 20(4), 236-236.
Naseer, M. M., & Mahmood, K. (2009). Use of bibliometrics in LIS research. LIBRES: Library and Information Science Research Electronic Journal, 19(2).
Pendlebury, D.A. (2009). Whitepaper: Using bibliometrics: A guide to evaluating research performance with citation data. Philadelphia: Thomson Reuters.
Rehn, C., & Kronman, U. (2008). Bibliometric handbook for Karolinska Institutet. Huddinge: Karolinska Institutet.
Rehn, U. K. C. (2007). Bibliometric indicators–definitions and usage at Karolinska Institutet.
Smith, L. C. (1981). Citation analysis. Library trends, 30(1), 83-106.