20 Performance evaluation

1. INTRODUCTION

In this competitive environment only effective organizations can prosper and survive. Emergence of computer and communication technology and widespread use of Internet has brought revolutionary changes in every sphere. Public libraries are no exception to this change and are continuously transforming and evolving to survive. Libraries have to do so in order to maintain prominent position as centres of knowledge, culture, education and lifelong learning for the community they are required to serve.

In this profound societal and technological change, regular performance evaluation of libraries is must, to preserve the visibility and viabilities of the public libraries. There are stakeholders, who believe that in this paperless society traditional libraries have no role to play. In such cases, evaluation of a library and demonstrating its benefits become crucial for the survival of the library itself. Here, it becomes necessary to go beyond the evaluation of specific library services and attempt to assess the total impact of the library on the lives of its user community. An evaluation of the total library or library as a system demonstrates how effective the library is to its user community.

Secondly, funding any library is an expensive proposition. Funding covers cost of print and electronic collection, building, equipment and ongoing cost of recruiting and training staff. Benefits gained from funding any library are increasingly being questioned by the funding agencies. The questions raised by these agencies are the following:-

Does the investment in libraries represent value for money?
Are there demonstrable tangible benefits that arise from library use?
Does the library meet the needs of its user community?

The primary means of describing the value of the library have occurred in the areas of accomplishment, economic benefits and, for public libraries, social impact of the library on the lives of users.

This module deals with measuring performance of a public library as an organization and covers need and purpose of library evaluation, methods and techniques available for evaluation and how to apply these techniques to evaluate library as a whole, its collection, staff, and services.

2. PERFORMANCE MEASUREMENT OF A LIBRARY

Performance measurement of a library as a whole deals with monitoring and reviewing the jobs accomplished by a library, progress made towards its pre-established goals and the impact of the library on its user community. Evaluation of any service, process or product, usually refers to „determining its worth‟ or to know its utility for the target population. The need for measurement and evaluation is felt in libraries, particularly, in public libraries which are supported by public funds. Evaluation is required to increase the efficiency of a library,measure quality, utility, and timeliness of a product or service, and its productivity in relation to costs incurred during its production.

The role of performance measurement in libraries is to:

Measure library‟s performance in relation to pre-established goals,
Measure performance of the staff in meeting users‟ requirements,
Measure efficiency of various departments and their processes,
Find out adequacy of printed and electronic resources in meeting users‟ needs,
Measure quality and utility of a product or service in relation to cost,
Determine satisfaction level of the target users as well as that of library staff,
Measure library‟s performance in getting and increasing funding resources,
Measure library‟s ability in sustaining qualified and dedicated staff,
Measure efficiency of the library in adapting to technological changes, and
Increase the efficiency of the library.

Performance measurement in libraries helps in detecting and correcting problems in achieving goals and objectives of a library, improving the efficiency of various products and services offered, improving the collection, efficiency of staff and various departments. Thus, the overall objective of the performance measurement is to increase the efficiency of a library.

3. PERFORMANCE MEASUREMENT MODELS

Several useful models are available to measure organizational effectiveness. These models include the Goal

Model, Critical Constituencies Model, the Resource Model, the Human Resource Model, and Natural and Open

System Model, the Decision Process Model and the Customer Service Model. These models and the key

questions they address are summarized in the Table 1.

MODEL	KEY QUESTIONS
Goal Model	Has the established goals of the library been met?
Critical Constituencies Model	Have the needs of constituents been met?
System Resource Model	Have necessary resources been acquired?
Human Resource Model	Is the library able to attract, select, and retain
	quality employees?

Open System Model	Is the library able to maintain the system, adapt to threat, and survive?
Decision Process Model	How are decisions made and evaluated?
Customer Service Model	How satisfied are the customers with the library?

Table 1: Models of Effectiveness

(Source: “Library Evaluation”. Edited by Danny P. Wallace and Connie Van Fleet)

3.1 GOAL MODEL

In goal based model, the function of the organization is to accomplish the goals that have been established. Goal may be established by either library board along with administration, or by library and management staff. In some public libraries, various government bodies may set up the goals. This model is considered a rational model and is very popular model for organizations in private and public sectors. It is commonly used in libraries, particularly, in public libraries.

However, this model has some limitations. As this model specifies on achieving the set goals, it may place extra burden on staff and resources. Moreover, even if a library accomplishes its set goals, it may be considered ineffective by its users. As, this model does not take into account users‟ opinion.

3.2 CRITICAL OR CRUCIAL CONSTITUENCIES MODEL

In critical constituency‟s model, organizational effectiveness is measured based on the extent to which the needs of critical constituencies are met. Constituencies are persons acting in their own interest or working as representative on behalf of others. These constituencies take many forms, including the library board and administration, middle management, library staff, accrediting bodies, funding agencies, regulatory agencies, and the general public, or subgroups within the general public.

Critical constituencies are those that shape the policies and practices of the organization the most. In reality, judgments concerning organizational effectiveness are usually seen from the perspective of the dominant elite groups, who are usually perceived as the critical constituencies (Starbuck and Nystrom, 1983). Some theorists have suggested that the dominant constituency, such as, library board, determines the library‟s effectiveness. Library staff is also Avery important constituency, which by training and tradition promotes and delivers services to external constituencies.

To use critical constituencies model effectively, libraries must carry out following activities:-

Identify critical external and internal constituencies;
Determine the resources and services needs of external constituencies;
Identify the needs of internal constituencies;
Assess the current ability of the library to meet the needs of external constituencies and build up resources and design services accordingly;
Assess the current ability of the library to meet the needs of internal constituencies and make organizational changes to meet them; and
Determine whether changes made for internal and external constituencies are meeting their needs or not.

However, in this model sometimes needs of internal and external constituents may conflict. Such conflicts may generate considerable tension.

3.3 SYSTEM RESOURCE MODEL

In this model, the central mission of an organization is to acquire needed resources, both, printed as well as electronic resources. Effectiveness of a public institution, such as, a public library directly depends on its resources. However, for libraries, the source of funding is not from individuals who use the resources, but the bodies that provide the funds. Hence, measuring organizational effectiveness using the resource model means that substantial attention is to be paid to the relationship of the library to those bodies that provide funding.

Assessing organizational effectiveness under the resource model means to:-

Measure the effectiveness of the process of obtaining and increasing fiscal resources;
Measure the degree to which influential funding sources are positively disposed to the library;
Measure the degree to which the library is able to monitor threats to fiscal resources or degree to which opportunities for new resources are exploited; and
Measure the degree to which a productive relationship is maintained with those supplying material resources such as books, audio visual materials, computers, etc.

3.4 HUMAN RESOURCE MODEL

Within the context of the human resource model, organizations are considered effective “when they attract and select diverse kinds of people who are able and willing to comprehend what an organization‟s goals should be and behave in ways that push the organization towards the future (Schneider 1983)”. In this model, effectiveness is measured by the success of the organization in effectively carrying out human resource practices.

The effectiveness measures should include the following:-

Measure the employee turnover rate and reasons for leaving,
Measure level of job satisfaction and worker commitment among employees,
Measure the success of the hiring system to attract productive workers, and
Measure the employees‟ degree of belief that the reward system is fair and that there are positive effects to display of personal motivation and ability.

3.5 NATURAL OR OPEN SYSTEM MODEL

This model perceives the organization as similar to a natural system, i.e., a living organism. This model views that the living organism are constantly interacting with the environment, making adaptations, and responding to threats in the environment. The purpose of living organism is to survive, so it is with the organization. Many things may threaten public libraries like funding deprivation, failures on the part of material and service suppliers, low use by the public and legal regulations. The effectiveness of the organization is measured by its ability to maintain the system, adapt to threats, and survive.

Assessing effectiveness using this model requires measuring how library adapts to threats in its environment. This can be accomplished by:-

Measuring the library‟s ability to fill positions quickly with qualified personnel;
Measuring the library‟s ability to create a positive public image among its users;
Measuring the organization‟s ability to react productively to crisis and promote programmes and services successfully;
Measuring the administration‟s success in creating among funding bodies a positive attitude toward the library; and
Measuring library‟s effectiveness in dealing with suppliers.

3.6 DECISION PROCESS MODEL

In this model, organizations develop their own methods for collecting, organizing, disseminating, processing and using information. Measuring organizational effectiveness in this context focuses on how communications occur and decisions are made. Decision quality becomes the crucial factor here. Among the possible sources of evaluation are:-

Measuring levels of participation or levels of group involvement in making decisions, and
Measuring the extent to which decision processes and rules produce good results.

At Westlake Porter Public Library, a study was conducted by Paula J. Miller implementing total quality management and focusing on five years activity involving all staff members, from 1993 to 1998. The purpose of the study was to change the management and organizational climate at the library and making a shift from traditional organizational culture to a quality organization that focuses on customers by continuously improving processes.

Implementing total quality management efforts resulted in marked internal as well as external impact. Internally major changes and innovations were renaming and changing the logo of the organization to identify with the community in a better way; eliminating annual performance appraisal in favor of a continuous feedback process; initiating training programmes for the new employees as well as development and competencies programmes for long-term employees, and initiating reward system on fulfilling specific training and activity requirement. External impact was also very good. There was significant increase in library usage level and community support.

3.7 CUSTOMER SERVICE MODEL

This model focuses on the extent to which customers‟ needs are satisfied. Evaluating customers‟ attitude is a complex process. Some suggest that the perception of service quality is based on the following five factors:-

Tangibles: Appearance of physical facilities, equipment, personnel, and communication materials;
Reliability: Ability to perform the promised service dependably and accurately;
Responsiveness: willingness to help customers and provide prompt service;
Assurance: knowledge and courtesy of employees and their ability to inspire trust and confidence;
Empathy: caring, individualized attention provided to the customers.

Based on these factors, measurement of customers‟ satisfaction can be accomplished in many ways. Basic perceptions of library service that may be examined should include) customer‟s view of attractiveness of physical facilities and equipment; ii) helpfulness of staff and informational material; and iii) customer‟s views on the dependability and quality of library services.

One of the important implications of all the above mentioned models is that the choice of any one model will capture only a portion of the organization described by the model. Other segments of the organization will not be included in the measurement and thus it will not reflect the performance of total organization. By approaching organizational effectiveness from many perspectives would provide more accurate picture of the true performance of an institution.

Goal Model and System Resource Model are most commonly used models by the librarians to demonstrate the value or effectiveness of a library. This is because librarians have an internal focus, seeing themselves as doing good, and are less concerned about assessing the outputs and impacts. On the other hand, the library‟s funding agencies have an external view and rate the library efficient and effective only when it meets the needs of the community of users. As public library is a social institution, librarians need to establish the effectiveness of a library from the many perspectives.

Thomas Childers and Nancy Van House (1989-1990) conducted library effectiveness study in a public library, with the goal of identifying the perception of seven groups viz. users, friends, trustees, local officials, community leaders, library managers, and library service staff, about what performance measures indicate the effectiveness of a library. An analysis of responses showed that following six items were in the top ten preferences for all the groups:

Convenience of hours
Range of materials
Range of services
Staff helpfulness
Services suited to community
Materials quality

People in most of the groups surveyed recognized that perception of effectiveness of a library are multidimensional and numbers of measures will be needed while measuring library effectiveness.

4. INTERNATIONAL STANDARD FOR PERFORMANCE MEASUREMENT

International Standard ISO11620:2008-Information & Documentation- Library Performance Indicators.

This International standard specifies the requirements of a set of performance indicators for measuring performance of libraries and establishes a set of indicators to be used by libraries of all types. The standard was initially developed for academic libraries. The current version covers public libraries as well. The standard specifies four broad areas for measuring the effectiveness of a library. These areas are: library resources and infrastructure; acceptance of library services; efficiency of library services; and potentials of library for future development. The standard provides a list of 40 indicators grouped in four areas as follows.

A. Resources, Infrastructure (10 indicators)

B. Use (12 indicators)

C. Efficiency (13 indicators)

D. Potentials and Development (5 indicators).

A. Resources, Infrastructure

This aspect is represented by 10 indicators. The library‟s effectiveness as a place for learning and research is defined by the size of user area, the availability of user workplaces and by opening hours of the library (3 indicators). The quality of collection is measured by expenditure on information provision per capita, availability of required titles, percentage of rejected sessions, ratio of requests received to the requests sent out in interlibrary lending and immediate availability of requested media (5 indicators). Staff resources and website quality are represented by one indicator each (2 indicators).

B. Use: How are the services accepted?

This aspect deals with how the services are accepted. This perspective is represented by 12 indicators. Market penetration, user satisfaction and number of library visits per capita are used as general indicators (3 indicators). Seat occupancy is the indicator for library as a place (1indicator). The attractiveness of physical collection is evaluated by using loan data (4 indicators, viz. collection use, percentage of stock not used, loans per capita, and percentage of loans to the external users). Evaluation of electronic collection is measured by using download data (1 indicator, number of content units downloaded per capita). Adequacy of library‟s information services are measured against attendances at user training and reference questions per capita (2 indicators). Attendances are also used as criterion for attractiveness of events (I indicator).

C. Efficiency: Are the services offered cost-effectively?

This aspect deals with the cost effectiveness of the services. The perspective “efficiency” includes both, indicators for measuring cost-effectiveness and indicators for quality of processes. This perspective is represented by 13 indicators. Here, the library‟s operating expenditure is set in relation to users‟ visits and total collection use. Acquisition costs are compared to staff cost in order to assess whether a sufficient part of the budget is spent on the collection (4 indicators, viz. cost per user, cost per visit, cost per use and ratio of acquisition cost to staff cost). The efficiency of collection building is calculated as cost per download and cost per document processed (2 indicators). The efficiency of processes is evaluated as per speed and reliability of processes (7 indicators, 5 dealing with speed and 2 with reliability of processes). Speed indicators cover speed of acquisition, media processing, and staff productivity in media processing, lending and interlibrary loan (5 indicators). Reliability of processes cover reference fills rate and shelving accuracy (2 indicators).

D: Potentials and Development: Are there sufficient potentials for future development?

This aspect evaluates library‟s potentials for development. This perspective is important in times of constant change, as it assesses library‟s capability to cope with such changes. The potential for development is measured on one hand against library‟s input in electronic services, and on the other hand against the library‟s success in obtaining funds from funding agencies as well as in generating fund from other sources. This perspective is represented by 5 indicators. Two indicators for electronic services (1 indicator for percentage of acquisition expenditure spent on electronic services and 1 indicator for percentage of library staff providing and developing electronic services), one indicator for staff development (attendance at training sessions per staff members) and 2 indicators for budget (1 for percentage of library means received from special grant and/or income generation and 1 indicator for percentage of institutional means allocated to the library).

This standard provides detailed guidelines on how to implement these performance indicators in libraries. It provides definition of each performance indicator along with its concise description and methods of collecting and analysis of data along with suitable examples and related bibliography. The standard can be used for measuring the performance as well evaluation of public libraries.

In the present day societal change and technological change context, Glen Holt (2005) has suggested seven measurable criteria for defining a “great library”

Great libraries provide measurable superior service.
Great libraries have great funding.
Great library train and retrain their staff.
Great libraries integrate the marketing of virtual, place and outreach services.
Great libraries serve both the weakest and strongest among the constituents.
Great libraries provide constituents with education and entertainment.
Great libraries use virtual tools to offer full range of timely information and services.

5. PERFORMANCE EVALUATION

The evaluation is the process of determining the worth, or value of something. It consists of comparing “what is” to “what ought to be”. Implicit in the comparison is the need to select one or more measures, often called performance measures, to use as the basis for comparison. For example, if a program or service of a library is evaluated, then the experience of a group of library users is evaluated in order to draw some conclusion. If overall performance of the library is being evaluated, then evaluation would compare one library to a group of peer libraries. In general, performance evaluation of library involves comparing functions and services of a library to a set of standards or norms established for such functions and services. Evaluating functions and services of a library forms an important part of the management of any library. Vast amount of literature is available pertaining to all aspects of library and information services evaluation. A number of models have also been developed and many of them are useful in evaluating library functions and services. Choosing an evaluation model can assist the library in better understanding of the relationship between the resources it has and the outputs and outcomes it achieves. Listed below are three models which are used for carrying out different types of evaluation studies.

5.1 PERFORMANCE EVALUATION MODELS

A broad and generic evaluation model has been suggested by Blaise Cronin (1982). He developed the evaluation matrix as shown in the Fig. 1. He suggested that a library could focus its evaluation efforts on costs, benefits and effectiveness. In this model, choice of a particular perspective influences the expectations of library staff members as well as those of library‟s stakeholders. In this model, an evaluation would focus on a single cell in the matrix at a time.

Jose-Marie Griffiths and Donald King developed a similar evaluation matrix, which suggested five evaluation perspectives viz., the library, user, organization that the library is part of, industry and society. They suggested that entire library, its functions, services/products, activities and resources could be evaluated using the evaluation matrix as shown in the Fig. 2. This particular model is suitable in the corporate environment.

Fig. 2Griffiths’s and King’s Evaluation Matrix

One of the oldest, and most frequently cited, evaluation models in library literature was developed by Richard Orr in 1973. His Input – Process – Output-Outcome model as shown in Fig. 3 is applicable in public library setting.

Orr’s Evaluation Model:

As you are aware, when a library is established it is provided with a set of resources. These resources are organized and directed so that they become transformed and have capability to provide set of services. These capabilities are utilized. Once used, the information and service that have been provided have the potential to make a positive, beneficial impact or effect on the individual, community or organization.

Input measures in the library can be broadly grouped into five categories, viz. budget, staff, collections, facilities and technology. Input measures can be quantified and are easy to measure.

Process measures or productivity measures focus on activities that transform resources into services/ products offered by the library.

Output measures indicate the degree to which library and its services are being utilized. This is often measured by counting the volume of activity in the library, such as use of services, use of collection, use of facilities, visits to the library‟s website, etc.

Outcomes indicate the effect of library and its services on the user. Outcomes occur first in the individual and then in the larger context- the organization or community. Outcomes that may occur in an individual include-changes in attitude, skill, knowledge, behavior, and status or condition.

Outputs are measurable and can be compiled, counted or gathered, while outcomes or impacts are often not easily measured.

In a library setting, outcome-based evaluation (OBE) assesses the extent to which a particular program or service has achieved its goal. In a library, a program or service is provided to satisfy people‟s need; hence OBE focuses on the following:

Need: A condition, want, or requirement common to a group of people.

Solution: A program or service that will change attitude, skills, knowledge, behavior, status or condition.

Desired results: The Change or improvement that is expected.

Most public libraries prefer to use Orr‟s Input-Process-Output-Outcome evaluation model. This is because, they are accustomed to reporting inputs and outputs to a variety of agencies and expanding these measures to include process and outcome measures is easy.

6. METHODS OF EVALUATION

Apart from house-keeping functions, a library provides range of reference and information services to its user community. This involves numerous activities, to evaluate these activities effectively, requires choosing the appropriate methodologies, administering them effectively, analyzing the findings and implementing changes to improve the process or service.

Evaluation methods used in such activities can be grouped into two broad categories as shown below:

Quantitative Methods
Qualitative Methods

6.1 QUANTITATIVE METHODS

Quantitative methods gather statistical data by using known quantities to collect data. The resultant data is then subjected to analysis. Tools used for quantitative evaluation methods are counting, measuring, surveys and transaction log analysis.

Libraries have been counting things for a long time for example; the counts are reflected as input measures, such as budget, number of staff, number of workstations, size of library collections, size of building, etc. or as output measures, such as count of circulation, number of reference questions answered, number of people visiting the library, etc. A variety of means are used to capture the counts, reports may be generated by an automated library system, a physical counter may be located at the gate, or staff member may make a tick marks on a form.

Measuring deals with understating and improving a particular process. Examples of processes are cataloguing and physical preparation of new material, re-shelving of items returned from circulation, loaning of materials, etc. When attempting to understand and improve a process, a method known as DMAIC (Define, Measure, Analyze, Improve and Control) is recommended.

Most computer systems keep a log of all transactions- input from the user and system outputs. These transaction logs are analyzed. All transactions that occurred during a particular period of time say a day, a week, a month, or a year can be used for analysis because the computer can quickly calculate the data.

6.2 QUALITATIVE METHODS

Qualitative methods do not quantify a library service, but attempt to capture the views, feelings, and attitudes of individuals towards particular service. Qualitative studies use surveys, observations, interviews, and case studies methods to collect data. As qualitative methods use smaller samples, the findings cannot be generalized, i.e., these cannot be applied to a larger group. The quantity of raw data generated is large and difficult to categorize.

6.3 EVALUATION TOOLS

Tools used for evaluation studies are surveys, sampling, focus groups, interviews, observations and case studies.

Surveys are used to gather information about different aspect of services in a variety of library settings. These usually take the form of questionnaires given to a portion of library users. Survey instruments are helpful in determining opinions and attitudes of users about services. They are not helpful in describing why a library system is functioning effectively.

Sampling refers to the selection of a random or targeted population to survey or include in a study. This technique is useful for analyzing components of reference services, such as types of questions users ask, who ask the questions and how questions are resolved.

Focus group involves gathering selected individuals to provide qualitative data through focused discussion led by an impartial facilitator.

Interviews involve personal interaction between the librarian and the user. Interviewees can include experts, focus groups as well as individuals. In structured interview, same questions and instructions are repeated with different groups and this provides reliable data. Unstructured interview closely resembles natural conversation. Neither the questions, nor the response categories are determined in advance. Questions emerge from the interview as it develops. This method has great degree of flexibility. Respondents are free to express themselves in language that suits them. An unstructured interview is useful in exploratory studies and for determining the actual feelings of people.

Observation is a methodology that notes factual conditions or behavior through personal scrutiny. It is used in both obtrusive and unobtrusive studies. In reference services evaluation, observations are often used in studies that utilize anonymous researchers to ask typical reference questions from reference librarians and then record the results. Observation technique can be used along with survey method.

Case studies thoroughly describe a single situation by utilizing a variety of methods, such as interviews, use of documents or records, and obtrusive and unobtrusive observations. Case study provides an in-depth understanding of one particular group or organization.

7. EVALUATING LIBRARY COLLECTION

Collection evaluation is concerned with determining the strength and weaknesses of the library collection in terms of quality, use and value to the library‟s users and potential users. There are several collection evaluation techniques. Some techniques focus on the nature of the data, i.e., quantitative and qualitative. Some categorize measures by focuses which are either collection centered or user centered.

To evaluate size of collection, ratios are used to demonstrate the relationships between two counts and may include parameters as shown below:

Size of collection in relation to population (volumes per capita),
Collection budget in relation to total budget or other lines in budget (percentage of budget allocated to acquisition), and
Collection budget categories in relation to one another (percentage collection budget allocated to audiovisual materials or online database subscription).

To evaluate cost of providing materials, efficiency ratios are used to examine relationship between cost and count, such as expenditure per capita and average cost per item.

To evaluate collection from performance point of view, three areas, viz. use, user satisfaction and accessibility measures are used. Use measures are generally quantitative, such as circulation data and in-house use measures. User satisfaction measures focus on the perception of library users. These measures find out directly from users their views on the quality of collection through surveys or interviews. Accessibility measures determine whether the materials are available to users on demand. Accessibility measures use data, such as number of requests received from the users to the percentage of requests satisfied from the collection.

8. EVALUATING REFERENCE AND INFORMATION SERVICES

Provision of reference service involves a user, a librarian, the interaction between the two, and access to a set of resources that can be used to provide the answer to a query. The user interact with the librarian using a variety of methods such as, in person in the library, over the telephone, via e-mail, Instant Messaging, or other electronic methods.

Reference services are assessed on the basis of volume, cost, benefit and quality. Analysis of total number of reference transactions handled during a specified period provides an insight into the use and demand for reference services.

Analysis of reference services by type, location, subject, and sources required can be utilized for collection development, staff training and budget allocation.

Cost, benefit and quality assessment of reference services provide a useful feedback for improvement of services, requirement for staff training and continuing education.

In cost benefit analysis, cost is compared to the benefit derived by the users served. User benefits may be measured by actual or perceived outcome, such as goals and satisfaction achieved, time saved, money saved, etc.

Perceptions and needs of users are important measures of the quality and impact of reference services. Surveys, observations and focus groups are used to assess the reference service performance and users needs.

WOREP (Wisconsin-Ohio Reference Evaluation Program) instrument-called the Reference Transaction Assessment Instrument, is used to assess the quality of the reference transactions and identify factors related to success or failure of a transaction. WOREP instrument consists of two-part form, one part to be filled out by the user and the other by the librarian. Reference transaction is marked successful only when the user reports exactly what was wanted, marks being fully satisfied, and does not check any of the nine listed reasons for dissatisfaction. This tool has been successfully used for over 85 public libraries and 100 academic libraries. WOREP instrument was developed by Charles Bunge and Marjorie E. Murfinin 1988.

The Reference Assessment Manual (1995) of American Library Association provides a wide range of evaluation instruments in assessing reference service effectiveness.

Assessing Behavior of Reference Service Providers

The library profession has developed two guidelines for recommended behavior of reference service providers:

i) Facets of Quality of Digital Reference Services and ii) RUSA Guidelines for Behavioral Performance of Reference and Information Service Providers. The second publication lists identifiable attributes of reference professionals in five areas, viz. approachability, and interest, listening/inquiring, searching and follow-up activities. According to these guidelines, the reference librarian should be approachable, show interest in user‟s queries, should have good communication skill to keep user engaged during reference transaction, be efficient in searching and must follow up to find out if user‟s questions have been answered fully or not, and suggest other possible alternatives. The guidelines are applicable for evaluating reference librarians who are engaged in reference transaction face-to-face as well as remotely with the users.

9. EVALUATING TECHNICAL SERVICES

Technical services in the library involve cataloguing, classification and processing of physical materials that are added to the library‟s collection. Activities that encompass technical services are important because they support the services that library‟s user interact with, such as library catalogue, electronic collection etc.

Evaluation of technical services covers aspects such as time and cost of processing, workflow analysis (dealing with how materials flow in technical services area), and comparing efficiency of operations to a set of peer libraries.

Among the methods that have been used to evaluate technical services are activity surveys to gather information about time spent on a specific set of tasks, workflow analysis, and desk work analysis to determine costs, productivity, and accuracy of records.

10. EVALUATING ONLINE SERVICES

Evaluation of the library‟s online catalogue and website is important because this is the user‟s first point of interaction with the library. Improving its usability will make the user more successful in interacting and encourage him/her to use the library regularly.

Library‟s online public access catalogue can be evaluated on the basis of layout, functionality, ease of use and terminology employed. Examining the standard reports available from a library‟s catalogue can reveal information about how frequently an index was used, how often a search failed, and the point at which a search was abandoned.

A number of studies have been conducted to find out how users search OPACs. It was observed that users of a Web based online catalogue can often move from screen to screen quite easily, and yet their actions can lead to the wrong conclusion. Susan Augustine and Courtney Greene (2002) measured the amount of time and number of clicks to perform a given task by the users and compared these measures with those of an “expert”. They found that users employ trial-and-error method while searching an online catalogues, are frequently unable to interpret the information they retrieve, and struggle to understand commonly used terminology.

Karen Markey and her colleagues (1999) examined end-user understanding of subject headings to determine the extent to which children and adults understood subdivided subject headings. Children and adults from three Michigan Public libraries participated in the study. The study found that about 36 percent of the meanings users gave to subject headings were correct. End-user understanding was not affected by context or subject heading order.

A library website offers access to online catalogue as well as other electronic resources accessible through the library. It provides information about the library, including its location, hours, upcoming events, etc. In addition, a library website may provide user the facility to reissue and reserve a book, submit a request for a purchase, change a mailing address, interact directly with a reference librarian, etc.

The usability of a website is tied to a variety of factors, including how the site is organized and what navigation features, such as buttons, tabs, menus, links, graphics, site maps, and a site engine, are provided. When users visit a website, they have a set of expectations, in order to be successful; the website must either meet or exceed those expectations.

The more services and functionality are added to the library website, more important the usability becomes. The interaction between an individual and an automated system is called an interface. A useful definition of usability states that “The usability of an interface is a measure of effectiveness, efficiency and satisfaction with which specified users can achieve specified goals in a particular environment with that interface.” (ISOStandard-13407). Jacob Nielson suggests that there are five attributes of a usable interface, it is easy to learn, efficient to use, easy to remember, causes few errors and pleasant to use.

A variety of qualitative and quantitative methods are available for assessing the usability of library‟s website. Some of the qualitative methods are experts‟ opinion, interviews, focus groups, checklists, diary, observation, failure analysis and complaints. A library may keep a log of all users‟ complaints pertaining to library‟s website and online catalogue. These complaints will reveal the problems if log is reviewed and the problems categorized. Similarly, a computer system can keep a log of search requests that failed and analyzing these will provide an insight into what went wrong and how it can be fixed.

Quantitative methods include visualization tools, transaction log analysis and information retrieval analysis, surveys, experiments, etc. Visualization tools map a website so that a library can see how the site is structured and how many clicks it takes to get from one location to another. The library can track the actual usage of online system by analyzing the transaction logs that are kept by the system. The transaction log analysis will report the timings and actions of the uses, such as clicking on objects and links, mouse movements, typing, and so forth.

Quantitative analysis provides very specific and detailed information about how an online system is actually used. This information can be used to complement the data gathered using qualitative methods. It is suggested that refining and improving an online system, particularly library‟s website, should be done on regular basis, rather than occasionally.

11. SUMMARY

Public libraries supported by public funds are under obligation to report to the funding agencies about their performance. Increasing costs of print as well as electronic resources on one hand and associated budget constraints on the other hand, put great pressure on the libraries to find additional funding. Libraries attempt to achieve this by demonstrating their value to the funding agencies by reporting about their performance and the impact they are making on the lives of user community. To measure their performance and evaluate their functions and services, libraries use various performance measurement and evaluation models. Most commonly used performance measurement models by public libraries are Goal Based Model and System Resource Model.

Of various performance evaluation models, Orr‟s Input-Process-Output-Outcome Evaluation Model is preferredthe most by public libraries. Libraries use both, quantitative and qualitative methods to measure and evaluate their performance. ISO standard on Performance Indicators, specifies a set of 40 performance indicators in 4 areas, viz., i) Resources, infrastructure; ii) Use of services; iii) Efficiency of services; and iv) Potentials and development, to measure the performance of a library. Performance measurement and evaluation of a library, helps in detecting and correcting problems in achieving goals and objectives of a library and improving efficiency of various products and services. Performance evaluation, therefore, should be carried out on continuous basis rather than as one time activity.

12. REFERENCES

Augustine, Susan and Greene, Courtney.“Discovering How Student Search a Library Web Site: A usability case Study”. College & Research Libraries 63,no. 4 (July 2002): 354-65.
Chan, Donna C. “Core Competencies and Performance Measurement in Canadian Public Libraries.” Library Management 27, no.3 (2006):144-153.
Childers, Thomas and Van House, Nancy.“Dimensions of Library Effectiveness.”Library and Information Science Review, 11, (1989): 273-301.
Childers, Thomas and Van House, Nancy.“Dimensions of Library Effectiveness II.” Library and Information Science Review, 12, (1990): 131-53.
Cronin, Blaise. “Taking the measure of service.”ASLIB Proceedings. 34. no. 6/7 (1982): 273-94.
Evaluation of Reference and Adult Services Committee, Reference and Adult Services Division, American Library Association, Comp. and ed. The Reference Assessment Manual.Ann Arbor, MI: Pierian Press, 1995.
Facits of Quality for Digital References Services. June 2003. (http://www.vrd.org/facets-06-03.shtml)
Griffiths, Jose-Marie and King, Donald. Special Libraries: Increasing the Information Edge. Washington, DC:SLA, 1993.
Holt, Glen E.: “What makes a library great?” Public Library Quarterly. 24, no. 2 (2005): 83-89.
ISO 13407: Human–Centered Design Processes for Interactive Systems. Geneva: International Organization for Standardization, 1999.
Jacob Nielson. Multimedia and Hypertext: The Internet and Beyond. Boston: Academic Press, 1995, 281p.
Katz, William A.: Reference Service Policies and Evaluation- Introduction to Reference Work. Vol. II. 7thed. New York: McGraw Hill, 1997.
Lancaster, F. W.: If You Want to Evaluate Your Library. 2nd ed. Urbana: University of Illinois. Graduate School of Library and Information Science, 1993.
Lynch, Beverly P. and Yang, Wenxiang. “Evaluation of public libraries: The 2001 IFLA Standards and the 2003 Standards for Provincial Libraries in China.” Libri 54, (2004): 179-189.
Markey, Karen et al. “End-User Understanding of Subject Headings in Library Catalogs.”Library Resources and Technical Services. 43, no. 3 (July 1999): 140-60.
Matthews, Joseph R. The Evaluation and Measurement of Library Services. London: Libraries Unlimited, Inc., 2007.
Miller, Paula J. “Implementing total Quality Management”. pp. 25-40. in. LibraryEvaluation- A Casebook and Can-Do-Guide.Danice P. Wallace and Connie Van Fleet. Eds. Englewood: Libraries Unlimited, Inc., 2001.
MOUSS Management of reference Committee. “Guidelines for Behavioral Performance of Reference and Information Service Providers. June 2004. (http://www.ala.org/rusa/rusaprotools/referenceguide/guidelinesbehavioral.htm)
Murfin E. Marjorie and Gugelchuk, Gary.“Development and Testing of a Reference Transaction Assessment Instrument.”College and Research Libraries. 48, (July 1987): 314-36.
Murfin, Marjorie E., and Bunge, Charles A. “Responsible Standards for Reference Service in Ohio Public Libraries.”Ohio Libraries 1 (April/May), 1988.
Orr,Richards. “Measuring the Goodness of Library Services.” Journal of Documentation, 29, no. 1 (1973): 315-52.
Parkar, Sandra. “The Performance Measurement of Public Libraries in Japan and the UK.” Performance Management and Metrics 7, no.1(2006):29-36.
Reference and User Services Association (RUSA), “RUSA guidelines for Behavioral Performance of Reference and Information Services Professionals.” RQ 36 (Winter), 1996:200-203.
Schneider, Benjamin. “An Interactionist Perspective on Organizational Effectiveness”, pp. 27-54. in.
Organizational Effectiveness: A Comparison of Multiple Models, eds. Kim, S.Cameron and DavidA. Whetten.Eds. New York: Academic Press, 1983.
Starbuck, William H., and Nystrom Paul C. “Pursuing Organizational Effectiveness That is Ambiguously Specified,” pp. 135-61 in Organizational Effectiveness: Acomparisonof Multiple Models. Eds. Kim S. Cameron and David A. Whetten. New York: Academic Press, 1983.
Wallace, Danny P. and Van Fleet, Connie.eds. Library Evaluation- A case book and can do guide. Englewood: Libraries Unlimited Inc., 2001.

Learn More:

Websites