26 Evaluation for Fine Tuning

Dr. Rajesh

Content Outline

1. Introduction

2.Objectives

3. Key consideration for evaluation of fine-tuning.

4. Evalution of fine-tuning for

5. Fine tuning of established program

6. Reappraising Objectives and outcomes

7. Reputability Assessments

8. Program Replanting and Redesign

9. Overview of Evaluative activities of Fine- Tuning

10. Conclusion

Introduction

Once programs are under ways, it is often important to test variation in the ways they operate. The major reason to do so is to improve either their efficiency – to increase their impact or to decrease their costs per unit of impact or to provide equitable service delivery – that is to see that a program services are delivered to its target population just as these services are delivered to its target population in the general population.Often the line between a new intervention and the refinement of an existing program through fine-tuning is a thin one. Likewise, fine-tuning evaluations is costs and magnitudes of effort often are difficult to distinguish from the full-scale evaluation of established programs. Indeed, program managers, on the basis of ongoing evaluationinformation, may make day to day administrative andtechnical changes that are quite extensive and that are subject to systematic evaluation but that we would classify as “fine-tuning”. The distinction rests with whether or not the program modification emerges from qualitative changes in the rationale that has guided program activities. Fine tuning of a program fundamentally leaves its impact model intact.

Objectives

After going through this unit the learners will be able to:

1. Fine-tuning means and its importance.
2. Fine tuning of established program.
3. Identify the Objectives behind the evaluation of fine-tuning.
4. Role fine tuning toreshape the program.

Evaluation is a systematic process for an organization to obtain information on its activities, its impacts, and the effectiveness of its work, so that it can improve its activities and describe its accomplishments.

Fine-tuning refers to circumstances when the parameters of a model must be adjusted very precisely in order to agree with observations. Fine tuning is a process to take a network model that has already been trained for a given task, and make it perform a second similar task.

Key consideration for evaluation of fine-tuning–

• Consider the following key questions when designing a program evaluation.
• For what purposes is the evaluation being done, i.e., what do you want to be able to decide as a result of the evaluation?
• Who are the audiences for the information from the evaluation, e.g., customers, bankers, funders, board, management, staff, customers, clients, etc?
• What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences, e.g., information to really understand the process of the product or program (its inputs, activities and outputs), the customers or clients who experience the product or program, strengths and weaknesses of the product or program, benefits to customers or clients (outcomes), how the product or program failed and why, etc.
• From what sources should the information be collected, e.g., employees, customers, clients, groups of customers or clients and employees together, program documentation, etc.
• How can that information are collected in a reasonable fashion, e.g., questionnaires, interviews, examining documentation, observing customers or employees, conducting focus groups among customers or employees, etc.
• When is the information needed (so, by when must it be collected)?
• What resources are available to collect the information?

Program evaluation can help:

1. Understand, verify or increase the impact on clint – These “outcomes” evaluations are increasingly required by various funders as verification that the programme is indeed helping their constituents. Too often, service providers (for-profit or non-profit) rely on their own instincts and passions to conclude what their customers or clients really need and whether the products or services are providing what is needed. Over time, these organizations find themselves in a lot of guessing about what would be a good product or service, and trial and error about how new products or services could be delivered.
2. Improve delivery mechanisms to be more efficient and less costly – Over time, product or service delivery ends up to be an inefficient collection of activities that are less efficient and more costly than need be. Evaluations can identify program strengths and weaknesses to improve the program.
3. Verify that you’re doing what you think you’re doing – Typically, plans about how to deliver services, end up changing substantially as those plans are put into place. Evaluations can verify if the program is really running as originally planned.

Evalution of fine-tuning for–

Improvement: Evaluations help to assess how well their programs are working by estimating the extent to which desired outcomes are being achieved and by identifying whether improvements are needed to increase effectiveness with respect to objectives. Evaluation activity helps programmer proactively optimize their programs’ performance. Evaluations provide actionable information on program activities that are not being performed as intended, outputs that are not as effective as they were expected to be, customer needs or expectations that are not being met, and outcomes that are below projections. Managers can use this information to request additional resources or to modify the program design to improve its effectiveness. A program that undertakes evaluation activities to obtain timely information during the planning, budgeting, implementation, and other management cycles to make continuous improvements will achieve greater success than one that does not.

• Accountability: Evaluations help program managers demonstrate internal and external accountability for the use of public resources. This includes demonstrating fiscally responsible management, establishing evidence that goals are being met or services are being delivered as promised, and quantifying “realized” impacts of the program. Accountability is, in part, motivated by a need to addressinternal and external Evalution requirement and expectations. A program that can account for its public investments by demonstrating progress and results is more likely to be viewed as successful by the appropriators and stakeholders

Fine tuning of established program

Program fine tuning typically occurs because program sponsors and staff are dissatisfied with the effectiveness or the efficiency of their interventions, or both. The basis for implementing such changes may be the findings of systematic evaluation studied of a monitoring or impact type, the outcome of evaluability assessment or more impressionistic evidence of dissatisfaction with the way efforts are being conducted.

All the proceedings can be understood with an example- In the mental health field, community mental health centre isareas, where there are significant Spanish- speaking population, have become aware of the barriers to access that these face because of language problems. A variety of approaches have been tried to increase access for a person of Hispanic background. One approach has been to increase the proportion of staff fluent in Spanish and to employ persons of Hispanic background to contact key religious and voluntary groups in the Spanish community. A second approach has been to establish satellite “fedder” program in heavily Hispanic neighborhoods. Staff, materials, and interpersonal relations reflect the special cultural character of the resident , short term diagnosis and treatment are available and patients needing more extensive care are referred to the parent center.At issue, is whether these approaches, in fact,increase access for the target population, and which of them proves more effective.

Another example of fine- tuning concerns the large number of initiatives in connection with neighborhood community health centers, most of which begin as “free-standing” organizations. Groups and large foundations have adopted avariety of the program to link community health centre to the hospital and medical school. Although the basic character of a medical care provider in the centres is not changed the initiatives are seen as mechanisms that reduce the undesirable overlap of service between the center and hospital facilities and that may increase continuity of care for patients. Other similar efforts to reduce fragmentation of care have been made single neighborhoodcenters, for instance, offer welfare recipients and other persons of marginal income avariety of services that previously were available in separate offices at some distance from each other.

Fine- tuning basically requires three related sets of activities one part of sponsors, program staff, and evaluators : reappearing program objectives and outcome; undertaking reputability assessments, and planning / redesigning the program.

Reappraising Objectives and outcomes

Like the innovative program, fine-tuning efforts are responses to existing conditions. In fine tuning efforts are responses to the existing condition. In fine tuning, however, the action focuses on conditions adhering to the program itself, rather than a new and untreated problem. Often awareness that a program has failed to meet community concerns requires some modification of the programs objectives and outcome criteria. If take the example of community mental health centers may have been provided to a range of diagnostic, emergency, and short term treatment program to the catchment area resident. The objectives, as originally stated, did not include special consideration for the ethnic and cultural backgrounds of the target population. Fine-tuning in the face of failure to provide access to persons of particular social and cultural backgrounds require a refinement of the objective.

The need to refine objectives often becomes apparent after a program has been implemented, particularly as innovative programs stabilize and emerge as established venture in human service field. In some case modifications in objectives steam from the experiences and exchange of idea of staff that occur as part of the day to day work activities. In other case evaluators undertake special studies either as independent contractors or as staff members to obtain to aid program personnel in revising objectives. Sometimes evaluator and programs also staffs have disposal ongoing management and service information system that provide data on the extent to which current objectives are being met. Manager of social program also encourage the expression of different views on program activities and operations at staff meeting, retreats, and so on, and these may led to fine-tuning efforts. In some organizations, as in the business sector, “quality circle” activities borrowed from Japanese industry has been initiated.

Reputability Assessments

We use the term reputability assessments to refer to systematic effort to obtain from relevant stakeholders, particularly targets, data on their experiences with, and assessment of program as a basis on to judge the extent of programs success in meeting its objectives. Reputability assessment can be viewed as consumer surveys similar to those conducted market research for consumer products. Some program provides questionnaires to clients, either as a part of the special study or as part of an ongoing monitoring effort, in order to obtain information about their satisfaction with programs. The survey includes questions on various aspects of treatment: waiting time, relation with practitioners, cost and fees, and like. They may also invite clients to indicate unmet needs and desired services. In addition, and particularly for larger program evaluators may survey the providers of their services.

Often reputability assessment will point to fine tuning efforts that are comparatively simple such as providing the feedback to teachers on what is being done for students, areas in which special programs are needed andway to collaborate effectively in order to deal with students’ problems. Other times, the information may highlight the need for considerable program modification.

Reputability information also comes, less formally, from advocacy pressure exerted by stakeholders and community groups, as well as probing by persons in the mass media and in political life. When systematic reputability assessments are conducted in advance of these pressures, program managers may be able to fine tune intervention and avoid becoming subject to harassment.

Program Replanting and Redesign–

Implementing refinements and fine tuning requires a return to the various steps and activities discussed in our section on evaluating innovative programs. The problem must be identified and described, objectives operationalized, a revised impactmodel developed the targetedpopulation redefined, the delivery system redesigned, and plans made for whatever revision are required in monitoring impact and efficiency.

It should be stressed that not only are fine tuning efforts much more commonplace than innovative intervention in the human service field. But the consequences of these efforts are extensive. For example in many high schools, some students in such a courses asEnglish and mathematics are informalprepared to take SATs or similar college admission examinations. If it can be demonstrated that these students obtain higher scores, the school may fine tune its program and incorporate formal preparatory sessions into the regular high school curriculum, since maximizing students opportunities and choices for college education is an objective consistent with one of the general goals of higher education, it would be sensible, however, to evaluate the addition of the preparatory session; such curriculum modification may increase educational competence gap between students with college aspiration and those without them because of less emphasis on other aspects of the educational program.

In terms of fine tuning, then, the evaluator is involved in the following tasks:

1. Reappraising objectives
2. Identifying program modification by drawing on the data of previous evaluations, as well as information about program progress gathered as part of the service delivery.
3. Undertaking reputability assessments
4. Participating in program planning and redesign.
5. Planning and implementing evaluation designs to monitor the program changes and their impact.

Conclusion

There is no clear-cut dividing point between innovative and fine tuning or modification efforts. Sometimes the change being tested are minor and clearly modificatory.Other time; however, they are costly and may have broadramifications for human service network. For example, fine tuning thatintegrates formerly free-standing community health centre with teaching hospital does not change the basic concept underlying delivery of medical care to the low-income person. However, it may have major consequences for the cost of such care on a national basis and may markedly alter the quality of services received. So evaluation of fine tuning is very technical in nature.