28 Impact of Government programs

Dr. Abhay Chawla

Contents:

1. Introduction to Government Programs
2. Impact Evaluation
3. Evaluating an Impact
4. Summary
5. Conclusion

Learning Outcomes:

After studying this module you will:

•  be able to understand what is a government program and why do the programs need to be evaluated;
•  learn when a program need evaluation and what are the various evaluation methods; and
•  be able to identify why impact evaluation methods have shortcomings.

1. Introduction to Government programs

A government program is a program or activity sponsored or administered by local, state or national government. A majority of such programs fall in the realm of welfare. The term welfare is loosely defined as the minimum level of well being and social support of its citizens by the modern nation state.

Britannica Online Encyclopedia defines a welfare state as a concept of government in which the state plays a key role in the protection and promotion of the social and economic well-being of its citizens. It is based on the principles of equality of opportunity, equitable distribution of wealth, and public responsibility for those unable to avail themselves of the minimal provisions for a good life. The general term may cover a variety of forms of economic and social organization.

Even though welfare systems differ from country to country, the common thread is about providing for individuals who are unemployed, have illness or disability, elderly or those with dependent children, and veterans. Welfare can take many forms like monthly payments, housing and other assistance, subsidies and vouchers.

One of the first program launched by the Government of India after independence was the Community Development Program (CDP). It was launched in 1952. The CDP was a follow up to the pilot launched in 1948. The program was launched for 55 districts each covering about 300 villages or a population of 30,000, aiming at the socio-economic transformation of rural people. The National Extension Service was inaugurated on October 2, 1953, with a view to provide necessary manpower for the implementation of the community development program throughout the country.

Consequently the government has launched various programs some of which are:

•  1960-61- Intensive Agriculture Development program (IADP) to provide loan for seeds and fertilizers to farmers;
•  1966-67- High yielding variety program (HYVP) to increase the productivity of food grains by adopting latest varieties of inputs of crops;
•  1972-73 – Accelerated Rural water Supply Program (ARWSP) for providing drinking water in villages;
•  1975- Command Area Development Program (CADP) for better utilization of irrigational capacities;
•  1980 – Integrated Rural Development Program (IRDP) for overall development of rural poor; 1983- Rural Landless Employment Guarantee Program (RLEGP) for employment to landless farmers and laborers;
•  1986- Self Employment Program for the Poor (SEPUP) for Self employment through credit and subsidy;
•  1995- Prime Minister Integrated Urban Poverty Eradication program (PMIUPEP) to eradicate urban poverty and
•  2004- National Food for Work program (NFWP) to supplementary wage as food grains for work.

Impact evaluations of government program are needed to give a feedback to policy makers on a range of issues which include efficiency of a program, need of scaling up working interventions, adjusting program benefits and difference between various program alternatives. They are also particularly effective when applied to pilot programs that are testing a new and unproven approach that looks promising on paper.

2. Impact Evaluation

Gertler et al. (2011) defined Impact evaluations as a particular type of evaluation that seeks to answer cause-and-effect questions. Unlike general evaluations, which can answer many types of questions, impact evaluations are structured around one particular type of question: What is the impact (or causal effect) of a program on an outcome of interest?

As impact is defined as marked effect or influence, when we ask a question i.e. what is the impact of a particular government program we need to first determine the field in question and define the metrics for measurement of performance. For example one can look at the impact of the National Food for Work program (NFWP) in respect to the following treatments:

Increasing nutrition levels in households Increase in school enrollment

The new community structures spawned

The changed nature of politics in the village

Increase of household saving in the formal banking sector

Each of the above falls under a different field i.e. nutrition will fall under health, school enrolment under education and so on.

After having determined the field of treatment one has to set the metrics for measurement, for example under nutrition are we looking at all girl children in the age group 0-5 and from what level of current nutrition level to what nutrition level does the program target to make an impact?

So most of the time when a user used the term “Impact” he/she means what the program produced. Impact actually on the other hand in the purview of rigorous evaluation means something else. Impact means the causal effect that a program has on an outcome that we care about. Taking the above National Food for Work program (NFWP) example, one can say the program is a success as it benefitted (say) a million households and one can say with confidence that the program is functioning. But this metric does not tell us whether the program has the impact that we desire because what the government really cares about for which is program runs (say) is to increase the nutrition intake of parents and children and increase food security of households and that people are not skipping meals. Those are the impacts the program should care about. Hence defining “what the government cares about” is the start and the first part to determining the impact of the program. The second part of the study being what is called “causal impact” or what the program is leading to directly, for example does the (say) National Food for Work program (NFWP) lead families to totally depend on the government and not supplement their income from other sources?

Lance, P., D. Guilkey, A. Hattori and G. Angeles. (2014, p1 ) in their article How do we know if a program made a difference, state:

We would like to estimate program impact by measuring the difference in outcomes for a sample of individuals holding everything but their program participation constant. Unfortunately, this is not possible: we cannot observe an individual’s outcomes while varying their participation status in a program and only their participation status in that program. Within the framework of varying exposure for the same individuals, we are therefore reduced to at most considering differences in outcomes over time for the same individual as their program participation varies.

They then argue the alternatives to the above methodology i.e. see how the program impacts two different individuals, one a participant and other a non participant. However recording this isn’t simple either because unlike controlled laboratory conditions, humans in a real world interact with situations based on conscious decisions based on cost benefit analysis. Hence both the participants and non participants differ in many more ways than just the experience of the program participation. This in itself would introduce a difference in measurement of impact of the program.

The First step hence in any program impact evaluation is to characterize the parameter one wishes to capture.

Let’s simplify the above concept by taking the example of a drug test to find the impact of a particular drug on (say) lab rats. The impact testing is done by separating rats into two groups, one on which the drug is tested and the other, called control group, on which the drug is not tested. Assignment of rats between the two groups is typically completely random. This random assignment insures that the rats are alike on average, so that their peculiar or individual differences are statistically canceled out when comparing average outcomes between the two groups. There is then attempts made to insure that their experiences in the course of the experiment vary only in terms of exposure to the drug with no differences between the groups in terms of food levels, veterinary attention, etc. during the experiment. Hence to assess the average impact of the drug it is then necessary only to compare mean (i.e. average) in the control and treatment samples of the parameter one wishes to capture.

The above simplistic example on impact analysis in lab conditions is turned on its head in a messy real life human world as one can longer control variables that apply on participants besides the constant application of a program. Even the guaranteed constant and uniform implementation is itself a suspect in real life conditions based on human subjectivity and other implementing conditions. A case in point is the Midday Meal Scheme of Government of India, a school meal programme designed to improve the nutritional status of school-age children nationwide.

The Midday Meal program was launched by the Indian government in August of 1995 to boost enrollment, retention, and attendance rates for children, while also improving nutrition and health outcomes. Supreme court in its order dated 28 November 2001 under Right to Food Campaign mandated all government and government-assisted primary schools to provide cooked midday meals.

A decentralized scheme, the meals are cooked on-site by local cooks and helpers or self-help groups. The decentralized model helps serve local cuisine besides providing jobs in the area and minimizing waste as well as better monitoring by parents and teachers. However inspite of years of experience and the advantages of the decentralized Midday meal scheme, in July of 2013, 20 children in a school in Bihar died after consuming the midday meal presumably from grain contaminated with organophosphates negating confidence in the scheme (Kumar, 2013) . How do we then measure the impact of the midday meal scheme then?

3. Evaluating an Impact

Most evaluations of impact chose a methodology that relies on an estimate called counterfactual, i.e. what the outcome would have been for program participants if they had not participated in the program. As detailed above even though it is not the perfect method to measure impact, in real life conditions it is impossible to keep other variables same except the participation or non participation in the program. Hence evaluators normally find a comparison group to estimate what would have happened to the program participants without the program.

The basic evaluation question asked is the impact or causal effect of a program on an outcome of interest? This can be applied to many contexts for example Does Food for Work Program (FFWP) improves nutritional levels in a household? Does FFWP increase school enrolment? Or Does FFWP decrease infant and child mortality?

Gertler et al.(2011) list out the reasons why a program needs an evaluation i.e.

Innovative: It is testing a new, promising approach.

• Replicable: The program can be scaled up or can be applied in a different setting.
• Strategically relevant: The program is a flagship initiative; requires substantial
  • resources; covers, or could be expanded to cover, a large number of people; or could generate substantial savings.
• Untested: Little is known about the effectiveness of the program, globally or in a particular context.
• Influential: The results will be used to inform key policy decisions.

Impact evaluations can be done in two ways i.e. prospective or retrospective. A prospective evaluation involves determining criterion designed and built into program implementation with baseline data collected prior to program implementation for both the implementation and comparison groups. On the other hand Retrospective evaluations assess program impact after the program has been implemented with no baseline data.

A simple way to calculate the impact or causal effect of a program P on an outcome of interest X is given by the basic impact evaluation formula:

                    = (X | P = 1) – (X | P = 0).

This formula says that the causal impact ( ) of a program (P) on an outcome (X) is the difference between the outcome (X) with the program (P = 1) and the same outcome (X) without the program (P = 0). For example, if P denotes a Food for Work program (FFWP) and X denotes income, then the causal impact of the FFWP (α) is the difference between a person’s income (X) after participating in the program (i.e. P = 1) and the same person’s income (X) at the same point in time if he or she had not participated in the program (i.e. P = 0).

Hence the evaluator would like to measure income at the same point in time for the same unit of observation (a person) but in two different states of the world. The evaluator observes how much income the same individual would have had at the same point in time both with and without the program and thus concludes that the only possible explanation for any difference in that person’s income would be the program. By comparing the same individual with herself at the same moment the evaluator manages to eliminate any outside factors that might also have explained the difference in outcomes and concludes that the relationship between the FWWP and income is causal. This basic impact evaluation formula is valid for anything that is being analyzed i.e. a person, a household, a community or any other unit of observation that may receive or be affected by the program. The formula is also valid for any outcome (X) that is plausibly related to the program at hand. Once we measure the two key components of this formula—the outcome (X) both with the program and without it program impact questions can be answered.

However there is still a problem for the evaluator. He can observe and measure the outcome (X) for program participants (X when P = 1), there are no data to establish what their outcomes would have been in the absence of the program (X when P = 0). The term (X | P = 0) represents the counterfactual, when the participant had not participated in the program or in the absence of the program. To do this the evaluator uses a comparison group, also called a control groups.

After setting the program and the control group it is important how the evaluator interprets the results. Depending on what the treatment and the control group actually represent, the interpretation of the impact of a program will vary. Also the method of selecting control group has a high risk of bias by the evaluator.

The most important parameter of impact evaluation method depends on the rules the program follows for enrolling participants. One of the most reasonable and transparent way for allocating scarce resources among equally deserving populations is by giving everyone who is eligible an equal opportunity to participate in the program. A randomized selection is the one which typically reduces the risk of a bias and the best way to evaluate the impact. Randomized assignment is essentially a lottery system with every eligible unit having an equal probability of selection.

A randomized assignment also ensures both internal and external validity of impact evaluation, provided the evaluation sample is large enough. Internal validity means that the estimated impact of a program is the net sum of all potential confounding factors and the control group representing the true counterfactual. This ensures that once the program starts, the comparison group is exposed to the same set of external factors over time except the program. Hence any differences in outcomes between the treatment and comparison groups can only be due to the existence of the program. The external validity on the other hand means that the impact estimated in the evaluation sample can be generalized to the population of all eligible units or the evaluation sample a representative of the population of eligible units. Thus the evaluation sample must be selected from the population by using one of several variations of random sampling.

Some of the impact evaluation methods are randomized assignment, randomized promotion, regression discontinuity design, Difference-in-Differences and Matching. These methods could also be combined by the evaluator. Each method has its own pros and cons especially the risks of bias which gets reduced by combining methods.

Complications arise when programs have multifaceted aspects of implementation or multiple treatments. Evaluations of such programs are done using crossover design or cross-cutting design.

Some of the Impact evaluation case studies that one can read are Impact of Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA) on Agriculture using the Difference-in-Differences method (Varshney, Goel & Meeenakshi 2014); Impact of MGNREGA on Rural Agricultural Wages in SAT India (Nagaraj, Bantilan, Pandey & Roy 2014) and IIM Ahmedabad Impact Assessment study of E-Government projects in India (2007).

4. Summary

• A government program is a program or activity sponsored or administered by local, state or national government. A majority of such programs fall in the realm of welfare.
• Even though welfare systems differ from country to country, the common thread is about providing for individuals who are unemployed, have illness or disability, elderly or those with dependent children, and veterans.
• One of the first program launched by the Government of India after independence was the Community Development Program (CDP). It was launched in 1952.
• Impact evaluations of government program are needed to give a feedback to policy makers on a range of issues.
• Impact is defined as marked effect or influence. It means the causal effect that a program has on an outcome that we care about.
• Ideal way to estimate program impact is by measuring the difference in outcomes for a sample of individuals holding everything but their program participation constant.
• Unlike controlled laboratory conditions, humans in a real world interact with situations based on conscious decisions based on cost benefit analysis, so recording impact is tricky.
• Most evaluations of impact chose a methodology that relies on an estimate called counterfactual.
• A program may needs an evaluation if it is Innovative, Replicable, Strategically relevant, Untested or Influential.
• Impact evaluations can be done in two ways i.e. prospective or retrospective.
• The most important parameter of impact evaluation method depends on the rules the program follows for enrolling participants.
• A randomized assignment also ensures both internal and external validity of impact evaluation. Impact evaluations are a costly exercise and not all programs need an impact evaluation.
• Evaluations cannot be conducted in isolation from other sources of information like the nature and content of the program to contextualize evaluation

5. Conclusion

Impact evaluations are a costly exercise and not all programs need an impact evaluation. A program affecting a few people or not needing a huge budget like a volunteer program may not need an impact evaluation.

Such evaluations cannot be conducted in isolation from other sources of information like the nature and content of the program to contextualize evaluation results else certain results may or may not be achieved. Also impact evaluations are not typically designed to provide insights into program implementation though are guided by information on how, when, and where the program under evaluation was being implemented.

As Lance, Guilkey, Hattori & Angeles in their 2014, A Guide to Statistical Methods for Program Impact Evaluation state:

The aim of program impact evaluation is to learn whether and to what degree a program altered outcomes from what otherwise might have prevailed. Measuring what might “otherwise have prevailed” is a challenging task. (p.1)

There is no “Gold Standard” method of impact evaluation. All of the methods involve assumptions, not all of which are testable, that allow one to interpret the estimates generated by them as program impact (or, more broadly, as reflecting a causal relationship). Some present inherent limitations in terms of the parameters that can be estimated.

This is why careful impact evaluation work is so important. One must have a good sense of the institutional and environmental framework in which a program operates, as well as a good understanding of the design and procedures of the program itself and the types of populations motivated to participate and why they would be. This allows the evaluator to have an informed sense of what assumptions are (probably) reasonable, and hence which impact evaluation methods might be preferred and how much weight to assign to the estimates generated by them.

It is true that even then assumptions (or, at the least, certainly untestable assumptions) are glorified opinions, but they will at least be informed opinions. (p. 319)