UNITED NATIONS ENVIRONMENT PROGRAMME
Final Evaluation of the UNEP GEF project
"Regional Program of Action and Demonstration of Sustainable Alternatives to DDT for
Malaria Vector Control in Mexico and Central America"
Project No. GF/2732-03-4680 PMS: GF/4030-03-22
Alberto Narváez Olalla
Evaluation Office
November 2009

CONTENTS
Executive summary................................................................................................. 3
Background................................................................................................................................... 3
Major findings............................................................................................................................... 3
Achieved objectives.................................................................................................................. 3
Sustainability ............................................................................................................................ 4
Catalytic Role ............................................................................................................................... 5
Achievement of outputs and activities..........................................................................................5
Assessment of Monitoring and Evaluation Systems.....................................................................5
Assessment of processes that affected attainment of project results ............................................ 6
Recommendations and the way forward....................................................................................... 6
Lessons learned............................................................................................................................. 7
GLOSARY............................................................................................................... 8
1. INTRODUCTION AND BACKGROUND ....................................................... 9
1.1. Project identification.............................................................................................................. 9
1.2. Project rationale ..................................................................................................................... 9
Background............................................................................................................................... 9
Project aims and components ................................................................................................... 9
2. OBJECTIVE AND METHODS....................................................................... 11
2.1. OBJECTIVE AND SCOPE OF THE EVALUATION .......................................................11
2.2. METHODS .......................................................................................................................... 11
2.2.1. Sources of Evidence and Codes Used........................................................................... 11
2.2.2. Variables, questions and codes ..................................................................................... 12
2.2.3. Study Population and Units of Analysis ....................................................................... 13
2.2.4. Data Analysis................................................................................................................ 13
3. PROJECT PERFORMANCE AND IMPACT ............................................... 16
3.1. Attainment of objectives and planned results ...................................................................... 16
3.1.1. Effectiveness................................................................................................................. 17
3.1.2. Relevance...................................................................................................................... 31
3.1.3. Efficiency...................................................................................................................... 32
3.2. Assessment of sustainability of project outcomes ............................................................... 34
3.2.1. Financial sustainability ................................................................................................. 35
3.2.2. Socio political sustainability......................................................................................... 37
3.2.3. Institutional framework and governance ......................................................................39
3.2.4. Environmental sustainability ........................................................................................40
3.2.5. Suggestions for long term impact ................................................................................. 40
3.3. Catalytic role........................................................................................................................ 41
3.4. Achievement of outputs and activities................................................................................. 43
3.5. Assessment of Monitoring and Evaluation Systems............................................................ 46
3.5.1. M&E design.................................................................................................................. 48
3.5.2. M&E plan implementation ........................................................................................... 50
3.6. Assessment of processes that affected attainment of project results. .................................. 53
3.6.1. Preparation and readiness ............................................................................................. 55
3.6.2. Country ownership / Driveness..................................................................................... 56
3.6.4. Stakeholder involvement .............................................................................................. 57
1

Executive summary
Background
The project had three main goals: to implement demonstration projects of malaria vector control
without DDT or other persistent pesticides that can be replicable in other parts of the world; to
strengthen national and local institutional capacity to control malaria without the use of DDT; and
to eliminate DDT stockpiles in the eight participating countries. The project involved eight
countries: Mexico, Belize, Costa Rica, El Salvador, Guatemala, Honduras, Mexico, Nicaragua,
and Panama and nine sites for demonstration projects were selected in each country. The
project begun in August 2003 and was finished by July 2008 (except for the POPs Disposal
component). The project was extended twice beyond its initial duration of three years.
A final evaluation was conducted from September 2008 to January 2009 as an in-depth
evaluation using a participatory approach. It was a descriptive multicase study, using several
sources of information. The project evaluation was conducted at the regional level and at three
levels; national, demonstrative areas and localities, in four countries, namely Costa Rica,
Guatemala, Mexico and Panama.
Major findings
Objectives achieved
6,0
5,2
Processes affected results
Sus tainability
5,0
5,2
0,0
M&E Sys tems
Catalyzed Role
5,3
4,1
5,4
Achievement of outputs
Highly Satisfactory Satisfactory or Moderately
or Highly Likely
likely
satisfactory
or
5,5 to 6
4,5 to 5,4
moderately likely
3,5 to 4,4
Achieved objectives
The achievement of objectives was satisfactory, because the objective 3 (elimination of DDT
stockpiles) was not met. All countries adapted techniques of vector control without using
persistent insecticides in their demonstrative areas.
The establishment of a regional network was developed and there was an active exchange of
best practices and lessons learned among neighbouring countries. A major planned outcome
was to increase government and local community awareness of DDT and other pesticide
hazards to the environment and human health. Through the project, the personnel of national
and local teams, leaders, community agents and teachers became informed, trained and strongly
3

empowered and mobilized, and the project succeeded in increasing community participation in
vector control activities without insecticides.
Apart from effectiveness, the achievement of objectives was "satisfactory". In fact, there was a
considerable reduction in the number of malaria cases registered and in the Annual Parasite
Index (API) in all of the 202 pilot communities. From 2002 to 2007, malaria cases reduced by
63% and the API decreased from 15,92 to 8,36 cases per thousand inhabitants. Several
communities and several departments and provinces where the demonstration areas were
located reduced the incidence close to the API of 1 (pre elimination of malaria).
The project used a combination of control methods according to the Roll Back Malaria policy:
diagnosis and prompt treatment, elimination of mosquito breeding sites by physical media called
EHCA (Elimination of habitat and mosquito breeding sites of anopheles), larvae-eating fish,
cleaning of houses and patios and whitewashing houses (painting with lime) through community
participation. Risk approach (focalization of intervention in areas of high historical transmission)
and selective control of vectors were strategies used to improve the cost effectiveness. The
countries have adapted the model into the conditions, resources and local capabilities.
All participating countries executed activities to promote the public alert about health and
environmental risks due to the use of DDT and other POPs. Experts from all countries were
trained and studies about environmental impacts were executed, and the national laboratories
now have the necessary equipment and trained personnel for this purpose.
The relevance of the project was rated highly satisfactory because the results were consistent
with the proposed malaria control strategies. DDT and other POPs were not used, only
insecticide impregnated bed nets were used in some localities. In addition the project contributed
to achieving the Millennium Development Goals, the Roll Back Malaria goal and the possibility to
eliminate the malaria in these localities and also in the entire communities of demonstrative
areas.
The efficiency of the project was satisfactory. Implementing methods of vector control activities
through community mobilization (cleaning houses and patios, control of mosquito refugees and
breeding sites) were cost-effective, resulting in savings (0.4, 0.64 and 0,003 USD cost per case
averted in three places studied). The major costs were those from payments of National
Coordinators (NAP) and travel expenses for supervision and community support.
Sustainability
The overall rating on sustainability was "likely". Only the regional level assessed the sustainability
as "moderately likely" and the rest of the countries rated it "likely". The influence of hurricanes
and tropical storms, the reduction of financial resources, outcomes of the global crisis, and the
high migration in frontier areas, are factors that might affect the sustainability of the project to
medium term.
Financial sustainability was evaluated as "likely". Guatemala rated it "highly likely", Panama as
"moderately likely" and the rest, including regional "likely". The budget will not be likely to change
significantly, although the global crisis could have an impact. Only the national levels of Mexico
and Costa Rica considered that sustainability could be affected by socio-political factors,
particularly because of the 2009 presidential elections, which could change health policies and
health authorities. In relation to the institutional framework and governance, the rating is
"likely". Guatemala and Panama rated it "likely".
4

The aspect that was considered to affect the environmental sustainability most was the
presence of hurricanes, tropical storms and floods, which are frequent in the demonstration
areas. Nevertheless, in Guatemala, Costa Rica and Panama, health services and communities
reacted very well against the effects of tropical storms occurred at the end of 2008. By January,
2009, no epidemics were reported in demonstration areas of these three countries.
Catalytic Role
Catalytic role was one of the most successful elements of the project and it was rated as "highly
satisfactory". Indeed, the control strategy on the initiative of the community leaders and health
workers has been considerably replicated in other neighbouring localities and municipalities. In
Mexico, Guatemala, Nicaragua and Honduras the replication was very extensive; in the last three
countries alliances were forged, among others, with the Global Fund Projects. In Costa Rica, the
strategy is being replicated in all Atlantic areas by the Ministry of Health in order to eliminate
malaria transmission.
Achievement of outputs and activities
The project reached all programmed outputs, both in terms of quantitative and qualitative
measures. The timelines at the community level were also achieved. In several cases, the
accomplishments exceeded the expectations and, in other cases, unplanned activities were
implemented. All countries developed institutional capacity through the following activities: i)
training of national and local personnel, community members and the delivery of equipment, ii)
formulation and implementation of guidelines on malaria control to develop legal capacities, and
iii) constitution of Steering Committees, National Committees and Local Committees. The local
committees were inserted in the structures of the Ministry of Health, using technical and
management experience of the malaria control programs. There was an appropriate inter and
intra institutional coordination.
The project used Echo Health approach, with four elements: i) a strategy of prevention an
integral control, emphasizing Integral Vector Control recommended by WHO, ii) multidisciplinary
and multisectorial approach, iii) community participation as the central axis of the control
activities, iv) equity, with priority in areas with the following characteristics: rural, predominance of
indigenous population, critical poverty and malaria persistence.
The approach used allowed for follow-up not only of operational activities but also expected
results. The rapid impact achieved at the start of the project allowed the model to have wide
support from the community and local workers. The evidence generated by scientific studies and
the systematization of experiences gave the project a scientific authority and credibility to
influence the formulation of malaria policies and decision makers.
Assessment of Monitoring and Evaluation Systems
One of the less developed components was the M&E (moderately satisfactory), since it did not
have a person or unit devoted exclusively to this activity. The M&E Design was only "moderately
satisfactory" and it had two documents of reference: i) The M&E system, described by project
document referred to the aims and activities developed with a qualitative approach, with few
goals or thresholds. This system worked adequately but it did not allow quantifying activities,
products and results. ii) The technical guide related to strategies of malaria control implemented
and to surveillance malaria cases. This system turned out to be excessive and very few
indicators were gathered in the base line and at the post evaluation end of the project. The most
important advances of the information system have been the development of a Geographical
Information System (GIS). These applications demonstrated visually the power that the GIS has
for the monitoring and evaluation of malaria situation and its determinants.
5

M&E
Plan
Implementation
(use
for
adaptive
management)
was
satisfactory.
The
implementation of the M&E system and the preparation of quarterly reports facilitated monitoring
of outputs and outcomes throughout the project implementation. In addition, the progress of
project performance was presented in the Regional Technical Committees and used for decision-
making. At the country level, targets were set, and the monitoring of all activities, deadlines and
objectives of the project were done regularly. Only a few indicators defined in technical guide for
monitoring malaria control strategies and malaria epidemiological situation were used to evaluate
the results and impact of the project.
Budgeting and Funding for M&E activities was "moderately satisfactory". There were enough
funds to assure M&E activities, but the budget for M&E activities was integrated in the budgets of
each activity.
Assessment of processes that affected attainment of project results
Officially, this project started in May of 2003, but at the country level, it started at different times
in each country, beginning from April 2004. The project implementation was delayed because of
several factors, including institutional arrangements and the adaptation of human resources
management mechanisms and financial management into the local and national realities.
Moreover, there was a delay in the appointment of Regional Coordinator (June 16th, 2004),
National Coordinators and focal points. Many administrative arrangements were done during
project implementation. According to stakeholders, the project had great flexibility which did not
only facilitate implementation of new malaria control strategies but also promoted adaptation of
the plan to the context of each demonstrative area.
As assessed by interviewees preparation and readiness achieved a "satisfactory" rating, but
due to the fact that coordination and institutional arrangements were not in place at the start of
the project, the score is only "moderately satisfactory". In addition, some actors considered that
preparation and readiness was insufficient, because many administrative arrangements and the
stakeholder involvement were made during project implementation. The preparation of the
project included evaluations on feasibility, base line, and contacting and involving stakeholders.
Country ownership and driveness was "highly satisfactory". At the national level, Technical
Committees were constituted with delegates from several institutions (health and environment
fundamentally), universities or researchers. At demonstration sites, technical local committees
were constituted with participation of municipalities and NGOs. At the community level, the
participation of delegates from the majority of the community organizations (Committees of
Malaria Control or Groups of Health) was high. Empowerment of community leaders and health
workers was high at the local level. In all levels, there was a very good stakeholder' participation.
Stakeholders´ involvement was "highly satisfactory".
The financial planning was evaluated as "highly satisfactory". Administration of funds was done
by the PAHO/WHO of each country. National Coordinators in 7 of the 8 countries were hired for
the management of the project. No other supplementary personnel were hired for the
management of the project, because existing resources were employed, and there was no need
to create additional administrative structures in any of the countries.
Recommendations and the way forward
1. In order to maintain the political and financial support, PAHO has to make advocacy to
Ministries of Health in order to declare the eradication of malaria as a medium-term goal
in Mesoamerica
6

2. PAHO could design new projects to replicate the strategy in other areas of high incidence
of malaria transmission and maintain the surveillance of localities where malaria
transmission was stopped. The Mesoamerican Health System (Plan Puebla Panama) is a
great opportunity to replicate the strategy in other areas.
3. Malaria Control Programs in every country keep track of the areas and towns as a
strategy of long term impact evaluation and homogenize and improve surveillance
systems, monitoring and evaluation of the Mesoamerican countries.
Lessons learned
It is possible to control or even eliminate malaria with environment-friendly methods and without
the use of persistent insecticides. The main condition is intersectoral and community
participation. The strategies needed for this kind of intervention, such as the control of mosquito
breeding sites, are easily adopted by the community. They also contribute to the empowerment
of the community and to change the conception about their participation in malaria control
activities.
Malaria control requires a multi methodological approach with the combination of interventions:
diagnosis and complete treatment (compliance), plasmodium reservoir elimination (active search
for asymptomatic and febrile persons), control of mosquito breeding sites with physical and
biological methods (larvae eating fishes), control of typical mosquito hiding places (house and
yard cleaning) and creation of barriers between people and mosquitoes (Insecticide Impregnated
Bed Nets).
Before implementing new demonstrative projects, development of comprehensive protocols is
required. They should include systematization of experiences, impact assessment, cost
effectiveness analysis, and surveillance and monitoring systems adapted to the strategy.
7

GLOSARY
ABER: Annual Blood Examination Rate
API: Annual Parasite Rate
COLVOL: malaria volunteer collaborators
CODODES: Development Community Councils
CEC: Commission for Environmental Cooperation of North America
CCAD: Commission for Environmental and Development for Centroamerica (Comisión
Centroamericana de Ambiente y Desarrollo)
DDT: dichlorodiphenyltrichloroethane
GEF= Global Environment Facility
GIS = Geographic Information System
IR= Increase Ratios
MCP: Malaria Control Program
M&E: Monitoring and Evaluation
MTE; Mid Term Evaluation
NAP: National Professional
PAHO: Pan American Health Organization
POPs: Persistent Organic Pollutants
SDT: Single Doses Treatment
TREDI: Treatment, Revaluation and Elimination of Industrial Mailing (Tratamiento,
Revalorización y Eliminación de Desechos Industriales)
UNEP: United Nations Environment Programme
WHO: World Health Organization
8

1. INTRODUCTION AND BACKGROUND
1.1. Project identification
Project title: POPS projects: Regional Program of Action and Demonstration of Sustainable
Alternatives to DDT for Malaria Vector Control in Mexico and Central America
UNEP Project Number: IMIS No.: GFL-2328-2760-4680, PMS No.: GF-4030-03-22
Responsible
Office:
Pan
American
Health
Organization,
PAHO
(Area
of
Sustainable
Development and Environmental Health, SDE) and executing countries: Ministries of health and
the environment of Belize, Costa Rica, El Salvador, Guatemala, Honduras, Mexico, Nicaragua
and Panama.
Project starting date: September 2003
Project completion date: December 2008 (with authorized extensions)
Reporting period: September 2003 December 2008
Total Budget ($US): 14.4936; UNEP contribution 7.165.000
1.2. Project rationale
Background
During the last decade Mexico and Central American countries have gradually discontinued DDT
sprayings for malaria vector control. Malaria, however, still poses a serious risk for the population
of these countries. The initial project duration was 36 months starting in August 2003 and ending
in July 2006. The Project was extended to December 2008 in order to enable continuous
collection of field data, to sustain project results, and to enable important international diffusion of
achievements.
This project complemented the "Contaminant-based" Operational Program No 10 and aimed to
"help demonstrate ways of overcoming barriers to the adoption of best practices that limit
contamination of the International Waters environment". The proposed activities were also
consistent with several provisions of the recently adopted Stockholm Convention on POPs, and
with the draft Operational Program on POPs under development. Seven of the participating
countries have already signed the POP´s convention.
Project aims and components
The overall objective of the project was to demonstrate that methods for malaria vector control
without DDT or other persistent pesticides are replicable, cost-effective and sustainable, thus
preventing the reintroduction of DDT in the region. Human health and the environment were
expected to be protected in Mexico and Central America by promoting new approaches to
malaria control, as part of an integrated and coordinated regional program. The establishment of
a regional network was expected to facilitate the exchange of best practices and lessons learned
among neighbouring countries. A major outcome was an increase in government and local
community awareness of DDT and other pesticides hazards to the environment and human
health, and adjustment of future behaviour regarding the use of persistent pesticides.
9

The project activities were organized under the following four components:
Component 1: Demonstration Projects and Dissemination. The objective was to implement,
evaluate, and disseminate the alternative strategies of malaria vector control without the use of
DDT, which were developed during the PDF-B phase. The main outcome planned was to avoid
future reintroduction of DDT or other persistent pesticides in national malaria control programs.
Component 2: Strengthening of national institutional capacity to control malaria without
DDT. The objective was to strengthen national and local institutional capacities to control malaria
with methods that do not rely on DDT or other persistent pesticides. The planned outcome of this
component was strengthened national capacities of malaria risk assessment, development of
analytical laboratory infrastructure, community participation and training regarding malaria vector
control and pesticide management.
Component 3: Elimination of DDT stockpiles. This component addressed the existing problem
of DDT stockpiles in six of the eight participating countries. All activities were to be documented
and management plans were to be put into place to prevent further accumulation of stockpiles of
pesticides.
Component 4: Coordination and Management. A regional coordinator was to be hired for this
project under terms of reference established by the steering committee. Each country was to
have a national coordinator. This component also included three annual meetings of the steering
committee, three regional meetings for planning and evaluation of activities, and three regional
annual reports.
10

2. OBJECTIVE AND METHODS
2.1. OBJECTIVE AND SCOPE OF THE EVALUATION
The objective of this terminal evaluation was to determine the extent to which the project
objectives were achieved, or are expected to be achieved, and assess if the project has led to
any other positive or negative consequences. The extent and magnitude of the project as well as
the possible future impacts are now documented. The evaluation also assesses project
performance, through comparison of planned project activities/outputs and actual results.
2.2. METHODS
The final evaluation was conducted as an in-depth evaluation using a participatory approach.
This is a multiple comparative descriptive study of cases (Yin R 1994). The evaluation was
carried out in several locations in four countries. In each country, three levels were visited:
National, department or provincial, and community (see Annex 1 for more details). Strategies to
assure quality assurance, validity, representativeness, validation of information sources and bias
control were defined (See Annex 1).
2.2.1. Sources of Evidence and Codes Used
The complexity and extension of the subject and the multi-theoretical approach chosen for this
study, presupposes methodological pluralism. Therefore, a combination of strategies and
research techniques were used. In this study, five different sources of evidence were used: four
of them involved qualitative methods (workshops to attend a presentation of final evaluation by
stakeholders, semi-structured collective interviews, documentation, and participant observation)
Comment [p1]: ?
and one quantitative method (archival records). In the following paragraphs, the Collective
Interview is described; the other procedures are described in detail in Annex 1.
Collective Interview
Semi-structured collective interviews with groups of actors or key informants were carried out at
three levels (see Annex 1). All the interviews were tape-recorded and transcribed. The main
questions of the collective interviews were organized into sections according to the project
evaluation parameters (see next section). Each section has questions formulated in order to
complete all the parameters suggested by UNEP (UNEP 2009). For each question four cells
were filled by the evaluator (see Figure 1).
1. Answers, which were evaluated by interviewers with four alternatives: Yes completely,
Partially, No at all, N/A (no applicable)
2. Evaluator notes
3. Existence of supporting documentation
11

Figure 1. Example of Interview formulary
TERMINAL EVALUATION DDT-GEF PROJECT DDT-GEF IN MEXICO Y CENTROAMERICA
Area programatica:
MaControl de la Malaria
Level:
Years in the
Participantes names :
Rol
project
Answers
n
of
tio
e
a
Yes - completely
Evaluator notes
nt
Questions
nc
Partially
e
?
te
m
No - not at all
is
u
x
c
N/A= no aplicable
E
do
A2. Relevance
were the project's outcomes consistent with the focal
A16
areas/operational program strategies and country priorities
Yes - completely
were the project's outcomes consistent with the country
A17
priorities
Partly
What was the contribution of the project outcomes to the wider
A18
portfolio of Operational Programme on POPs
No - not at all
Aditional Coments
2.2.2. Variables, questions and codes
The main variables of study and questions have also coded as follow:
Section
Item
Code
A. Attainment of
i) Effectiveness
A1 to A7
objectives and planned
ii) Relevance
A8 to A10
results
iii) Efficiency
A11 to a14
i) Financial resources
B1 to B4
B. Assessment of
ii) Socio-political
B5 to B7
Sustainability of project
iii) Institutional framework & governance
B8 to B11
outcomes
iv) Environmental
B12 to B14
B15 to b17
v) Recommendations
C. Catalytic role
C1 to C5
i) Soundness and effectiveness of the
D1
methodologies
ii) Legal, institutional, technical and
D2
D. Achievement of
financial capacities and mechanisms
D3
outputs and activities
iii) Produced outputs have the weight of
scientific authority / credibility of project
outputs
D4
iv) Delivered outputs
i) Design
E1 to E12
E. Assessment of
ii) Implementation plan
E13 to E16
Monitoring and
iii) Budgeting and Funding for M&E
E17
Evaluation Systems
iv) Long-term Monitoring
E18
F. Assessment of
i) Preparation and readiness
F1 to F5
processes that affected
ii) Country ownership/Driveness
F6 to F12
attainment of project
iii) Stakeholder involvement
F13 to F21
results
iv) Financial planning
F22 to F23
F24 to F29
v) Structure and organization
A. "PO": participant observation;
12

C. Documents:
Document
Code
Trimester Report
TR
Mid Term Evaluation
MTE
Ramirez E, Henao. Final Report, December 2008
FR
Ramirez
et.
al,
(2009).
DDT-free
Malaria
Control
in
Ramirez et.al 2008
Mesoamerica: focalized control and management of breeding
sites as basic strategies.
These codes will be quoted throughout the text to ensure that the source of information is clear. The
quotations of documents and interviews were translated from Spanish to English by the author.
2.2.3. Study Population and Units of Analysis
The project constituted of 202 localities (see Annex 2). For the evaluation, three levels were
visited in each of the four countries. In the current study, the unit of observation is the malaria
demonstrative project of each country and in each of its three levels:
1. Central Level: Headquarters of PAHO and Ministry of Health (MOH).
2. Local Level: Headquarters of demonstrative projects.
3. Community Level: localities directly involved in the project.
The following table presents the countries, places and levels where the interviews took place,
and number of collective interviews held:
COUNTRY
PLACE AND LEVEL
Num. interviewees Code
Guatemala
Regional
2
R
Mexico
National
4
MN
Chiapas Estate
8
ME
Neguatzotcoyol community
4
MNEG
Nueva Macedonia community
3
MNM
Guatemala Total
National
3
GN
Peten Department
4
GP
Ixacan Department
2
GI
La Felicidad community
5
GF
Sta. Maria Dolores community
5
GM
Costa Rica Total
National
3
CRN
Talamanca Municipality
3
CRT
Matina community
2
CRM
Panama Total
National
3
PN
Bocas Province
2
PB
Puente Blanco community
8
PT
Las Tablas community
5
PPB
2.2.4. Data Analysis
Four methods of analysis were used according to the type of evidence and the variables involved
in the study:
13

1. Analysis of collective interviews. For each section mean and standard deviation was
computed. In order to do this the categories of answers were transformed in ordinal
values: yes completely = 2, yes partially = 1 and no at all = 0. Non applicable was not
quantified.
A score of the level of satisfaction was computed for each question, country and place.
The procedure was:
Compute observed values for every category: sum the values obtained in every
question that forms part of the category.
Compute expected values: sum all the expected values in every question.
Compute the score by multiplying the observed values by 6 and dividing by expected
values. See the following example:
Observed
Expected
VARIABLES
Score
values
values
A Attainment of
29
30
(29 * 6) 30 = 5,80
objectives
A1. Effectiveness
15
16
(15 * 6) / 16= 5,63
A1
2
2
A1A
2
2
A2
2
2
A3
1
2
A4
2
2
A5
2
2
A6
2
2
A7
2
2
The quantitative scores were transformed into levels of satisfaction as follows:
Highly
Satisfactory
Moderately
Moderately
Unsatisfactory Highly
Satisfactory
Satisfactory
Unsatisfactory
Unsatisfactory
5,5 to 6
4,5 to 5,4
3,5 to 4,4
2,5 to 3,4
1,5 to 2,5
0 to 1,4
Comment [p2]: Shouldn't this be 2,4?
For sustainability sub-criteria on each of the dimensions of sustainability of the project outcomes
was rated as follows.
Highly Likely
Likely (L)
Moderately
Moderately
Unlikely (U)
(HL)
Likely (ML)
unlikely (MU)
5,5 to 6
4,5 to 5,4
3,5 to 4,4
2,5 to 3,4
0 to 2,5
Comment [p3]: 2,4?
Highly Likely (HL): There are no risks affecting this dimension of sustainability.
Likely (L): There are small risks that affect this dimension of sustainability.
Moderately Likely (ML): There are moderate risks that affect this dimension of sustainability.
Moderately Unlikely (MU): There are significant risks that affect this dimension of sustainability.
Unlikely (U): There are severe risks that affect this dimension of sustainability.
14

2. Analysis of document contents. The content of each document was analyzed using a
matrix of content analysis. The most relevant findings were classified and grouped
according to the research variables.
3. Meaning categorization. The contents of interviews (evaluator notes) and meetings
records were classified and grouped by the same procedure as the document analysis.
4. Statistical analysis. Epi-Info 6.04 was used to process the quantitative data.
5. Epidemiological Data. For a descriptive analysis of epidemiological, service production
and performance data, the number of events, percentages, rates and ratios were used.
Increase Ratios (IR) were calculated to demonstrate increase or reduction (Dever A
1991).
The Annual Parasite Rate (API) was standardized using the case detection effort (ABER) for the
year 2004 (Roberts D et al. 1997) (See Annex 3):
15

3. PROJECT PERFORMANCE AND IMPACT
3.1. Attainment of objectives and planned results
The attainment of objectives and planned results is "satisfactory", although the objective 3 could
not be obtained completely (elimination of DDT stockpiles) (Table 1). The third objective
involved an inventory of the existing DDT reserves, the transport of reserves to places of
accumulation in each country, repacking of the material as required, and transportation and
elimination of the DDT reserves. The first two activities, under the responsibility of the
participating countries, were fulfilled. However, there was breach of the contract by the company
contractor concerning the three final activities. The majority of the survey respondents rated
partially to the attainment of the objective 3.
Mexico received the lowest rating on the attainment of objectives (moderately satisfactory). This
was mainly due to that the interventions in Chiapas's State (which was selected for the
evaluation) was not developed fully and only during the last months of 2008 the activities were
increased. In Costa Rica and Guatemala the attainment of objectives was "highly satisfactory",
but in Panama was rated as only "satisfactory".
Two components of this parameter, effectiveness and efficiency, obtained a "satisfactory" rating.
Relevance was rated "highly satisfactory". Guatemala has the best results with a "highly
satisfactory" rating for all of the three components and in the majority of the six regions
evaluated.
Regarding effectiveness, the global score was "satisfactory". Mexico scored a "moderately
satisfactory" rating, due to the low scores achieved at state and community levels. Although in
Panama, the total score achieved was "satisfactory". The rest of the countries scored "highly
satisfactory" ratings. Efficiency had similar trends than effectiveness. In Relevance, the global
score achieved was "highly satisfactory", but in Mexico it was only "satisfactory".
16

Table 1. Evaluation of attainment of objectives and planned results. Final Evaluation.
Mesoamerica 2008-2009.
PLACE
Attainment of
Effectiveness
Relevance
Efficiency
objectives
TOTAL
S (5,24)
S (5,11)
HS 5,52
S (5,26)
Regional
HS 5,8
HS 5,63
HS 6
HS 6
Mexico Total
MS 4,13
MS 3,88
S (4,8)
MS 4,07
National
S 5,2
S 5,25
S 5
S 5,25
Guajaca State
MS 3,8
MS 3,38
MS 4
S 4,5
Neguatzotcoyol community
MU 3
MU 2,25
S 4,5
MU 3
Nueva Macedonia community
4
3,75
HS 6
MS 3
Guatemala Total
HS 5,8
HS 5,8
HS 6
HS 5,67
National
HS 5,6
HS 5,63
HS 6
S 5,25
Peten Department
HS 5,57
HS 5,57
HS 6
S 5,25
Ixacan Department
HS 6
HS 6
HS 6
HS 6
La Felicidad community
HS 6
HS 6
HS 6
HS 6
Sta. Maria Dolores community
HS 6
HS 6
HS 6
HS 6
Costa Rica Total
HS 5,62
S 5,4
HS 5,63
HS 6
National
S 5,2
S 4,5
HS 6
HS 6
Talamanca Municipality
HS 6
HS 6
HS 6
HS 6
Matina community
HS 5,67
HS 6
S 4,5
HS 6
Panama Total
S 5,17
S 5,09
HS 5,5
S 5,14
National
S 5
S 4,88
S 5
S 5,25
Bocas Province
S 4,5
S 4,29
S 5
S 4,5
Puente Blanco community
HS 6
HS 6
HS 6
HS 6
Las Tablas community
HS 5,67
HS 6
HS 6
S 5
Score
Highly
Satisfactory
Moderately
Moderately
Unsatisfactory Highly
Satisfactory
(S)
Satisfactory
Unsatisfactory (U)
Unsatisfactory
(HS)
(MS)
(MU)
(HU)
5,5 to 6
4,5 to 5,4
3,5 to 4,4
2,5 to 3,4
1,5 to 2,4
0 to 1,4
3.1.1. Effectiveness
The first objective (question A1) received the highest score on effectiveness with a mean of 2
(yes - completely). Only in Mexico, the impact (Annex 2, Table 1, questions A1A) obtained a
rating of yes - partly (score=1), the rest of the countries evaluated had a score of 2 (yes-
completely).
The second objective received a mean rating of 1,5 in Mexico and Panama, and 2 in the other
countries. Objectives 3 and 4 received the lowest ratings. For objective 3, regional, Mexico and
17

Guatemala obtained a score of 1, which represented a partial achievement of the objectives, and
Costa Rica obtained a score of 1,5.
Three questions assessed to what extent the results of this project have informed regional
(question A5), national (question A6) or international (question A7) processes. Regional refers to
Mesoamerica region and International means other regions of America or other continents. At the
national level, Mexico and Costa Rica obtained a good performance score (more than 1,5), and
the rest of the countries obtained scores of 2 (completely). At the regional level, Mexico and
Panama rated a score of "partially" on performance (1 and 1,25). At the international level, most
countries obtained a score higher than 1,5 (intermediate between partially and completely), with
an exception of Panama with a mean of 2 (completely).
The following sections present a description of the strategies applied for every component.
Component 1: Demonstration Projects and Dissemination
This component was designed as the most important and complex component of the project,
thereby receiving most of the resources, as well as great deal of institutional and community
effort. It was successfully implemented in 202 demonstration communities and 52 municipalities.
A total of nine demonstration projects were launched, one in each country, that directly benefitted
a total of 159,018 inhabitants and indirectly a population of 6, 845,000 people. This accounts for
29% of the population living in high risk malaria areas of Mesoamerica (FR).
All countries have adopted technical alternatives of vector control at the demonstrative areas, not
only without using DDT, but also without the use of persistent insecticides. This is why the
component is rated as "highly satisfactory". Only Panama, sprayed PH 40% Sumithion in one of
the demonstrative communities (Barranco Montańa Adentro) because of the presence of a
malaria outbreak at the beginning of the project. Once the epidemic was under control, the use
was ceased (MTE). The use of insecticides was restricted to the Insecticide Treated Nets in
Guatemala and Nicaragua, due to the presence of Global Fund Projects, which was providing
them (Impact evaluation).
Positive and negative effects
All the interviewees described positive effects of the implementation of malaria control strategy.
The most important positive effects are related to: impact on other diseases, inter sectorial
participation; scientific publications; no use of insecticide for the control of malaria; high
community mobilization and participation; improved performance of the malaria control program
and of the vector control workers; reduction of costs and improvement of the cost effectiveness
of the interventions, and community self-replication. There were no negative effects.
Impact
There was a considerable reduction in the number of cases and the Standardized Annual
Parasite Rate in all of the 202 demonstration communities. It registered a reduction of 63% of
malaria cases and APIs decrease from 15,9 to 8,36 cases per thousand inhabitants from (Graph
1) (Ramirez et.al. 2008).
18

Graph
1.
Annual
trend
of
malaria
transmission
in
demonstrative
localities.
DDT/PNUMA/GEF/OPS Project, Central America, 2004 - 2007.
When we compare the percentage of reduction in malaria cases in the countries, with the
demonstrative localities, the average reduction in the countries was 25% while in the
demonstrative localities it was 63% (Table 4). Only the demonstrative localities in Costa Rica
experienced an increase in malaria cases, which was due to a new locality that was included in
the project in 2007 (Estrada, Matina) and which was affected by an outbreak (Ramirez et. Al,
2008).
Table 2. Number of malaria cases and the percentage of malaria case reduction in the countries,
and demonstration localities. DDT/PNUMA/GEF/OPS Project. Mexico and Central America, 2004
2007.
Number of Cases in the Country
Number of Cases in Demonstrative localities
Countries
2004
2006-7*
reduction %
2004
2007
reduction %
Belize
1,057
844
20%
376
128
66%
Costa Rica*
1,289
1,223
5%
99
112
-13%
El Salvador
76
49
36%
26
0
100%
Guatemala
35,349
31,093
12%
265
92
65%
Honduras
14,813
11,561
22%
521
105
80%
Mexico
6,861
2,514
63%
902
456
49%
Nicaragua*
5,095
2,514
51%
94
16
83%
Panama*
3,406
1,281
62%
156
5
97%
Total
67,946
51,079
25%
2,439
914
63%
Source: Pan American Health Organization, Health Analysis and Statistics Unit. Regional Core
Health Data Initiative; Technical Health, 2007
* Cases reported in year 2007
Among the countries visited, several communities registered zero malaria cases during the past
two years (2007-2008). Moreover, in all of the departments and provinces, where the
19

demonstration areas were located, incidences were reduced near the API of 1, which is
considered the standard to qualify as the elimination of malaria. Matina is an example of a rapid
control of an epidemic without insecticides; the number of cases reported in 2006 was 286, in
2007 there were 99 reported cases, and in 2008 only 12. (A1A.CRNAC). In Panama in 2008, 90
% of the controlled localities registered zero local transmission (autochthonous) cases of malaria
(A1A.PNAC, A1A.PNB, A1A.T).
A similar situation prevailed in Guatemala, where both of the demonstration communities visited
presented no cases in 2008. In Guatemala, the factors that contributed to the success were: "the
project was implemented using health promotion of health principles through the structures of
local power (COCODE) and involving local stakeholders (municipalities), empowerment of
community through analyzing data, organization of action community groups, reinforcing the
governance and leadership of Ministry of Public Health. There was a permanent presence of the
MSP civil servants in supervision" (A1GNAC).
There was a positive impact, not only in demonstration communities, but also in all
demonstration areas. Actually, in Guatemala there was considerable reduction of malaria cases
in the whole country (A1A.PGAC), particularly in the three demonstration areas, due to the
alliance with the Global Fund Project and Health Action Project. In all three districts of Bocas Del
Toro in Panama, cases reduced from 700 in 2004 to 13 cases in 2007 due to the influence of the
project (A1A.PNAC, A1A.PNB).
Mexico was the only country in which the cases were not reduced: the number of cases
remained the same in 2008 in comparison to 2007 (A1A.MNM). Chiapas State reported only a
reduction of 3% of the cases in 2008 (A1AMNAC). The reasons for this are: i) the presence of
hurricane Stan in 2004 (A1A.MN); ii) the low efficiency of control of the breeding-sites in localities
surrounded by rain forests due to the difficulties in identifying and controlling the breeding-sites
(A1B.MN), iii) the presence of many new workers who had not been trained about the application
of this strategy, iv) the low performance of administration of treatments and the work with
communities (many workers were about to retire and there was a lack of discipline), v) the
reluctance of people to give blood samples for diagnosis and to work in community activities, vi)
the lack of a systematic supervision in all levels (A1ME).
Impact on other health problems
All interviewees agreed that the project had transformed towns (A1B.I). All the towns visited
were clean, without rubbish or weeds in the open areas and the houses visited were clean and
tidy. It can be stated that all towns in the demonstration areas are now examples of healthy
communities (PO).
Several actors considered that through this strategy, other types of illnesses have also been
controlled, for example parasitosis (A1B.I) and diarrhoea. There has also been a decrease in
mosquito stings: "before it was impossible to stand near lakes". In Guatemala, leaders thought
that due to the "clean houses" strategy, there were positive impacts on other vector born
diseases like dengue (Aedes) and dermatological diseases (scabies). Other impacts include the
reduction of common house fly, the improvement of community safety (because it is possible to
see intruders from a longer distance) (A1B.GF), basic environmental sanitation and
improvement of the streets (A1B.GM, A1BGN).
Several leaders of the towns visited, thought that the strategy has improved general health
issues, such as the reduction of malnutrition, tuberculosis and child/maternal mortality, due to
20

the training, health education and workshops (A1B.MNM). There has also been a decrease in
diarrhoea cases because of latrine building and safe water actions; both results of strengthening
of community organizations (A1B.PPB). In Panamá leaders said that "although the interest was
focused in malaria control, there was also interest in safe water and latrines. People were very
interested and motivated to follow the program (A1B.PB)". The generalized opinion is that the
empowerment of the community allowed the interventions to extend to other health problems
(A1B.PPB).
The floods that occurred in Panama, Guatemala and Costa Rica (years 2008 and 2009), were
followed by a very strong reaction of the community (elimination of mosquito breeding sites and
refugees - cleaning houses and patios- and active search for febrile patients), which enabled the
prevention of malaria outbreaks without the use of insecticides. It has also prevented other
illnesses related to these climatic phenomena (OP). In Panama, the population of a locality
(Puente Blanco) has acknowledged that as a result of the project, there is "more working union,
more people come to visit us, Europeans come to see this native community, because there is
no malaria or other illnesses" (A1B.PB).
Intersectorial participation
The strategy allowed not only to reduce the burden of the disease, but also to promote
community participation and collaboration between governmental institutions, communities and
other actors. These multiple actors were involved in the organization, planning and execution of
sustainable health promotion activities.
In Panama, the project involved local leaders and mayors (A1b.R). Another relevant aspect was
the participation of the municipalities which collaborated in malaria control through public policy
regulations, environmental sanitation, infrastructure and logistical support to the communities
involved (A1B.MN). In addition, several universities were involved in research projects.
There was a synergy between GEF-DDT project and other projects such as those of the Global
Fund. In these cases, the project strategy was to integrate the efforts of both institutions
(A1B.GP, A1B.GI). Alliances were also made with the education sector (A1B.GP, A1B.GI).
The experience with the project showed that the municipalities play a very important role in the
implementation of malaria control strategies. The reasons are the following: i) they can act in a
wide range of levels and create a proper environment for successful local interventions; ii) they
can support community participation and intersectorial collaboration iii) they can offer funding for
infrastructure, logistical support and personnel; iv) the municipal governments can act as agents
for structural change (social, cultural and physical change) through the formulation of public
policies.
The experience also showed the potential of the private sector to act as an important partner in
human development processes, specifically, in the area of health promotion. The intervention of
the private sector helped rationalize the use of resources and to improve the coordination of
actions for malaria prevention and control. Their participation in the project strengthened the
social responsibility and commitment with the health of the population and the conservation of
the environment.
Scientific publications
21

Eight scientific articles were prepared and have been sent to scientific journals for publication
(A1.R, FR). Documents with systematization of the experiences were distributed in several
meetings, workshops or scientific congresses (FR, A1CRN).
Community participation
The inhabitants learned to organize themselves in order to cope with health problems, not only
malaria. People got involved with the project (A1B.PNAC). There has been a visible change in
the level of participation of the community, seen through the different attitudes regarding the
protection of the family (A1B.PT). The opinion is that political changes (in the government or the
local authorities) will not affect the application of the strategy and the model developed by the
project (A1B.PNAC).
The opinion of majority of the interviewees was that the reduction of malaria cases and mosquito
stings has contributed to the credibility of the strategy and the improvement of community
participation (A1B.MN). This opinion is generalized in all the areas visited (OP).
In the state of Chapas (Mexico), there were political problems associated with the zapatist
guerrilla, who refused to take part in the malaria control activities, pointing out that this was the
state's responsibility. However, the project was restarted in several areas, where the population
is again taking part in the project activities (A1B.ME).
The project strengthened local organizations. For example it helped Development Community
Councils (COCODES) to focus on public health and health policy subjects, because they had
been acting with other topics. The project allowed these groups to increase their power and have
influence over the population (A1B.GN).
Improvement of performance of the malaria control program and of the vector control workers
In Mexico, the interviewees at the national level thought that the project contributed to the
effectiveness of the national malaria and dengue control programs because it brought resources
to the control of these illnesses. On the other hand, the health workers benefited of the
experience in community work and new strategies of vector control, which are the base of
dengue control (A1B.MN). The model became an example to other programs like immunization
(A1B.ME).
Model and approach
The terminal evaluation confirmed that all countries in the demonstration areas have adopted
alternative methods for malaria vector control, not only without the use of DDT, but also without
using persistent pesticides. A very important aspect stressed by the mid term evaluation was
related to the ecological and systemic approach applied, which is characterized by five elements:
A prevention strategy and integrated control based on epidemiological models of the
health sector.
A multidisciplinary and multisectoral approach when involving the environment and
education sectors with the health sector.
Community participation as central axis of the control activities.
Equity in prioritizing rural areas with a majority of indigenous populations in critical
poverty and the persistence of malaria.
22

Combination of control methods according to the Roll Back Malaria initiative adapted
to the local conditions.
Strategies implemented
The project used a combination of control methods that covered all the necessary steps to
control malaria, surpassing the practices focused on the use of insecticides for the control of
adult mosquitoes (MTE). The methods used are summed up in the table below:
Table 3. Implemented Malaria Control Interventions DDT/PNUMA/GEF/OPS Project.
Mexico and Central America, 2004 - 2007.
EFFECT
CONTROL INTERVENTIONS
1. Parasite Elimination
Early diagnoses and prompt and complete treatment
Compliance of treatment supervision: counselling for complete treatment,
graphic prescriptions, direct observed treatment, treatment completeness
supervision and blood smear control at the day planned to receive the last
doses of treatment (day seven o fourteen).
2. Reservoir Elimination Single Dose Treatment SDT (In Mexico)
Treatment for household contacts of malaria cases
3. Contact reduction
Insecticide impregnated bed nets
between humans and
Nets on doors and windows
vectors
Repellent trees
White washing of houses (painting with lime)
4. Elimination of
Physical control of breeding sites: filling and drainage.
breeding sites
Biological control: native fishes and sporogonic bacteria ( in Guatemala,
Costa Rica, Nicaragua and Honduras)
Chemical control: ethoxilated alcohol against Anopheles albimanus, (in
Mexico)
5. Elimination of vector
Cleaning houses and backyards
refuges and reduction of
Vector refuges elimination on household surroundings
attractors
Personal hygiene promotion
Source: Adapted from Najera et al (1992:14)
The countries have made adaptations to the control strategies in terms of conditions, resources
and national capabilities, which has helped the replication in diverse scenarios. The
characteristics of the control strategy used, coincided with the technical elements of the Global
Malaria Control Strategy (WHO 1993) and the Roll Back Malaria initiative (Academia Nazionale
dei Lincei. 1998;Alnwick D 2001). These can be summarized in the following aspects:
In Guatemala, where considerable impact was registered, a combination of majority of the
strategies showed in Table 3 was implemented. The use of larvae eating fish was evaluated and
documented, obtaining a high efficiency and efficacy and allowing the reduction of the periodicity
of the malaria cases. In places called "aguadas" (places for watering farm animals and cattle),
the frequency of malaria cases in the community was reduced from once every month to once
every three months.
23

Risk approach and focalization of the interventions
The first step of the stratification allowed identifying the towns with higher index of historical
transmission and persistent malaria (that were prioritized as demonstration areas of the project).
The second step identified the malarious houses (defined as houses with the presence of one or
more cases in the last two years) and the repeated cases. The interventions directed to improve
the hygiene of houses (cleaning houses and patios and painting houses with lime) and personal
hygiene, have been focused on malarious houses. The treatments of malaria cases and family-
contacts of malaria patients with Single Dose Treatment (SDT) were also concentrated in the
malarious houses. In this way, a more cost-effective intervention was achieved (MTE).
In Guatemala, a new methodology of stratification was developed in order to improve the
interventions of Global Fund Project, with success (PO, MTE).
This strategy of stratification was used to focus the interventions to a few localities of high
transmission and improve the efficiency and the efficacy of the control strategy. This approach
was applied in all demonstration areas, but modifications have been made in some places
(MTE).
Opportune diagnosis and prompt treatment
At the preparation phase, early diagnosis and prompt treatment was not considered a priority
intervention, but when the Technical Guidelines were formulated (Mendez J, Betanzos A, &
Tapia R 2004), there was no agreement on what scheme to use, so each country used its own
one.
During the mid term evaluation, one of the problems identified was that several countries used
schemes not recommended by PAHO/WHO. Actually, for the year 2004, only Belize and
Nicaragua used schemes approved by PAHO: chloroquine for three days and primaquine for 7
days or chloroquine for three days and primaquine for 14 days. The other countries (El Salvador,
Costa Rica and Guatemala) used primaquine for less days and with smaller doses. Mexico had
been using the single dose treatment (SDT 3x3x3) schemes before 2004 (MTE). Due to the
project influence, Guatemala, Honduras and Panama changed their schemes for those
recommended by PAHO during 2007 (Ramirez et.al 2008).
There was an important progress in the opportune diagnosis and prompt treatment, especially in
Guatemala where in 2004, only 15% of the treated cases went through laboratory diagnostics.
The strategy implemented in the participating countries allowed the patients to be treated within
the first 24 hours, which is one of the goals of the Roll Back Malaria Initiative (MTE) (Alnwick D
2001).
Locating inhabitants with fever was a central aspect of the strategy that aimed at improving
opportunities and coverage of diagnostics and treatment of malaria patients. However, there was
no uniformity among the countries over this procedure at the project mid term. Similarly, there
were no guidelines on the follow up of treated patients, thus each country had its own policy at
project mid term. At the end of the project, all countries adopted the procedure of active search
for febrile patients (MTE, PO).
Reservoir Elimination
Another important element of the model was the strategy to treat malaria infected patients thus
preventing the development of the parasite, particularly of Plasmodium vivax. All countries
24

adopted the strategy of locating and treating symptomatic or asymptomatic cases of malaria in
contacts with positive cases, including Mexico (MTE). Mexico had adopted an explicit strategy,
Single Doses Treatment (SDT) 3x3x3. Costa Rica modified this strategy and applied it only for
one year (2005). Nicaragua implemented SDT as part of a multicentric study to evaluate the
efficacy (MTE).
Vector control Interventions
Actions carried out for larvae control were mainly physical means, such as mud removal,
cleaning of edges, and filling or drainage of the breeding sites. As biological means, native larvae
eating fishes were used extensively; sporogonic bacteria were used in some areas of Nicaragua,
Honduras and Panama. In El Salvador, Temefos® granulated chemical was used once in 2005
(Ramirez et. al 2008).
As a means of controlling densities of adult mosquitoes, the localities implemented strategies
related to the reduction of mosquito attracts through cleaning of houses and patios and
eliminating refuges in surrounding areas. To a lesser degree, whitewashing of houses was
performed, nets were installed on doors and windows, and repellent trees were planted.
(Ramirez et. al 2008) (Graph 1).
Graph 1. Number of demonstrative localities according to Developed Interventions.
DDT/PNUMA/GEF/OPS Project. Mexico and Central America, 2004 - 2007. (N=202)
25

Component 2. Building Capacity
The project reinforced the national programs of malaria control in the Mesoamerican countries.
From 2004 to mid 2008 it maintained a technical training program in epidemiological surveillance,
entomology, social participation, participative planning and evaluation of risk factors due to
exposure to DDT and other POPs, geographic information systems (GIS) and other technical
areas complemented with guidelines and manuals generated by the project's experience (FR).
With resources provided by the project, many documents were edited and published in order to
support training, exchange and dissemination programs as part of the strategy to strengthen
institutional and communal capabilities for malaria vector control without using DDT. Fifty sets of
educational materials were prepared by the participating countries with resources from the
regional component (FR).
Through meetings, consultation meetings, and training workshops, it was possible to share
information, knowledge and experiences among the countries that facilitated the decision making
processes. Reports from the countries recorded 888 meetings, workshops or seminars. These
were financed with project resources and held from December 2003 to December 2008. About
75% of these meetings were held at the local level (668 meetings), within or very close to the
communities. The rest were held at the regional or national level (Table 7). A total of 21,306
participants were reported to have participated, from which 54% where community personnel
(11,459) and the rest institutional personnel, all from the education, environment and health
sectors, including municipal mayors and other key stakeholders (FR).
This combination of strategic actions enabled the revitalization of national malaria programs and
the placing of malaria issues on the political agendas of the participating countries. The joint
action of the health and environment sectors was also a benefit, as well as the performance of
other external cooperation projects which were investing in malaria programs supported by the
model proposed by this project (FR).
Table 4. Meetings and participants per country. Project DDT/UNEP/GEF/PAHO. 2003-2008
Num. of
Countries
meetings
Type of meeting
Level
Num. of participants
Total
1
2
3
Local
National
Regional
Community Institutional
Belize
124
1
17
106
89
35
0
591
676
1267
Costa Rica
28
3
4
21
23
5
0
488
468
956
El Salvador
79
1
2
76
60
19
0
757
721
1478
Guatemala
231
3
47
181
186
45
0
4246
2828
7074
Honduras
63
2
5
56
50
13
0
261
818
1079
Mexico
181
3
29
149
161
20
0
2917
1764
4681
Nicaragua
94
0
16
78
62
32
0
1284
1332
2616
Panama
47
1
5
41
37
10
0
623
305
928
Regional
Component
41
27
7
7
0
0
41
292
935
1227
Total
888
41
132
715
668
179
41
11,459
9,847
21,306
Source: Final Evaluation
Does not include meetings of the Steering Committee (5) and the Regional Operational Committee (3)
Type of meeting: (1) Inter-governmental meeting (2) Expert group meeting (3) Training workshop-seminar
26

The most relevant activities for the development of institutional capabilities were as follows:
Malaria control training
The technical guide was prepared through participation of the countries and PAHO experts. One
thousand copies of the guide were printed and distributed to the eight participating countries, to
other international events and to the strategic partners of the project. Training courses and
workshops have been carried out using the guide's contents.
Strengthening of the capability of toxicology laboratories and environmental studies
The project strengthened capacity of the network of gas chromatography laboratories in
Mesoamerica, with technical capability for monitoring and evaluating environmental and health
risks due to the exposure of POPs in the region. It was possible to build the Central American
laboratories' network. The regional reference laboratories (the Toxicology Laboratory of the
University of San Luis Potosí, Mexico and the Regional Institute of Toxicology of the National
University of Heredia, Costa Rica), allowed the exchange and development of the inter-
laboratory capabilities and the analysis of DDT compounds from soil, sediment, fish and blood
(FR) samples.
The laboratories were equipped to develop risk assessments of the exposure to residual DDT.
Two national laboratory professionals from each country were trained on gas chromatography
and preparation of protocols, site selection and collecting environmental and biological samples.
Thus, the personnel was trained for the evaluation of other POPs (FR). Therefore, it was
important that the network of laboratories analyzed the samples and prepared the country reports
in a reliable manner, which were then used for drafting the report for Mesoamerica and the
publication "Environmental Health Risk Assessment of DDT in Mexico and Central American
Countries" (FR).
Component 3: elimination of DDT reserves
During the project's PDF phase (2000-2001) it was estimated that the stockpiles of DDT in the
eight participating countries were 135 tonnes. In order to have a more accurate assessment of
these reserves, an inventory was carried out during the project's first year in co-operation with
the national authorities and the ministries of health and environment. The inventory revealed a
stockpile of 136.7 tonnes of DDT and 64.5 of other POPs (toxaphene, chlordane HCB, aldrin,
dieldrin, and mirex) (Table 5).
Based on the information provided by FAO, 15 specialized companies were invited by PAHO
through a public biding to implement the component of elimination of DDT reserves. SEMTREDI
was selected and contracted in March 2007 for $ 500,000, in order to repackage, transport and
eliminate 200 tonnes of DDT and other POPs, as well as to advise the countries to prepare the
transit documents.
On October 2007 60 tonnes of POPs in El Salvador were repackaged and in February 2008, 55
tonnes of DDT in Mexico (FR). In Costa Rica, Guatemala and Panama the DDT reserves were
collected, concentrated and stored in places which did not comply with international standards,
and in Guatemala, this resulted in a high risk of contamination (PO).
27

Although attempts were made, it was not possible to export and eliminate the stockpiles of DDT
and other POPs. There were many issues that affected the achievement of this component:
Between July and August 2007 the European Union enacted new regulations in relation
to notification and request of transit.
Weak support of TREDI to advise the countries to prepare transit documents.
TREDI requested an increase in the contract amount.
By October 25, 2008 none of the countries had received transit authorization from the
European Union. On October 29, 2008, the Program Officer from the United Nations
Environment Programme (UNEP/GEF) informed PAHO "that, as of now, it is impossible
to import chemical waste to countries of the European Union".
Therefore, UNEP/GEF and PAHO are analyzing alternatives to adequately solve the
implementation of this key component.
28

Table 5. Regional Program of Action and Demonstration of Sustainable Alternatives to DDT for Malaria Vector Control in
Mexico and Central America. Updated Inventory of DDT and other POPs in Mexico and Central America. Update period: June
2004 to August 2005.
DDT*
Container conditions
Transportability
Stockpile
Unknown
Minor
Major
Country
sites
10%
75%
94.20%
100%
concentration
Subtotal
Good
damages
damages
Unknown
Yes
No
Unknown
Belize
1
13.000
13.000
13.000
13.000
Costa Rica
1
0.028
4.060
4.533
8.621
0.028
8.593
0.028
8.593
El Salvador
1
4.672
4.672
4.672
4.672
Guatemala
4
15.058
15.058
0.150
14.907
0.150
14.907
Honduras
1
3.539
3.539
3.539
3.539
Mexico
53
42.043
45.269
87.312
58.055
13.137
11.385
4.735
71.192
11.385
4.735
Nicaragua
2
0.003
0.003
0.003
0.003
Panama
1
4.545
4.545
4.545
4.545
Total
64
4.573
59.103
4.672
64.859
3.541
136.749
67.274
13.316
47.885
8.274
80.590
47.885
8.274
Percentage
3.34%
43.22%
3.42%
47.43%
2.59%
49.20%
9.74%
35.02%
6.05%
58.93%
35.02%
6.05%
Other POPs*
Container conditions
Transportability
Stockpile
Minor
Major
Country
sites
Toxaphene
Chlordane
HCB
Aldrin
Dieldrin
Mirex
Subtotal
Good
damages
damages
Yes
No
Belize
1
0.008
0.008
0.008
0.008
Costa Rica**
2
0.120
0.002
0.122
0.002
0.120
0.002
0.120
El Salvador
5
36.636
7.802
1.814
46.252
46.252
13.776
32.476
Guatemala
Honduras
1
12.490
12.490
12.490
12.490
Mexico
Nicaragua**
4
5.640
0.003
0.004
5.647
0.007
5.640
5.647
Panama
Total
13
42.276
12.493
7.802
1.814
0.120
0.014
64.519
0.007
18.140
46.372
31.923
32.596
Percentage
65.52%
19.36%
12.09%
2.81%
0.19%
0.02%
0.01%
28.12%
71.87%
49.48%
50.52%
29

Component 4. Project's management and coordination
The project was implemented by UNEP and executed by the Pan American Health
Organization (PAHO) under the overall responsibility of the Director, Division of Health and
Environment and Ministries of Health. The UNEP Division of GEF Co-ordination, in
association with PAHO, monitored activities undertaken during the execution of the project.
The Director, Division of Health and Environment of PAHO was responsible for maintaining
systematic overview of the implementation of the project through monthly project monitoring
meetings or consultation and preparing to monitor reports.
Regional Coordination provided an accurate and permanent technical communication among
those involved in administering, managing and implementing the project, verifying the flow of
communication to all participating communities.
Human resources
For the coordination and management of the project, PAHO hired a regional coordinator with
office in Guatemala and seven national coordinators (NAPs) located at PAHO's offices in
each country (except for Costa Rica, where the duties were undertaken by an international
consultant form SDE/PAHO upon decision from the national authorities). Each country had a
national focal point for this project, appointed by the executing ministry.
NAPs solved the effects derived from the instability from institutional national focal points and
provided continuity to the project. They also provided technical cooperation to the countries
to develop community participation, harmonize the linkage between institutions, and
supervise, implement and evaluate the demonstration projects in their respective countries.
The project organizations; Ministries of Health and the Environment in the Region benefited
from the influence, leadership and infrastructure of PAHO (FR).
The regional coordinator was located in the headquarters of the Institute of Nutrition of
Central America and Panama (INCAP) PAHO's centre located in Guatemala. Regional
management
was
based
on
the
principles
of
management
and
development
of
administration, focusing on organization, planning, execution, monitoring, and evaluation.
Management was highly participatory and inclusive, prioritizing joint decisions with the
highest degree of consensus among the eight countries (FR).
Organization of the project's coordination and management process
The project was organized in five levels of coordination and management:
1. The Steering Committee was the highest body of the project and met five times to approve
working plans of the countries and other duties to ensure the project's success. Every
significant change to programs and budget were approved at this level (FR).
2. The Regional Committee was composed of the Regional Coordinator, the focal points from
health and environment ministries, the national coordinators, PAHO, CEC, CCAD, and
members of the civil society in the demonstration projects. This Committee met three times;
in Mexico (24-28 May 2004), Costa Rica (12-14 September 2005) and Panama (25-27 April
2006) in order to be informed of the project's progress, to propose technical adjustments and
to exchange experiences (FR).
3. In all countries, National Operational Committees were organized under the leadership of
the Ministries of Health and PAHO, with participation of the ministries of the environment,
30

universities, and institutions. The operational committees met at least six times a year to
reach agreements in the project's operations in each country (FR).
4. The demonstration area groups operated in each country close to the demonstration
projects and were formed by the national coordinator, the health focal point representing
each
governmental
administrative
unit
(departments,
districts
and
municipalities),
environment, education and other sectors at the local level, representatives from NGOs, and
the civil society involved in the project DDT/GEF/PAHO (FR).
5. Community working groups were organized in all demonstration areas, taking advantage
of current structures in order to avoid creating parallel organizations. In addition malaria
volunteer collaborators (COLVOL) were involved in all countries (FR).
3.1.2. Relevance
The project's relevance was rated as "highly satisfactory", because the results were coherent
with the strategies used (Table 9). The project did not only contribute towards bringing down
the use of DDT and other POPs, but also reduced the use of other insecticides. Moreover,
the project contributed to achieving the Millennium Development Goals (MDG), and the Roll
Back Malaria goals.
Most of the interviewees thought that the results of the project were "completely" consistent
with the specific strategies used in the project (Annex 2, Table 2, question A8); objective 3
was the only one for which the strategy was seen as weak (A8GN). Only in the Chiapas
State (Mexico) and Matina (Costa Rica), the results were considered as "partially relevant".
Most of the interviewees considered the malaria control strategy as having been validated in
practice and can be replicated in other areas.
One of the more controversial aspects of the project was the single dose' treatment (SDT,
TDU in Spanish) used in Mexico which aimed at eliminating the reservoir of plasmodium (see
the rationale of this intervention in Mid Term Evaluation report). At the state- and local levels,
people interviewed thought that it was necessary to make changes because it was difficult to
carry SDT in a high migration zone. Moreover, studies carried out in Mexico to compare the
efficiency of SDT with radical treatment of 3 days of chloroquine and 14 of primaquine
showed, in an initial report, that the radical treatment presented fewer cases of repeated
malaria than SDT (A8ME).
Integral vector control with the elimination of habitats (refugees) and mosquito breeding sites
(EHCA in Spanish) was a highly valued strategy by all actors in this evaluation and in others
performed previously. At the beginning of the project, there was no published evidence of the
impact of this strategy, but studies made in Mexico showed that the impact of the elimination
of mosquito breeds (clean house, clean garden) in malaria transmission is considerable.
Additionally this strategy has high compliance among the community and families and
considerable impact on other health problems (A8ME).
All people interviewed remarked the efficiency of the strategy used of involving the
community in malaria control (A8PB). A study carried out in Panama about the relation
between culture and malaria control revealed the fact that when people understood the links
of breeding sites, vectors and malaria, a behavioural change occurred with better compliance
and participation of indigenous communities in diagnostic, treatment and vector control
(Salinas V & Narváez A 2009).
31

Only in one country did the interviewees think that the results of the project were "partially
consistent" with country priorities; in all other countries they were seen as "completely
consistent" (Annex 2, Table 2, question A9). The project was concentrated in towns in which
malaria was persistent; therefore, all the interviewees considered malaria a priority (PO).
A significant issue that weakens the efforts of malaria control in Mesoamerica, and in the
case of Mexico, the aims of eliminating malaria, is that dengue fewer is seen as a more
significant risk since it is endemic in economically important tourist areas and because
explosive outbreaks of hemorrhagic dengue are prevalent (A9ME).
The question "To which extent the results of the project contributed to the Operational
Program of elimination of the use of Organic Persistent Pesticides (POPs)?" was only made
in regional, national and provincial levels. Almost half of the interviewees answered that the
contribution of the project to the elimination of the pesticides was "partial", the rest answered
"completely" (Annex 2, Table 2, question A10). The project reinforced the determination of
the countries of not using DDT or other POPs (A10R, A10CRN). Guatemala and Costa Rica
signed the Stockholm Agreement (A10GI).
3.1.3. Efficiency
According to the interviewees, the project efficiency was rated as "satisfactory". Payment of
NAP, workshops, meetings, supervision and community support represented the highest
amount of the total investment. There was no investment in human resources because the
project employed existing structures and resources.
Most of the interviews assessed the project as completely cost-effective (Annex 2, Table 3,
question A11). The general agreement among the civil servants and health workers, who had
experience using insecticides for control of malaria vectors, was that the cost-effectiveness
of the strategy was superior to spraying (OP, A11CRN). On the other hand, community
participation could be expensive at the early stage, but the costs would diminish during the
course of the intervention (A11PB).
As part of project activities, a study on cost effectiveness of the project strategy was carried
out in demonstration areas in El Salvador, Honduras and Nicaragua (4). This study showed
that implementing methods of vector control activities through community mobilization
(cleaning houses and patios, control of mosquito refugees and breeding sites) were cost-
effective, resulting in savings. The costs per malaria case in the three countries studied
during the two years of intervention were highest in Nicaragua, but the costs per case
prevented were also lowest in this country.
32

Table 6. Costs and cost effectiveness of interventions of the GEF-DDT project.
Indicators
El Salvador
Honduras
Nicaragua
Population at risk in study place
100,28
1,469
13,003
Program costs 2005-2007 (USD$)
133.680
24,878
287,430
Program costs 2005-2007 ($ internationals)
289,376
68,894
845,308
Costs/person 2005-2007 (USD$)
13
17
22
Costs/person 2005-2007 ($ internationals)
29
47
65
Cases averted
26
8
19
Costs/(saving) per case averted (USD$)
(0,40)
(0,64)
0,003
Costs/(saving) per case averted ($
(0,79)
(1,51)
0,008
internationals)
DALYs averted /100000 persons
6
269
5
Costs/(saving) per DALYs averted (USD$)
(1,72)
(0,02)
0,01
Costs/(Saving) per DALYs averted ($
(3,43)
(0,04)
0,03
internacional
DALYs averted /1,000 persons
0,6
2,69
0,5
Costs/ (saving) per DALYs averted /1000
(17,22)
(1,90)
(44,10)
persons (USD$)
Cost/(saving) DALYs averted /1000 persons
(34,33)
(4,48)
(129,71)
($ internacional)
* For the estimated cost-effectiveness assume that happen 5% of complicated cases in the three countries and
5% of fatalities in Nicaragua and Honduras.
The majority of the interviewees thought that the project was the least cost option. Only in the
communities of Mexico, was the project assessed as "partially efficient" (Question A12).
Although there were delays in disbursements during the first two years, the majority of
interviewees thought that the project delay did not affect cost-effectiveness (Question A13).
There was good capacity for execution of the project (A13PN), because many activities were
done with little money (A13PB).
The interviewees from national levels of three countries and Chiapas's State in Mexico
thought that the delays in the project partially affected cost-effectiveness. In Mexico, the
project suffered from interruptions due to the delays (A13MN). In Panama, delays affected
the confidence of local actors (A13CRN), the credibility of external actors and breached the
agreements with the communities, particularly concerning the planned interchanges' trips
(A13PN).
Majority of the locations were highly successful in obtaining additional resources. However,
in few locations, obtaining additional resources was partially successful (Annex 2, Table 3,
question A14). In fact, many alliances were made, which facilitated the mobilization of
additional funds from: PAHO, the National Governments, the Sanitary Districts, the
Municipalities, communities, NGOs, private companies and churches (A14R). Impregnated
bed nets and funds were obtained from the Global Fund and NGOs to extend the strategy to
other localities (A14GI).
The Malaria Control Programs contributed to infrastructure, human resources and transport
(A14MN). In Mexico, the National Institute of Health contributed funds to the study of
relapses and efficiency of the SDT (A14BMN). Municipalities assigned resources (A14CRN)
for the purchase of lime, tools, pipes for drainage (A14GF, A14GM) and machinery (A14B-
R). (A14MN). They also hired staff to clean and control the mosquito breeding sites (A14PN),
and to build permanent infrastructure for the drainage of the breeding sites (A14GN).
33

3.2. Assessment of sustainability of project outcomes
In this evaluation sustainability is understood as the probability of continued long-term
project-derived outcomes and impacts after the GEF project funding ends. Four aspects of
sustainability were addressed: financial, socio-political, institutional frameworks and
governance, and ecological.
The general evaluation of sustainability is "likely". The influence of hurricanes and tropical
storms, the reduction of financial resources due to the global crisis and the high level of
migration in frontier areas, are factors that the actors consider might affect the sustainability
of the project in the medium term. In summary, although the evaluation rates sustainability as
likely, there are places with high vulnerability.
Financial, socio-political and ecological sustainability obtained a "likely" rating, where as the
ratings for institutional sustainability were "highly likely". Panama received the lowest score in
financial sustainability. Regarding ecological sustainability, all countries with an exception of
Panama, recognised that the influence of hurricanes, tropical storms or flows could
undermine the sustainability of benefits after the project ends.
Other important factors that contributed to the project's sustainability were: I) the high
perception of success achieved during the implementation of the project, demonstrated by a
marked reduction in malaria cases; the rapid adherence of the countries to the project's
objectives,
which
was
achieved
through
participation
in
the
design,
execution,
implementation, monitoring and final evaluation phases, ii) the strategy which prioritized
vector elimination during its immature phases through low-cost techniques independent of
external resources. Community interventions for vector control were adopted by most of the
demonstration communities, iii) costs for the state, community and other key stakeholders
were lower due to the targeting strategy used, as well as the selective interventions, and the
community contribution, which could have accounted for a reduction of more than 50% to
that of operational costs using chemical substances, iv) the trans-disciplinary, inter-sector
and intercultural approach of the project with participation of the municipalities, indigenous
organization and researchers were maintained throughout interventions (Narvaez A 2008).
34

Table 7. Evaluation of sustainability of the project outcomes. Final Evaluation. Mesoamerica
2008-2009.
PLACE
Sustainability
B.1. Financial
B.2. Socio-
B.3.
B.4.
political
Institutional
Ecological
frameworks
and
governance
TOTAL
L (5,0)
L (4,8)
L (5,1)
HL (5,5)
L 4,64
Regional
ML (4,3)
L (4,5)
L (5, 0)
L (4,5)
MU (3,0)
Mexico Total
L (4,9)
L (4,5)
L (4,5)
HL (6)
L (4,5)
National
L (5,1)
L (5,3)
ML (4,0)
HL (6)
L (5,0)
Chiapas State
L (4,5)
ML (3,8)
MU (3,0)
HL (6)
L (5,0)
Neguatzotcoyol
community
L (4,8)
L (4,5)
L (5,0)
HL (6)
MU (3,0)
Nueva Macedonia
community
L (5,1)
L (4,5)
HL (6)
HL (6)
MU (3,0)
Guatemala Total
L (5,3)
HL (5,6)
HL (5,6)
L (5,3)
L (4,6)
National
L (5,4)
L (5,3)
HL (6)
HL (6)
ML (4,0)
Peten Department
HL (5,6)
HL (6)
HL (6)
L (4,5)
HL (6)
Ixacan Department
L (5,4)
HL (6)
HL (6)
L (4,5)
L (5)
La Felicidad
Community
L (5,1)
L (5,3)
L (5,0)
HL (6)
MU (3,0)
Sta. Maria Dolores
Community
L (5,1)
L (5,3)
L (5,0)
HL (6)
MU (3,0)
Costa Rica Total
L (4,9)
L (5,3)
L (4,3)
L (4,8)
ML (5,1)
National
L (4,1)
ML (3,8)
MU (3,0)
L (5,25)
ML (4,0)
Talamanca
Municipality
L (5,4)
HL (6)
L (5,0)
L (4,5)
HL (6)
Matina Community
L (5,4)
HL (6)
L (5,0)
L (4,5)
HL (6)
Panama Total
L (5,3)
ML (4,3)
HL (5,8)
HL (5,8)
HL (5,6)
National
L (5,6)
L (4,5)
HL (6)
HL (6)
HL (6)
Bocas Province
L (5,1)
ML (3,8)
HL (6)
HL (6)
L (5,0)
Puente Blanco
Community
L (4,8)
L (4,5)
L (5,0)
L (4,5)
HL (6)
Las Tablas
Community
L (5,4)
L (4,5)
HL (6)
HL (6)
HL (6)
Highly Likely
Likely (L)
Moderately
Moderately
Unlikely (U)
(HL)
Likely (ML)
unlikely (MU)
5,5 to 6
4,5 to 5,4
3,5 to 4,4
2,5 to 3,4
0 to 2,5
Comment [p4]: 2,4?
3.2.1. Financial sustainability
Financial sustainability was evaluated as "highly likely" in Guatemala, "likely" on the regional
level, Mexico and Costa Rica and "moderately likely" in Panama. Although it does not
diminish the influence of the global crisis, it was considered that the budget that was
assigned for next years should not change significantly from what will be needed to support
the reached achievements.
35

Regarding the question to what extent are the outcomes of the project not dependent on
continued financial support, 8 of the 17 locations stated "partly not dependent". The other
locations stated that the project was "completely not dependent" on financial support (Annex
2, Table 4, question B1). In several municipalities, specific funds have been destined for the
control of malaria and dengue, as alliances with Global Fund projects in Nicaragua,
Honduras and Guatemala (B1R).
In Guatemala, Panama and Costa Rica, the strategies continued without finances from the
project for the last 6 months of the year 2008. The community continued with the malaria
interventions because it was a habit that they had already adopted (B1GM). The floods in
September 2008 and January 2009 were a test of sustainability, which indicated that the
ending of the project will not affect the continuation of the strategies considerably. It could
have some affect regarding the follow-ups of the communities and the quality of the
interventions at local level (B1GN).
Regarding the likelihood that any required financial and economic resources will be available
to sustain the project outcomes/benefits once the GEF assistance ends, the interviewees
from nine locations considered the likelihood to be "partial", where as the rest considered it to
be "completely likely" (Annex 2, Table 4, question B2). In all of the four countries evaluated,
there are funds (partially or totally) to give continuity to the interventions and to assure the
sustainability of the results reached by the project. The major suppliers of funds will be the
Ministries of Health, the municipalities, Global Fund (GF) in Nicaragua, Guatemala and
Honduras (B2R, B2CRM). The municipalities have included malaria in their regular budgets
in almost all of the countries (B2CRT, B2GF, B2GM).
A probable financial source is the Mesoamerican Health System (Plan Puebla Panama
Initiative), which will be financed by the Foundations of Bill and Melinda Gates and Carson, in
order to eliminate malaria in Mexico and Central America (B2MN). Due to the high impact
reached in Guatemala, there is a possibility of obtaining a third phase from the Global Fund
projects (B2GP, B2GI) and a proposal for 9th round has been prepared with an aim of
replicating the model at national level (B2GN).
In eleven places (from 17) the interviewees considered that the project was successful in
identifying and leveraging co-financing (Annex 2, Table 4, question B3). However in all
locations of Mexico and in two of Panama, the interviewees considered that the project was
partially successful (B3MNEG, B3MNM). (See Annex 2, Table 4, question B3). Besides of
PAHO, the Ministries of Health and municipalities offered a great contribution. In general,
there was an increase in the budget for malaria control by governments (B8MN) in all of the
project countries. In Nicaragua, Guatemala and Honduras the project was financed by Global
Fund Projects. In Panama, Nicaragua and Costa Rica also private industries financed
activities and human resources (B3R).
All interviewees thought that several long term impacts resulted from the project (completely
score) (See Annex 2, Table 4, question B3). They are listed in the following paragraphs:
Implementation of laboratories for toxicological studies, development of research
capacity, and monitoring the impact of POPs on environment and human health
(B4R).
The pre -elimination phase of malaria transmission was achieved in Mexico, El
Salvador, and Costa Rica (B4CRM), maintaining malaria case reduction tendency in
Mexico and Salvador (B4R, B4ME) B4MNM B4MNEG. There is a high probability of
reaching the pre-elimination phase in Guatemala and Nicaragua over the next 3 to 4
years (B4GP, B4GI, B4PT).
Strengthening of the regulatory role of Ministries of Health (B4CRN) and vector
control programs (B4MN).
36

Strengthening of the local capability to carry out research and epidemiological
surveillance of malaria (B4CRN). GIS´s use for malaria surveillance and study of
pesticides (B4GN).
Reduction in pesticides used (B4GN).
Malaria reduction could have an important impact on improving health conditions and
reducing poverty, allowing families to improve the quality of life (B4GM).
Awareness and community participation in environmental conservation.
Empowering communities and building their capacity on claiming and negotiating in
order to resolve other health problems, such as lack of latrines and roads.
3.2.2. Socio political sustainability
Respondents of nearly half of the locations (8 / 17) stated that the outcomes of the project
are not dependent of socio-political factors. (Annex 2, Table 5, Question B5). At regional and
national levels of Mexico and Costa Rica, it was believed that sustainability could be affected
partially by socio-political factors such as presidential elections (2009), which may cause a
change in public health policies and health authorities. At the regional level, political
instability, expressed as constant changes in government, is identified as the main threat for
sustainability and a cause of failure in achieving planned objectives. Changes in
national/central governments that imply changes at the local level were perceived as a
threat/menace to the stability of employment.
The global economic crisis was identified by respondents as a negative issue; a good
example is the bankruptcy of the shipping companies as an additional factor that delays the
elimination of stockpiles of DDT and other POP's (B5R).
In Guatemala, the general opinion was that as the strategy is based on health promotion, the
outcomes and impacts will probably be independent of changes in government.
Nevertheless, such changes may modify the implementation of strategy in other places
/regions (B5GN). In Panama, interviewees' opinions were that political changes will not affect
the implementation strategy due to the existence of a defined budget for vector control (a
budget reduction for the control of malaria is not considered) (B5PB). People working in
vector control have job stability and awareness about the subject, and they continue
attending meetings even after the project finishes. Moreover, it is hoped that the new
government's policies are still the same, especially on issues of citizen participation.
In Mexico, because of the change in mayors, the connections with municipalities were
interrupted. Changes in high level authorities in the National Institute of Health affected the
national program since the new managers had different perceptions and policies. (B5ME).
It is believed that internal migration is also a threat (B5MN). There are some communities
which are transit areas for emigrants to United States. These immigrants pose a risk of
reintroduction of cases into areas that are malaria free (B5MNEG).
The opinion of Costa Rica and Mexico was that the changes in municipal authorities will
partially affect the local levels. Even though, they believe that the existence of career civil
servants would guarantee the continuity of the process. Furthermore, at the municipal level
of Costa Rica, there are committees or commissions that have a statutory obligation or
commitment (B5CRT) and do not depend on the mayor (B5CRM, B5GF, B5GM).
In most locations (14/17), the interviewees considered that the level of stakeholder
ownership will allow to sustain the project outcomes/benefits completely (Annex 2, Table 5,
Question B6). However, at the regional level, the level of stakeholder ownership is
considered to sustain the project outcomes/benefits only partially. At the regional level it was
37

considered that five more years is required in order to achieve real sustainability. Currently,
"there is a high level of adherence to the strategy on behalf of parents and children who will
apply the strategy for the rest of their lives because they saw the changes and the benefits"
However monitoring is required. Mayors and municipal governments joined the strategy and
there is high probability that political will would be maintained under community demands.
Support by the technical level of public institutions is lower (B6R).
"If you keep on track for five years, there will be no decrease in the use of DDT, but must be
achieved to eliminate stockpiles. The commitment of technical workers in public institutions is
inferior, and is related to the time they will stay in determinate job position. If
monitoring/assessment is maintaining, there will be no decrease in the use of DDT, but
stockpiles must be eliminated" (B6R).
In Mexico, a high adhesion by the community to the model has been demonstrated, but
environmental conditions and a few social factors (migration) are unfavourable (B6MNEG).
The lack of empowerment and training (B6MN) among workers and the size of the territory
that exceeds the capabilities of workers are aspects that limit continuity. The existing
resources are sufficient to maintain a low number of cases, but it is still difficult to eliminate
malaria (B6MN).
Community involvement in countries was very variable. In some places, there was broad
participation, but in others it was quite limited. For example, in Mexico in the Zapatista area
(Chiapas) the population seemed to be more reluctant to participate because of the rejection
by
the
government
(B6MN).
In
addition,
a
subsidy
program
called
"Community
Opportunities", weakened community participation, because of the sabotage of its
authorities. (B6ME)
In Guatemala, it is believed that the strategy has been integrated only into 25% of the
population (B6GF, B6GM). With the change of government and the new authorities, they
might have to start from the beginning at the national level. However this would not be the
case at the community level, which will be the first to assert and enforce support if the
malaria cases increase (B6PN). Empowerment among health care workers was good, but
the project has not been able to incorporate personnel from other places that are distant from
the demonstration areas. (B6PB)
In Guatemala, Panama and Costa Rica the level of empowerment of health workers and
communities and their capacity to respond to natural disasters was tested in the recent
floods. The level of organization and collective hard work was high in order to eliminate
rubble and repair the damages in public places. Each family was responsible for maintaining
their houses and yards clean (B6PPB, B6PT).
The majority of interviewees (14/17) thought that there was sufficient (completely)
public/stakeholder awareness in support for the term objectives of the project (Annex 2,
Table 5, question B7). Only in the national and state levels of Mexico and national level of
Costa Rica, the stakeholder awareness was partially sufficient. At the regional level,
awareness in demonstration areas of the use of insecticides was high, particularly in
indigenous areas because of their holistic concept to protect the ecosystems (B7R). In
Guatemala and Panama, high levels of sensitivity and awareness of communities and
stakeholders was reported (B7PB, B7GN). "They know, accept and participate (B7PN)
thanks to work in every house" (B7PPB, B7PT).
In Costa Rica, the base line survey revealed that people had not been empowered about
malaria and the use of DDT. Although there were no other surveys done, the situation itself,
according to the interviewees, changed (B7CRN), and the awareness is now high especially
in the municipalities and communities (B7CRT). Nevertheless, nowadays there is social
38

pressure to use insecticides for other problems as dengue (B7CRN), which is a risk to
reintroduce POPs for vector control.
3.2.3. Institutional framework and governance
Institutional framework and governance was rated as "highly satisfactory" in 13 localities
evaluated, with the exception of a locality in Panama, two localities in Guatemala and one in
Mexico, where the parameter was rated as "partially satisfactory".
The majority of the interviewees (14/17) thought that there were issues relating to institutional
frameworks and governance that favoured completely the extent of the outcomes of the
project (Annex 2, Table 6, question B8). Only the regional level and two departments of
Guatemala considered it as "partial". The malaria control strategy integrated and
strengthened several policies such as the Millennium Development Goals, the social and
community participation (B8MNEG, B8CRN), decentralization, protection of individual rights
(B8ME), intersectoral coordination (B8PB, B8PT, B8CRT), and promotion of healthy schools
and healthy municipalities (B8GN).
In seven of 17 places the answer to the question of the likelihood that institutional and
technical achievements, legal frameworks, policies and governance structures and
processes will allow for, the project outcomes/benefits to be sustained was "completely
likely'. In the Regional level, all levels of Costa Rica rated the aspect as "partially likely" (See
Annex 2, Table 6, question B9).
According to the regional interviewees there was initial resistance to the new strategy and it
required more than one year of work for the Ministries to join (B9R). The strategy was a clash
against tradition, so that during the first implementation phase, the execution was not
smooth, but the project reversed this problem. In some countries, there was rapid change
and the practice spread to other areas (B9R).
In several countries the decentralization process carried out during the 1990´s, weakened
the malaria control programs, but the program were, however passed (B9MN). The project
reinforced malaria control programs despite their structure, because the local committees
have joined the Ministry of Health structures and specially at the vector control programs,
taking advantage of the technical experience and the structure that remain from the vector
elimination program.
In Mexico, a specialized semi autonomous structure of the program still prevails, but the
integration with the general health services is good (PO). In Panama, the specialized
structure persists, but because of the health service's decentralization process (began in
1996), the Malaria Control Program (MCP) disappeared and the old structure was weakened
because the retired vector workers were not replaced. Through the project, the integration of
the MCP to the general health services is being reinforced.
In Guatemala, the vector control structure persists, but it is under the Health Regional Chief´s
leadership. It can be qualified as an integrated structure, but it is not a horizontal structure
yet.
Costa Rica had the most decentralized intervention. There was no parallel or independent
structure for vector control, so the area chief was, at the same time, responsible of the
preventive activities. The area has a multidisciplinary team, with director, epidemiologists,
teacher and also vector inspectors who work in the communities. There was some
coordination between the general health services and the health staff.
39

In majority of the locations (8/9) the interviewees thought that there is a complete likelihood
that institutional and technical achievements, legal frameworks, policies and governance
structures and processes will allow for, the project outcomes/benefits to be sustained (Annex
2, Table 6, question B10).
In Mexico, the model has been adopted to the entire country (B10MN). Nicaragua,
Guatemala, El Salvador, Honduras and Panama reviewed national guidelines for malaria
control taking as reference the Technical Guide of the Project (OP, B10GP). All countries
have ascertained that they will not use DDT for Public Health. There were changes in
treatment patterns and structures of community participation (B10R).
All the interviewees considered that the systems for accountability and transparency and the
required technical know-how were completely in place (Annex 2, Table 6, question B11).
PAHO has a strict management system control of funds to ensure transparency and proper
resource management (B11GN, B11PB, B11CRN, B11CRT). In PAHO headquarters and in
each country level PAHO´s agencies have internal and external audit mechanisms (B11R). It
is difficult to deviate from resources because the audit mechanisms are strict.
Only in Guatemala, did the community delegates' participate in planning, implementation and
auditing of funds, through a committee called "table of health"· (B11GI). In other countries,
the community participation in the audits was weak (PO).
3.2.4. Environmental sustainability
The aspect that was considered to affect sustainability the most was the presence of
hurricanes, tropical storms and floods, which are frequent in the demonstrative areas.
Nevertheless, in Guatemala, Costa Rica and Panama the health service and the
communities reacted very well against the effects of tropical storms that occurred at the end
of 2008. By January 2009 no outbreaks or epidemics had occurred and the threats were
controlled without the use of any kind of insecticides (B12R, B12PB).
In more than half of the places the interviewees (9/17) noted that there will be no
environmental risks that can undermine the future flow of the project's environmental benefits
(Annex 2, Table 7, question B12). In all places in Mexico and, in two places in Guatemala the
opinion was that the impact of these climate phenomena will be partial. At the national level
of Costa Rica, the opinion was that the impact will be severe (score= 0).
In seven of the nine places studied, the interviewees considered that there are no risks to the
ecological sustainability of this project (Annex 2, Table 7, question B13). According to the
regional level, if the elimination of DDT is not completed, there is a risk of re-use in case of
epidemics (B13R).
Generally, activities that have become threats to the project are not identified (Annex 2,
Table 7, question B14). The regional level confirms that failure to complete the component 3
is a threat, which would take out merit from the project (B14R).
3.2.5. Suggestions for long term impact
The interviewees made the following suggestions to improve future impacts of the project at
regional or country level (question B15). It is important to find new funds to support a new
small project (one million U.S. dollars); in order to develop unexpected needs such as
laboratories, disseminate research, to maintain the monitoring of communities, to replicate
the experience, and to generate valid scientific evidence (B15CR, B15PB), which not only
40

ensures the sustainability, but would also help to eliminate malaria in Mesoamerica (B15R,
B15ME, B15GI). The regional level suggests that it is necessary to find mechanisms to
disseminate the achievements and experiences widely in the region and beyond (B15R).
Another way to ensure sustainability is to strengthen the health system, including the
information system, operational research and the formulation of a strategic plan to improve
efficiency and effectiveness. (B15MN).
The best channels and means to ensure the long-term impact (question B16) are to maintain
the partnership between PAHO, Ministries, Municipalities, communities, universities and
collaborating centres (B16R). The creation of the Mesoamerican Health System Initiative
(Rodriguez M 2009) is a potential means to sustain these partnerships (one of its work areas
is the vector control) (B16R).
It is necessary to find the mechanisms to ensure sustainable community participation
(B16MN). One of the prospective ways of obtaining such participation is to insert the malaria
control project in the broader multi-purpose development projects (B16PN). It is also
necessary to address nutritional problems and high mortality from preventable diseases
(B16PN). In indigenous areas, it is necessary to strengthen an intercultural approach, for
instance, to motivate health workers to speak native languages and to produce educational
materials in these languages (B16PN).
There were several limitations to evaluate the impact of the project. Although there is an
evaluation chapter in the Technical Guide, the methodology that was to be used to evaluate
the impact was not clearly defined. After mid term evaluation, a protocol was formulated and
during 2008 a retrospective study was carried out. However, a problem occurred concerning
the selection of control localities in each of the participating countries. The condition for the
appropriate selection of control localities is that they share similar characteristics as the
demonstration communities but should not be placed near them. Unfortunately, these kinds
of localities could not be found because the vector workers have the same areas of influence
and the project control strategy was replicated in these communities, so they were not real
control localities.
An alternative to solve the lack of control locations in the impact assessment study could be
to maintain several communities where it would be continued to record cases as
demonstration sites (question B17). Nicaragua, Guatemala, El Salvador and Mexico have
good epidemiological surveillance systems to achieve this objective. In other countries, it is
necessary to strengthen surveillance systems (B17R, B17CRN, B17GI).
3.3. Catalytic role
Catalytic role was one of the key evaluation parameters with major success. It was rated as
"highly satisfactory"; since there was an extension of the interventions to other neighbouring
localities and municipalities due to an initiative of the community leaders and health workers.
Only in México one locality had not replicated the model to other communities. In Mexico,
Guatemala, Nicaragua and Honduras the replication was very extensive. In the last three
countries there were alliances with the Global Fund Projects and other projects.
Table 8. Evaluation of Catalytic Role. Final Evaluation. Mesoamerica 2008-2009.
PLACE
Score of Catalytic
Role
TOTAL
HS (5,5)
41

Regional
HS (6,0)
Mexico Total
HS (5,5)
National
HS (6,0)
Chiapas State
HS (6,0)
Neguatzotcoyol community
MU (3,0)
Nueva Macedonia community
HS (6,0)
Guatemala Total
HS (5,6)
National
HS (6,0)
Peten Department
HS (6,0)
Ixacan Department
HS (6,0)
La Felicidad community
S (4,5)
Sta. Maria Dolores community
S (4,5)
Costa Rica Total
HS (5,4)
National
S (4,5)
Talamanca Municipality
HS (6,0)
Matina community
HS (6,0)
Panama Total
HS (5,3)
National
HS (5,3)
Bocas Province
S (4,5)
Puente Blanco community
HS (6,0)
Las Tablas community
HS (6,0)
Health workers and community leaders spontaneously replicated the strategy in other
neighbouring areas (Annex 2, Table 8, question C1). Although most countries replicated the
strategy in several locations, it is important to highlight the extent of these replicas in
Panama (C1PB, C1PPB, C1PT). There were only two communities (one in Mexico and one
in Guatemala) in which the leaders did not replicate the strategy in other localities, but
malaria workers did it. Partial replication was performed in only one in Panama.
Comment [p5]: One community?
Currently, with the exception of Belize, the model has been applied by all countries in other
endemic areas (C1R, C1CRN, C1CRT). Nicaragua, Honduras and Guatemala are using this
model in the areas of intervention of the Global Fund Projects. Guatemala is the country with
most replicas (600 towns) (C1PN).
The impacts of these replicas have reduced malaria cases in all communities of the
demonstration areas (C2R, C2CRN) and in the case of Guatemala, in the whole country. The
main effect is that the involved institutions have adopted the strategy and methodology,
specially the approach to work with the community and environment as main axes (C2R,
C2CRN). In all countries, the demonstration areas are an example of a successful
experience (question C2).
Only one community in Mexico responded that there were no lessons or experiences applied
in other areas (Annex 2, Table 8, question C3). The rest of interviewees answered that this
aspect was "completely" fulfilled.
The project strategy was extended to Andean countries. In the frame of the PAMAFRO
Project, supported by Global Fund, a mission from Mexico presented the model and shared
experiences in the Andean Area (C3R). The author of this evaluation also applied the model
in Ecuador with success (Narváez A & Cańas M 2007). PAHO is implementing the model in
Veraguas-Panama in the frame of the project "Faces, voices and places" (C3PN).
42

Concerning the question are there lessons or experiences that have been developed as
results of the project that can be replied or extended to other places or to the design and
implementation of other projects, all the interviewees answered "completely" (Annex 2, Table
8, question C5). In fact, experiences from this project could be applied in other projects, for
example:
An education experience with puppets in Costa Rica and El Salvador (C5CRM)
The strategy of community participation and a variety of activities of social
sensitization (C5CRT)
The intercultural approach for indigenous and afro descendent populations, adapted
in each country (C5GN). An interviewee says: "When we go to some areas with our
ideas to impose them, it does not work. Here we started with a different approach: we
made meetings where the people prioritized their health problems and took
advantage of the popular conception that malaria is an illness that affects normal day
work. This makes a difference with other projects that want to impose, because in this
project we let people choose what to do. Knowledge is not useful if the community
does not participate and is not aware of what they are doing" (C5PB).
3.4. Achievement of outputs and activities
The achievement of outputs and activities obtained a "satisfactory" rating at the overall level.
Only at the regional level, the rating was "moderately satisfactory" because it was not
possible to eliminate the existing DDT stockpiles (30% of implementation), Belize did not
carry out the risk assessment, and two activities of component were not implemented (Anexx
4). At the national level of Costa Rica, the component was rated as "moderately
unsatisfactory", because the accomplishment of the four criteria formulated to evaluate this
component was partial.
43

Table 9. Evaluation of Achievement of outputs and activities. Final Evaluation.
Mesoamerica 2008-2009.
Score of
Achievement of
PLACE
outputs
TOTAL
S (5,4)
REGIONAL
MS (3,8)
MEXICO TOTAL
S (5,4)
National
S (5,3)
Chiapas State
S (5,3)
Negualzotcoyol Community .
S (5,0)
Nueva Macedonia Community
HS (6)
GUATEMALA TOTAL
HS (5,8)
National
S (5,3)
Peten Department
S (6)
Ixacan Department
S (6)
La Felicidad Community
S (6)
Sta. Maria Dolores Community
S (6)
COSTA RICA TOTAL
S (4,9)
National
MU (3,0)
Talamanca
HS (6)
Matina Community
HS (6)
PANAMA TOTAL
HS (6)
National
HS (6)
Bocas del Toro Province
HS (6)
Puente Blanco Community
HS (6)
Las Tablas Community
HS (6)
Interviewees from regional level, Costa Rica national level and a Mexican community
qualified the soundness and effectiveness of the methodologies as "partially satisfactory".
The rest rated this parameter as "completely satisfactory" (Annex 2, Table 9, question D1).
According to some interviewees the effectiveness and soundness of the methodologies used
to develop the actions were satisfactory in the components 1, 2 and 4, but not in objective 3,
related to the elimination of DDT stock piles (D1CRM, D1GN). The methodological structure
of the implementation was proper and allowed a follow-up methodology in operative actions
and expected results (D1MN).
The methodology used and the impact achieved in objective 1 has already been extensively
explained in previous paragraphs. An interviewee said "the methodology is satisfactory,
because before health workers used to think only in fumigating and giving treatment, but did
not educate people. Now, the most important is the education to change behaviours and
control breeding sites without chemicals" (D1GI), "...the key was not imposing, but
interacting" (D1PN).
The organization of working groups at community level and at higher levels was another
important aspect. One or more leaders were trained and they expanded the knowledge to
their working group or to the Health Committee. "They went to talk to every family. Families
44

received a lot of support from vector workers through lectures and advise on what to do in
each case "(D1PPB).
It should be noted that both, the Single Dose Treatment (SDT), and whitewashing of homes,
were not supported by scientific evidence. The last intervention was strongly welcomed by
the community in all countries where it was implemented. In the case of SDT, the evaluation
carried out in Mexico showed less effectiveness in reducing relapse compared to radical
treatment (Cerón L 2009).
Organization of the project execution in phases was also a success. The first phase was to
validate the model in practice. During second extension, research and documentation were
prioritized, although there was not enough time to complete all the activities planned
(D1CRN).
Organizing the operative committees at regional, national and demonstrative areas under
objective 4 was rated as "successful" (D1R). The methodology used for exchanging
experiences through phone-conferences, meetings of technical committees, and internships
of leaders and health workers in other locations had a significant role in the success of the
project. (D1CRT).
All interviewees assessed the legal, technical and financial capacities and mechanisms
implemented in the project as "completely satisfactory" (score 2) (See Annex 2, Table 9,
question D2). All countries have developed in building institutional capacity activities through:
i) training of national and local personnel and community members, and the delivery of
equipment, ii) strengthening of legal capacities through the formulation and implementation
of guidelines for malaria control, iii) constituting of Steering Committee, the technical teams,
National Committees and Local Committees (MTE, OP).
The project was adapted to the national regulatory frameworks (D2R), but administrative and
financial adjustments were made to meet PAHO requirements to the donor (D2MN). In
general, there were no significant changes, mainly because the same structure of the
Ministry and PAHO were used (D2PN).
At the municipal and community levels of all countries, laws and ordinances were formulated
to encourage the implementation of the project, particularly the breeding site control and
environmental sanitation (D2MNEG, D2CRM). Local authorities (in Mexico Commissioners
and in Panama Aldermen), appointed by the people in meetings, encouraged participation
and participated themselves in activities and sometimes punished people who did not clean
houses and patios (D2GF, D2GM, D2PPB, D2PT).
According the interviewees the project outputs have the complete weigh and scientific
credibility. Only the national level of Panama assessed it as "partially" complete, because at
Ministry of Health Headquarters, some civil servants did not support the new strategy (Annex
2, Table 9, question D3). The rapid impact achieved at the start of the project allowed the
model to achieve wide support from the community and local workers. The approach "test-
error" used in the implementation of control strategies determined that the implementation of
strategies were highly flexible (D3GP).
The evidence generated by scientific studies and the systematization of experiences have
given the project a scientific authority and credibility to influence the formulation of malaria
policies and decision makers (D3CRT, D3GN).
In terms of participation, social stakeholders and communities were convinced about the
benefits of the strategy, because of their previous experiences. The results of the studies on
the impacts of DDT on environment and human health constituted a scientific and practical
45

contribution, because many people believed that they would not find metabolites of DDT
(D3CRN), but results showed contamination in several places (Díaz-Barriga F 2009).
Interviewees from four places thought that the delivery of outputs was "partial", but in
general, achievement of the activities and planned results was "complete" (Annex 2, Table 9,
question D4). In several places the goals planned were surpassed and in other cases
activities not planned were implemented, such as the studies to validate the strategies
(D4MN). In general, respondents felt that the project was successful in achieving each of the
products in quality, size and utility; particularly in objectives 1, 2 and 4 (D4MNEG, D4MNM,
D4CRT, D4CRM), but not in objective 3. There were problems in time management due to
late payments and administrative changes, but this was overcome in most cases thanks to
the countries and PAHO, which provided money when resources were not available due to
delays in disbursements (D4R).
3.5. Assessment of Monitoring and Evaluation Systems
As mentioned previously, one of the less developed features was the M&E, which obtained a
"moderately satisfactory" rating.
M&E design, budgeting and funding, and long term
monitoring obtained only "moderately satisfactory" rating, but implementation plan was
"satisfactory". Mexico shows least performance, particularly at community levels in almost all
the criteria evaluated (Table 11).
46

Table 10. Evaluation of Monitoring and Evaluation Systems. Final Evaluation.
Mesoamerica 2008-2009.
E- M&E
E.1. M&E
E.2. plan
E.3.
E.4. Long-
performance
design
implemen-
Budgeting
term
tation
and
Monitoring
PLACE
Funding
TOTAL
MS (4,0)
MS (3,9)
S (4,6)
MS (3,6)
MS (3,8)
REGIONAL
MS (4,0)
MS (3,8)
S (5,3)
MU (3,0)
MU (3,0)
MEXICO TOTAL
MS (3,2)
MS (3,3)
MU (2,5)
U (1,5)
U (2,3)
National
MS (4,2)
MS (4,0)
S (4,5)
MU (3,0)
HS (6)
Chiapas State
MS (3,3)
MS (3,3)
MS (3,8)
MU (3,0)
MU (3,0)
Negualzotcoyol
Community
U (1,7)
U (1,5)
U (2,0)
HU (0,0)
MU (3,0)
Nueva Macedonia
Community
U (1,7)
U (1,5)
U (2,0)
HU (0,0)
MU (3,0)
GUATEMALA TOTAL
S (4,9)
S (4,7)
HS (5,7)
S (4,8)
MS (4,2)
National
S (4,2)
MS (3,5)
HS (6)
HS (6)
MU (3,0)
Peten Department
S (4,7)
S (5,0)
S (4,5)
MU (3,0)
MU (3,0)
Ixacan Department
S (5,0)
S (5,0)
HS (6)
MU (3,0)
MU (3,0)
La Felicidad Community
HS (6)
HS (6)
HS (6)
HS (6)
HS (6)
Sta. Maria Dolores
Community
HS (6)
HS (6)
HS 6
HS (6)
HS (6)
COSTARICA TOTAL
MS (3,8)
MS (3,5)
S (4,9)
MS (4,0)
MU (3,0)
National
U (2,3)
U (2,0)
MU (3,0)
MU (3,0)
MU (3,0)
Talamanca Municipality
S (5,2)
S (5,0)
HS (6)
HS (6)
MU (3,0)
Matina community
S (4,3)
MU (3,0)
HS (6)
MU (3,0)
MU (3,0)
PANAMA TOTAL
S (4,6)
S (4,5)
S (4,9)
S (4,5)
MS (3,8)
National
S (4,5)
MS (4,3)
S (5,3)
HS (6)
MU (3,0)
Bocas del Toro Province
S (4,8)
MS (4,3)
HS (6)
HS (6)
HS (6)
Puente Blanco
community
HS (5,6)
HS (6)
S (5,0)
HS (6)
HS (6)
Las Tablas community
MU (3,0)
HS (6)
MU (3,0)
HU (0,0)
HU (0,0)
Score
Highly
Satisfactory
Moderately
Moderately
Unsatisfactory Highly
Satisfactory
(S)
Satisfactory
Unsatisfactory
(U)
Unsatisfactory
(HS)
(MS)
(MU)
(HU)
5,5 to 6
4,5 to 5,4
3,5 to 4,4
2,5 to 3,4
1,5 to 2,4
0 to 1,4
47

3.5.1. M&E design
The M&E system had two documents of reference; the Project document Section 5,
Monitoring and Reporting (GEF 2003) and the technical guide (Mendez J, Betanzos A, &
Tapia R 2004). The project document was referring to the aims and activities planned, but
the guide was related with malaria epidemiological surveillance and monitoring of the
strategies of control implemented.
According to the Project document, monitoring of the GEF-DDT Project would consist of: (i)
quarterly progress reports referring to the aims and activities developed with a qualitative
approach, with few goals or thresholds; (ii) terminal report; (iii) Substantive Reports
presented and (iv) financial reports. This system design was assessed as adequate.
The Technical Guide defined an extended list of indicators (more than 1209) and it covered
procedures, products, results and the impact.
Eight of 17 places evaluated rated the statement "the project has a sound M&E plan to
monitor results and track progress towards achieving project objectives" as "complete". The
rest (nine places) answered "partially", particularly in Mexico, Guatemala and Costa Rica
(See Annex 2, Table 10, questions E1).
The M&E section of the Project Plan did not contain M&E of malaria control interventions and
epidemiological surveillance of the malaria situation. This was developed in the Technical
Guide, which presents a chapter "Demonstrative Projects Evaluation". In this chapter, four
evaluation aspects are described: the impact, the process, the effectiveness, and efficacy.
Some of the key interventions did not have monitoring indicators, as the coverage and
treatment compliance.
The complexity of the malaria surveillance system presented in the technical guide and the
heterogeneity of the surveillance systems of the countries determined that each country used
their own surveillance and monitoring systems (PO). During 2008 Mexico, Guatemala and
Nicaragua designed and implemented, surveillance and monitoring systems consistent with
the needs and strategies of the malaria control model implemented by the project (E1M,
E1GP, E1GI); in other countries, this is still a challenge.
Lack of uniformity of the surveillance systems for malaria in the countries caused serious
difficulties in evaluating the impact and results of the project in relation to the incidence of
malaria and control of mosquito breeding sites and refugees. The base line was carried out
at the beginning of the project but not at the end (E1CRN). In the participating countries, an
explicit monitoring plan was not designed at the national level. The project was set to the
regional level (E1GN).
Several countries designed and implemented situation rooms at the community and health
district levels, but there was no uniformity (OP, E1CRM, E1PB). The use of GIS in the
situation room was a privileged tool.
Interviewees from three of the nine places assessed this question as "partially", and the rest
thought that the project met the minimum requirements for project design of M&E (SMART)
and the application of the Project M&E plan. (Annex 2, Table 10, question E2).
At the regional level and in each country there were no specific units or persons responsible
for M&E, which was recognized as one of the limitations of the project (PO) (Annex 2, Table
10, question E3). Because of that, in three of ten places the interviewees assessed this
48

aspect as "partially" (Chiapas State, Guatemala National and Panama National), the rest
assessed it as "complete".
The regional project coordinator, NAPs and focal points were responsible of the M&E
activities (E3R, E3MN, E3ME, E3CRT, E3PN). At the local level, the focal points of the
project (epidemiologist of demonstration areas or the heads of vectors) were the ones who
brought up this activity (E3CRT, E3GN, E3GP, E3GI, E3PB).
In Bocas Del Toro - Panama the interviewees assessed the quality, application and
effectiveness of project monitoring and evaluation plans and tools as "no - no at all" (0
score). In six places the rating was "partially" and only three said "completely" (Annex 2,
Table 10, question E4). The M&E project was carried out through quarterly reports using the
instrument designed in the project formulation. This assessment focused on compliance of
products and cannot report results and impacts (PO). Each country prepared a quarterly
report that was sent to the regional level. The regional coordinator of the project then
consolidated these quarterly reports and sent them to the donor. A total of 16 quarterly
reports were prepared, but there was no feedback from the donor, except for a one
comment. The Mid-Term Evaluation (MTE) was an important element in defining the
necessary changes to ensure project success (E4R). Before MTE, instruments for the
systematization were not defined. After instruments for systematizing, monitoring and
evaluation were discussed and designed, they were implemented during the last year of the
project (E4CRN).
The definition of risk management was done for the project as a whole, but a risk
assessment based on management assumptions of each country were not formulated
(E4GN). Therefore, some management risks were not identified in the project documents
(E4MN).
With exception of Chiapas in Mexico, the baseline was carried out in all places. Mexico
carried out the baseline only in Oaxaca demonstrative area (Annex 2, Table 10, question
E5). Seven of the participating countries published a report of the baseline. A comprehensive
guide and the baseline were formulated (E5R) containing socio economics, demographic and
ecological indicators, knowledge, attitudes and practices, prevalence of malaria and
evaluation of mosquito breeding sites (E5CRT). A guide for the inventory stock of DDT was
also designed (E5CRN).
In more than half of the places (6/10) evaluated the interviewees said that the schedule of
M&E activities and standards was "partially" defined, the rest answered "completely"
(question E6). Although a timeline was defined at the regional level and was adjusted to the
conditions of the program (E6MN, E6GN, E6PB), no explicit schedules were defined at the
country level. Time and goals were defined with regional coordination, but at the country
level they were not explicit (E6CRN, E6CRT). There were no minimum standards to ensure
products (E6R). There were reporting formats for each area of work (E6CRN). Quarterly
reports had a definition of the level of compliance: complete, incomplete or in process.
Quarterly reports were complemented by annex when required (E7PN).
In three places the answer was "no not at all" and in six places it was "partially". Indeed,
there was not a specific document for collection, aggregation, analysis and decision making
for each level of the regular reporting system. (Annex 2, Table 10, question E7). Mexico,
Guatemala, Nicaragua and Panama are using the ones that control programs had. (E7GN,
E7ME, E7MN, E7PB). For impact assessment and baseline study specific documents were
developed to meet this requirement (E7R).
There were no specific written procedures for handling late, incomplete or inaccurate
information of surveillance system at the regional level and at country levels of Guatemala
49

and Costa Rica (OP, E8CRT E8CRM E8GN E8PN). Mexico national, Chiapas state level and
Panama National level answered "partially" (Annex 2, Table 10, question E8).
This
represented a constraint to the impact evaluation study, as inconsistencies were found in
reports from several countries (E8MN). In the four countries visited, the review for mistakes
and inconsistencies was carried out eventually (E8ME).
Only two places answered "completely" to the question "how inconsistencies have been
resolved when discrepancies were found in surveillance system?" Interviewees from six
places said that there was "no not at all" and one said "partially" (Annex 2, Table 10,
question E9). Indeed, the procedures to resolve discrepancies had not been documented. At
the local levels of each country, a regular review of data and correcting the discrepancies
was done directly by health workers (E9GN, E9GI), but there were no documents that specify
these procedures (E9MN, E9ME, E9CRT, E10MN).
Related to the availability of a systematic feedback for the different levels of information
about data quality of the reports produced (Annex 2, Table 10, question E10), there were
different answers. At the national level of Costa Rica, the answer was "no not at all", at the
regional level and in national and state levels of Mexico, the answer was "partially". These
opinions were based on information from malaria surveillance systems, but not on quarterly
reports, which received systematic feedback. One of the most important mechanisms for
feedback and discussion of progress were phone-conferences and the Regional Technical
Committee meetings (OP, MTE).
At each country level and demonstration area, there were several meetings, workshops and
other feedback activities each month (E10CRT, E10ME), but there was no systematic
procedure to evaluate accuracy and completeness (E10GN E10GI E10PB). In relation to
timeliness, the participating countries defined that the reports for all the surveillance systems
for malaria were delivered weekly (PO).
Only in five of the nine places, the interviewees said that there was quality control of data
done in the field for the reports written (question E11). Interviewees of the national level of
Costa Rica answered "no not at all" and four said "partially" (regional, Mexico national and
state and Panama national). Data quality controls on impact evaluation study were done on
regional level (E11MN). In Costa Rica and Guatemala the entomological surveys in the field
were done routinely (E11CRT, E11GN, E11GP). In Panama and Mexico monitoring visits
were done during supervision visit, but there were no registration instruments (E11PN,
E11PB).
The regional level used to review the quality and completeness of information of the quarterly
reports of each country, and then it consolidated them and published on the web site and e-
mailed to countries (E10R).
The regional level, national and provincial levels of Panama, and national level of Costa Rica
had no available computer system with a clear and active documentation of the procedures
for database administration. In four places the answer was "partially" (Annex 2, Table 10,
question E12).
3.5.2. M&E plan implementation
The
following
monitoring
activities
were
implemented:
i)
baseline
study
in
each
demonstrative area about malaria epidemiological situation, environmental risk and
knowledge, attitudes and practices ; ii) quarterly progress reports of every country and a
regional summary according to the parameters established by UNEP; iii) Substantive reports
of Steering Committee and Technical Committees meetings where progress and adjust work
50

plans, and the general interest of the core team in achieving quality were discussed. The
reports of Executive Committee meetings convey detailed and in-depth discussions on
implementation and delivery issues; iv) a mid-term evaluation undertaken with the
participation of the PAHO and Regional Coordinator to diagnose problems and suggest
necessary corrections; v) a final evaluation undertaken by independent consultant from
November 2008 to February 2009.
Monitoring and evaluation of project document was adequately addressed, contributing to the
project's effective implementation, although it did not allow quantifying the activities, the
products or results. Baseline study and mid-term evaluation were carried out as planned,
with detailed responses prepared to each of the recommendations, all of which were
considered by the Regional and Country Boards.
On the other hand, the M&E system of the Technical Guide turned out to be excessive and
very few indicators were in use which was gathered in the base line and at the end of the
project. After the mid term evaluation, the Regional Coordinator defined 20 core indicators,
but not all were reported in final impact evaluation; only Guatemala, Nicaragua and El
Salvador reported all the indicators defined for the final evaluation.
The budget for M&E was integrated in each of the activities.
In all countries, a surveillance system for reporting malaria cases was implemented at
community levels. Before that, the information was only aggregated at provincial or
departmental levels. Nowadays, there is good quality surveillance system at community
level, which will allow evaluating the impact in the long term.
The most important advance of the information system has been the GIS development. GIS
applications were developed in all countries, using the SIG-EPI package of PAHO (E12CRT,
E12GN). In Guatemala, Costa Rica, Mexico, Nicaragua, El Salvador and Panama, the
regional, national and local staff has achieved useful applications to take decisions (base line
diagnosis, the monitoring interventions). These applications enable seeing the power that the
GIS has for the monitoring and evaluation. In this sense, it is evident that the capability which
the vector and the epidemiological staff have reached in making epidemiologic analysis was
enhanced by the maps made in GIS.
In eleven places, a M&E system was in place, and it facilitated tracking of results and
progress towards project objectives throughout the project implementation period (Annex 2,
Table 11, question E13). One third of the places (6/17) answered "partially" to this question,
Chiapas State and both communities in Mexico, Peten District in Guatemala and both
communities of Panama.
The opinion of the interviewees was that the implementation of the M&E system and the
preparation of quarterly reports facilitated monitoring of outputs and outcomes throughout the
project implementation (E13ME E13CRN). In addition, the progress of project performance
was presented in the Regional Technical Committees and used to take decisions (E13R). At
the country level, goals to be achieved were defined, and monitoring of all activities,
deadlines and objectives of the project were done regularly and in extent (E13CRT, E13GF,
E13PN).
In ten places (from 17) the quarterly reports of the project were complete and accurate, and
well justified rates were achieved. In four places, the answer was "partially" and in three "no
not at all" (Annex 2, Table 11, question E14). No annual reports were prepared, but only a
quarterly and a final report, which were complete and contained information that was
requested in the guidance of procedures provided by the donor (E14R, E14MN, E14CRT,
E14GN, E14GP, E14GI).
51

The majority of the interviewees (13/17) said that the information from M&E system was used
to improve project performance (Annex 2, Table 11, question E15). The standard instrument
used for monitoring allowed identifying the progress easily, making it a useful tool to improve
performance, and adapt it to the necessity of change at regional and country levels (E15R).
However, the absence of entomological and malariometric information was a limitation in the
first phase, which was corrected in the second phase and the extensions.
Information from the mid term evaluation is considered as one of the most important inputs
that allow improving performance and quality of the project (E15R). At local levels
(demonstrative areas) evaluation of the epidemiological and entomological situation was
used to improve the intervention (E15CRN, E15CRT, E15CRM, E15GI). In this sense, the
experience of Guatemala about decision making with community participation for the
stratification and evaluation of impact of interventions was successful (E15GP).
In six places the interviewees said that the project has a M&E system in place with proper
training for parties responsible for M&E activities to ensure data will continue to be collected
and used after project closure (Annex 2, Table 11, question E16). In three out of nine places
the answer was "partially" (Chiapas State in Mexico, Panama national and Peten in
Guatemala). According to regional interviewees there were training sessions at the beginning
of the project. However, a training program was not designed at the outset, but people were
trained when problems appeared (E16R). One of the interviewed said "everyone was
learning on the fly as required" (E16PN).
Budgeting and Funding for M&E activities
Three communities from Mexico and one from Panama responded that "no not at all" to the
question if the project had adequate budget provisions for M&E. Six places, including
regional interviewees, responded "partially" and in ten places, the answer was "completely"
(Annex 2, Table 12, question E17).
There was a specific budget to carry out the baseline study from UNEP (E17CRN). The rest
of financial resources for M&E execution were taken out from funds allocated by PAHO as
part of the counterpart contribution. Because of that, there were no problems in timeliness
(E17R). At the country level, specific budget was not defined, but the resources for M&E
were integrated into the programming of each component. That is why interviewees said that
the resources allocated were adequate (E17MN, E17ME, E17CRT E17GP E17GI). In
several countries, funds for local monitoring (per diem and transportation) were provided by
the Health Ministries or Vector Control Programs.
52

Long-term Monitoring
Only five places answered "yes completely" to the question "Was long-term monitoring
envisaged as an outcome of the project?". One answered "no not at all" and the rest
"partially" (Annex 2, Table 12, question E18). According to the regional interviewees, an
evaluation was planned to be carried out, but the resources were not provided (E18R).
The opinion of many interviewees was that the best way of evaluating long-term impacts is to
maintain a surveillance of malaria cases in the participatory localities or define some
localities as sentinel sites. Mexico, Guatemala, Nicaragua and El Salvador have
implemented a new surveillance system, which allows long term monitoring in demonstrative
communities (E18ME, E18MN, E18GF, E18CRT, E18CRN, E18PM). In other countries,
there are monitoring systems that should be strengthened to enable a long-term monitoring,
but achieving it requires additional funds (E18R).
A specific protocol must be designed in order to evaluate: entomological, demographic
(migration) and ecological variables as hurricanes, storms and floods, and intersectoral
community participation and other factors related to sustainability.
3.6. Assessment of processes that affected attainment of project
results.
Officially, this project started in May 2003. At the country level, the project started at different
time in each country. Of the evaluated countries, Panama and Costa Rica were the first to
begin the activities (April 2004). The rest started the project at national levels in June of
2004. The activities of institutional arrangements and the adaptation of the mechanisms of
human resources and financial management into the local and national realities delayed the
implementation process of the project by one year.
Once the project started, the implementation was "highly satisfactory", and there were no
important factors that affected the attainment of project results. Mexico received only, a
"satisfactory" overall score, with a "moderately satisfactory" score in Chiapas. The lowest
score, "satisfactory", was received from the preparation and readiness - parameter. The
score was influenced by results mainly in Mexico, Costa Rica and Panama which received
"satisfactory" and "moderately satisfactory" scores in several places. Country ownership,
stakeholder involvement, financial planning and the structure and organization achieved a
"highly satisfactory" score. Only the state level in Mexico obtained an "unsatisfactory" rating
in stakeholder involvement.
53

Table 11. Assessment of processes that affected attainment of project results. Final
Evaluation. Mesoamerica 2008-2009.
F.
F.1.
F.2.
F.3. Stake-
F4.
F.5.
Processes
Preparation
Country
holders
Financial
Structure
PLACE
affected
and
ownership
involvement
Planning
and
attainment
readiness
organiza-
of results
tion
TOTAL
S (5,3)
S (4,8)
HS (5,6)
HS (5,7)
HS (6)
S (5,2)
REGIONAL
HS (5,7)
S (4,8)
HS (6)
HS (6)
HS (6)
HS (5,5)
MEXICO
TOTAL
MS (4,3)
S (4,7)
HS (5,5)
S (4,6)
HS (6)
MS (4,3)
National
S (4,6)
S (4,8)
HS (6)
HS (6)
HS (6)
S (4,5)
Chiapas State
MU (2,8)
MS (3,6)
S (4,7)
2
HS (6)
MS (4,0)
Negualzotcoyol
Comm.
HS (5,8)
HS (6)
HS (5,5)
HS (6)
Nueva Macedonia
Comm.
HS (6)
HS (6)
HS (6)
HS (6)
GUATEMALA
TOTAL
HS (5,6)
S (5,2)
HS (5,5)
HS (5,8)
HS (6)
S (5,3)
National
S (5,4)
S (4,2)
S (5,1)
HS (6)
HS (6)
S (5,0)
Peten Department
HS (5,7)
S (5,4)
HS (5,6)
HS (6)
HS (6)
HS (5,5)
Ixacan Department
HS (5,9)
S (5,4)
HS (6)
HS (6)
HS (6)
HS (5,5)
La Felicidad
Comm.
HS (5,8)
HS (6)
HS (6)
S (5,3)
Sta. Maria Dolores
Comm.
S (5,3)
HS (6)
S (5,0)
S (5,3)
COSTARICA
TOTAL
HS (5,7)
S (4,5)
HS (6)
HS (6)
HS (6)
HS (5,8)
National
HS (5,5)
MS (3,6)
HS (6)
Hs (6)
HS (6)
HS (5,5)
Talamanca
Municipality
HS (5,7)
S (4,8)
HS (6)
HS (6)
HS (6)
HS (6)
Matina Community
HS (6)
HS (6)
HS (6)
HS (6)
PANAMA
TOTAL
HS (5,5)
S (4,7)
HS (5,7)
HS (5,5)
HS (6)
S (5,0)
National
HS (5,7)
S (4,8)
HS (6)
HS (6)
HS (6)
S (5,0)
Bocas del Toro
Province
S (4,4)
MS (3,6)
S (4,7)
S (4,3)
HS (6)
S (5,0)
Puente Blanco
Community
HS (6)
HS (6)
HS (6)
HS (6)
Las Tablas
Community
HS (6)
HS (6)
HS (6)
HS (6)
54

3.6.1. Preparation and readiness
Despite an extensive preparation phase including evaluations on feasibility, base line, and
contacting and involving stakeholders, several actors thought that it was not sufficient
because many administrative arrangements and stakeholder involvement were done along
the implementation of the project thus causing a year's delay in starting activities.
In fourteen (14/17) places the interviewees said that the project's objectives and components
were "completely" clear, practicable and feasible (Annex 2, Table 13, question F1). The
regional and national levels of the 4 evaluated countries answered that it was "partial",
because the scheduled time was insufficient; and the project required two extensions to
complete the scheduled activities (F1R, F1CRN, F1GN). Despite all the programmed
activities were executed there were delays (F1MN, F1MNEG, F1MNM).
In half of the places, the interviewees though that the capacities of the executing institutions
and counterparts were only "partially" considered when the project was designed, the rest
answered that they were considered "completely" (Annex 2, Table 13, question F2).
According to regional level the consideration was assessed "completely" at PDF Phase
(Project Formulation) (F2R). The interviewees from Mexico, who responded "partially" said
that there were several problems: i) internal conflicts in malaria control program, ii) poor
communication with national level about resources that were not authorized to be receive
them by the donor (F2MN), iii) Insufficientinappropriate linkage to the Education Ministry
Comment [p6]: ?
because in some communities, there are no teachers, or they are not involved (F2ME). In
Guatemala the commitment was made with former government and the project began with
another and the transition was not considered (F2GN).
In Costa Rica, mayors and community leaders were not involved in project formulation.
Several stakeholders participated in the formulation of the technical guide (F2CRN).
However, the participation of the entomological component was weak despite having great
importance (F2CRT).
Sixteen respondents said that the lessons from other relevant projects were "completely"
incorporated in design (Annex 2, Table 13, question F3). Only in one place (Panama, Bocas
del Toro) the answer was "partially". In general, Mexico's experience in 2003 was used as a
basis for designing the strategy, but country specific experiences were incorporated and
adapted by others (F3CRN, F3GI), such as the experience of Guatemala related to the use
of larvae eating fish in a previous project funded by PAHO (F3GN). Mexico used the
experience of PLAXSALUD financed by DANIDA Denmark about insecticides (F3R, F3MN).
Six of ten respondents said that the partnership arrangements were partially identified and
the roles and responsibilities negotiated prior to implementation (Annex 2, Table 13, question
F4). At the regional level, the response was completed (F4R). At the country level
commitment papers were signed as a prerequisite for initiating the project, principally with
institutions related to environment and education. Alliances were made after, but formal
statements were made through letters of agreement as a requirement of the project (F4GN).
At local and community levels, many of the arrangements were made during the project with
local authorities, teachers, private companies, and development institutions (F4CRN, F4GP).
In half of the places, the opinion was that the availability of counterpart resources (funding,
staff, and facilities), passage of enabling legislation, and adequate project management
arrangements in place at project entry was "partial" (Question F5). In all countries, the
Ministry of Health contributed funds for the regular implementation of the model (F5ME) and
particularly for salaries for personnel, funds to cover travel expenses and vehicles (F5GN).
The project had the support of vector control personnel, Governing Areas of the Health
55

Ministry staff, the Social Security Institute and the Municipalities (F5CRT). There were
problems in several countries for the delivery of equipment, particularly vehicles (F5CRN),
due to specific legislation of each country to receive donations.
3.6.2. Country ownership / Driveness
According to the opinion of the interviewees, reports and participant observation, there was
high empowerment of the local level health workers and of community leaders. In all levels,
the level of stakeholder participation was very good, through national and local committees
and community working groups.
According to all interviewees the project design was completely in-line with the national
sector and development priorities and plans (Annex 2, Table 14, question F6). All
participating countries have joined to the Millennium Development Goals and adopted Roll
Back Malaria Initiative, as well as, they are signatories of the conventions for the elimination
of POPs. (F6R, F6CRN). Except for Mexico and El Salvador which had successfully reduced
malaria transmission before the start of project, the countries, considered the disease as a
public health priority. In Guatemala, which bore 30% of malaria cases throughout
Mesoamerica, malaria was in fifth place on the priority list (F6GN).
The interviewees considered that the project outcomes were contributing completely to
national development priorities and plans, particularly to sectoral development plans (Annex
2, Table 14, questions F7) (F7R).
The majority thought that the involvement of the relevant country representatives, from
government and civil society was complete (Question F8). Only in two places, the
interviewees considered the involvement to be "partial". Indeed, civil servants of the Vector
Control Program or the Epidemiology Department were focal points at national levels and in
demonstration areas. National committees were formed by representatives from various
ministries and universities. The second-level community organizations (COMUDE in
Guatemala, Comarca in Panama), mayors, police, churches, firemen, etc. participated in the
local committees at demonstration area levels (F8R, F8GN) with an excellent performance
(F8CRT).
In all places, the interviewees thought that the recipient governments maintained completely
their financial commitments to the project (Question F9) (F9R). Despite of temporary
problems (F9MNl), the Health Ministries (F9CRN) and municipalities (F9CRT). never denied
time nor resources). The contributions were mainly in human resources, local transportation
(F9GN) and technical support (F9GI).
The opinion in every country was that government approved policies and regulatory
frameworks were in-line with the project's objectives (Question F10). The approved national
standards and guidelines were in line with the Project Guide (F10R, F10CRT, F10GN). In
Mexico, these regulations were already in place before the project start (F10MN). At
municipal and community levels, the local authorities (mayors, commissioners, oldermen)
adopted regulations to ensure that people complied with malaria control activities. At the
community level, assemblies took decisions to strengthen the implementation of the project
(F10MNEG, F10MNM, F10CR).
Regarding the level of ownership and commitment of the country, only in one place the
interviewees considered the ownership to be "no not at all" and in four places "partial"
(Annex 2, Table 14, question F11). The majority though that it was "complete" (F11CRM,
F11CRT, F11GI). At the regional level, the weakest was Belize (F11R). In some countries,
56

including Panama, Guatemala and Mexico, there were weaknesses at the central level due
to the change of authorities, but it was very appropriate to the local and departmental level
(F11GN). At local levels, there were difficulties when the mayors belonged to a different
political party than the central government (F11ME).
The majority of the interviewees though that the project was completely effective in providing
and communicating information that catalyzed action in participating countries to improve
decisions relating to the use of alternative strategies of malaria vector control without the use
of DDT (Annex 2, Table 14, question F12) (F12R). Some respondents felt that there was no
defined Information Education and Communication plan in each country (F12ME) and that
there was a lack of communication through mass media (F12CRN). Communication at
community level was very good (F12GI).
3.6.3. Structure and organization
No additional structures or organizations were created on to the existent ones. In addition,
there were no supplementary personal agreements devoted to the project in hardly any
country (Annex 2, Table 15, question F24). All the interviewees rated this aspect as
"complete". The local committees joined to the structures of the Ministry of Health, thus,
taking advantage of the technical experience and the structure that remained from the
vertical elimination program.
Except for Mexico, all the interviewees thought that the coordination mechanism between
general health services and malaria control services were complete (Question F25). In
Mexico and Panama, there is still a specialized semi autonomous structure of the program,
but there is a good integration with the general health services (MTE). Only in Costa Rica
and Guatemala the vector workers were completely integrated into the general health
service's structure; in other countries, this integration was "partial" (Question F29).
In Costa Rica, Honduras and El Salvador there was no parallel or independent structure for
vector control, so the area chief was at the same time responsible for the preventive
activities. The Environmental Sanitary Workers (former malaria inspectors) was a
multipurpose unit for all vector borne diseases (dengue, malaria, Chagas, etc.) (PO).
In Guatemala and Nicaragua, the vector control structure persisted, but it was integrated in
Health District at demonstrative areas (it is under the Regional Chief´s leadership). It can be
considered as an integrated structure, but it is not a horizontal structure yet. In Guatemala
and Panama, some interviewees stated that "the vertical structure costs more. It was better
where the intervention was integrated with general health services" (F25N, F25P, F25I).
Most interviewees had the opinion that there was a clear definition of the structure and
organization of the project and of the levels of control and supervision (Question F26).
In all countries visited, there was personnel dedicated to vector control activities, not only
malaria (Annex 2, Table 15, question F27). It was assessed that in Costa Rica the personnel
responsible for vector control carried out activities related to Environmental Primary Health
"completely". The majority said that this involvement was "partial" (Question F28). However,
all vector control workers in demonstrative areas used the project strategy that involved
environmental health activities such as clean house and clean patio.
3.6.4. Stakeholder involvement
57

Stakeholder participation was good at all levels. At the national level, Technical Committees
were constituted of researchers and delegates from universities and several institutions
(mainly Health and Environment). At demonstration areas, with the exception of the state of
Chiapas in Mexico, Technical Local Committees were constituted with participation of
municipalities and NGOs. At the community level, there was high participation of delegates
from majority of the community organizations, which were those who named the members to
the Committees of Control of Malaria or to the Groups of Action in Health.
Almost all interviewees said that the project involved the relevant stakeholders "completely"
through information sharing, consultation and by seeking their participation in project's
design, implementation, and monitoring and evaluation. Only in three places did they state
that involvement was "partial" (Annex 2, Table 16, question F13). Indeed, all relevant
stakeholders were involved in the project formulation, execution and evaluation through
National, Local and Community Committees (F13R). In few places, such as Bocas del Toro
(Panama) and Chiapas (Mexico), the municipalities and Costa Rica, the institutions that
promote tourism did not participate, although they were invited (F13CRN, F13CRT). Some
national civil servants had weak participation in Guatemala (F13GN)
The answer to the question "did the project implement appropriate outreach and public
awareness campaigns?" was similar to the former question (Annex 2, Table 16, question
F14). However, these campaigns were focused on demonstration areas, and were limited to
the national level (F14R F14GN). At the community level, the primary mechanisms were
community assemblies, workshops and person-to-person conversation (F14MN, F14MNM,
F14GN).
Almost all interviewees assessed the statement that the project consulted and made use of
the skills, experience and knowledge of the appropriate government entities, NGOs,
community groups, private sector, local governments and academic institutions in the design,
implementation and evaluation of project activities (Annex 2, Table 16, question F15) as
"completely". Only in Chiapas was the answer "partially". There was an important
participation of the universities and researchers in the studies of impacts of DDT on the
environment and human health, entomology and GIS (F15CRN F15CRT). At the municipal
level, participation of local authorities such as commissariats, aldermen, municipal agents
and health promoters was important (F15MNEG, F15MNM). In Guatemala, three students of
a Masters Public Health Program were involved in health system studies (operational
studies) (F15GN).
Most considered that the perspectives of those that would be affected by decisions, those
that could affect the outcomes and those that could contribute information or other resources
to the process were taken into account completely while taking decisions (Annex 2, Table 16,
question F16). Only in one place did the interviewees state that the consideration was
"partial". Perspectives and opinions of community leaders, majors, managers of private
enterprises, Global Fund projects, and Collaborative Centres (IRET, INSS Mexico and
University of San Luis Potosí) were taken into account (F16R, F16CRN). An interviewee said
"always took into account the thoughts of everyone involved" (F16CRT).
Fourteen interviewees (of 17) thought that the relevant vulnerable groups and the powerful
groups of the supporters and the opponents of the processes were properly involved (Annex
2, Table 16, question F17). There were no opponents: "some persons were indifferent to the
strategy, but little by little with house to house visits, people were sensitized; the project was
easy to sell" (F17CRN). At a community level, all decisions were discussed with leaders and
family members, including children (F17MNEG, F17MNM). In Guatemala, the discussion of
malaria situation and the results of intervention (Situation Room) in the monthly assembly
was a favoured mechanism (F17GN).
58

In eight of ten places, interviewees had the opinion that the mechanisms put in place by the
project for identification and engagement of stakeholders in each participating country were
defined completely and were successful (Annex 2, Table 16, Question F18 and F19). The
Steering and Regional Technical Committees, the National and Local Technical Committees,
the technical teams, the national operative committees, the local operative groups and the
community groups were the better mechanism to identify and engage stakeholders (F18R,
F18CRN).
In eight of the nine places, the interviewees thought that the degree and effectiveness of
collaboration/interactions between the various project partners and institutions during
implementation of the project was completely sufficient (Annex 2, Table 16, question F20).
In general, the interviewees thought that the degree and effectiveness of various public
awareness activities that were undertaken during implementation of the project, both within
the countries as well as in the international context, were "complete". The response was
"partial" in one instance (Annex 2, Table 16, question F21). As a result, no insecticides for
malaria control were used in most of the countries (F20CRN). Many individuals and health
workers did not know the effects of the chemicals at base line study (F21GN). An effective
campaign against the use of POPs was developed, and it enhanced knowledge in all
institutions involved (F20CRT).
3.6.5. Financial planning
According to all interviewees the project had the appropriate financial controls, including
reporting and planning, that allowed management to make informed decisions regarding the
budget and allowed for timely flow of funds (Annex 2, Table 17, Question F22). PAHO at a
central level and in every country had a very well constructed and strict financial control
system (F22CRN, F22MN, F22GN, F22GI).
The interviewees at regional level thought that financial controls were very high "sometimes
in excess". There were changes in the allocation of the funds; new activities not foreseen
were included, particularly for the extension of the project. Nevertheless, any changes in
budget lines were authorized by Managerial Committees and by the donor (F22R). The
expenses were registered in the quarterly reports (F22CRT). In Peten Guatemala, there was
a community audit for delegates of communities (group of 15 leaders) (F22GN).
All interviewees thought that the financial controls, including reporting and planning were
completely strong and useful (Annex 2, Table 17, question F23). However, they allowed the
project management to make informed decisions regarding the budget and allowed a proper
and timely flow of funds for the payment of project deliverables. The controls were invaluable
in order to see which area was working and which not; and also to define where reforms or
corrections should be done (F23GN).
3.6.5. UNEP Supervision and backstopping.
As already mentioned, the start of the project was delayed by more than a year and thus
UNEP authorized two extensions. Thanks to PAHO and UNEP permanent relation, the
problems were identified on time. With exception of the elimination of DDT stockpiles, there
were no serious problems.
PAHO staff provided quality support and advice to the project at country and local levels,
where the intervention of UNEP staff was not necessary. UNEP approved modifications and
59

restructures in time, particularly the approbation of the two extensions. According to regional
interviewees, UNEP's support improved from 2007. However, one of the problems was the
lack of feedback to the quarterly reports, which for the 16 reports, was received only once
(E1R).
3.6.6. Co-financing and Project Outcomes & Sustainability
There was a considerable difference in the level of expected and actual co-financing.
Multilateral agencies, including PAHO and CEC actual contribution in kind was ten times
(92%) more than was programmed. Furthermore, the contribution from central governments
was 13% more (See Table 12).
The contribution from private sector and municipal governments was not foreseen. The
municipal governments financed important infrastructure constructions and provided
logistical and human resources as a counterpart to the project. Their contribution as co-
financing was estimated to be approximately 180,000USD (Table 12). The figures for co-
financing shown in Table 13 were estimated based on interviews with key informants.
According to all interviewees the community contribution was enormous, but it was not
estimated. Indeed they contributed with voluntary work, food, transport and other inputs.
Thanks to these additional contributions, it was possible for the project to be replicated in
other localities and areas. Nevertheless, it was impossible to assure that these contributions
would be ongoing for medium and long term, thus it is necessary to search for new funds.
60

Table 12. Co-financing Project GEF-DDT. September 2003 - December 2008.
Source
In-kind (Mill US$)
Other type of
TOTAL
Difference
resource - PDF-B
(Mill US$)
Budget
Actual
Budget
Actual
Budget
Actual
(Rate per 100)
Multilateral
854.000
12.321.000
440
440
854.440
12.321.440
92,26
Agency (Non-GEF)
Central
51.164
58.838
51.164
58.838
13,04
Governments
Local
0
180
0
180
0,00
Governments
Private Sectors
0
80
0
80
0,00
Total
905.164
12.380.098
440
440
905.604
12.380.538
16,05
*PAHO: Approximately $US 7.848.000; * CEC: Approximately $US 4.473.000
Source: Final Report.
It was not possible to compare the final actual project expenditure by activity to be supplied by the
UNEP Fund management Officer in Terms of Reference (UNEP 2009) because the date of this report
was until 06 of November 2008 and the cost reported by activity at final report by Regional Coordinator
was until December 2008.
Component
Final Report (Ramirez et. Al,
Terms of Reference
2008)
Component 1
2841423
1768022
Component 2
977618
3218255
Component 3
633556
52692
Component 4
2100107
45308
Total
6552704
5084277
3.6.7. Delays and Project Outcomes & Sustainability.
As it was explained above, the institutional arrangements delayed the implementation of the
project by one year, but with the two extensions the project was completed successfully, with
the exception of component 3.
61

4. CONCLUSIONS AND RATINGS
The project has demonstrated at global, regional, national and local levels that methods for
malaria vector control without DDT or other persistent pesticides are cost effective, highly
replicable, and sustainable.
There were not studies of cost effectiveness of the DDT use, but comparing the study of cost
effectiveness of the strategy used in this project with studies of bed nets cost-effectiveness in
Togo (5) (a program integrated to measles immunization campaign) and Tanzania (6) the
cost per person in the GEF-DDT areas is greater than those reported in African studies; but
the cost per case prevented is much lower in Central America than in Africa (5). In relation to
cost per Disability Adjusted Life Year (DALY) averted per 1000 persons, El Salvador and
Honduras showed lower costs, but Nicaragua reported costs twice as high as the Togo
study.
Table 13. Comparison of costs of the GEF-DDT project and bed nets.
GEF-DDT
BED NETS
COSTS
El Salvador Honduras
Nicaragua
Togo
Tanzania
Cost per person 2005-2007
13
17
22
5,95
7,57
(USD$)
Cost per case averted (USD$)
0,4
0,64
0,003
4,4
13
Cost
per DALY averted x
17,22
1,9
44,1
22,1
1000 persons (USD$)
The project was successful in: i) implementing new integrated vector control techniques, ii)
developing a coordinated regional program and iii) improving national capacities to control
malaria and evaluate the impact of POPs, in human health and environment. The project has
strengthened regional, national and local institutional technical and community capacity to
control malaria with methods that do not rely on DDT or other persistent pesticides.
The project had a high impact with a 63 percent reduction of malaria cases in the
demonstration areas from 2004 to 2007. The cases of malaria caused by Plasmodium
falciparum, the species that causes the most severe morbidity and mortality in the world,
reduced by 86,2 percent without any registered deaths due to this disease. Due to the
project's large impact on malaria control in demonstration areas, the reintroduction of DDT
for malaria control will be unlikely. The scaling up of this model could help to achieve the
objective of eliminating malaria from Mesoamerica (WHO 2007;WHO 2008).
The project has raised awareness on the hazards of DDT and other pesticides to the
environment and human health, particularly at demonstrative areas. Lessons learned were
documented and shared at local, national, regional and global levels.
Only one of the four components was not achieved. It was not possible to eliminate the DDT
stock piles in the participating countries, but due to the huge impact in malaria transmission,
the possibility of re-introduction of DDT for malaria control is unlikely.
All the expected results were achieved at three levels: i) At the national level, each one of
the 8 participating countries had documented the results and monitored demonstration
project; ii) At the regional level the lessons learned in each country were exchanged and a
regional consensus was built; iii) The model could be replicable in other places with similar
characteristics.
62

The project's success can be attributed to the implementation of a new model for control of
disease, whose main characteristics are:
1. Stratification to prioritized communities and intensive application of the
interventions in territories and prior groups (more affected, higher risk, higher
vulnerability) (RABREDA-AMI 2007).The methodology of stratification was used to
limit the interventions to a few localities of high transmission and predominance of
vulnerable groups. It improved the efficiency and the efficacy of the control strategy.
In Mesoamerica, most of the malaria cases are concentrated in few localities; the
methodology of stratification developed by Mexico and Guatemala allowed
interventions to be carried out in few communities, thus reaching high cost
effectiveness. The same situation was also discovered by the evaluator in the
Andean area, and it is possible that it is also similar in other continents.
2. The simultaneous combination of several control measures (diagnosis and
treatment, elimination of breeding sites, elimination of human reservoirs), adapted to
specific reality (selective vector control) (Nyarango P et al. 2006).
3. Interventions with impact on other health problems. Clean house, clean patio and
mosquito breeding site control probably served to control other vector borne
diseases, dermatological problems and improve quality of life.
4. Ecosystem approach, with five characteristic elements (Level J 2003):
A control and prevention strategy based on an epidemiological model for health
fields (Dever A 1991), that covers interventions on four fields: i) the biological field
with the clinical management of cases and the elimination of the plasmodium
human hosts, ii) the modification of lifestyles such as
the clean house, clean
patio-strategy and improvement of the personal hygiene, iii) environmental
modification, through the EHCA interventions and the elimination of the use of
persistent insecticides, iv) the improvement of the provision of diagnostic and
treatment services, as well as the integration of the general services in this
activity.
Transdisciplinary approach (Level J 2003), which involved multidisciplinary and
intersectoral approaches with the integration of several professionals from
different
disciplines
(doctors,
biologists,
nurses,
educators,
etc),
several
institutions and organizations, municipalities, universities and investigation
institutes. The experience with the project showed that the municipalities play a
very important role in the implementation of malaria control strategies. The
reasons are the following: i) they can act in a wide range of levels and create a
proper environment for successful local interventions; ii) they can support
community participation and intersectorial collaboration iii) they can offer funding
for infrastructure, logistical support and personnel; iv) the municipal governments
can act as agents for structural change (social, cultural and physical change)
through the formulation of public policies.
Community participation which in the project was situated to a central axis of the
vector control activities, with participation in planning, execution, and monitoring of
activities.
Equity: Due to the fact that areas chosen as demonstration sites were the ones
with persistence of malaria and most of them were rural areas with highly
vulnerable (critical poverty) native population, the concept of social equity was
accomplished. Additionally, the focalization of interventions at the malarious
houses, allowed the neediest people to receive major interventions.
Environmental protection, through the integral strategy for malaria control without
using persistent toxic substances.
5. Strengthen the local capability for basic information (monitoring, evaluation
and surveillance), and investigation.
63

6. Sustainable and replicable interventions. The interventions are ecologically
friendly, highly sustainable, easy to adopt by communities and health workers, and
easy to replicate with small resources.
64

Table 14. OVERALL RATINGS
Evaluator's
Criterion
Evaluator's Summary Comments
Rating
A. Attainment
All countries have adapted techniques of vector control without using
of project
persistent insecticides. However, objective 3 (elimination of DDT
objectives and
stockpiles) has not been achieved completely. The establishment of a
Satisfactory
results (overall
regional network was developed and there was an active exchange
rating)
of best practices and lessons learned among neighbouring countries.
Effectiveness
A reduction of 63% of malaria cases and API decrease from 15,9 to
8,4 per 1000 in between 2002 to 2007 was registered. Several
communities and several departments or provinces where the
Satisfactory
demonstrative areas were located reduced the incidence near the
API of 1, which is considered the standard to qualify the elimination of
malaria. Objective 3 has not been achieved completely
Relevance
The results were coherent with the strategies used. Not only was the
use of DDT and other POPs stopped, but also other types of
insecticide were not used. The project contributed to progress
Highly
towards the achievement of the Millennium Development Goals, the
Satisfactory
Roll Back Malaria goal and the possibility to eliminate the malaria
transmission.
Efficiency
According to the study the project was cost effective. It did not invest
in human resources because it used the existing ones. The major
costs were those of payment of National Coordinators (NAP) and
travel expenses for supervision and community support. But the
Satisfactory
interviewees from national levels of three counties and Chiapas's
State in Mexico thought that the delays in the project partially affected
cost-effectiveness. In Mexico, the project suffered from interruptions
due to the delays"
Sustainability
Although the evaluation rates sustainability as highly satisfactory,
of Project
there are places with high vulnerability. The influence of hurricanes
outcomes
and tropical storms, the reduction of financial resources due to the
Likely
(overall rating)
global crisis and the high level of migration in frontier areas, are
factors that the actors consider might affect the sustainability of the
project"
Financial sustainability was evaluated as "likely" on the regional level
and in Guatemala and "moderately likely" in Panama, Costa Rica and
Mexico. Although it does not diminish the influence of the global
Financial
Likely
crisis, it was considered that the budget that was assigned for next
years should not change significantly from what will be needed to
support the reached achievements"
Only Mexico and Costa Rica National levels consider that the
sustainability could be affected by socio-political factors, particularly
Socio Political
Likely
because in the year 2009, there will be presidential election, which
can cause change of health policies and health authorities.
Institutional
All countries strengthened institutional technical capacity at a regional
framework and
scale for malaria control and the capacity to assess and research
Likely
governance
impact of POP in human health and environment.
Although the presence of hurricanes, tropical storms and floods,
which are frequent in the demonstrative areas, in Guatemala, Costa
Rica and Panama the health services and the communities reacted
Ecological
very well against the effects of tropical storms occurred at the end of
Highly Likely
Comment [p7]: I would still rate this
2008; until January, 2009 epidemics were not reported in the
as likely
demonstrative areas.
Catalytic Role
There was a high scaling up of the control strategy to other
neighbouring localities and municipalities on own initiative of the
Highly
community leaders and health workers. In Guatemala, Nicaragua
Satisfactory
and, Honduras the replication was very extensive for the alliances
that were done with the Global Fund Projects and other projects.
Achievement
In general, respondents felt that the project was successful in
of outputs and
achieving each of the products in quality, size and utility; particularly
activities
in objectives 1, 2 and 4 but not in objective 3. All countries have
Satisfactory
developed
in
building
institutional
capacity
activities.
The
methodology implemented is based on ecosystems approach and
65

Evaluator's
Criterion
Evaluator's Summary Comments
Rating
Roll Back Malaria Strategy. The rapid impact achieved at the start of
the project allowed that the model had a wide support from the
community and local workers. The evidence generated by scientific
studies and the systematization of experiences give the project a
scientific authority and credibility.
Monitoring and
The M&E system of the project worked adequately but it did not allow
Evaluation
quantifying activities, products and results. The surveillance system
(overall rating)
and M&E for control strategies data were gathered in the base line
Moderately
and at the post evaluation end of the project. The most important
Satisfactory
advances of the information system have been given at the GIS
development. These applications allow seeing the power that the GIS
has for the monitoring and evaluation.
At the regional level and in each country there were no specific units
or persons responsible for M&E, which was recognized as one of the
limitations of the project (PO).
Moderately
M&E Design
Lack of uniformity of the surveillance systems for malaria in the
Satisfactory
countries caused a serious difficulty to evaluate the impact and
results of the project
M&E Plan
Quarterly report was the main mechanism for monitoring the process,
Implementation
and result of the objectives planned. Only a few indicators defined in
(use for
technical guide were used to evaluate the results and impact of the
Satisfactory
adaptive
project in malaria epidemiological situation.
management)
Although there were enough funds to assure M&E activities, three
Budgeting and
communities from Mexico and one from Panama responded that "no
Moderately
Funding for
not at all" to the question if the project had adequate budget
Satisfactory
M&E activities
provisions for M&E. Six places, including regional interviewees,
responded "partially" and in ten places, the answer was "completely."
Preparation
There was a long preparation phase which included evaluations of
and readiness
feasibility and base line, contact and involving stakeholders. This
preparation was not sufficient, many administrative arrangements
Satisfactory
and involving stakeholders were done along the implementation of
the project and caused a one year delay in starting activities.
Country
At the national level, Technical Committees were constituted of
ownership /
delegates
from
several
institutions
(Health
and
Environment
driveness
fundamentally), universities or researchers. At demonstrative areas
Technical Local Committees were constituted with participation of
Highly
Municipalities, NGOs. At the Community level, there was a high
Satisfactory
participation
of
delegates
of
the
majority of
the
community
organizations (Committees of Malaria Control or Groups of Health).
Stakeholders
There was high empowerment of the health workers (vectors and
Highly
involvement
environment) at the local levels and of community leaders. In all
Satisfactory
levels, the stakeholder participation was very good.
Financial
The administration of funds was made by the PAHO/WHO of each
Highly
planning and
country.
Satisfactory
cofinancing
UNEP
Thanks to PAHO and UNEP permanent relation the problems were
Supervision
identified on time and there were no serious problems. PAHO staff
and
provided quality support and advice to the project at country and local
backstopping
level, because that was no longer necessary the intervention of
UNEP staff at these levels. UNEP approved modifications and
Satisfactory
restructures in time, particularly the approbation of the two
extensions. According to the regional interviewees, the UNEP
support improved from 2007. One of the problems was the lack of
feedback to the quarterly reports, only one time in 16 reports (E1R).
Overall Rating
Only one objective was not achieved completely due to limitation non
Satisfactory
attributable to project management.
66

5. LESSONS LEARNED
The main lesson learnt was that it is possible to control or even eliminates malaria with
environment-friendly methods and without the use of persistent insecticides. The main
conditions are the combination of control strategies, the intersectorial approach and
community participation. The strategies needed for this kind of intervention, such as the
control of mosquito breeding sites, cleaning houses and patios are easily adopted by the
communities. They also contribute to the empowerment of the communities and to the
change of the understanding about their participation in malaria control (Salinas V & Narváez
A 2009).
Control of mosquito breeding sites and refugees are strategies that are easiest to adopt by
the community, and it is a means for the communities to change their conception about
malaria and the empowerment of malaria control activities. The personal education in every
house in prior localities and the Situation Room are better strategies to educate, to involve
and to empower the communities.
Malaria control requires a multi methodological approach with the combination of
interventions related to: diagnosis and complete
treatment (compliance), reservoir
plasmodium elimination (active search for asymptomatic and febrile persons), control of
mosquito breeding sites with physical and biological methods (larvae eating fish), control of
mosquito refugees (clean house and clean yard), barrier mosquito-persons (Insecticide
Impregnated Bed Nets).
Most malaria cases are located in specific towns and houses. That is why epidemiological
stratification allows focusing the interventions in a few locations and makes the intervention
more effective.
The design of a new demonstration project requires the development of a comprehensive
protocol before the start of execution of: the systematization of experiences, impact
assessment, cost effectiveness evaluation and surveillance, and monitoring system adapted
to the strategy.
Relating to the elimination of DDT stockpiles, although PAHO has handled the component
(including all difficulties) very well, a more experienced organization to conduct this type of
highly specialized activity would have been more efficient. FAO is the UN entity with
comparative advantage and experience in dealing with these specific activities.
For future complex and multi country projects, the duration of the project should be six years,
considering one year for the organization and institutional arrangements, four for
implementation and one for evaluation and preparation of final reports.
67

6. SUGGESTIONS FOR SIMILAR PROJECTS
PAHO and UNEP have to make advocacy with Ministries of Health to declare the eradication
of malaria as a medium-term goal in Mesoamerica to avoid reducing the political and
financial support. PAHO could help countries to design new projects to replicate the strategy
in other areas of high incidence of malaria transmission and hold the localities where it has
been able to eliminate the indigenous transmission. National Malaria Control Programs have
to keep track of the areas and towns as a strategy of long term impact evaluation. It is
important to homogenize and improve surveillance, monitoring and evaluation systems of the
Mesoamerican countries.
UNEP, PAHO and country partners have to identify the strategies, as well as the national
and regional sources of funding that could support the countries in the region of the Americas
and throughout the world to expand the model in other areas with high transmission.
UNEP and PAHO have to continue the promotion and dissemination of the project
achievements and experiences with regional and global reach. The publication of scientific
papers is an important task.
PAHO and UNEP should fund a new study to assess the impact of the project strategy used,
correcting the problems that presented the evaluation of project impact, particularly the
absence of control locations.
Once there is evidence regarding the success of the application of the control model, it is
necessary to initiate a process for gathering scientific evidence, for which it would be
suggested to strengthen alliances between national programs with universities and research
centres. Following is a description of some studies that should be performed in order to
obtain scientific evidence of the model:
1. Impact assessment of the model in the improvement of quality of life of families.
2. Comparative entomological studies amongst localities intervened with the DDT-GEF
model and control localities, with the objective to demonstrate that the first strategy
not only impacts in the reduction of cases, but also in the risk of reintroducing
malaria. The measurement indicators are: vector species, vector habitats, vector
density of adult mosquitoes and mosquito bites index.
3. Assessment of the process, product, results and impact of community participation
with intercultural approach in the decrease of malaria.
4. Assess the impact of a clean house and patio in the transmission of malaria and
dengue, in order to define its contribution on the effectiveness of the model. This is
important because this intervention, as is the daily bath and clothing change, are
based in one case study not published in Mexico.
5. Evaluation or the process, products and results of the replication in the rest of the
localities of the demonstrative areas.
Extend or replicate the OPS-DDT-GEF strategy to the rest of the localities of the municipality,
department or province. Expand from the locality and demonstrative areas to the
demonstrative municipalities and departments, which has already been done spontaneously.
Costs are relatively small compared to the costs to maintain spraying to reduce vector
density.
68

In this sense it is necessary to broaden the objectives or problems to be solved by the
communities. Integrate the interventions in developmental programs or integral disease
control programs. For example the promotion of family gardens in the yards would allow
greater sustainability to the clean patio strategy and additionally improve food safety and
intake of vegetables and beans which is already reduced in malaria communities.
The
experience of Community Epidemiology in Borbon, Ecuador may be taken as reference,
where the communities participated in the control of other prevalent diseases prioritized by
the communities, which allowed maintaining the motivation of leaders and volunteers
(Tognoni, 1998).
At the preparation phase, early diagnosis and prompt treatment were not considered a
priority intervention, but when the Technical Guidelines (Mendez J, Betanzos A, & Tapia R
2004) were formulated, there was no agreement on what scheme to use, so each country
used its own. Consequently all the countries, except Mexico, changed their protocols to
those recommended by PAHO.
Additionally in every country, mainly Guatemala and
Nicaragua, strategies to improve the adherence to treatment were adopted: supervised
mouth treatment or graphic prescription to reduce loss of memory and control of compliance
of complete treatment through visits and supervision of the last dose taken. According to the
evaluator, these interventions are the foundation of the control strategy and have a great
weight in the control of malaria, reason why in any malaria control project or program,
improvement of treatment adherence should be prioritized as main element of the universal
interventions in all the localities disregarding its priority level.
Strengthen health systems: mainly the access to adequate and complete treatments; and
increase laboratory networks, which will not only allow the control and elimination malaria
transmission, but will also rapidly detect reintroduction of transmission in the area in which
native transmission was eliminated. The improvement of laboratory networks will contribute
to a need to improve surveillance in low transmission areas. In order to improve efficiency,
laboratory staff should not only be trained to work with malaria, but also with other health
problems, such as tuberculosis, leischmaniasis, river blindness, HIV, etc.
Maintain monitoring and elimination of breeding sites through community work, followed by
periodical assessments of the sites by vector control workers.
Evaluate the convenience of the use of insecticide impregnated bed nets or residual spraying
in localities where native transmission of malaria has not been eliminated; mainly in those
localities where large breeding sites exist and which cannot be controlled by the community,
nor vector control workers. In some places with temporary high migration (farm workers), a
residual spraying cycle with piretroids may be sufficient, without violating the objective of the
malaria control project without persistent insecticides.
Regarding the parasite reservoir elimination strategy in the current situation for the reduction
of transmission, in order to reduce the efforts and assure sustainability, it is necessary to
review the national norm in three aspects, mainly in Panama and Nicaragua:
Revise the massive population treatment strategy. In Nicaragua, one dose is given to
farmers and in Panama they are providing massive treatments to neighbouring
localities of the demonstrative areas. Prevalence studies should be performed to
asymptomatic patients and its association with the presence of febrile episodes
during the last 12 months using the quality assurance sampling technique by lots in
order to provide a more selective intervention.
69

Focalized treatment. Assess the prevalence of asymptomatic patients in collateral
families and in houses nearby to the malaria cases, in order to reduce the treatment
radius without diagnosis which is currently performed to control the focus.
Integrate to surveillance, the analysis of age of the patients and the time span
between the onset date of symptoms and the beginning of the treatment, in order to
evaluate if the transmission is native or imported, evaluate its transmission intensity
and the opportunity of initiating treatment to redefine focus control strategies.
Continue ongoing training of health workers (mainly new employees of the Ministry of Health
and new epidemiologists) in the control and epidemiological surveillance of malaria.
Establish lobbying activities at national and local level of every country in order assure
continuity of temporary hired personnel in prioritized areas and improve the surveillance,
monitoring and evaluation system for malaria.
In those localities where the number of malaria cases have been reduced to zero,
diversification of functions should be suggested for vector control workers and volunteers, so
they can become promoters of primary environmental care and could also perform
monitoring of other activities, such as vaccination, food safety, etc.
Assess the strategy used to provide complete treatment to family and neighbour contacts
without laboratory diagnosis in currently low transmission areas. Besides the IPA, the use of
ILP, cases of children under 10 years (Kazembe L et al. 2006) of age and the presence of
gametocytes in positive blood smears for P. falciparum, are indicators that should be
included in the surveillance and decision making for the treatment of collateral cases without
microscopic confirmation (Narvaez A 2007).
Advance from the surveillance of cases to the surveillance of meteorological conditions,
breeding sites, and the risk for malaria reintroduction. For this latter objective, active
surveillance should be maintained and screening goals should be determined in those
localities as strategy to early detect outbreaks and epidemics, and the prevention of
reintroduction of new cases.
Strengthen mechanisms within the Municipalities in order to continue supporting efforts and
seek their involvement in the expansion of the strategy to other localities.
For the replication of the model in new localities, joint stratification should be performed
every 6 months or every year, in order to concentrate efforts in new prioritized places.
The influence of global warming is a fact that should be addressed and re-evaluated, since
community and services response was evaluated in disasters such as floods and storms, but
not in situations as drought and hunger, as occurred towards the end of 2009 in the
intervention areas of the project in Guatemala. Population poverty could also reduce the
sustainability of the model.
70

TERMS OF REFERENCE
Terminal Evaluation of the UNEP GEF project
"Regional Program of Action and Demonstration of Sustainable Alternatives to DDT for
Malaria Vector Control in Mexico and Central America"
Project No. GF/2732-03-4680 PMS: GF/4030-03-22
1. PROJECT BACKGROUND AND OVERVIEW
Project rationale
During the last decade Mexico and Central American countries have gradually discontinued
DDT sprayings for vector control. Malaria, however, still poses a serious risk for the
population of these countries. The project aimed at preventing reintroduction of DDT for
malaria control by promoting new integrated vector control techniques and implementing a
coordinated regional program to improve national capacities. Major project components were
to be: the implementation of demonstration projects of vector control without DDT or other
persistent pesticides that can be replicable in other parts of the world and which are cost-
effective, environmentally sound, and sustainable; the strengthening of national and local
institutional capacity to control malaria without the use of DDT; and the elimination of DDT
stockpiles in the eight participating countries.
There was a need to strengthen institutional technical capacity at a regional scale for
assessment and control of malaria disease vectors. Countries with less capacity to address
malaria control without DDT needed help from their neighbours who had had successful
experiences. Only a long-term regional cooperative program could help deter some countries
from returning to use DDT or using other persistent pesticides to control endemic malaria
vectors. The participating countries were committed to developing and implementing
comprehensive management practices that were to build and strengthen awareness about the
importance of environmental conservation and sound water management in the control and
prevention of endemic diseases with the active participation of local communities, particularly
in immigration corridors. The principles which formed the basis for the proposed project
were: integrated inter-institution and inter-sectoral (environment and health) approaches;
broad community participation in all steps of the project; integration of the work to existing
national institutions so that no parallel structures were created; technical, financial and
organizational sustainability of the new approaches to malaria control; and widespread
dissemination of the information generated by the project.
The overall objective of the project was to demonstrate those methods for malaria vector
control without DDT or other persistent pesticides are replicable, cost-effective and
sustainable, thus preventing the reintroduction of DDT in the region. Human health and the
environment were expected to be protected in Mexico and Central America by promoting new
approaches to malaria control, as part of an integrated and coordinated regional program. The
establishment of a regional network was expected to facilitate the exchange of best practices
and lessons learned among neighbouring countries. A major outcome was to increase
government and local community awareness of DDT and other pesticides hazards to the
71

environment and human health, and adjustment of future behaviour regarding the use of
persistent pesticides.
The results of this project was expected be felt at three levels: (i) At the national level, each
one of the 8 participating countries was to have the documented results of a well monitored
demonstration project of malaria vector control without DDT or other persistent pesticides;
(ii) At the regional level the lessons learned in each country were to be exchanged and a
regional consensus built; (iii) At the global level the results of this project were expected to
define replicable models for malaria control based on cost effective, environmentally sound
and sustainable strategies. These models which were to be thoroughly tested and documented
in a series of interconnected demonstration projects were to constitute a set of best practices
which may have been applied in other regions of the world.
Relevance to GEF Programmes
This project conforms with the "Contaminant-based" Operational Programme No 10 and was
to "help demonstrate ways of overcoming barriers to the adoption of best practices that limit
contamination of the International Waters environment".
The proposed activities were
expected to be also consistent with several provisions of the recently adopted Stockholm
Convention on POPs, and with the draft Operational Programme on POPs under development.
Five of the participating countries have already signed the POPs convention: El Salvador,
Honduras, Mexico, Nicaragua and Panama. The other three countries have expressed their
intention to sign it.
Executing Arrangements
The project was implemented by UNEP and executed by the Pan American Health
Organization (PAHO) under the overall responsibility of the Director, Division of Health and
Environment and National Executing Agencies (i.e. Ministries of Health). The UNEP
Division of GEF Co-ordination, in association with PAHO, monitored activities undertaken
during the execution of the project. The Director, Division of Health and Environment, PAHO
was responsible for maintaining systematic overview of the implementation of the project
through monthly project monitoring meetings or consultation and preparing monitoring
reports.
A regional coordinator contracted by PAHO and residing in Guatemala was
responsible for the day-to-day operation of the project, including, soliciting national reports,
preparation of consolidated progress and financial reports, preparation of substantive reports
and review, etc. Each country had a national focal point for this project, appointed by the
executing ministry, and a national coordinator selected and contracted by PAHO in
consultation with the governments and UNEP for the duration of the project.
At the beginning of the project implementation National Steering/Operational
Committees (NOCs) were established in each country as a mechanism for coordination of
national project activities. The NOCs were coordinated by the national focal point with the
participation of the other ministries, technical coordinator and representative of community
organizations and NGOs involved in the project. The technical body, the Regional
Operational Committee chaired by the regional coordinator would include national focal
72

points (Ministries of Health and national technical coordinators) were expected to meet to
discuss, plan and evaluate the technical activities of the project.
Project Activities
The initial project duration was 36 months starting August 2003 and end in July 2006. The
Project was extended to December 2008 to enable continuous collection of field data, the
sustaining of the already achieved project results and important international out-reach of
achievements.
The project activities were organised under the following four components:
Component 1: Demonstration Projects and Dissemination. The objective was to
implement, evaluate, and disseminate the alternative strategies of malaria vector control
without use of DDT which were developed during the PDF-B phase. The main outcome was
to avoid future reintroduction of DDT or other persistent pesticides in national malaria control
programs.
Component 2: Strengthening of national institutional capacity to control malaria without
DDT. The objective was to strengthen national and local institutional capacities to control
malaria with methods that do not rely on DDT or other persistent pesticides. The outcome of
this component was to be strengthened national capacities of malaria risk assessment,
development of analytical laboratory infrastructure, community participation and training
regarding malaria vector control and pesticide management.
Component 3: Elimination of DDT stockpiles. This component was to address the existing
problem of stockpiles in six of the eight participating. All activities were to be documented
and management plans were to be put into place to prevent further accumulation of stockpiles
of pesticides
Component 4: Coordination and Management. A regional coordinator was to be hired for
this project under terms of reference established by the steering committee. Each country was
to have a national coordinator. This component also included three annual meetings of the
steering committee, three regional meetings for planning and evaluation of activities, and
three regional annual reports.
Budget
The total budget is estimated at USD 13,905,400 of which USD 7,495,000 is from the GEF
and USD 5,026,400 in co-financing from national budgets for malaria control programs
specifically oriented to the population of the demonstration project areas. USD 654,000 is in
kind contribution from PAHO and CEC is contributing USD 200,000 to be directed to
assessment of pesticides residues in the two demonstration project areas in Mexico. The initial
budget was adequate to cover the extension period.
73

TERMS OF REFERENCE FOR THE EVALUATION
1. Objective and Scope of the Evaluation
The objective of this terminal evaluation is to determine the extent to which the project
objectives were achieved, or are expected to be achieved, and assess if the project has led to
any other positive or negative consequences. If possible the extent and magnitude of any
project impacts to date will be documented and the likelihood of future impacts will be
determined. The evaluation will also assess project performance and the implementation of
planned project activities and planned outputs against actual results. The evaluation will focus
on the following main questions:
Assess how the project has demonstrated at global, regional, national and local
levels that methods for malaria vector control without DDT or other persistent
pesticides are replicable, cost effective and sustainable, thus preventing the
reintroduction of DDT in the region
Has the projected eliminated the DDT stock piles in the participating countries and
prevented its re-introduction for malaria prevention?
Has the project strengthened national and local institutional technical capacity to
control malaria with methods that do not rely on DDT or other persistent
pesticides?
Has the project raised awareness on the hazards of DDT and other pesticides to the
environment and human health, documented lessons learned and established
mechanisms to share the lessons at national, regional and global levels?
2. Methods
This terminal evaluation will be conducted as an in-depth evaluation using a participatory
approach whereby the UNEP/DGEF Task Manager, key representatives of the executing
agencies and other relevant staff are kept informed and regularly consulted throughout the
evaluation. The consultant will liaise with the UNEP/EOU and the UNEP/DGEF Task
Manager on any logistic and/or methodological issues to properly conduct the review in as
independent a way as possible, given the circumstances and resources offered. UNEP/EOU is
responsible for contracting and logistic matters. The draft report will be circulated to
UNEP/DGEF Task Manager, key representatives of the executing agencies and the
UNEP/EOU. Any comments or responses to the draft report will be sent to UNEP / EOU for
collation and the consultant will be advised of any necessary revisions.
The findings of the evaluation will be based on the following:
A desk review of project documents including, but not limited to: The project
documents, financial and monitoring reports such as progress and financial
reports to UNEP and GEF annual Project Implementation Review reports, the
quarterly and semi-annual reports to UNEP, and relevant correspondence; as
well as other M&E reports, if available.

Desk review of reports and minutes of meetings of the Steering and
Operational Committees.
Field visits to Guatemala, Mexico, Costa Rica and Panama, including meetings
with national focal points, national coordinators, local governments, national
committee members, demonstration site coordinators.
74

Interviews by phone or in person of members of the project Committees.
Phone interviews with project staff in PAHO in Washington, D.C. and telephone
conference with UNEP/DGEF in Nairobi.
Key Evaluation principles.
In attempting to evaluate any outcomes and impacts that the project may have achieved,
evaluators should remember that the project's performance should be assessed by considering
the difference between the answers to two simple questions "what happened?" and "what
would have happened anyway?". These questions imply that there should be consideration
of the baseline conditions and trends in relation to the intended project outcomes and impacts.
In addition it implies that there should be plausible evidence to attribute such outcomes and
impacts to the actions of the project.
Sometimes, adequate information on baseline conditions and trends is lacking. In such cases
this should be clearly highlighted by the evaluator, along with any simplifying assumptions
that were taken to enable the evaluator to make informed judgements about project
performance.
3. Project Evaluation Parameters
A. Attainment of objectives and planned results:
The assessment of project results seeks to determine the extent to which the project
objectives were achieved, or are expected to be achieved, and assess if the project has led
to any other positive or negative consequences. While assessing a project's outcomes the
evaluation will seek to determine the extent of achievement and shortcomings in reaching
the project's objectives as stated in the project document and also indicate if there were
any changes and whether those changes were approved. If the project did not establish a
baseline (initial conditions), the evaluator should seek to estimate the baseline condition
so that achievements and results can be properly established (or simplifying assumptions
used). Since most GEF projects can be expected to achieve the anticipated outcomes by
project closing, assessment of project outcomes should be a priority. Outcomes are the
likely or achieved short-term and medium-term effects of an intervention's outputs.
Examples of outcomes could include but are not restricted to stronger institutional
capacities, higher public awareness (when leading to changes of behaviour), and
transformed policy frameworks or markets. The evaluation should assess the extent to
which the project's major relevant objectives were effectively and efficiently achieved or
are expected to be achieved and their relevance.
Effectiveness: Evaluate how, and to what extent, the stated project objectives have
been met, taking into account the "achievement indicators" specified in the project
document and logical framework1 together with any additional monitoring tools. In
particular, the analysis of outcomes achieved should include, inter alia, an
assessment of whether and to what extent the results of this project have informed
national, regional or international processes.
1 In case in the original or modified expected outcomes are merely outputs/inputs then the evaluators
should assess if there were any real outcomes of the project and if yes then whether these are
commensurate with the realistic expectations from such projects.
75

Relevance: In retrospect, were the project's outcomes consistent with the focal
areas/operational program strategies and country priorities? The evaluation should
also assess whether outcomes specified in the project document and or logical
framework are actually outcomes and not outputs or inputs. Ascertain the nature
and significance of the contribution of the project outcomes to the wider portfolio
of Operational Programme on POPs.
Efficiency: Cost-effectiveness assesses the achievement of the environmental and
developmental objectives as well as the project's outputs in relation to the inputs,
costs, and implementing time. Include an assessment of outcomes in relation to
inputs, costs, and implementation times based on the following questions: Was the
project cost-effective? Was the project the least cost option? Was the project
implementation delayed and if it was then did that affect cost-effectiveness? The
evaluation should assess the contribution of cash and in-kind co-financing to
project implementation and to what extent the project leveraged additional
resources.
B. Assessment of Sustainability of project outcomes:
Sustainability is understood as the probability of continued long-term project-derived
outcomes and impacts after the GEF project funding ends. The evaluation will identify
and assess the key conditions or factors that are likely to contribute or undermine the
persistence of benefits after the project ends. Some of these factors might be outcomes of
the project, e.g. stronger institutional capacities or better informed decision-making. Other
factors will include contextual circumstances or developments that are not outcomes of
the project but that are relevant to the sustainability of outcomes. The evaluation should
ascertain to what extent follow-up work has been initiated and how project outcomes will
be sustained and enhanced over time.
Four aspects of sustainability should be addressed: financial, socio-political, institutional
frameworks and governance, and ecological. The following questions provide guidance
on the assessment of these aspects:
Financial resources. To what extent are the outcomes of the project dependent
on continued financial support? What is the likelihood that any required
financial and economic resources will be available to sustain the project
outcomes/benefits will be sustained once the GEF assistance ends (resources
can be from multiple sources, such as the public and private sectors, income
generating activities, and market trends that support the project's objectives)?
Was the project successful in identifying and leveraging co-financing?
Socio-political: To what extent are the outcomes of the project dependent on
socio-political factors? What is the likelihood that the level of stakeholder
ownership will allow for the project outcomes/benefits to be sustained? Is there
sufficient public/stakeholder awareness in support of the term objectives of the
project?
Institutional framework and governance: To what extent are the outcomes of
the project dependent on issues relating to institutional frameworks and
governance? What is the likelihood that institutional and technical
achievements, legal frameworks, policies and governance structures and
processes will allow for, the project outcomes/benefits to be sustained? While
responding to these questions consider if the required systems for
76

accountability and transparency and the required technical know-how are in
place.
Environmental: Are there any environmental risks that can undermine the
future flow of the project's environmental benefits? Are there any risks to the
ecological sustainability of this project? The Terminal Evaluation should
assess whether certain activities in the project area will pose a threat to the
sustainability of the project outcomes. For example, construction of dam in a
protected area could inundate a sizable area and thereby negatively impact the
biodiversity related gains made by the project or, a newly established pulp mill
might jeopardise the viability of nearby protected forest areas by increasing
logging pressures.
As far as possible, also assess the potential longer-term impacts considering that the
evaluation is taking place upon completion of the project and that longer term impact is
expected to be seen in a few years time. Frame any recommendations to enhance future
project impact in this context. Which will be the major `channels' for longer term impact
from the project at the national and international scales? The evaluation should formulate
recommendations that outline possible approaches and necessary actions to facilitate an
impact assessment study in a few years time.
C. Catalytic role
The terminal evaluation will also describe any catalytic or replication effect of the
project. What examples are there of replication and catalytic outcomes that suggest
increased likelihood of sustainability? Replication approach, in the context of GEF
projects, is defined as lessons and experiences coming out of the project that are
replicated or scaled up in the design and implementation of other projects. Replication
can have two aspects, replication proper (lessons and experiences are replicated in
different geographic area) or scaling up (lessons and experiences are replicated within the
same geographic area but funded by other sources). If no effects are identified, the
evaluation will describe the catalytic or replication actions that the project carried out.
D. Achievement of outputs and activities:
Assess the soundness and effectiveness of the methodologies used for developing
regional programme of action
Assess the establishment of the necessary legal, institutional, technical and financial
capacities and mechanisms within the region which should have allowed the effective
joint implementation of the project.
Assess to what extent the project outputs produced have the weight of scientific
authority / credibility, necessary to influence policy and decision-makers, particularly
at the national or regional levels.
Delivered outputs: Assessment of the project's success in producing each of the
programmed outputs, both in quantity and quality as well as usefulness and timeliness.
E. Assessment of Monitoring and Evaluation Systems:
M&E design. Did the project have a sound M&E plan to monitor results and
track progress towards achieving project objectives? The Terminal Evaluation
will assess whether the project met the minimum requirements for project design
of M&E and the application of the Project M&E plan (Minimum requirements
are specified in Annex 4). The evaluation shall include an assessment of the
77

quality, application and effectiveness of project monitoring and evaluation plans
and tools, including an assessment of risk management based on the assumptions
and risks identified in the project document. The M&E plan should include a
baseline (including data, methodology, etc.), SMART (see Annex 4) indicators
and data analysis systems, and evaluation studies at specific times to assess
results. The time frame for various M&E activities and standards for outputs
should have been specified.
M&E plan implementation. Was an M&E system in place and did it facilitate
tracking of results and progress towards projects objectives throughout the
project implementation period. Were Annual project reports complete, accurate
and with well justified ratings? Was the information provided by the M&E
system used during the project to improve project performance and to adapt to
changing needs? Did the Projects have an M&E system in place with proper
training for parties responsible for M&E activities to ensure data will continue to
be collected and used after project closure?
Budgeting and Funding for M&E activities. Were adequate budget provisions
made for M&E made and were such resources made available in a timely fashion
during implementation?
Long-term Monitoring. Was long-term monitoring envisaged as an outcome of
the project? If so, comment specifically on the relevance of such monitoring
systems to sustaining project outcomes and how the monitoring effort will be
sustained.
F. Assessment of processes that affected attainment of project results.
The evaluation will consider, but need not be limited to, consideration of the following
issues that may have affected project implementation and attainment of project results:
i.
Preparation and readiness.
Were the project's objectives and components
clear, practicable and feasible within its timeframe? Were capacities of the
executing institutions and counterparts properly considered when the project was
designed? Were lessons from other relevant projects properly incorporated in
design? Were the partnership arrangements properly identified and the roles and
responsibilities negotiated prior to implementation? Was availability of
counterpart resources (funding, staff, and facilities), passage of enabling
legislation, and adequate project management arrangements in place at project
entry?
Ascertain to what extent the project implementation mechanisms outlined
in the project document have been closely followed. In particular, assess
the role of the various committees established and whether the project
document was clear and realistic to enable effective and efficient
implementation, whether the project was executed according to the plan
and how well the management was able to adapt to changes during the life
of the project to enable the implementation of the project.
Evaluate the effectiveness and efficiency and adaptability of project
management and the supervision of project activities / project execution
arrangements at all levels (1) policy decisions: Steering Group; (2) day to
day project management; (3) GEF guidance: UNEP DGEF
78

ii.
Country ownership/Driveness. This is the relevance of the project to national
development and environmental agendas, recipient country commitment, and
regional and international agreements. Examples of possible evaluative questions
include: Was the project design in-line with the national sectoral and
development priorities and plans? Are project outcomes contributing to national
development priorities and plans? Were the relevant country representatives,
from government and civil society, involved in the project? Did the recipient
governments maintain its financial commitment to the project? Have the
governments approved policies or regulatory frameworks been in-line with the
project's objectives? Specifically the evaluation will:
iii.
Assess the level of country ownership and commitment. Specifically, the
evaluator should assess whether the project was effective in providing and
communicating information that catalyzed action in participating countries
to improve decisions relating to the use of alternative strategies of malaria
vector control without use of DDT.
iv.
Stakeholder involvement. Did the project involve the relevant stakeholders
through information sharing, consultation and by seeking their participation in
project's design, implementation, and monitoring and evaluation? For example,
did the project implement appropriate outreach and public awareness
campaigns? Did the project consult and make use of the skills, experience and
knowledge of the appropriate government entities, NGOs, community groups,
private sector, local governments and academic institutions in the design,
implementation and evaluation of project activities? Were perspectives of those
that would be affected by decisions, those that could affect the outcomes and
those that could contribute information or other resources to the process taken
into account while taking decisions? Were the relevant vulnerable groups and the
powerful, the supporters and the opponents, of the processes properly involved?
Specifically the evaluation will:
Assess the mechanisms put in place by the project for identification and
engagement of stakeholders in each participating country and establish, in
consultation with the stakeholders, whether this mechanism was successful,
and identify its strengths and weaknesses.
Assess the degree and effectiveness of collaboration/interactions between
the various project partners and institutions during the course of
implementation of the project.
Assess the degree and effectiveness of any various public awareness
activities that were undertaken during the course of implementation of the
project both within the countries as well as in the international context.
v.
Financial planning. Did the project have the appropriate financial controls,
including reporting and planning, that allowed management to make informed
decisions regarding the budget and allowed for timely flow of funds?
Specifically, the evaluation should:
Assess the strength and utility of financial controls, including reporting,
and planning to allow the project management to make informed decisions
regarding the budget and allow for a proper and timely flow of funds for
the payment of satisfactory project deliverables throughout the project's
lifetime.
Present the major findings from the financial audit if one has been
conducted.
79

Did promised co-financing materialize? Identify and verify the sources of
co-financing as well as leveraged and associated financing (in co-operation
with the IA and EA).
Assess whether the project has applied appropriate standards of due
diligence in the management of funds and financial audits.
The evaluation should also include a breakdown of final actual project
costs by activities compared to budget (variances), financial management
(including disbursement issues), and co- financing. This information will
be prepared by the relevant DGEF Fund Management Officer of the project
for scrutiny by the evaluator (table attached in Annex 1 Co-financing and
leveraged resources).
vi.
UNEP Supervision and backstopping. Did UNEP Agency staff identify
problems in a timely fashion and accurately estimate its seriousness? Did UNEP
staff provide quality support and advice to the project, approved modifications in
time and restructure the project when needed? Did UNEP and Executing
Agencies provide the right staffing levels, continuity, skill mix, frequency of
field visits?
vii.
Co-financing and Project Outcomes & Sustainability. If there was a
difference in the level of expected co-financing and actual co-financing, then
what were the reasons for this? Did the extent of materialization of co-financing
affect the project's outcomes and/or sustainability, and if it did affect outcomes
and sustainability then in what ways and through what causal linkages?
viii.
Delays and Project Outcomes & Sustainability. If there were delays in project
implementation and completion, the evaluation will summarise the reasons for
them. Did delays affect the project's outcomes and/or sustainability, and if so in
what ways and through what causal linkages?
The ratings will be presented in the form of a table with each of the categories rated
separately and with brief justifications for the rating based on the findings of the main
analysis. An overall rating for the project should also be given. The rating system to be
applied is specified in Annex 1.
4. Evaluation Report Format and Review Procedures
The report should be brief, to the point and easy to understand. It must explain; the purpose of
the evaluation, exactly what was evaluated and the methods used. The report must highlight
any methodological limitations, identify key concerns and present evidence-based findings,
consequent conclusions, recommendations and lessons. The report should provide information
on when the evaluation took place, the places visited, who was involved and be presented in a
way that makes the information accessible and comprehensible. The report should include an
executive summary that encapsulates the essence of the information contained in the report to
facilitate dissemination and distillation of lessons.
Evidence, findings, conclusions and recommendations should be presented in a complete and
balanced manner. The evaluation report shall be written in English, be of no more than 50
pages (excluding annexes), use numbered paragraphs and include:
80

i)
An executive summary (no more than 3 pages) providing a brief overview of
the main conclusions and recommendations of the evaluation;
ii)
Introduction and background giving a brief overview of the evaluated
project, for example, the objective and status of activities;
iii)
Scope, objective and methods presenting the evaluation's purpose, the
evaluation criteria used and questions to be addressed;
iv)
Project Performance and Impact providing factual evidence relevant to the
questions asked by the evaluator and interpretations of such evidence. This is
the main substantive section of the report and should provide a commentary on
all evaluation aspects (A - F above).
v)
Conclusions and rating of project implementation success giving the
evaluator's concluding assessments and ratings of the project against given
evaluation criteria and standards of performance. The conclusions should
provide answers to questions about whether the project is considered good or
bad, and whether the results are considered positive or negative;
vi)
Lessons learned presenting general conclusions from the standpoint of the
design and implementation of the project, based on good practices and
successes or problems and mistakes. Lessons should have the potential for
wider application and use. All lessons should `stand alone and should:
Specify the context from which they are derived
State or imply some prescriptive action;
Specify the contexts in which they may be applied (if possible who
when and where)
vii)
Recommendations suggesting actionable proposals regarding improvements
of the current project. They may cover, for example, resource allocation,
financing, planning, implementation, and monitoring and evaluation.
Recommendations should always be specific in terms of who would do what,
provide a timeframe, and a measurable performance target. In general,
Terminal Evaluations are likely to have very few (only two or three) actionable
recommendations;
viii)
Annexes include Terms of Reference, list of interviewees, documents
reviewed, brief summary of the expertise of the evaluator / evaluation team, a
summary of co-finance information etc. Dissident views or management
responses to the evaluation findings may later be appended in an annex.
Examples of UNEP GEF Terminal Evaluation Reports are available at www.unep.org/eou
Review of the Draft Evaluation Report
Draft reports submitted to UNEP EOU are shared with the corresponding Programme or
Project Officer and his or her supervisor for initial review and consultation. The DGEF staff
and senior Executing Agency staff are allowed to comment on the draft evaluation report.
They may provide feedback on any errors of fact and may highlight the significance of such
errors in any conclusions. The consultation also seeks agreement on the findings and
recommendations. UNEP EOU collates the review comments and provides them to the
evaluators for their consideration in preparing the final version of the report.
All UNEP GEF Evaluation Reports are subject to quality assessments by UNEP EOU. These
incorporate GEF Office of Evaluation quality assessment criteria and are used as a tool for
providing structured feedback to the evaluator (see Annex 3).
81

5. Submission of Final Terminal Evaluation Reports.
The final report shall be submitted in electronic form in MS Word format and should be sent
to the following persons:
Segbedzi Norgbey, Chief, Evaluation and Oversight Unit
UNEP, P.O. Box 30552
Nairobi, Kenya
Tel.: (254-20) 7623387
Fax: (254-20) 7623158
Email: segbedzi.norgbey@unep.org
With a copy to:
Maryam Niamir-Fuller
Director
UNEP/Division of GEF Coordination
P.O. Box 30552
Nairobi, Kenya
Tel: + 254-20-7624165
Fax: + 254-20-624041/4042
Email: Maryam.Niamir-Fuller@unep.org
Jan Betlem
UNEP/GEF Task Manager
United Nations Environment Programme (UNEP)
Division of GEF Coordination (DGEF)
P. O Box 30552
Nairobi, Kenya
Tel: 254 20 7624607
Fax: 254 20 7624041/2
Email: Jan.Betlem@unep.org
Carmen Tavera
Portfolio Manager
United Nations Environment Programme (UNEP)
Division of GEF Coordination (DGEF)
P.O Box 30552
Nairobi, Kenya
Tel: 254 20 7624153
Email:Carmen.Tavera@unep.org
The final evaluation report will be printed in hard copy and published on the Evaluation and
Oversight Unit's web-site www.unep.org/eou. Subsequently, the report will be sent to the
GEF Office of Evaluation for their review, appraisal and inclusion on the GEF website. In
addition the final Evaluation report will disseminated to: The relevant GEF Focal points,
Relevant Government representatives, UNEP DGEF Professional Staff, The project's
Executing Agency and Technical Staff. The full list of intended recipients is attached in
Annex 5.
82

1. Resources and schedule of the evaluation
This terminal evaluation will be undertaken by an international evaluator contracted by the
Evaluation and Oversight Unit, UNEP. The contract for the evaluator will begin on 8th
December 2008 and end on 9th March 2009 (2 months spread over 3 months). The evaluator
will submit a draft report on 15th February 2009 to UNEP/EOU, the UNEP/DGEF Task
Manager, and key representatives of the executing agencies. Any comments or responses to
the draft report will be sent to UNEP / EOU for collation and the consultant will be advised of
any necessary revisions. Comments to the final draft report will be sent to the consultant by
28th February 2009 after which, the consultant will submit the final report no later than 9th
March 2009.
In accordance with UNEP policy, all UNEP projects are evaluated by an independent
evaluator contracted by the EOU. The evaluator should not have been associated with the
design and implementation of the project. The evaluator will work under the overall
supervision of the Chief, Evaluation and Oversight. The evaluator should have the following
minimum qualifications: (i) technical experience in the area of pesticides and insecticides; (ii)
expertise in human and environmental health related issues; (iii) experience with
implementation and management of GEF projects, in particular demonstration projects
(desirable); (iv) experience in project evaluation and (v) working experience in the region.
Excellent command of spoken and written English and Spanish a requirement.
The evaluator will travel to office of regional coordinator located in Guatemala. He will also
travel to demonstrations sites in Guatemala, Mexico, Costa Rica and Panama. Mexico has the
two largest demonstration sites, and the Guatemala site is close to the Mexican border. Costa
Rica reportedly has a high level local participation and prepared well for demonstration site
activities. Costa Rica and Panama have developed border cooperation on malaria control.
7. Schedule Of Payment
Lump-Sum Option
The evaluator will receive an initial payment of 30% of the total amount due upon signature
of the contract. A further 30% will be paid upon submission of the draft report. A final
payment of 40% will be made upon satisfactory completion of work. The fee is payable under
the individual Special Service Agreement (SSA) of the evaluator and IS inclusive of all
expenses such as travel, accommodation and incidental expenses.
In case, the evaluator cannot provide the products in accordance with the TORs, the
timeframe agreed, or his products are substandard, the payment to the evaluator could be
withheld, until such a time the products are modified to meet UNEP's standard. In case the
evaluator fails to submit a satisfactory final product to UNEP, the product prepared by the
evaluator may not constitute the evaluation report.
83

Annex 1. OVERALL RATINGS TABLE
Evaluator's
Summary
Evaluator
Criterion
Comments
's Rating
Attainment of project objectives
and results (overall rating)
Sub criteria (below)
Effectiveness
Relevance
Efficiency
Sustainability of Project
outcomes (overall rating)
Sub criteria (below)
Financial
Socio Political
Institutional framework and
governance
Ecological
Achievement of outputs and
activities
Monitoring and Evaluation
(overall rating)
Sub criteria (below)
M&E Design
M&E Plan Implementation (use
for adaptive management)
Budgeting and Funding for M&E
activities
Catalytic Role
Preparation and readiness
Country ownership / driveness
Stakeholders involvement
Page 84 of 99

Evaluator's
Summary
Evaluator
Criterion
Comments
's Rating
Financial planning
UNEP Supervision and
backstopping
Overall Rating
RATING OF PROJECT OBJECTIVES AND RESULTS
Highly Satisfactory (HS): The project had no shortcomings in the achievement of its
objectives, in terms of relevance, effectiveness or efficiency.
Satisfactory (S): The project had minor shortcomings in the achievement of its objectives, in
terms of relevance, effectiveness or efficiency.
Moderately Satisfactory (MS): The project had moderate shortcomings in the achievement of
its objectives, in terms of relevance, effectiveness or efficiency.
Moderately Unsatisfactory (MU): The project had significant shortcomings in the
achievement of its objectives, in terms of relevance,
effectiveness or efficiency.
Unsatisfactory (U): The project had major shortcomings in the achievement of its objectives,
in terms of relevance, effectiveness or efficiency.
Highly Unsatisfactory (HU): The project had severe shortcomings in the achievement of its
objectives, in terms of relevance, effectiveness or efficiency.
Please note: Relevance and effectiveness will be considered as critical criteria. The overall
rating of the project for achievement of objectives and results may not be higher than the
lowest rating on either of these two criteria. Thus, to have an overall satisfactory rating for
outcomes a project must have at least satisfactory ratings on both relevance and effectiveness.
RATINGS ON SUSTAINABILITY
Sustainability will be understood as the probability of continued long-term outcomes and
impacts after the GEF project funding ends. The Terminal evaluation will identify and assess
the key conditions or factors that are likely to contribute or undermine the persistence of
benefits after the project ends. Some of these factors might be outcomes of the project, i.e.
stronger institutional capacities, legal frameworks, socio-economic incentives /or public
awareness. Other factors will include contextual circumstances or developments that are not
outcomes of the project but that are relevant to the sustainability of outcomes..
Rating system for sustainability sub-criteria
On each of the dimensions of sustainability of the project outcomes will be rated as follows.
Likely (L): There are no risks affecting this dimension of sustainability.
Page 85 of 99

Moderately Likely (ML). There are moderate risks that affect this dimension of
sustainability.
Moderately Unlikely (MU): There are significant risks that affect this dimension of
sustainability
Unlikely (U): There are severe risks that affect this dimension of sustainability.
All the risk dimensions of sustainability are critical. Therefore, overall rating for sustainability
will not be higher than the rating of the dimension with lowest ratings. For example, if a
project has an Unlikely rating in either of the dimensions then its overall rating cannot be
higher than Unlikely, regardless of whether higher ratings in other dimensions of
sustainability produce a higher average.
RATINGS OF PROJECT M&E
Monitoring is a continuing function that uses systematic collection of data on specified
indicators to provide management and the main stakeholders of an ongoing project with
indications of the extent of progress and achievement of objectives and progress in the use of
allocated funds. Evaluation is the systematic and objective assessment of an on-going or
completed project, its design, implementation and results. Project evaluation may involve the
definition of appropriate standards, the examination of performance against those standards,
and an assessment of actual and expected results.
The Project monitoring and evaluation system will be rated on `M&E Design', `M&E Plan
Implementation' and `Budgeting and Funding for M&E activities' as follows:
Highly Satisfactory (HS): There were no shortcomings in the project M&E system.
Satisfactory(S): There were minor shortcomings in the project M&E system.
Moderately Satisfactory (MS): There were moderate shortcomings in the project M&E
system.
Moderately Unsatisfactory (MU): There were significant shortcomings in the project
M&E system.
Unsatisfactory (U): There were major shortcomings in the project M&E system.
Highly Unsatisfactory (HU): The Project had no M&E system.
"M&E plan implementation" will be considered a critical parameter for the overall
assessment of the M&E system. The overall rating for the M&E systems will not be higher
than the rating on "M&E plan implementation."
All other ratings will be on the GEF six point scale.
GEF Performance Description
Alternative description on
the same scale
HS
= Highly Satisfactory
Excellent
S
= Satisfactory
Well above average
MS
= Moderately Satisfactory
Average
MU
= Moderately Unsatisfactory
Below Average
Page 86 of 99

U
= Unsatisfactory
Poor
HU
= Highly Unsatisfactory
Very poor (Appalling)
Page 87 of 99

Annex 2. Co-financing and Leveraged Resources
Co-financing (basic data to be supplied to the consultant for verification)
IA own
Government
Other*
Total
Total
Co financing
Financing
Disbursement
(Type/Source)
(mill US$)
(mill US$)
(mill US$)
(mill US$)
(mill US$)
Planned
Actual
Planned
Actual
Planned
Actual
Planned
Actual
Planned
Actual

Grants

Loans/Concessio
nal (compared to
market rate)

Credits

Equity
investments

In-kind support
90,000
654,000
744,000

Other (*)
7,165,000
5,026,000
200,000
12,391,000
-
-
-
-
-
7,165,000
7,165,000
5,116,000
5,116,000
854,000
13,135,000
Totals
* Other is referred to contributions mobilized for the project from other multilateral agencies, bilateral development cooperation
agencies, NGOs, the private sector and beneficiaries.
Leveraged Resources
Page 88 of 99

Leveraged resources are additional resources--beyond those committed to the project itself at the time of approval--that are mobilized
later as a direct result of the project. Leveraged resources can be financial or in-kind and they may be from other donors, NGO's,
foundations, governments, communities or the private sector. Please briefly describe the resources the project has leveraged since
inception and indicate how these resources are contributing to the project's ultimate objective.
Table showing final actual project expenditure by activity to be supplied by the UNEP Fund management Officer. (insert here)
Global Budget: Regional Program of Action and Demonstration of
Sustainable Alternatives to DDT for Malaria Vector Control in Mexico and
ACTUAL EXPENDITURES AS AT 06.11.2008, PAHO ARE YET TO REPORT JULY TO DEC 2008
Central America
EXPENDITURES
2003
2004
2005
2006
2007
2008
Total
ACTUAL
ACTUAL
ACTUAL
ACTUAL
ACTUAL
ACTUAL
1101 National Project Coordinator
82,110
-82,110
0
1102 Regional Coordinator
0
107,752
93,085
141,082
149,473
114,594
605,985
1103 Technical Consultant for Admin & Techn Support
15,134
18,486
1,987
30,366
51,586
6,888
124,447
Total
15,134
126,238
95,072
171,448
283,169
39,372
730,433
1201 Baseline and technical evaluation of Demo Projects
0
6,829
0
0
61,753
9,075
77,657
1202 Assesment of Environment and human exposure
375
-375
-1
1203 Communication plan to promote public awareness and participation
Demo
14,940
-14,940
0
1204 Strenghtening of National malaria reference centres for data
gathering
4,284
-4,284
0
1205 Devpt of Technical Guide for Demo Projects
0
3,469
13,715
0
1
17,185
1206 Webpage and internet maintenance of info and results of Demo
Projects
0
23,121
0
4,520
9,743
37,384
1207 Establishment of a national/regional malaria surveillance system
0
0
0
0
0
0
0
1208 Rapid Test Validation for assessment of human exposure to DDT
0
0
0
0
0
0
1209 Inter-Laboratory quality control program and capacity building
0
0
0
0
0
0
1210 Implementation of GIS application with maps and data of Demo
projects
0
49,999
16,805
61,519
31,243
8,148
167,714
Page 89 of 99

1211 Updating of national inventories of DDT stockpiles and evaluation of
the problem at regional level.
0
22,338
-9,538
1,651
49,253
63,704
1212 Identification and mapping of areas perviously sprayed with DDT
0
0
0
0
4,443
4,443
1281 Terminal Evaluation (to be paid directly by UNEP)
0
0
0
0
176,034
0
176,034
Total
0
105,756
20,982
67,690
176,034
-2,376
368,085
1381 Project Support Cost (PAHO 8%)
4,779
43,841
51,255
97,109
186,309
54,343
437,637
1382 Project Preparatory Costs Recovery
38,380
0
0
-384
0
4
38,000
Total
43,159
43,841
51,255
96,726
186,309
54,347
475,637
1601 Travel of local participants (ntl coordinators and local leaders)
0
0
0
1,311
94,597
12,643
108,551
1602 Short term travel for malaria technicians
0
2,630
0
-2,630
1,834
-1,834
0
1603 Regional Staff travel
3,710
10,299
11,061
26,174
20,000
15,072
86,316
Total
3,710
12,929
11,061
24,855
116,431
25,881
194,867
1999 Component Total
62,003
288,764
178,370
360,719
761,943
117,223
1,769,022
2101 Environment field interventions and analyticals costs
0
166
0
-166
151,619
-151,619
0
2201: Belize
0
34,954
38,660
43,656
23,937
29,512
170,719
2202: Costa Rica
0
16,653
35,159
45,082
54,100
25,391
176,384
2203: El Salvador
0
34,892
40,451
62,414
62,035
60,897
260,689
2204: Guatemala
0
31,531
67,430
72,114
90,934
135,089
397,098
2205: Honduras
0
36,452
86,067
55,709
133,859
82,475
394,562
2206: Mexico
0
77,485
56,554
384,352
411,107
349,390
1,278,888
2207: Nicaragua
0
24,248
67,670
95,652
84,473
78,319
350,362
2208: Panama
0
45,426
52,816
17,718
17,987
55,606
189,553
Total
0
301,807
444,807
776,531
1,030,051
665,059
3,218,255
2999 Component Total
0
301,807
444,807
776,531
1,030,051
665,059
3,218,255
3101 Fellowship for qualified lab technicians for training in center of
excellence
0
0
0
0
0
3,620
3,620
Total
0
0
0
0
0
3,620
3,620
3201 Training courses and educational activities
81,197
-81,197
0
3202 Technical Regional Workshop to discuss Technical Guide and
Activities of Demo. Projects
0
11,473
-2,221
0
2,244
-2,244
9,252
3203 Training courses for demo projects with local communities
464
-464
0
Page 90 of 99

Total
0
11,473
-2,221
0
83,904
-83,905
9,251
3301 3 Steering Committee Meetings
40,324
-10,462
23,651
36,471
63,272
47,107
200,364
3302 Regional Meetings of Operational Committee
0
0
22,013
55,625
1,780
0
79,418
3303 Local meetings with Demo Projects communities
0
0
27,992
52,964
14,210
-1,218
93,947
3304 Evaluation of demo projects with locals
15,169
-15,169
0
Total
40,324
-10,462
73,656
145,060
94,431
30,720
373,729
3999 Component Total
40,324
1,011
71,435
145,060
178,336
-49,565
386,601
4101 Repackage of DDT and other persistent pesticides in UN approved
containers
120,556
0
120,556
4201 Basic equipment for monitoring Demo Projects
0
257
4,607
11,845
5,604
725
23,039
4202 Strengthening of analytical infrastructure and technical capacity for
assessing pesticide exposure
0
0
13,592
7,521
359,476
2,736
383,325
Total
0
257
18,199
19,366
485,637
3,461
526,920
4999 Component Total
0
257
18,199
19,366
485,637
3,461
526,920
5201 3 Regional Annual Reports
0
0
0
0
5,189
8,578
13,767
5202 Publication of Final Report (CD, Video, Book) on alternative
strategies of malaria control
0
0
2,498
0
6,791
22,252
31,541
Total
0
0
2,498
0
11,980
30,830
45,308
5999 Component Total
0
0
2,498
0
11,980
30,830
45,308
GRAND TOTAL
102,327
591,839
715,309
1,301,676
2,467,946
767,009
5,946,106
Previous Budget
102,327
591,839
715,309
5,186,561
568,964
0
7,165,000
Variance Rev. 03
0
0
0
-3,884,885
1,898,982
767,009
-1,218,894
Page 91 of 99

Annex 3
Review of the Draft Report
Draft reports submitted to UNEP EOU are shared with the corresponding Programme or
Project Officer and his or her supervisor for initial review and consultation. The DGEF
staff and senior Executing Agency staff provide comments on the draft evaluation
report. They may provide feedback on any errors of fact and may highlight the
significance of such errors in any conclusions. The consultation also seeks agreement
on the findings and recommendations. UNEP EOU collates the review comments and
provides them to the evaluators for their consideration in preparing the final version of
the report. General comments on the draft report with respect to compliance with these
TOR are shared with the reviewer.
Quality Assessment of the Evaluation Report
All UNEP GEF Mid Term Reports are subject to quality assessments by UNEP EOU.
These apply GEF Office of Evaluation quality assessment and are used as a tool for
providing structured feedback to the evaluator.
The quality of the draft evaluation report is assessed and rated against the following
criteria:
GEF Report Quality Criteria
UNEP EOU Assessment
Rating
A. Did the report present an assessment of relevant outcomes
and achievement of project objectives in the context of the
focal area program indicators if applicable?
B. Was the report consistent and the evidence complete and
convincing and were the ratings substantiated when used?
C. Did the report present a sound assessment of sustainability
of outcomes?
D. Were the lessons and recommendations supported by the
evidence presented?
E. Did the report include the actual project costs (total and per
activity) and actual co-financing used?
F. Did the report include an assessment of the quality of the
project M&E system and its use for project management?
UNEP EOU additional Report Quality Criteria
UNEP EOU Assessment
Rating
G. Quality of the lessons: Were lessons readily applicable in
other contexts? Did they suggest prescriptive action?
H. Quality of the recommendations: Did recommendations
specify the actions necessary to correct existing conditions or
improve operations (`who?' `what?' `where?' `when?)'. Can
they be implemented? Did the recommendations specify a goal
and an associated performance indicator?
I. Was the report well written?
(clear English language and grammar)
J. Did the report structure follow EOU guidelines, were all
requested Annexes included?
K. Were all evaluation aspects specified in the TORs
92

adequately addressed?
L. Was the report delivered in a timely manner
GEF Quality of the MTE report = 0.3*(A + B) + 0.1*(C+D+E+F)
EOU assessment of MTE report = 0.3*(G + H) + 0.1*(I+J+K+L)
Combined quality Rating = (2* `GEF EO' rating + EOU rating)/3
The Totals are rounded and converted to the scale of HS to HU
Rating system for quality of terminal evaluation reports
A number rating 1-6 is used for each criterion: Highly Satisfactory = 6,
Satisfactory = 5, Moderately Satisfactory = 4, Moderately Unsatisfactory =
3, Unsatisfactory = 2, Highly Unsatisfactory = 1, and unable to assess = 0.
93

Annex 4
GEF Minimum requirements for M&E
Minimum Requirement 1: Project Design of M&E2
All projects must include a concrete and fully budgeted monitoring and evaluation plan
by the time of Work Program entry (full-sized projects) or CEO approval (medium-
sized projects). This plan must contain at a minimum:
SMART (see below) indicators for project implementation, or, if no indicators are
identified, an alternative plan for monitoring that will deliver reliable and valid
information to management
SMART indicators for results (outcomes and, if applicable, impacts), and, where
appropriate, corporate-level indicators
A project baseline, with:
a description of the problem to address
indicator data
or, if major baseline indicators are not identified, an alternative plan for
addressing this within one year of implementation
An M&E Plan with identification of reviews and evaluations which will be
undertaken, such as mid-term reviews or evaluations of activities
An organizational setup and budgets for monitoring and evaluation.
2
http://gefweb.org/MonitoringandEvaluation/MEPoliciesProcedures/MEPTools/meptstan
dards.html
94

Minimum Requirement 2: Application of Project M&E
Project monitoring and supervision will include implementation of the M&E plan,
comprising:
Use of SMART indicators for implementation (or provision of a reasonable
explanation if not used)
Use of SMART indicators for results (or provision of a reasonable explanation if not
used)
Fully established baseline for the project and data compiled to review progress
Evaluations are undertaken as planned
Operational organizational setup for M&E and budgets spent as planned.
SMART INDICATORS GEF projects and programs should monitor using relevant
performance indicators. The monitoring system should be "SMART":
1. Specific: The system captures the essence of the desired result by clearly and
directly relating to achieving an objective, and only that objective.
2. Measurable: The monitoring system and its indicators are unambiguously
specified so that all parties agree on what the system covers and there are
practical ways to measure the indicators and results.
3. Achievable and Attributable: The system identifies what changes are
anticipated as a result of the intervention and whether the result(s) are realistic.
Attribution requires that changes in the targeted developmental issue can be
linked to the intervention.
4. Relevant and Realistic: The system establishes levels of performance that are
likely to be achieved in a practical manner, and that reflect the expectations of
stakeholders.
5. Time-bound, Timely, Trackable, and Targeted: The system allows progress
to be tracked in a cost-effective manner at desired frequency for a set period,
with clear identification of the particular stakeholder group to be impacted by
the project or program.
95

Annex 5
List of intended additional recipients for the Terminal Evaluation
Name
Affiliation
Email
Mail list
UNEP DGEF Professional
Evelyn.Machasio@UNEP.org
staff
Aaron Zazuetta
GEF Evaluation Office
azazueta@thegef.org
Government Officials
(none; distribution of the
Terminal Evaluation
Document to the respective
Government institutions will
be left to the Executing
Agency
GEF Focal Point(s)
Costa Rica, El Salvador, Guatemala,
Operational Focal Point: Constituency of
Claudia_grayeb@hacienda.gob.mx
Honduras, Mexico, Nicaragua, Panama:
Costa Rica, El Salvador, Guatemala,
Claudia Grayeb Bayata
Honduras, Mexico, Nicaragua, Panama
Belize:
Political Focal Point: Belize
belizemfa@btl.net
Alexis Rosado
(no Operational Focal Point
provided)
Executing Agency
Pan American Health
GEF Executing Agency
pradohug@paho.org
Organization (PAHO):
Dr. H.Prado
World Health Organization
GEF Executing Agency of
NeiraM@who.org
(WHO):
the Global DSSA Program
Dr. Maria Neira
96

7. REFERENCES
1. Academia Nazionale dei Lincei. 1998, "Roll Back Malaria: Section III: Background,
Conceptual Framework and Strategic Considerations.," Academia Nazionale dei
Lincei., ed., Academia Nazionale dei Lincei., Rome.
2. Alnwick D. Roll Back Malaria, State of the Movement. WHO/RBM. 2001.
Ref Type: Electronic Citation
3. Cerón L 2009, Comparación de la eficacia de dos esquemas de tratamiento para
casos de paludismo por Plasmodium vivax en comunidades con transmisión persistente
de los Estados de Oaxaca y Chiapas, México (PNUMA-OPS-OMS). Mexico.
4. Dever A 1991, Epidemiología y Administración de Servicios de Salud, Primera edn,
OPS.
5. Díaz-Barriga F "Evaluación del riesgo por exposición al DDT residual en Meso
América. Proyecto DDT/PENUMA/GEF/OPS", Universidad de San Luis Potosí, ed.,
OPS-OMS, Mexico.
6. GEF. United Nations Environmental Program. Global environment facility. Project
document. 10-4-2003. GEF.
Ref Type: Generic
7. Kazembe L, Kleinschmidt I, Holtz T, & Sharp B 2006, "Spatial analysis and
mapping of malaria risk in Malawi using point-referenced prevalence of infection
data.", International Journal of Health Geographics.International Journal of Health
Geographics, vol. 5, no. 41.
8. Level J 2003, Health, an ecosystem approach. Ottawa-Canada.
9. Mendez J, Betanzos A, & Tapia R 2004, Guía para la Implementación y
demostración de alternanativas sostenibles de control integrado de la malaria en
méxico y américa central. Programa Regional de Acción y Demostración de
Alternativas Sostenibles para el Control de Vectores de la Malaria sin Uso de DDT en
México y América Central. México. Primera edn, OPS, Mexico DF.
10. Narvaez A. Sostenibilidad y replicabilidad de la estrategia control de malaria sin
insecticidas persistentes en localidades demostrativas de mesoamerica. OPS and
DDT_GEF. 2008.
Ref Type: Unpublished Work
11. Narvaez A 2007, Guía de estratificación de la malaria en Guatemala. OPS,
Guatemala.
12. Narváez A & Cańas M. Evaluación de la ejecución del Proyecto PAMAFRO
Ecuador de Enero a Septiembre 2007. FUNSAD. PAMAFRO. 2007.
Ref Type: Electronic Citation
97

13. Nyarango P, Gebremeskel T, Mebrahtu G, Mufunda J, Abdulmumini J,
Ogbamariam A, Kosia A, Gebremichael A, Gunawardena D, Ghebrat Y, & Okbaldet Y
2006, "A steep decline of malaria morbidity and mortality trends in Eritrea between
2000 and 2004: the effect of combination of control methods.", Malaria Journal, vol. 5,
no. 33.
14. RABREDA-AMI 2007, Estratificación para evaluación de desempeńo y calidad de
las intervenciones. Estrategia para racionalizar la toma de decisiones en control
vectorial en malaria., OPS.
15. Roberts D, Laughlin L, Hsheih P, & Legters L 1997, "DDT, Global Strategies, and a
Malaria Control Crisis in South America", Emerging Infectious Diseases, vol. 3, no. 3.
16. Rodriguez M 2009, Mesoamerican Health System Initiative.
17. Salinas V & Narváez A. Las construcciones culturales de la malaria en un proyecto
de salud en Panamá. OPS. 2009.
Ref Type: Unpublished Work
18. UNEP. TERMS OF REFERENCE: Terminal Evaluation of the UNEP GEF project
"Regional Program of Action and Demonstration of Sustainable Alternatives to DDT
for Malaria Vector Control in Mexico and Central America". Project No. GF/2732-03-
4680 PMS: GF/4030-03-22. 1-25. 2009. Nairobi, UNEP.
Ref Type: Generic
19. WHO 1993, Implementation of the Global Malaria Control Strategy, W.T.S.S. 839.
World Health Organization, Geneva.
20. WHO 2007, Malaria elimination: a field manual for low and moderate endemic
countries, 1 edn, Geneva.
21. WHO 2008, Global malaria control and elimination: report of a technical review
WHO, Geneva.
22. Yin R 1994, Case Study Research: design and methods. SAGE Productions, USA.
98

Document Outline

ţ˙
- ţ˙
- ţ˙
  - ţ˙
  - ţ˙
- ţ˙
- ţ˙
- ţ˙
- ţ˙
- ţ˙
- ţ˙
ţ˙
ţ˙
- ţ˙
- ţ˙
  - ţ˙
  - ţ˙
ţ˙
- ţ˙
- ţ˙
  - ţ˙
    - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
ţ˙
- ţ˙
  - ţ˙
    - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
    - ţ˙
      - ţ˙
      - ţ˙
    - ţ˙
    - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
      - ţ˙
  - ţ˙
  - ţ˙
- ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
- ţ˙
- ţ˙
- ţ˙
  - ţ˙
  - ţ˙
    - ţ˙
    - ţ˙
- ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
  - ţ˙
ţ˙
ţ˙
ţ˙
- ţ˙
ţ˙
- ţ˙
- ţ˙