Towards a Convention and Action Programme for the
Protection of the Caspian Sea Environment
A project of:
The Governments of:
Azerbaijan, The Islamic Republic of Iran, Kazakhstan,
The Russian Federation and Turkmenistan
And
The Global Environment Facility,
The United Nations Development Programme and
The United Nations Office of Project Services.
Final Evaluation Report
October 2007
Dennis Fenton
Jeffrey Griffin
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Table of Contents
Table of Contents..................................................................................................................................... 2
Acronyms and Abbreviations ................................................................................................................. 3
Acknowledgements .................................................................................................................................. 4
Executive Summary ................................................................................................................................ 5
1. Introduction to the Evaluation ........................................................................................................... 9
1.1 Project Context .............................................................................................................................. 9
1.2 Stakeholders in the Project and in the Evaluation ................................................................... 10
1.3 Evaluation Methodology ............................................................................................................. 10
1.4 Evidence ....................................................................................................................................... 12
1.5 Limitations ................................................................................................................................... 12
2. Project Results .................................................................................................................................. 13
2.1 Background.................................................................................................................................. 13
2.2 The Project's Overall Objective and Project-specific Goal .................................................... 13
2.3 Results under Project Objective I .............................................................................................. 14
I.
Baseline ............................................................................................................................... 14
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective I. ..................................................................................................................................... 15
2.4 Results under Project Objective II ............................................................................................ 20
I.
Baseline ............................................................................................................................... 20
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective II. .................................................................................................................................... 20
2.5 Results under Project Objective III........................................................................................... 24
I.
Baseline ............................................................................................................................... 24
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective III. ................................................................................................................................... 25
2.6 Results under Project Objective IV ........................................................................................... 28
I.
Baseline ............................................................................................................................... 28
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective IV. .................................................................................................................................. 28
2.7 Conclusion ................................................................................................................................... 32
3. Assessment of Sustainability of Project Outcomes ......................................................................... 33
3.1 Assessment of Project Outcomes ............................................................................................... 33
3.2 Assessment of Four Dimensions of Sustainability .................................................................... 34
3.3 Conclusion ................................................................................................................................... 37
4. Monitoring and Evaluation (M&E) System .................................................................................... 37
4.1 M&E Planning and Design ......................................................................................................... 37
4.2 M&E Implementation ................................................................................................................. 38
4.3 M&E Funding ............................................................................................................................. 40
4.4 Long Term Monitoring ............................................................................................................... 40
5. Processes that Affected Project Results ........................................................................................... 41
5.1 Preparation and readiness .......................................................................................................... 41
5.2 Country ownership/driven-ness ................................................................................................. 45
5.3 Stakeholder involvement ............................................................................................................ 47
5.4 Financial Planning ...................................................................................................................... 49
5.5 Implementing and Executing Agency (IA/EA) Supervision and Backstopping .................... 49
5.6 Co-financing ................................................................................................................................ 50
5.7 Delays ........................................................................................................................................... 52
6. Lessons, Recommendations and Examples of Good Practice ........................................................ 53
6.1 Recommendations and Lessons Pertaining to Future GEF Projects ..................................... 53
6.2 Recommendations and Lessons Pertaining to CEP-SAP Project Stakeholders .................... 54
6.3 Examples of Best Practice. ......................................................................................................... 56
Annexes: ................................................................................................................................................. 57
2
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Acronyms and Abbreviations
ABC
Aquatic Bio-resources Commission of the Caspian Sea
CCSI
Caspian Coastal Site Inventory
CEP-SAP
Strategic Action Programme for the Caspian Environment Programme (short
name for this project)
CoP
Conference of the Parties (to the Tehran Convention)
EQO
Environmental Quality Objectives
EU
European Union
FSU
Former Soviet Union
GEF
Global Environment Facility
GPA
Global Pollution Assessment
IA
Implementing Agencies
IAEA
International Atomic Energy Authority
IMO
International Maritime Organisation
IPIECA
International Petroleum Industry Environment and Conservation Association
IW
International Waters
M&E
monitoring and evaluation
MEG
Micro environmental grant
MPPA
Matched grant and public participation advisors
MSGP
Matched small grants programme
NCAP
National Caspian Action Plan
NFP
National Focal Points
NGO
Non-Governmental Organization
PCU
Project Coordination Unit
PC
Program Coordinator
PIR
Project implementation review
POPs
Persistent organic pollutants
PTS
Persistent toxic substances
RAG
Regional Advisory Group
RSC
Regional Project Steering Committee
SAP
Strategic Action Programme
SAPIC
SAP Implementation Coordinator
SMART
Specific, Measurable, Attributable, Relevant and realistic, Time-bound
TACIS
Grant financed technical assistance program of the EU
TC
Tehran Convention or Framework Convention for the Protection of the
Maritime Environment of the Caspian Sea
TDA
Transboundary Diagnostic Analysis
TPR
Tri-Partite Review
UNDP
United Nations Development Programme
UNEP
United Nations Environment Programme
UNOPS
United Nations Office for Project Services
WB
World Bank
3
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Acknowledgements
The Evaluators would like to acknowledge the guidance and insight provided by the UNDP, UNOPS
and PCU staff in enabling this evaluation.
Dr. Vladimir Mamaev, UNDP-GEF Regional Technical Advisor, Dr. Andy Hudson, UNDP-GEF
Principal Technical Advisor for International Waters, and Dr. Frits Schlingeman, Senior Advisor,
UNEP Regional Office for Europe provided much appreciated big picture perspectives.
In particular, we`d like to thank Dr. Hamidreza Ghaffarzadeh, Manager of the PCU. His assistance with
organizing the evaluation and his extensive knowledge of the project and the CEP were an important
contribution to this evaluation.
Thanks and recognition go also to the PCU staff who so openly shared their insights and work with us.
Special thanks go to the SAP Implementation Coordinators in each of the five Caspian countries: Mr.
Rasim Sattarzadeh of Azerbaijan, Mr. Serik Akhmetov of Kazakhstan, Ms. Parvin Farshchi of the IR-
Iran, Ms. Gozel Orazdurdyyeva of Turkmenistan, and Ms. Tatyana Butylina of the Russian Federation.
Their experience and expertise as well as logistical support in their respective countries were critical to
this evaluation.
We are grateful to Mahir Aliyev and his colleagues at UNOPS for their able and prompt logistical
support, without which we may not have been able to connect the travel dots in the Caspian region.
Thanks and appreciation also go out to the project partners in each of the five Caspian Countries who
took the time to talk to us about this project and the overall Caspian Environment Program. Thanks go
also to the project`s international partners and experts who endured web-based conference calls from
the Evaluators and so openly shared their experiences and insights.
4
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Executive Summary
Introduction
This is the terminal evaluation of the GEF/UNDP/UNOPS Project Towards a Convention and Action
Programme for the Protection of the Caspian Sea Environment` (hereafter referred to as the Project` or
CEP-SAP`). The Project constitutes the second stage of GEF support to coordination and cooperation
amongst the five Caspian countries in order to protect and sustainably manage the environmental
resources of the Caspian Sea. The five participating countries are Azerbaijan, the Islamic Republic of
Iran, Kazakhstan, the Russian Federation and Turkmenistan.
As set out in the Project Document, the Project`s primary objective was to support the countries to
consolidate and begin to implement the Strategic Action Programme for the Caspian Sea, including
filling gaps in information and developing capacity in the region for SAP implementation and project
execution. The Project became operational in April 2004, with GEF support of US$6.026 million, and
is scheduled to complete its work by January 2008.
The main purpose of this terminal evaluation is to promote accountability for achievement of GEF
objectives. This evaluation assesses and provides ratings for: the results of the Project according to their
relevance, effectiveness and efficiency; the likelihood of sustainability; and the Project`s monitoring
and evaluation system. It also analyses the factors and processes that affected the attainment of project
results and sets out important lessons learned and recommendations applicable to GEF`s larger portfolio
of projects.
It is far easier to evaluate than to be the one in the trenches working day in and day out. The credit goes
to those who have worked so hard to implement this project and achieve the results that it has. This
evaluation offers praise where praise is due and constructive criticism where improvement is possible.
The evaluation involved four phases of work planning, data collection, analysis and report writing and
consultation. Two independent experts conducted the evaluation during the period June-September of
2007. The Evaluators spent 14 days travelling among the five Caspian countries meeting with
stakeholders and conducting site visits as time and distance allowed.
Main Achievements
Overall, the Project excelled in its work to forge inter-governmental cooperation at the regional level,
enabling the adoption of the Tehran Convention and the countries` steady progression towards stronger
regional environmental collaboration.
The primary accomplishments of the CEP-SAP Project relate to how it served as a regional catalyst for
conservation of the Caspian Sea and its environs. The Project brought together international, regional,
national, governmental and non-governmental actors, contributing to priority setting and undoubtedly
generating synergies. Overall, the Evaluators consider this project to be a good example of the worth
and the importance of international assistance in the environmental arena.
The Project`s main achievement is to have sustained the Caspian Environment Programme (CEP) and
the outcomes of the 1st stages. These are notably the TDA/SAP/NCAP1 and the CEP regional and
national institutional structures and processes. With project support, these became the vehicles
supporting the ratification of the Tehran Convention or Framework Convention for the Protection of the
Maritime Environment of the Caspian Sea. This is the first ever formal, legal commitment among the
five Caspian countries, and it heralds a transformation in the region from a state of voluntary
adherence to one of legal obligation with respect to regional environmental commitments. This is a
1 Trans-boundary diagnosis analysis, Strategic Action Programme and National Caspian Action Plan
5
CEP-SAP Final Evaluation
Final Draft
30/04/2010
startling achievement and marks the beginning of a new stage of cooperation only recently beginning to
bear fruit.
The Project also facilitated the relatively rapid and ongoing negotiation of Protocols to the Tehran
Convention. To be sure, most of the credit for this belongs rightly with the five Countries and their
Parliaments who ratified the Convention as well as the first phase of GEF investment. However, this
project and its partners at UNEP rightly deserve accolades for the impressive work done in enabling this
to happen.
And finally, the most recent evidence of this new stage of cooperation is reflected in the decision in
May 2007 of the first Conference of the Parties to the Tehran Convention (TC). The decision paves the
way for a transition from the CEP to the Convention Secretariat, from SAP to a Convention Action Plan
and from a regional mechanism totally reliant upon outside funding to a mechanism funded at its basic
level by the countries themselves.
Other notable outcomes generated by the Project include:
Continuing and consolidating the systematic, transparent approach to regional environmental
problem solving in the region. This is characterized by the TDA, the SAP and the many high
quality planning documents that fed into these;
Continuing and consolidating the process to building inter-country trust and understanding and
facilitating a meaningful dialogue across countries;
The introduction and demonstration of best practices, methods and techniques, (e.g. for monitoring
and analyzing pollution, studying invasive species, identifying priority coastal sites);
Raising scientific and general understanding of the Caspian Sea as a region;
Continuing and consolidating the participation of oil consortia and other major private sector
stakeholders in the CEP process;
Enabling each of the five countries to develop National Caspian Action Plans (NCAP). These set
out national SAP implementation priorities and raised the profile of the important links between
development and environment e.g. wastewater treatment and water quality in the Caspian;
The links between the increasing level of investments and the NCAPs merit further analysis.
Moreover, the role of the project in leveraging these investments is difficult to determine and varies
across the five countries. But there is enough evidence to suggest that the CEP-SAP2 Project has
played a role in either stimulating or coordinating investments.
Overall, the Project`s range of achievements in four years is significant in a region as ecologically,
economically, socially, and politically dynamic and challenging as the Caspian.
Important Weaknesses
Contrasting with its clear successes at the inter-governmental level, the Project struggled more when it
came to fulfilling its mandate to catalyze changed practices or new outcomes at the national level.
The Project was not able to focus upon and measure adequately the impact it has had on stakeholder
practices at the national level. For example, the Project produced a large number of outputs such as
strategies, action plans, reports, studies, and media kits. The recurring issue with nearly all of these is,
Are they being used by stakeholders in the Caspian region? and too often there is no evidence that
they are yet. Some of the outputs were too theoretical to be of practical use to stakeholders.
There is a lack of evidence of on-the-ground impact, and a lack of activities that catalysed on-the-
ground impact. For example, the Project supported a series of small and micro-grants, but the
2 Both this stage and earlier stages.
6
CEP-SAP Final Evaluation
Final Draft
30/04/2010
effectiveness of most of these grants, and their net impact is unknown. The Project also supported
awareness raising and NGO strengthening activities, but the impact of these also seems limited.
A third area of weakness is in the fact that the NCAP, SAPIC and other project mechanisms were
unable to more directly catalyze institutional changes and physical investments in the participating
countries. Apart from the environment agencies, few government agencies have changed due to the
Project or benefited from the Project. In each country, ownership over the CEP has remained within too
small a group of stakeholders: these were not able to implement the NCAP, and their actual
commitment to regional cooperation or joint activities remains limited. The links between NCAPs and
relevant budgetary allocations is not clear, and they have not yet become effective coordination tools.
Sustainability
The evaluation finds that the Project has made important and impressive steps towards sustainability.
Chief among these steps are those taken with the Convention process. The ratification of the Tehran
Convention gives legal backing to many of the Project`s outputs, and this holds the promise of many of
the Project`s outputs being sustained in the future. Also promising are the preliminary commitments to
funding the TC Secretariat made by the delegates to the CoP-I in May 2007.
The CEP institutional structures and management track record provides a solid foundation upon which
to build new sustainable capacity for the TC. However. at project termination, CEP institutional
structures and CEP-inspired programs will not yet be sustainable. This is not surprising in that the
Project addressed some highly challenging issues. A lesson learnt from other international water body
initiatives in the world is that a process of up to 20 years may be necessary before sustainability is
achieved. Accordingly, the evaluation feels that continued international support to this process is
justified and critical. However, and critically, indicators and milestones to sustainability are important,
notwithstanding the difficult nature of the process.
Monitoring and Evaluation
The Project`s monitoring and evaluation (M&E) system was one of its main weaknesses. The Project
document contained many flaws with regards to monitoring and evaluation notably a poor logical
framework, inadequate indicators and inadequate resources allocated to M&E. Once under
implementation, insufficient effort was made to improve the M&E system.
Factors affecting project success
A detailed assessment of the factors affecting success both positive and negative - is provided in
Chapter 5. This assessment points to lessons learnt that could be applied to future projects. These
factors point to many things that the Project sponsors and managers did impressively well; they also
point to some areas where there is room for improvement.
One factor contributing to project success include the strong project stakeholder network that is well
anchored within national, regional and international organisations. This includes government agencies,
national experts, lawmakers, influential private sector stakeholders, and internationally reputed
technical and financing agencies.
A second important factor has been the quality and thoroughness of the work implemented by the
Project. The studies, surveys, and reports were all of high technical quality. The Project was able to
draw on and manage high quality expertise.
7
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The Project also successfully learned from other multi-country water body management processes. For
example, this helped in the critical process of getting countries to agree on the Tehran Convention, in
elaborating the national action plans in support of the TC, and in drafting the protocols under the TC.
The Project`s effectiveness was strengthened by the exemplary level of cooperation among GEF
Implementing Agencies (UNDP and UNEP), the EU, and other partners.
The Project`s exemplary collaboration with some of the region`s most prominent oil companies,
including Total, BP and KazMunaiGaz, also contributed to its success.
The main factors limiting success were:
Lack of clarity in the objectives of the project. This is apparent both in the project document and its
logical framework, and in a review of the activities implemented.
A focus on outputs` instead of outcomes` or impacts.
The over-ambitious nature of the Project: too many activities were attempted, in a diverse range of
fields, and often in sectors which require more resources (e.g. strengthening civil society) than were
available to the Project.
The low level of strategic reflection and adaptive management during implementation.
Summary of Ratings and Discussion
See Table A below for a compilation of overall ratings.
In Summary, this Evaluation finds the overall Results rating for the project to be Satisfactory. The three
main components of the Results rating -- Relevance, Effectiveness and Efficiency -- received
Satisfactory, Satisfactory and Moderately Satisfactory ratings respectively. For Objective-level
ratings, see Table 16 on page 32.
The Evaluation considered Sustainability with respect to project Outcomes and the Four Main
Dimensions of Sustainability (Financial, Socio-political, Institutional/Governance, and Environmental).
The likelihood of sustainability for project Outcomes is rated Moderately Unlikely and the likelihood of
sustainability with respect to the four dimensions is rated Moderately Likely. The project`s Monitoring
and Evaluation work is rated Moderately Unsatisfactory.
Table A: Summary of the ratings of the project.
Project Aspect
Rating
Overall Result:
Satisfactory
Results breakdown:
- Relevance
Satisfactory
- Effectiveness
Satisfactory
- Efficiency
Moderately Satisfactory
Sustainability
- Sustainability of Outcomes
Moderately Unlikely
- Four dimensions of Sustainability
Moderately Likely
M&E System
Moderately Unsatisfactory
8
CEP-SAP Final Evaluation
Final Draft
30/04/2010
1. Introduction to the Evaluation
1.1 Project Context
The UNDP/GEF Project Towards a Convention and Action Programme for the Protection of the
Caspian Sea Environment` (hereafter referred to as the Project` or CEP-SAP`) constituted the second
stage of GEF support to coordination, cooperation and action amongst the concerned Caspian countries
in order to protect and sustainably manage the environmental resources of the Caspian Sea. This second
stage project became operational in April 2004, with GEF support of US$6.026 million. The project is
supported under the GEF International Waters (IW) focal area.
As set out in the Project Document, the primary objective of the Project was to support the countries to
consolidate and begin to implement the Strategic Action Programme for the Caspian Sea, including
filling gaps in information and developing capacity in the region for SAP implementation and project
execution. The five participating countries are Azerbaijan, the Islamic Republic of Iran, Kazakhstan,
the Russian Federation and Turkmenistan. In order to reach this primary objective, the Project had four
immediate Objectives, namely:
1) To commence implementation of the Strategic Action Programme (SAP) in three priority areas:
Biodiversity, Invasive Species and Persistent Toxic Substances.
2) To continue with specific capacity building measures to ensure a regionally owned CEP
coordination mechanism capable of full implementation of the SAP and regional coordination of
the National Caspian Action Plans (NCAPs).
3) To strengthen the environmental legal and policy frameworks operating at the regional and the
national levels and where necessary improve implementation and compliance of those
frameworks.
4) To achieve tangible environmental improvements in priority areas by implementation of small-scale
investments supported by a small matched grants programme.
These immediate Objectives were to be achieved through a set of nine Outcomes, each to be produced
through a series of Outputs and Activities. Initially the project was to run for 3 years until early 2007.
This was recently extended by one year with no change in budget.
The Project implementation structure was established towards the end of the first stage activities, and
included (See Annex 9):
A Regional Project Steering Committee (RSC) and a regional CEP-SAP Program Coordinator (PC);
National Focal Points (NFP) and SAP Implementation Coordinators (SAPIC) in each country;
A regional Project Coordination Unit (PCU), consisting of managerial, technical and administrative
staff, and including a matched grant and public participation advisors (MPPA) for each country;
Technical support from UNDP and administrative and financial support from UNOPS.
The fieldwork for the final evaluation of the Project (hereafter referred to as the Evaluation) was
conducted during June July 2007. In line with UNDP and GEF guidelines, the Evaluation was
undertaken by independent evaluators and focussed on: results, sustainability, innovation, catalytic
nature, monitoring and evaluation, and the factors affecting project success. This report presents the
major findings of the Evaluation.
There are two notable socio-economic-political aspects to the Project`s context. First, until 1991, four of
the five countries were part of the former Soviet Union (FSU) and therefore Caspian Sea management
9
CEP-SAP Final Evaluation
Final Draft
30/04/2010
involved only two states (FSU and Iran). The collapse of the FSU led to: more complex negotiations
involving five participating states; severe economic and industrial decline in four of the states; and
losses of social capital. These aspects still affect development and environmental management in the
region. Second, in recent years, the region has seen spectacular growth in the proven reserves and
exploitation of hydrocarbon resources. Given prevailing high global oil prices, this has led to increasing
prosperity and growing government budgets in the region.
1.2 Stakeholders in the Project and in the Evaluation
The Project involved key stakeholders at the international, regional, national and local levels. At the
international level, these include UNEP, the World Bank, the International Maritime Organisation
(IMO), International Atomic Energy Authority (IAEA) and the International Petroleum Industry
Environment and Conservation Association (IPIECA). At the regional level, key stakeholders include:
the Tehran Convention3 (TC) and its Conference of Parties (CoP); the EU TACIS programme; the
Aquatic and Bio-resources Commission (ABC) and international companies active in the oil and gas
sectors.
At the national level, the main stakeholders were the governmental agencies responsible for
environment and natural resources. Other key stakeholders included: agencies responsible for
agriculture, water, energy, transport, foreign affairs; agencies responsible for Caspian issues; offices of
private sector organisations notably in the hydrocarbon sector; and NGOs and media. At the local
level, in addition to local government, key stakeholders included: NGOs, schools, institutes and
laboratories and media.
1.3 Evaluation Methodology
Two independent experts (hereafter referred to as the Evaluators`) undertook this Evaluation. The
initial Evaluation methodology is set out in the Terms of Reference (see Annex 1) and was further
elaborated by the Evaluators. The Evaluation consisted of the following steps4: planning, data
collection, validation of baseline and project targets, analysis, report writing and consultation.
Planning Phase:
The planning phase consisted of an initial documentation review, the scoping out of the main issues,
reviewing the ToR and proposing changes, determining the most appropriate mission itinerary,
collecting documentation and finalising logistical arrangements. This phase also consisted of a series of
brainstorming sessions between the Evaluators and consultations with the UNDP, UNOPS and the
PCU.
An important output of this phase was a conceptual model guiding the issues to be addressed in the
evaluation. Described in Annex 2, this conceptual model provides the structure for data collection and
analysis and guides documentation review. It also served as a framework of inquiry to help the
Evaluators ensure that relevant issues and questions were discussed with each stakeholder. The
Evaluators chose this flexible approach rather than use of a formal questionnaire. Given the nature of
the Project, this conceptual model was regularly refined as the Evaluators collected and analysed data.
Data Collection Phase:
3 Full name: Framework Convention for the Protection of the Maritime Environment of the Caspian Sea.
4 Although these steps are largely taken in chronological order, it is noted there is a significant amount of back and
forth and reiteration.
10
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The data collection phase included two main parts: (i) an in-depth review of documentation, and (ii) a
mission to the five countries5 to discuss the project with key stakeholders and to assess institutions and
project partners.
The aim of the mission to the five Caspian countries was to meet a diverse and representative group of
stakeholders. See Annex 3 for the mission itinerary and the list of people interviewed. In most cases the
interviews were bilateral involving uniquely the Evaluator(s) and the concerned stakeholder. In other
cases, project staff accompanied the meetings to provide translation. In other cases, in order to save
time, the Evaluators met several stakeholders at the same time. In addition to stakeholder meetings, the
Evaluators strived to include as many short field visits as possible to sites where the project has
supported on-the-ground action. See Annex 5 for a summary of field visits and key observations.
The in-depth review of documentation covered: background material on natural resources in the
Caspian region; outputs from previous stages of the CEP-SAP project; project planning and reporting
documents, and; project technical outputs. Annex 4 provides a list of the documentation reviewed.
Validating the baseline and target:
Ideally, the project document clearly describes the baseline, the benchmarks and the target indicators of
success, which the Evaluators use as the main objective tool to assess the project`s progress and level of
success. The Evaluators found that the description of the baseline in the document, with regards to
environmental management and to regional cooperation, was adequate.
However, the indicators and targets set out in the project document were not considered adequate for
measuring project progress. The indicators were not SMART6, they were too numerous, and the targets
were not clear. The indicators focussed mostly on outputs rather than outcomes and impacts to be
achieved by the project. Hence, during the Evaluation and with limited resources, the Evaluators had to
determine reasonable indicators and targets against which the project`s progress could be assessed. To
do this, the Evaluators drew heavily on: experience under the CEP-SAP during 1998- 20047; the
Evaluators` first-hand experience from many similar projects in the region and elsewhere; experience
with similar processes in other international water-bodies.
Analysis Phase:
The Analysis Phase consisted of two complementary components, in line with the two components of
the conceptual model (see Annex 2). First, the Evaluators reviewed progress towards each Objective,
Outcome and Output in the logical framework by estimating the percentage achievement of each
indicator. To do this, the Evaluators used the indicators from the PIR 2006 and PIR 2007 however the
Evaluators gave their own, independent assessment of the status of these indicators, taking into account
both achievement level and quality.
Second, as the Evaluators felt that the logical framework did not capture fully all the achievements and
opportunities of the Project, the Evaluators took other relevant issues into account. These issues related
either to the impact associated with the indicators or to complementary factors that were clearly an aim
of the project but were not fully captured in the project logical framework.
In line with GEF guidelines, the Evaluators used the evidence available in order to provide ratings for
Results (in terms of relevance, effectiveness and efficiency), Sustainability (financial, socio-political,
institutional/governance and environmental) and M&E (design and implementation). As in all
evaluations, triangulation was a key element of analysis this ensures that each finding of the
5 Both Evaluators visited Iran and Azerbaijan, only one Evaluator visited each of the other countries.
6 I.e.: specific, measurable, achievable & attributable, relevant & realistic and time bound & timely & trackable &
targeted.
7 Formally, the title CEP-SAP` was only used in the second stage of GEF support, post 2004. However, in this
paragraph, the term CEP-SAP is used to refer to all GEF support to the CEP since 1998.
11
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Evaluation is supported by at least two sets of data from two independent sources. For the important
step of providing ratings for the effectiveness of each Outcome, the Evaluators took three factors into
account: (i) the percentage of the stated indicators achieved (ii) the impact associated with the
indicators, and (iii) any mitigating factors associated with the concerned Outcome.
Attribution of success: Evaluators paid special attention to assuring that the achievements could be
reasonably attributed, at least in part:
to CEP-SAP support and did not result purely from the actions of other stakeholders. There are
many ongoing projects in the region (e.g. supported by governments or EU), and UNDP/GEF
cannot take credit for all progress. It is noted that it may be responsible for having catalysed or
coordinated the other achievements;
to CEP-SAP activities during 2004-2007. CEP-SAP has been active since 1995, and many early
activities are still bearing fruit. This cannot be attributed to the present Project. However, it may be
that CEP-SAP has provided critical support to ensure that activities implemented in earlier stages
bore fruit during the period 2004-2007.
1.4 Evidence
GEF guidelines require that sufficient and convincing evidence be collected to support each finding of
the Evaluation, notably with regards to Results. The Evaluators have made every effort to collect
independent, verifiable evidence in the time allowed. In cases where evidence was suggested or referred
to, but not directly witnessed by the Evaluators, the Evaluators were unable to take the evidence into
account.
1.5 Limitations
The lack of a clear logical framework and usable indicators was a significant limitation to this
evaluation. This, combined with the multi-country, multi-component aspects of the project, made this
evaluation particularly complex and challenging.
A second major limitation to this Evaluation was a shortage of time, particularly during the mission to
the participating countries. This had the following implications:
Typically, there was insufficient time to meet many key stakeholders. Insufficient time meant the
meetings were mostly with the stakeholders with whom meetings are logistically easier to organize
typically those who work for or in the project. These stakeholders are the most informed and the
most important to meet. However, they all tend to view the project from a similar perspective and
this limits diversity in data collection. Also, these stakeholders are well placed to observe project
successes, but perhaps less well placed to observe failures and missed opportunities. In many cases,
there was insufficient time to arrange meetings with more independent stakeholders and their
input was lower than hoped. Hence the Evaluation suffered from a lack of critical voices and
independent viewpoints. Paradoxically, if the Evaluators had been able to access more such
viewpoints, it may have been easier for the Evaluators to give the project a more resounding
approval.
Little time was available during the mission for the Evaluators to reflect upon findings and digest
findings. The mission rushed from meeting-to-meeting, city-to-city, and airport-to-airport. There
was little time to develop and validate hypotheses. Notably, the two Evaluators had different
itineraries for much of the mission, and there was inadequate time for interaction and reflection
between the two evaluators during the mission;
There was no opportunity for the Evaluators to present initial findings, in person, to the project
stakeholders, either as a regional group or in each country. This diminished the effectiveness of the
evaluation as a learning experience for project stakeholders.
12
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Other limitations include:
No independent experts from any of the participating countries were part of the Evaluation team;
The region does not have a culture conducive to learning lessons` from evaluations;
Language was a problem in a small number of interviews, as the Evaluators were unable to
communicate in all the Project`s working languages.
2. Project Results
2.1 Background
Results are defined as the positive and negative changes and effects produced by the Project. Results
can be both foreseen and unforeseen. Results include the direct Outputs of the Project. Results also
include the short and medium-term Outcomes resulting from project Activities and Outputs. This
includes the changes in the practices and behaviour of the people and organisations due to the Project.
Finally, Results may also include the longer-term impacts resulting, at least in part, from project
Activities and Outputs. These include global environmental benefits, replication effects and local
environmental or socio-economic effects.
This Chapter first provides a brief discussion of progress towards the Project`s Primary Objective. It
then provides a detailed discussion on progress and achievements under each of the four Immediate
Objectives identified in the project document. For each Immediate Objective, this Chapter provides:
A description of the baseline situation, as in 2004 and as if there had not been a GEF supported
project;
A summary discussion of the findings of the Relevance, Effectiveness and Efficiency of the related
Project achievements, followed by the Rating.
o Relevance: The Evaluators considered each Outcome, respective outputs and indicators and
their relevance to the project`s main objectives, to the IW focal area and to country priorities.
The Evaluators also considered whether the outcomes and their related outputs and indicators
were sufficiently high priority.
o Effectiveness: With respect to effectiveness, the Evaluators first considered to what degree the
main relevant indicators under each Outcome of each Immediate Objective were achieved.
Then, as explained above and in Annex 2, the Evaluators considered a series of other issues
pertinent to the concerned Objective.
o Efficiency: This Chapter includes an assessment of the Efficiency of the GEF support under
each Immediate Objective and relevant Outcome. Although information is very limited, this
includes a brief consideration of the cost-effectiveness of the GEF support based upon a
comparison of budgetary figures, relevant outputs and evidence of impact.
2.2 The Project's Overall Objective and Project-specific Goal
The Project Document states that the Project`s primary Objective is to support the countries to
consolidate and begin to implement the Strategic Action Programme for the Caspian Sea, including
filling gaps in information and developing capacity in the region for SAP implementation and project
execution. The Project Document does not provide a clear baseline with respect to this Objective, nor
does it set targets. During the lifetime of the Project, no evidence is available that the project team used
this primary Objective as either a monitoring or reporting tool. This Objective is not mentioned in the
PIRs produced by the Project. Hence, the Evaluators decided that it would not be useful to use the
primary Objective as a means to assess project progress.
13
CEP-SAP Final Evaluation
Final Draft
30/04/2010
However, as is standard in most GEF projects8, the Evaluators are of the opinion that it would have
been useful for the Project to have defined a single, overarching Project Objective to which the project
would contribute and a Project Goal, that it would be held accountable for achieving. The Project Goal
would be achievable within the lifetime of the project, through the activities supported by the project.
The absence of a measurable overall Objective and Goal mean that the discussion in this Evaluation on
impact and results has to focus at the Immediate Objective level and below. The absence of a
measurable overall Objective and Goal are examples of weakness in the project logical framework and
monitoring framework. These issues are discussed in more detail in Sections 4.1 and 5.2.
2.3 Results under Project Objective I
To commence implementation of the SAP in three priority areas: Biodiversity, Invasive Species and
Persistent Toxic Substances.
I.
Baseline
The following illustrate the baseline situation with regards to this Objective. The Transboundary
Diagnostic Analysis (TDA) and SAP had been finalised and approved prior to the project start-up.
These documents provided a basic scientific understanding of issues and a basis, albeit limited, for
planning and for action including joint action. With international support (e.g. from the EU, the
Darwin Initiative, NATO, etc), the countries in the region were participating in a range of projects to
monitor and to improve the environmental situation and to implement certain elements of the SAP.
These projects would have continued without GEF support under the second phase. Generally, given
slowly growing environmental awareness and increased government revenue, the countries in the region
were likely to increase the level of some SAP-related environmental investments through the baseline
period, for example in waste water treatment, pollution abatement, protected area systems and
monitoring. These investments mostly would have been implemented without cooperation among the
countries. Finally, it is likely that, in the absence of this project, the regional vision produced by the
TDA/SAP would have dissipated quickly; key regional issues such as invasive species, pollution, and
biodiversity would not have been investigated and discussed in a collaborative, transboundary manner.
In the baseline, the on-the-ground environmental situation with regards to the three priorities areas
could be characterised as slowly declining`, but with variations and based on very incomplete data and
information. In terms of biodiversity, the project document mentions habitat erosion, degradation, non-
sustainable use of key species, pollution and invasive species as major threats. It cites the loss and
reduction in hallmark species and habitat loss as the most visible aspect of biodiversity loss. In the
baseline, it is likely that biodiversity would have continued this slow decline. With regards to invasive
species, in the baseline there were few national or transboundary measures in place in the Caspian Sea,
and this threat would probably have grown in the baseline. However, the scale and nature of this threat
was well understood drawing from understanding in the FSU, but also based on CEP observations and
studies in the 1990s and early 2000`s. With regards to persistent toxic substances, the situation had
generally improved following the break-up of the FSU and subsequent economic and industrial decline.
However, information is very limited, and some problems were certainly present, notably the use of
pesticides and dangers related to the petroleum industry.
8 Although terminology may change, there is typically a single higher level objective, to which the project
contributes, and a single project level objective, which can be mostly achieved by the project, in the time frame of
the project.
14
CEP-SAP Final Evaluation
Final Draft
30/04/2010
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective I.
A.
Relevance:
The Project Document includes five Outcomes under this Objective. The five Outcomes address issues
identified in the TDA and SAP. Hence, they are all clearly relevant to the overall project objective and
to the GEF focal area. However, given that the TDA and SAP identify many activities and do not firmly
establish priorities, it is hard to assess if the Outcomes address priority issues. The Evaluators feel that
the Outcomes should address priorities in order to be highly relevant. The strategic process to identify
Activities under this Objective was not clear to the Evaluators, possibly it was a continuation of first
stage activities. Without this clarity, it is not possible to give a highly satisfactory` rating. The ratings
are provided in Table 1.
Table 1: Relevance of Outcomes under Immediate Objective I
Outcome
Relevance9
Outcome A: A quantitative assessment of habitat loss in the Caspian and its coastal zone
HS
and verification of critically threatened areas, and, the design and establishment of a
standardised monitoring methodology programme for the Caspian Sea in conjunction
with the oil and gas industry.
Outcome B: Preliminary implementation of the Caspian Biodiversity Action Plan,
S
focusing on compliance with existing nature protection regulations, implementation of
species and habitat protection conservation action plans and targeted public awareness
campaigns.
Outcome C: Implementation of the CEP invasive species action plan in close
S
coordination with the GEF Global Ballast Waters project to address the impact of
Mnemiopsis on the Caspian ecosystem.
Outcome D: Assessment of the pollution loading of the Caspian and determination of
S
distribution and composition of Persistent Toxic Substances (PTS) in the riverine waters
and sediments and coastal waters, in order to prioritise future interventions directed at
amelioration of the environment.
Outcome E: Regional and National Action Plans addressing the activities contributing to
S
transboundary PTS, including persistent organic pollutants, oil products, and heavy metal
pollution.
Total
S
B.
Effectiveness:
As described earlier, the discussion on Effectiveness is guided by two considerations:
Question #1: To what degree were the relevant indicators in the Project Document achieved?
To answer this question, the Evaluators reviewed each indicator under each Outcome in the 2006
/2007 PIRs. The evaluators assigned a value for the achievement of each indicator. These values can
be seen in Annex 6. The compiled percentages for each Outcome are in Table 2 below.
Question #2: What other issues should be considered, including evidence of the degree of impact with
respect to the indicators and relevant impacts not covered by the listed indicators? Question 2
supplements Question 1 in that it seeks to look further into the question of impact with respect to each
Outcome in order to arrive at a fair rating for project effectiveness.
9 In line with GEF guidelines, throughout this report, most ratings are in the following order: Highly Satisfactory
(HS), Satisfactory (S), Moderately Satisfactory (MS), Moderately Unsatisfactory (MU), Unsatisfactory (U) and
Highly Unsatisfactory (HU). Definitions of these are provided in: GEF Evaluation Office: Guidelines for
Implementing and Executing Agencies to Conduct Terminal Evaluation, GEFSEC, May 2007.
15
CEP-SAP Final Evaluation
Final Draft
30/04/2010
1. Discussion of Indicators and Outputs:
The PIR 2006/07 include 28 indicators for this Objective. Most are not SMART indicators, but rather,
are outputs or activities. Seventeen of the twenty-eight correspond to an Output from the Project
Document. These Outputs are presented in Table 2. The degree to which each PIR Indicator was
achieved, according to the Evaluators, is also shown in Table 2.
Table 2: Outcomes and Main Output/Indicators under Objective I
Outcomes
Main Outputs (from Logical Framework)
Percentage Achievement of
Associated Indicators10
Outcome A
Biodiversity Data base
79%
Interactive Maps
Caspian Coastal Sites Inventory
Environmental Monitoring Programme
Outcome B
Draft Seals Conservation Plan
66%
Seal population Surveys in 2005 & 2006
Anzali Lagoon Stakeholders Analysis and Situation
Analysis
(Several meetings of the BDIS RAG were also supported
under this Outcome)
Outcome C
Mnemiopsis Leide/Beroe Survey 2005
45%
Beroe Introduction Technical proposal
Survey of ballast Traffic in Volga
Draft Review of Legislations on Invasive Species
Outcome D
Ongoing studies on Global Pollution Assessment
40%
(GPA) /Rapid Assessment of Pollution in all five
countries
Volga flux study
Kura Flux study
Caspian Wide survey of Sediments.
Outcome E
POPs/PTS Regional Plan
67%
POPs Awareness Posters /literature
(Several micro grants and PTS RAG meetings were also
supported under this Outcome)
There are four pertinent observations regarding these Outputs:
Observation #1: The fact that the project produced 17 major outputs under Objective I in 3 years
testifies that the project team has been very industrious. This, in itself, is a major achievement.
Observation #2: The overall quality of these outputs is high, and is generally higher than the quality
of outputs in similar projects in this region and across the world. Good examples include the Coastal
Sites Inventory, the Anzali Lagoon studies, and outputs from the surveys of Mnemiopsis leidyi and
beroe ovata. Some outputs were of lesser quality. For example the biodiversity and environmental
monitoring programmes struggled to move from being studies and recommendations to the stage of
policy adoption and implementation by stakeholders. To be sure, they helped build capacity and
introduce new techniques, but cannot yet be considered programmes` as this would include
comprehensive and shared protocols, and shared information bases.
Observation #3: Stakeholders reported to the Evaluators that several of the reports and associated
surveys were catalytic and inspirational. For example, they introduced new practices, technologies
and methods. The sampling and analysis of the seabed sediment as part of the pollution monitoring
work was often cited as one of the best examples of this.
10 See Annex 6 for details.
16
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The process of conducting the sea-bed surveys and studies and producing the report with regional
scientists introduced new methods and technologies that should be considered a capacity building
result or impact, but the extent of this impact is not clear and not easy to ascertain because the project
did not focus on measuring this kind of impact.
Observation #4: The number and scope of the reports are very ambitious. Given the relatively limited
funding and time available, it is important for a project like this to be very clear on the main intended
purpose of each survey and main intended use of each report or action plan and focus on achieving
and documenting the use of these documents by key stakeholders. For example, in many cases, the
reports concentrated on desk studies, collecting and restructuring existing data and making it
available in an accessible format. This can be a very useful and valuable exercise. But, the Evaluators
had difficulty judging how useful these reports have been to local stakeholders. Also, translating such
a large number of reports and studies is a real challenge for regional projects such as this. Sometimes,
with the larger reports that have many technical annexes, it is not practicable to translate such reports
and this can diminish their impact.
2. Impact of the Project Activities under Objective 1
The Project Document (and 2006/07 PIRs) included 5 Outcomes, 19 Activities and 28 indicators under
this Immediate Objective. However, from these planning documents, the strategic thrust behind this
Immediate Objective was not clear and the Evaluators were unable to choose impact indicators.
Hence, in order to ascertain the impact of activities under Objective 1, the Evaluators sought evidence
that the Outputs are used by and are useful to specific environmental stakeholders in the Caspian (e.g.
evidence that the recommendations are adopted, or that the reports are incorporated into planning, or
that there are new budget allocations in response to reports, or common monitoring protocols are
implemented, or common databases are being used, etc.).
The Evaluators looked for evidence of impact by determining lines of influence: evidence that the
project activity catalysed further action and/or stimulated a process of change. Two positive examples
of the project`s influence are illustrated in the diagrams below. The project surveys on mnemiopsis
catalysed follow-up, as did the work on the Caspian Seal. Given the complexity of the situation, it will
always be very difficult to attribute the success to the project. The project`s work on Anzali lagoon and
sea level rise also had an interesting impact (see Annex 5 for more details).
Figure 1: Lines of Influence - Caspian Seals Action Plan
Main Output: Project supported preparation of
Main impact: likely
Caspian Seal Action Plan:
catalyzed larger
Background
Note: cannot be considered an action plan`;
follow-up project
studies on seals
Limited resources did not greatly increase
funded by Darwin
performed in
understanding;
Initiative and likely
earlier CEP-SAP
Probably raised awareness of seal issues;
raised awareness of
stages and by
Ongoing process probably was catalytic
seal issue by putting it
other Partners
facilitating new players and resources.
on the regional agenda.
Still no agreement on
What were the
the Caspian Seal
linkages?
problem among the
five countries,
e.g. During the project
hampering regional
lifetime, Azerbaijan
driven-ness and
established a national park
country buy-in.
on the Apsheron
Peninsula.
Figure 2: Lines of Influence - Mnemiopsis Leidyi surveys
17
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Ongoing
Main impact: CEP-SAP
review of
catalyzed and cultivated open
related
productive debate of this key
legislation in
issue, contributing to a
all countries.
process of change. The
Project
project continued a
financed
Main
systematic process of
Data continues
surveys and
Output:
Project RSC
investigating the problem and
to be collected,
monitoring
Project
is not
solutions and reporting to the
but very
of
RAG BDIS
convinced
RSC where the issue is
incomplete,
Mnemiopsis
technical
requested
discussed openly. This has
unstructured
Leidyi and
decision:
additional
helped to build trust and
and each
Beroe
recommend
work.
understanding.
country acting
Ovata.
introduction
of Beroe.
alone.
Agreement could not be
reached on this complex
Many
issue. Discussions ongoing
stakeholders
through process to develop
remain to be
the BD Protocol.
convinced
However, for the majority of activities under this Objective, there was little evidence of significant
influence or impact. Too many Outputs fail the important being used` test. For example, the Caspian
Coastal Site Inventory (CCSI) included significant data and information on a series of hotspots around
the Caspian. The information is up-to-date and well presented - including maps and socio-economic
data. However, no evidence was received from any of the five countries that planners, environmental
managers or decision-makers were actually using the CCSI. Many stakeholders had neither seen nor
heard of the CCSI. In Iran, whereas the government was initiating a process of integrated coastal
management and developing spatial plans for the 3 Caspian provinces, the methods and information in
the CCSI did not seem to feed into those governmental processes. Other Outputs, for which there was
no evidence of impact, include the POPs Regional Action Plan and the Biodiversity Database. It is
however noted that the project`s preparation of the Caspian region`s first POPs Regional Action Plan is
an innovative initiative from which lessons should be derived and applied to future work on this issue.
The Survey of Ballast Traffic lies somewhere in-between. It provides a clear introduction to this
subject, as well as setting out steps to take and possible actions. This can feed directly into decision-
making, but is not doing so yet. The report is being discussed and may yet have some impact in Russia,
whose Volga River and associated canal systems offer the only way into and out of the Caspian for sea-
going vessels. Russian stakeholders have some concerns with the report and even with the report`s legal
relevance, and discussion of the report in Russia and the FSU region is hampered because only its
Executive Summary is available in the Russian language.
In addition to the outputs listed as indicators, the Project`s targets under this Objective included
building trust, raising the overall level of Caspian understanding and developing capacity. It seems that
earlier stages of CEP-SAP had built trust and understanding across the region, as well as raising the
level of understanding. There is little evidence that this moved forward significantly during this stage of
the Project although it was maintained. With regards to capacity development, the outputs produced
under this Objective were typically prepared in a professional, systematic manner. To the extent that
local partners were involved in these activities, they benefited by being exposed to these thorough and
modern processes.
3. Rating Effectiveness
To commence, the Evaluators derived effectiveness ratings for each Outcome based upon answers to
the two questions stated above and repeated here: To what level were the indicators achieved? And,
what was the observed level of impact achieved under each Outcome? The findings are summarised in
Table 3. Detailed findings are provided in Annex 6.
18
CEP-SAP Final Evaluation
Final Draft
30/04/2010
From Table 3, it can be seen that overall most indicators were met and most outputs produced, with an
average achievement rate of 59%. These reports are above average quality for UNDP/GEF projects.
However, as also can be seen from the Table, there is little evidence that the Outputs are having an
impact, with an overall average of 27%. This means that, as of yet, most of the Outputs are not being
widely used and are not widely useful. This leads to a total average of only 43%.
When translating the percentage figures into Ratings, the Evaluators considered two factors both of
which tended to increase the Rating. First, the impacts anticipated under this Objective may start after
the Project is completed, maybe even several years after, and evidence may not yet be available. Hence,
it is not possible to have 100% for evidence of impact`. Second, as mentioned previously, the Project
undertook a large number of activities under Objective 1 in a short time and should not be penalized for
this. For example, a project with only one output that was fully achieved and with evidence of impact
would score 100% in Table 3. Accordingly, even though the total percentages ranged only from 33 to
52%, the Evaluators estimate the Effectiveness of the five Outcomes to be in the range Moderately
Satisfactory - Satisfactory, and the overall rating for Objective I to be Satisfactory.
Table 3: Effectiveness Ratings for Objective I
Outcome Indicator Evidence of Total Average Effectiveness
Achieved
Impact
Rating
A
79%
11%
45%
S
B
66%
30%
48%
S
C
45%
32%
39%
MS
D
40%
25%
33%
MS
E
67%
37%
52%
S
Total
59%
27%
43%
S
C.
Efficiency
The estimated expenditures on Outcomes under this objective are provided in Table 4. Overall
expenditure under this Objective is estimated at $2.45 million. Almost all outputs involved significant
transaction costs (i.e. participants from the five countries, regional travel and communications, and the
involvement of leading international experts). This increased the time and cost required to conduct the
Project`s work. So, the Evaluators give the Project the benefit of the doubt with respect to the cost-time
vs. outcome relationship.
There were some problems in the efficiency of project implementation. Finding and keeping good RAG
members involved in project work proved to be a significant challenge in the countries. Distributing
funds for project work also proved to be slower than expected due to bureaucratic delays and the
complexities of the banking system in the region. Despite these difficulties, the Project managed to
produce an impressive number of outputs in the region in a relatively short period of time and hence,
overall the use of funds has to be considered at least Satisfactory.
Efficiency is mostly about using the least inputs to obtain the identified outputs. Hence, to secure a high
efficiency rating, the Project should provide evidence of having considered different options for inputs
with reasons given for selecting a particular option. On the one hand, the Project made good use of both
international and national experts. It also made good use of expertise that had previously been involved
in the Black Sea Environmental Programme and earlier stages of CEP-SAP. On the other hand, there is
little evidence that alternatives were considered to contractors/approaches in some cases mostly the
people involved in earlier stages were involved in this stage. There is also insufficient evidence of close
monitoring of all the sub-contractors. For these reasons, the Evaluators are unable to provide the
Highly Satisfactory` rating.
19
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Table 4: Expenditures and Efficiency under Objective I11
Outcomes
Expenditure
Efficiency
(US$ 000)
Rating
A
539
S
B
339
HS
C
573
S
D
644
MS
E
358
S
Objective I Total
2,453
S
If efficiency is defined as producing the best result for the least cost and the least time, effectiveness
must also be considered in this discussion of efficiency. Take for example, Outcome D its MS
effectiveness rating certainly contributed to its MS efficiency rating.
2.4 Results under Project Objective II
To continue with specific capacity building measures to ensure a regionally owned CEP coordination
mechanism capable of full implementation of the SAP and regional coordination of the National
Caspian Action Plans (NCAP).
I.
Baseline
This Objective has two Outcomes. The first focuses on the need for effective inter-governmental
coordination mechanisms on environmental issues across the Caspian region and the related need for
national implementation structures within each country. In the baseline, a regional coordination
mechanism had been supported by earlier stages of CEP-SAP, with some support from EU-TACIS. In
addition, the SAP already existed as a tool for coordination and planning. However, in the baseline,
without GEF support, the regional coordination mechanism is very unlikely to have continued
functioning through the period 2004-2007, as no other stakeholders were able and willing to support it.
At a national level, in 2004, each country had an NCAP and a fledgling structure to implement the
NCAP. In the baseline, in each country, it is likely that the NCAP would have been used to guide and
plan some investments in most countries. In the baseline, the implementation structures consisted of
NFPs, committee and groups of expert, with varying strength from country to country. Many
stakeholders, both governmental and non-governmental recognised the importance of such a
coordination mechanism, and were willing to give some support to an existing mechanism. However, in
the baseline, in most countries it is very likely that the implementation structures would have declined
in strength, even stopped.
The second Outcome focused on increasing the participation of a wide range of stakeholders, notably
non-governmental, in the process to strengthen environmental management across the Caspian region.
The non-governmental participants identified in the project document were the general public, NGOs,
private sector and local authorities. In the baseline, non-governmental participation would have been
limited. Without UNDP/GEF support to CEP-SAP, only a very small number of private sector
organisations (i.e. a handful of multinational investors in the petrochemical sector) and some local
authorities would have been involved in the Caspian environmental processes.
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective II.
A.
Relevance:
11 Detailed budgetary information is provided in Annex 7.
20
CEP-SAP Final Evaluation
Final Draft
30/04/2010
This Objective includes two Outcomes. Both are fully in line with the findings of the TDA and the
guidance provided under the SAP. Hence, both Outcomes are relevant. However, given the challenges
in the region, and the complexities in establishing broad participation, and the scale of this Project, the
Outcome associated with participation (Outcome G) may not be considered a high priority or high
urgency for this project. At the outset, the complex cultural situation and difficulties in working with
NGOs meant that any attempt to enhance` civil society participation would be challenging, and would
require resources beyond the scope of this regional project. However, this Project, aware of these
challenges, boldly persisted with this goal, possibly as a requirement from the Project sponsors. This led
to a situation where the Project spent significant time and effort on NGO-related activities with little to
show for it in the end.
In addition, the Project did not adequately target specific stakeholder groups or define just what it meant
by stakeholder participation in this regional context and so the relevance of Outcome G and its work
was also diminished. For these reasons, Outcome G`s relevance is rated as Moderately Satisfactory.
Table 5: Outcomes and Relevance under Objective II
Outcome
Relevance
Outcome F: A sustainable, strengthened, and regionally owned coordination mechanism for
HS
development and management of the Caspian Sea environment, in the form of a newly formed
country-supported PCU located in the IR-Iran capable of execution of regional projects, strong
country-supported National Coordination Structures capable of executing national projects, and a
network of institutions addressing transboundary environmental issues in the NCAPs and SAP.
Outcome G: Enhanced and informed stakeholder and inter-sectoral participation in the management
MS
of the Caspian environment
TOTAL
S
B.
Effectiveness:
As described earlier, the discussion on Effectiveness is guided by two considerations:
Question #1: To what degree were the relevant indicators in the Project Document achieved?
To answer this question, the Evaluators reviewed each indicator included under each Outcome in the
2006/07 PIRs. The evaluators assigned a value for the achievement of each indicator. These values
can be seen in Annex 6. The compiled percentages for each Outcome can be found in Table 6 below.
Question #2: What other issues should be considered, including evidence of the degree of impact with
respect to the indicators and relevant impacts not covered by the listed indicators? Question 2
supplements Question 1 in that it seeks to look further into the question of impact with respect to each
Outcome in order to arrive at a fair rating for project effectiveness.
1. Discussion of Indicators and Outputs
The PIR 2006/07 includes 10 indicators for this Objective. Most are not SMART indicators, but rather,
are outputs or activities. Six of these indicators correspond to an Output from the Project Document.
These are presented in Table 6 below.
Each Output has been produced or implemented as planned, in a quality manner. At the outset of the
Project, the PCU was moved from Baku to Tehran. Despite this upheaval, the PCU continued
functioning well, with strong support from the Government of Iran. The PCU team has been competent
and professional. Likewise, other Outputs, notably the website and the awareness raising material, are
professionally prepared and appreciated by their users.
Table 6: Outcomes and Outputs under Objective II
Outcome
Main Outputs/Indicators
Percentage Achievement of
21
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Associated Indicators 12
Outcome F
Effective functional PCU
78%
Economic Valuation of Environment Workshop
CEP Website
Three TDA/SAP/NCAP revisit meetings
Outcome G
Stakeholders Analysis Revisited
50%
Awareness materials including E Bulletin ,
posters and brochures
Public Participation Plan
35 MEG were supported under this Outcome
2. Impact of Project Activities under Objective II
The Evaluators considered two additional factors (i) the impact of the Outputs listed in Table 6 and, (ii)
given that this is a second stage project, the Evaluators were expecting to see evidence that the countries
had taken ownership of the SAP and NCAPs. This ownership would be expected to translate into
institutional reforms, budget allocations and governance reforms13.
Outcome F. The PCU has played a strong role in lobbying, providing information and serving as the
regional communication hub. Its role has been greatly appreciated by the majority of stakeholders in all
countries and internationally. Several stakeholders, including EU TACIS, IPIECA, experts under the
Darwin Initiative and BP expressed that they continue to rely heavily on the services provided by the
PCU. An important PCU achievement has been maintaining the dialogue between private sector and
governments and with other CEP projects based in other cities in the region. It is also noted that the
PCU successfully took on two functions not envisaged in the project document: acting as interim
secretariat for RAGs and supporting the Programme Coordinator.
The professional design and the voluminous scope of documentation and information available on the
Project`s/CEP`s website reflects highly on the PCU and sets the standard for future websites of the
emerging TC Secretariat. However, an assessment of visitation to the site shows that it is used mostly
by international stakeholders. Consideration should be given to how to make its main target audience
decision makers and scientists in the participating countries important visitors to the site.
The revised NCAP-TDA-SAP have had moderate impact, particularly on the NCAPs, however the new
SAP has not been translated and does not seem to have found a niche beyond the previous SAP.
Going beyond the Outputs listed in Table above, Outcome F included strengthening the national
Caspian structures, through the NFP, the SAPIC and the NCAP. A related aim of this was to strengthen
inter-sectoral coordination in each country. There have been some related successes, notably:
The Ministry of Foreign Affairs in Iran is committed to the Project and emphasised how the Project
aided them in their work. The Project also contributes information to the Caspian Committee;
A recent Presidential Decree in Turkmenistan established an Inter-ministerial Commission on
Caspian Issues. The Project certainly influenced this process, although the linkages are not tangible.
The Kazakh Committee for Control of Industrial Safety plans to create a Caspian Enforcement
Committee with dedicated ships and infrastructure. The Project certainly had an influence in this
process as a result of the ratification of the TC, although the linkages are not directly tangible.
In Russia, key stakeholder agencies` (Rosshydromet, MNR, SOI) work with the Project`s
environmental monitoring prompted an internal assessment of their institutional framework and
agreement in principle on areas of responsibility in the Russian sector of the Caspian.
12 See Annex 6.
13 This means mechanisms to involve new stakeholders, such as civil society, the public and the private sector.
22
CEP-SAP Final Evaluation
Final Draft
30/04/2010
However, the in-country, inter-sectoral coordination on Caspian environmental issues remains one of
the greatest challenges. Mostly, the Project`s national Caspian structure focuses very much on one
person, the SAPIC, who was financed by the Project14. In general, the environment agencies strongly
lead most activities whereas the leadership of other agencies is required for many environmental
improvements and to enhance sustainability.
Outcome G. The Public Participation Strategy, although well prepared, does not seem to have a clear
target and is not action-oriented. The PCU`s efforts to support environmental awareness and public
participation lack strategic thrust, results-based measurement, and a focus on specific target groups, and
many seemed over-centralised. For example, the PCU produced many materials in Iran highlighting
Caspian issues for distribution to schools and communities Caspian-wide but faced real difficulty
distributing them in sufficient numbers.
Similarly, the purpose and intended use of the revised stakeholder analysis was not clear to the
Evaluators. On the positive side, the Evaluators saw some evidence that many of the micro
environmental grants (MEG) have had an impact at the micro level, typically raising awareness and
interest of the public through schools and exhibitions. Overall, the Project made many efforts to raise
awareness and involve the public (newsletters, posters, workshops), but there is little evidence of
awareness being raised, or of the public becoming more active i.e. little evidence of impact. This was
possibly beyond the scope of a regional project.
While there is some evidence of impact under Outcomes F and G, there is no verifiable way to measure
use or usefulness (i.e. impact) for most of the outputs. Success is not clearly defined and therefore hard
to ascertain. The Evaluators were able to elicit some evidence of stakeholders doing some things
differently as a result of the Project`s work under this Objective, but for the most part it remains on
paper, and changes have yet to appear on the ground.
3. Rating Effectiveness
In order to provide a rating for effectiveness under this Objective, the Evaluators considered each
indicator from PIR 2006/07 and (i) assessed whether it was achieved (ii) assessed evidence of
associated impact. The findings are summarised in the Table 7; the Total percentages range 38-74%.
It is noted that some of the aims under this Objective (notably strengthening inter-sectoral coordination
in-country and strengthening the national Caspian structures) were not covered by any indicator. The
Project set out to do these things but was not very successful. At the same time, the Project did produce
a largely effective PCU institutional mechanism under Outcome F. These strengths as well as
weaknesses were taken into account when providing the Ratings, and tended to moderate the Rating.
Accordingly, the overall effectiveness rating for this Objective is Moderately Satisfactory`.
Table 7: Effectiveness Ratings for Objective II
Outcome
Indicator
Evidence of
Total
Effectiveness
Achieved
Impact
Rating
Outcome F
78%
70%
74%
S
Outcome G
50%
26%
38%
MU
Total
64%
38%
51%
MS
C.
Efficiency
Based on the summary budgetary information (see Annex 7), the overall expenditures on this Objective
are estimated at $2.05 million. Table 8 provides estimated expenditures under each Outcome.
14 Interestingly, the status of the SAPIC varied from country to country, with some acting as project staff and
others fully integrated into governmental structures. An analysis of the advantages of these contrasting approaches
would be useful.
23
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Outcome F, the PCU, is the most costly Outcome of the Project. The overall cost of the PCU is $3.4
million. The Evaluators attribute half of this to Outcome F, and the other half is considered to have been
used in support of all other Outcomes and is distributed evenly across them. This includes the costs of
updating the TDA and SAP. The PCU seems to have been run in a cost effective manner. For example,
it did not maintain a car or a driver and its high quality offices were provided free of rent by the Iranian
Government. Moreover, given the high costs of travel and communication, and the high quality of the
PCU work, Outcome F is rated Satisfactory.
The overall cost of Outcome G (Enhanced and informed stakeholder and intersectoral participation in
the management of the Caspian environment) was $333,000, one of the Project`s least costly
Outcomes. The results of the work under this outcome were disappointing overall. The stakeholder
participation strategy was developed but not implemented. The media kit was developed and
distributed, but there is no evidence of whether it is being used or not. The Caspian Mayors Meeting
was held only recently, in the last year of project implementation, lessening the benefits of such
networking to the Project. The Caspian Coastal Concern Groups and the project`s NGO work never
really materialized as planned. Many activities were conducted under this Outcome and the Project did
succeed in helping to cultivate cross-sectoral collaboration. However, the efficiency of this Outcome
(value for money) was relatively poor. As a result, Outcome G is rated Moderately Unsatisfactory.
Table 8: Expenditures and Efficiency Ratings for Objective II
Outcome
Expenditure (US$'000) Efficiency Rating
Outcome F
1712
S
Outcome G
333
MU
Objective II Totals 2045
MS
Overall, the combined Efficiency rating for Objective II is Moderately Satisfactory.
2.5 Results under Project Objective III
To strengthen the environmental legal and policy frameworks operating at the regional and the national
levels and where necessary improve implementation and compliance of those frameworks.
I.
Baseline
This Objective relates mostly to the process to establish and implement a legal convention with
protocols. The generic process, divided into 7 steps, is illustrated in the following figure. Based on
experience in other regions (e.g. Rhine River, Black Sea), it may take as long as twenty years to fully
complete all 7 steps.
Figure 3: Process to implementing regional seas Conventions
A
B
7.
1.
2.
3.
4.
5.
6.
Developing
Scoping and
Consultation
Negotiating
Ratifying
Ratifying
Strengthen-
operational
scientific
and trust
Convention
Convention/
Protocols;
ing national
capacity in-
country and
assessment
building
Text and
Negotiating
Establishing
legislation
signing
protocols
and funding
and policies
investing in
implementat
Secretariat
-ion
24
CEP-SAP Final Evaluation
Final Draft
30/04/2010
At the outset of this Project, the countries had negotiated and signed the Framework Convention for the
Protection of the Maritime Environment of the Caspian Sea` (the Tehran Convention), and they had
initiated discussions on four protocols (i.e., in Figure 1, the process was at Point A, between step 3 and
step 4). In the baseline, without international support, the countries may have progressed to step 4, but
not beyond. However, the countries also benefited from 2.5 million of support from EU TACIS to this
process during the Project implementation period.
In addition to the national legislative and regulatory structure, each country should make a series of
investments in sustainable development and environmental protection under step 7. These should
respect the priorities as set out in NCAP. Hence, investments to implement the NCAP are considered
under this Objective. In the baseline, there would have been growing number of such investments,
however they would not be coordinated and regional aspects would not be given high consideration.
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective III.
A.
Relevance
There is only one Outcome (Outcome H) under this Objective. The six concerned Activities focus on
steps 4 through 7. Broadly speaking, the target under this Objective is to make as much progress as
possible on Steps 4-7, noting that one step may be initiated prior to the completion of the previous
steps. This Convention and the Protocols are central to the process of regional cooperation on
management of the Caspian Sea. Hence, in terms of relevance, this Objective (and Outcome) is
considered Highly Satisfactory.
Table 9: Relevance of Outcomes under Immediate Objective III.
Outcome
Relevance
Outcome H: Preparation of ancillary agreements to the Framework Convention and drafts of
HS
the major protocols targeting priority transboundary issues (biodiversity, PTS, invasive
species, land-based sources, marine and seabed pollution, EIA, data exchange).
TOTAL
HS
In parallel to the process to prepare the Tehran Convention (TC), the five Caspian countries are
negotiating a convention determining the legal status of the Caspian. This legal Convention has many
implications for borders, definitions and natural resources management. The fact that the countries have
not reached agreement on the legal Convention could be considered an obstacle to progress on the
Tehran Convention and related protocols. On the other hand, environmental issues covered by the TC
are less political and are shared priority where the countries can demonstrate progress in their regional
cooperation.
B.
Effectiveness
As described earlier, the discussion on Effectiveness is guided by two considerations:
Question #1: To what degree were the relevant indicators in the Project Document achieved?
To answer this question, the Evaluators reviewed each indicator included under each Outcome in the
2006/07 PIRs. The evaluators assigned a value for the achievement of each indicator. These values
can be seen in Annex 6. The compiled percentages for each Outcome can be found in the
Effectiveness ratings table below (Table 10).
Question #2: What other issues should be considered, including evidence of the degree of impact with
respect to the indicators and relevant impacts not covered by the listed indicators? Question 2
25
CEP-SAP Final Evaluation
Final Draft
30/04/2010
supplements Question 1 in that it seeks to look further into the question of impact with respect to each
Outcome in order to arrive at a fair rating for project effectiveness.
1. Discussion of Outputs/Indicators:
From the PIRs, two tangible Outputs are expected under this Objective, namely: (i) Four draft Protocols
on land based sources of pollution, biodiversity, oil pollution incidents and EIA in the Transboundary
context and (ii) Ratified Convention. The second Output has been fully accomplished. This is
considered by nearly all stakeholders interviewed by the Evaluators to be the greatest achievement of
the Project. Whereas this may appear straightforward with hindsight, it was a great challenge at the
Project outset, and the fact that it was finalised prior to the completion of negotiations on the legal
status is considered especially noteworthy. The Convention Secretariat is located temporarily in UNEP;
the next urgent challenge is to agree on the permanent location, and then implementation can take off.
The first Output has also been accomplished. Drafts have been prepared and discussed by the countries.
However, important questions remain: (i) given that the protocols are to be the teeth` to implement the
Convention, how effective is the wording in the drafts? (ii) when will the Protocols be completed and
signed? A review of the text of the draft protocols suggests the wording is effective and they compare
well to similar protocol in other seas. The biodiversity protocol seems to be the weakest of the
Protocols, and does not raise many new obligations beyond those under the global Convention on
Biological Diversity. The other three draft protocols have more articles that should lead to regional
cooperation and should regulate activities with trans-boundary impacts.
To summarise, compared to the baseline, at the end of the Project, Step 4 is completed, Step 5 is almost
complete, and Step is 6 has been initiated (as indicated by Point B in Figure 3).
2. Other Impacts of Project Activities under Objective III
The indicators do not cover two aspects to the work undertaken by the Project under this Objective.
These are: the work to strengthen the national implementation capacity, and the investments into NCAP
related activities. The Project recognised that only when these steps are well underway will there be
significant changes in behaviour and in environmental threats.
With regards to strengthening national capacity of countries to implement the Convention, this is
mentioned in the project document. The Evaluators observed limited major progress on this issue.
However, it is accepted that only limited progress can be made on steps 6 and 7 prior to step 5 being
completed.
With regards to investments in environmental protection and sustainable development, it is understood
that these should go ahead in parallel with development of the regulatory regime. As revenue has risen
in the countries, there has been a notable increase in such investments. These are understood to be
generally in line with NCAP15. The evaluation was not able to assess the environmental impact or
priority nature of these investments. These include:
Azerbaijan moved to implement its NCAP through a National Eco-Complex Action Plan, 2006-
2010. Typically, a new National Park with a focus on seals was established. Also, there is large-
scale investment programme to treat domestic and industrial wastewater around the Paterson
peninsula. Recently, a Presidential decree allocated US$ 25 million to this;
Turkmenistan is investing US$41 million in its fishery restocking programme and US$1 million
in fishery protection;
15 The Evaluators were not sure where to include a consideration of these investments. The links with project
activities are not clear, but it was probably through the support to the SAPIC and NCAP, which is mostly under
Objective II. However, these investments contribute to meeting standards and improving natural resources
management. They therefore seem most suited to Objective III.
26
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Kazakhstan has already incorporated NCAP/CEP-inspired activities into several Government
programs, including:
o Government Program for Protection of Environment of Kazakhstan.
o Measure requiring the capping of abandoned oil wells in flooded areas of Kazakh Caspian.
Under this measure at least 15-20 uncapped wells have been capped.
o Measure to clean up old oil storage sites along shore of Caspian. KazMunaiGaz and
partner oil companies in the region have implemented this measure.
The success in ratifying the Tehran Convention recently led to the first Conference of the Parties (COP
I). This marked some important achievements including adopting rules for COP procedures and
financial issues. These achievements also benefited greatly from the EU support through UNEP, and
cannot all be attributed to the Project.
Although the Project may not have played a direct role in catalysing many of these, it surely had an
indirect impact, through the SAPIC and NCAP process.
3. Rating Effectiveness
In order to provide a rating for effectiveness under this Objective, the Evaluators considered each of the
five indicators under Objective III from PIR 2006/07 and (i) assessed whether it was achieved (ii)
assessed evidence of associated impact (see Annex 6 for details). The Total figure is 64%. In addition,
on the strength of the ratified Convention, the draft protocols and project success in strengthening
national capacity and in coordinating NCAP implementation, the Evaluators provide an effectiveness
rating of Highly Satisfactory`.
Table 10: Effectiveness Ratings for Objective III
Outcome
Percentage
Evidence of
Total
Effectiveness
Achievement of
Impact
Rating
Associated
Indicators16
Outcome H
74%
54%
64%
HS
Total
74%
54%
64%
HS
C.
Efficiency
The estimated expenditures under this Objective are $625,000. It should be noted that a great deal of
additional support to this Objective came from EU TACIS through UNEP, hence not all the success can
be attributed to the UNDP/GEF support. If the effectiveness or impact of training activities were
monitored more closely, this could help to solidify and improve an efficiency rating. Accordingly,
despite the high Relevance and high Effectiveness ratings under this Objective, the Evaluators rate
Efficiency as Satisfactory.
Table 11: Expenditures and Efficiency under Objective III
Outcome
Main Outputs
Expenditure
Efficiency
(US$'000)
Rating
Outcome H Four draft Protocols on Land Based Sources of Pollution,
625
S
Biodiversity, Oil Pollution Incidents and EIA in Trans-
boundary context.
Ratified Tehran Convention.
Training for national structures.
TOTAL
625
S
16 See Annex 6.
27
CEP-SAP Final Evaluation
Final Draft
30/04/2010
2.6 Results under Project Objective IV
To achieve tangible environmental improvements in priority areas by implementation of small-scale
investments supported by a small matched grants programme.
I.
Baseline
The purpose of this component is to assist the implementation of small projects that address a national
priority, as identified in the NCAP, and that have a positive, multi-country impact in line with the issues
identified in the SAP. In the baseline, such projects may have been implemented with support from
governments, international partners or the private sector. In the baseline, few, if any, of these projects
would have a multi-country aspect.
II.
Findings With Respect to Relevance, Effectiveness and Efficiency under Immediate
Objective IV.
A.
Relevance
The matched small grants (MSG) programme follows-on from a similar programme implemented in the
earlier stages of CEP-SAP and managed by the World Bank. There is only one Outcome under this
Objective, which is fully in line with the Objective. The project team explained that the five criteria for
selecting grantees were: (i) relevance to TDA; (ii) trans-boundary nature; (iii) innovative and pilot
nature; (iv) leading to concrete results; and (v) matched` - i.e. at least an equal amount of co-finance.
These criteria are relevant. Hence, in terms of relevance, this Objective and its one Outcome are
considered Satisfactory.
Table 12: Relevance of Outcome under Immediate Objective IV.
Outcome
Relevance
Outcome I: Matched funding of small--scale investments from the NGO, public and private
S
sectors, which target common or transboundary Caspian issues identified as priorities in the
TDA/NCAPs/SAP and will result in tangible environmental improvements.
TOTAL
S
B.
Effectiveness
As described earlier, the discussion on Effectiveness is guided by two considerations:
Question #1: To what degree were the relevant indicators in the Project Document achieved?
To answer this question, the Evaluators reviewed each indicator under Outcome I in the 2006/7 PIRs.
The evaluators assigned a value for the achievement of each indicator. These values can be seen in
Annex 6. The compiled percentages for each Outcome can be found in the Effectiveness ratings table
(Table 14) below.
Question #2: What other issues should be considered, including evidence of the degree of impact with
respect to the indicators and relevant impacts not covered by the listed indicators? Question 2
supplements Question 1. It delves into the question of impact with respect to each Outcome in order to
arrive at a fair rating for project effectiveness.
1. Discussion of Indicators/Outputs:
28
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The 2006/7 PIRs included two indicators: that the grants are awarded and implemented. Implemented`
is understood to mean that the funds are transferred to the grantees and does not appear to refer to the
final quality or impact of the investment. Hence, both indicators are fully met. Yet, the Evaluators feel
that more ambitious targets are required accounting for the quality and tangible impact of the MSGP
projects, particularly as this is the second stage of a MSGP. The Evaluators feel that the small projects
should have an effect at the local level and should, in general, have some measurable impact in terms of
contributing to the SAP/NCAP`s objectives, replication and dissemination.
2. Other Impacts of Project Activities under Objective IV
12 MSG projects were supported under this Objective. The 12 projects were selected using the five
criteria listed above. The projects were distributed evenly across the five countries. These projects
addressed diverse issues such as artificial fish production, small-scale reforestation and plastics
recycling. The grantees were also diverse, including large-scale institutes, small NGOs and micro-
private enterprises. This diversity is an achievement.
The Evaluators visited four MSGP sites (see summary of visits in Annex 5) in three countries, met one
other grantee and reviewed documentation for the others. The overall results were mixed as seen in
Table 13.
Table 13: Match Small Grants Results
#
Matched Small Grants
Sustain-
Impact
Replic-
Relevance Score
ability
ability
to SAP
Azerbaijan
1* Promoting Sustainable Development in 217
1
1
2
6/20
local communities of Caspian region
through alternative sources of energy
2
Clean up of Oil-contaminated Soil
NE
NE
2
4
NA
Iran
3* Creation of a sperm bank for sturgeon
3
2
3
4
12/20
breeders in the south of Caspian
4* Artificial Spawning of Rutilus Kutum
3
4
2
4
13/20
5
Sustainable agro-ecosystem management NE
NE
4
4
NA
through participatory integrated crop
production & protection of rice fields
Kazakhstan
6
Cleaning the bottom of Ural River at the
NE
NE
NE
3
NA
west-Kazakhstan Oblast
7
Conservation & Restoration of Caspian
NE
NE
NE
4
NA
Sturgeon Stocks
Russia
8
Introduction of innovative technology of NA
NA
NA
NA
NA
marketable fish production in Tumak village
9
Supplying Sewage Purification Equipment NE
NE
2
4
NA
to Floating Hotels in Volga River
Incineration of POPs (cancelled)
NA
NA
NA
NA
NA
Turkmenistan
10
Recycling plastic waste materials in Too early
Too
Too
4
NA
*
Turkmenbashi city
early
early
11
Breeding Artemia Salina in Ponds
1
1
1
2
5/20
*
12 Potable Water for Inhabitants of Goyudijik NE
NE
1
1
NA2
Settlement
0
* = Evaluators were able to visit the site and/or speak with stakeholders.
Success in terms of the five selection criteria are discussed below:
17 Ranked on a scale 1-5, with 5 the highest.
29
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Relevance to TDA/SAP. All MSGP projects visited were considered relevant to TDA. However,
worldwide experience with small grants suggests that one of the greatest challenges is to ensure that
they are technically strong. Small grants typically address complex issues requiring good technical
input and experience. For example, one MSGP project included planting non-indigenous trees, which
require constant maintenance, and another involved catching wild individuals of rare fish species to
supply artificial breeding programmes. The technical soundness of these is a question beyond the remit
of the Evaluators. Even if technically successful, there are environmental risks associated with such
projects. The Evaluators were not able to confirm that the Project had been able to provide adequate
technical support and oversight to the MSGP projects. Also, relevance in this respect should also
consider relevance to the GEF incremental cost rule. Some funded MSGP deal with baseline activities
such as clean drinking water that traditionally are not GEF-eligible.
Trans-boundary18 nature All the projects reviewed involved issues common to most or all countries, and
issues relevant at many sights around the sea. However, there was little evidence of a strategic approach
to facilitating cooperation across countries or to disseminating findings around the region. In general,
stakeholders in one country were unaware of MSGPs in other countries and none of the projects
involved stakeholders from more than one country.
Level of Impact: All MSG projects reviewed were designed to lead to concrete results, in either
ecological or economic terms. Whether or not these projects had the intended impact is more difficult to
judge because MSG monitoring focussed on project selection and fund disbursement; impacts were not
monitored or measured. Consequently, it was impossible to determine objectively the impact of the
MSG projects.
The Evaluators visited MSG project Artificial Spawning of Rutilus frisii Kutum in Iran. This appeared
to have had a significant impact within the Iranian fishery research agency. Under the MSG project, the
agency successfully developed a methodology for artificial reproduction of the fish that it is now
operating successfully. The agency released two million fingerlings into Anzali Lagoon in a celebratory
event covered by local and national media. The project also dramatically increased the knowledge in
Iran of the specific autumn run of kutum.
Some MSG projects appear promising, but it is still too early to judge success. These include the tree
plantation project in Azerbaijan and the Gathering, Sorting & Processing of Plastic Waste Materials in
Turkmenbashi, Turkmenistan. The impact of other MSG projects in Turkmenistan is difficult to see.
The Evaluators were not able to visit any MSG project sites in Russia or Kazakhstan.
According to second and third party reports to the Evaluators, some MSG projects can already be
judged a success on the basis of impact. The integrated pest management MSG project in Iran is
reported to have yielded impressive results in terms of changing agricultural practices and improving
the economics of rice production. The Evaluators cannot cite this as an example as they were not able to
meet directly with the stakeholders involved, and cannot determine attribution.
To summarise the impact of MSG projects, most of the MSG projects visited or reviewed do not seek to
remove threats or root causes. However, some projects have been impressive in their specific
measurable results that help local stakeholders, fill knowledge gaps, and show people the benefits.
Matched funding. At least 50% of the financing for all MSG projects came from sources other than
GEF. The project was very successful on this account.
Finally, the Evaluators found little evidence of a clear strategy to determine the objectives and
approaches to MSG nor of impact targets and indicators, despite the program`s use of the five criteria in
reviewing MSG proposals. This applies both to individual projects and the programme as a whole.
18 This includes both trans-frontier issues and issues that are common to many points in several countries.
30
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Ultimately, the MSGP was a series of small, unconnected actions across a huge area each with some
likelihood of success, but with only a minor likelihood of replication or structured lesson learning.
Likewise, results-based monitoring and reporting seemed weak: both in terms of individual projects
and, more importantly, in terms of the overall MSGP. How is each MSG project related to the
respective NCAP? What is each MSG project`s specific tangible impact on the environment?
3. Rating Effectiveness
The 2006/07 PIRs have two indicators to measure progress towards this Objective. In order to provide a
rating for effectiveness under this Objective, the Evaluators considered each of these indicators and (i)
assessed whether it was achieved (ii) assessed evidence of associated impact, notably associated with
the Objective`s wording calling for tangible environmental improvements. The findings are
summarised in Table 14 and details are provided in Annex 6. It is noted that the two indicators are very
much focussed on the process of issuing grants, and do not address the quality or impacts of the MSGP.
Hence far more weight is given to the score on evidence of impact` than on indicator achieved`
hence the final rating is Moderately Satisfactory.
Table 14: Effectiveness Rating for Objective IV
Outcome
Indicator
Evidence of
Total
Effectiveness
Achieved
Impact
Rating
Outcome I
85%
40%
63%
MS
Total
85%
40%
63%
MS
C.
Efficiency
The Project seems to have been reasonably efficient in identifying, reviewing and selecting the MSGP
proposals, given the unwieldy nature of the 5-country review process. The process of selecting the
grantees was scientific, open and clear.
Efficiency was diminished under this Outcome, however, by several factors:
1) Stakeholders complained that the process of transferring funds and reporting requirements were
cumbersome and slow and caused delays in implementing MSG projects. This affected the
timeliness of the work, the confidence grantees had in the process, and in some cases also may have
affected the results.
2) The Project learned well from the first phase of the MSG program. However, there is no evidence
that it learned from UNDP`s significant body of experience with small grant management, design
and oversight, developed through the UNDP/GEF Small Grants Program.
Total expenditures under this objective are estimated at US$903,000. This was invested in 12 grants
averaging approximately $33,000. As is typical, with MSGP, the overheads are high, and this is
particularly understandable for a regional programme.
Table 15: Expenditures and Efficiency under Objective IV
Outcome
Main Outputs
Expenditure
Efficiency
(US$'000)
Rating
Outcome I
12 Matched Small Grant Program projects
903
MU
selected and implemented
TOTAL
MU
In summary, the Evaluators rate the Efficiency of GEF`s investment under Objective IV as Moderately
Unsatisfactory.
31
CEP-SAP Final Evaluation
Final Draft
30/04/2010
2.7 Conclusion
Table 16 provides the summary ratings for relevance, effectiveness and efficiency for the four project
objectives. In line with GEF guidance, the total` rating for each objective cannot be higher than its
rating for relevance and effectiveness. Hence, for the four objectives, two have a total rating MS, and
two have a total rating S. Aggregating these, the final rating for the Project`s relevance, effectiveness
and efficiency is S.
Table 16: Summary of ratings at the Objective level for relevance, effectiveness and efficiency
Objectives
Relevance
Effectiveness
Efficiency
TOTAL
Objective 1
S
S
S
S
Objective 2
S
MS
MS
MS
Objective 3
HS
HS
S
HS
Objective 4
S
MS
MU
MS
TOTAL
S
S
MS
S
The Evaluation`s findings with regards to the Project`s relevance, effectiveness and efficiency reveal a
project that has generated some impressive results, but suffers somewhat from a vague and unclear
vision of success. Examples of the Project`s strengths include the achievements under Objective 3,
mainly, the ratification by all 5 Caspian Countries of the Tehran Convention and the development of
draft protocols to the Convention, which if ratified by the countries, will pave the way for more
significant national-level commitment. The robust and systematic process introduced to the Caspian
region during the first phase and continued and strengthened in this CEP-SAP project is a second
strength. This systematic process of the TDA-SAP-NCAP supported by various field surveys, plans and
strategies, was mentioned by several high-level stakeholders as one of the enduring contributions of the
Project. The Project produced many high quality studies and reports, some of which catalysed follow-
up action.
The Evaluation also finds that the Project and its activities are very much relevant to the country
priorities in the region. This is not a minor achievement in this dynamic political geography. It seems
that the Project and the overall programme have helped to provide the Caspian countries with a
productive way to demonstrate progress in working together in the Caspian region.
On the other hand, the Evaluation reveals a project that was not focussed and failed to correct its lack of
focus. Many project activities, notably several surveys and studies and small grants, did not yield the
full results anticipated or those expected by a reasonable evaluation. The main reasons for this are
discussed in Chapter 6. Many of the successes in the period 2004-2007 were a result of action taken in
the first stage of CEP-SAP. Generally, this Project was about starting SAP implementation and securing
regional ownership of such, and the Project has not quite achieved this yet; there is still not a robust
self-sustaining regional process` as such.
Overall, the Project appears to have excelled in its role as regional catalyst of process meetings,
trainings, surveys, and planning, yet struggled in enabling stakeholders to generate changed situations
and changed practices. The evaluation finds a project that confused outputs with outcomes,
resulting in too much focus on producing things and not enough focus on impact and enabling people
to use these things to change practices. The Evaluation finds a project that struggled under at least two
of its four Immediate Objectives to be effective and to produce impact.
Some stakeholders commented that there were too many workshops, reports, letters and not enough on-
the-ground action, whereas other stakeholders greatly appreciated the numerous training and
coordination opportunities.
32
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Likewise some stakeholders commented that the Project had become too political` (and possibly
therefore losing some objectivity), whereas other stakeholders felt the Project`s greatest achievements
was to keep the political process moving.
Overall the CEP-SAP has been important. It has maintained its relevance. One of the lessons is that,
although GEF and UN support to the process is critical, leadership from the countries is essential and
the UN support should be more instrumental in stimulating this leadership.
3. Assessment of Sustainability of Project Outcomes
If CEP was to stop now, is the Project-inspired work sustainable? This Chapter assesses the likelihood
of sustainability of the Project`s nine main Outcomes. This Chapter gives special attention to the risks
and contextual factors that are likely to affect the persistence of Project Outcomes and addresses the
following four dimensions of sustainability: Financial, Institutional/Governance, Socio-political, and
Environmental.
Before starting the discussion, the Evaluators wish to draw attention to the difficulty in achieving
sustainability. As observed in other regions, it may reasonably take 20 years to establish sustainable
joint management mechanism for international water bodies the Caspian has only had 10 years.
3.1 Assessment of Project Outcomes
The following Table rates the sustainability of project outcomes. Sustainability is rated as follows:
Likely (L), Moderately Likely (ML), Moderately Unlikely (MU), Unlikely (U).
Table 17: Summary of Ratings for Sustainability of Project Outcomes
Outcome
Sustainability
Rating
Outcome A: A quantitative assessment of habitat loss and verification of critically
MU
threatened areas, and, the design and establishment of a standardised monitoring
methodology programme in conjunction with the oil and gas industry.
Outcome B: Preliminary implementation of the Caspian Biodiversity Action Plan, focusing
MU
on compliance with existing regulations, implementation of species and habitat conservation
action plans and targeted public awareness campaigns.
Outcome C: Implementation of the CEP invasive species action plan in close coordination
MU
with the GEF Global Ballast Waters project to address the impact of Mnemiopsis on the
Caspian ecosystem.
Outcome D: Assessment of the pollution loading of the Caspian and determination of
MU
distribution and composition of Persistent Toxic Substances (PTS) in the riverine and coastal
waters and sediments, in order to prioritise future interventions.
Outcome E: Regional and National Action Plans addressing the activities contributing to
MU
transboundary PTS, including persistent organic pollutants, oil products, and heavy metal
pollution.
Outcome F: A sustainable and regionally owned coordination mechanism in the form of a
MU
country-supported PCU, strong country-supported National Coordination Structures, and a
network of institutions addressing transboundary environmental issues of the NCAPs and
SAP.
Outcome G: Enhanced and informed stakeholder and inter-sectoral participation in the
MU
management of the Caspian environment.
Outcome H: Preparation of ancillary agreements to the Framework Convention and drafts of
ML
the major protocols targeting priority transboundary issues.
Outcome I: Matched funding of small--scale investments, which target common or
U
transboundary Caspian issues identified as priorities in the TDA/NCAPs/SAP and will result
in tangible environmental improvements.
TOTAL
MU
33
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Based on evidence provided and past experience with projects in other parts of the world and in the
region, and using the GEF guidance19, the Evaluators assessed the likelihood of the sustainability of
each of the Project Outcomes. The findings are displayed in Table 17. Eight of the nine project
Outcomes are rated Moderately Unlikely or Unlikely to be sustainable. The overall rating is
Moderately Unlikely. If external funding stops tomorrow, most of the Project-inspired research,
survey and planning work would not be sustainable.
If international support was to stop now, the regional process would drop-off significantly and
momentum would fall. The Evaluators consider it moderately likely that this would continue at a lower
level. It is also moderately likely that the process of the adopting the protocols would continue and that
some sort of regional Secretariat would be formed. Therefore Outcome H is the one Outcome whose
sustainability was rated Moderately Likely.
The countries have agreed to co-finance the future TC Secretariat from 2009, but funding has not yet
been allocated. While the funding levels are limited, this commitment is progress, and it hopefully
opens the door to higher levels of funding from the countries in the future. This is further evidence that
the regional collaboration process is not yet ready to fly under its own power.
More worrying, the robust regional PCU is not yet sustainable. With the closing of the Project, there
would be no regional coordinating mechanism to catalyze collaboration among the five countries.
Without the SAPIC at the national level, there will be no one to champion the SAP and the NCAP in
each country. Some project-inspired regional conservation work would continue under funding from
other partner sources, such as is the case with the Caspian Seal, but this would slow considerably.
The Evaluators recognise that the Project aimed to establish complicated inter-country processes and
mechanisms, and that long lead times are required for this. In these terms, it may not be realistic to
expect to achieve sustainability within the time span of the Project. It may have been possible to put a
process in motion and the Outcomes should have been designed and structured to reflect that.
3.2 Assessment of Four Dimensions of Sustainability
In this section, the Evaluation assesses the four dimensions of sustainability: financial, socio-political,
institutional governance, and environmental. The Evaluators, based on previous experience and
knowledge of the region, determined where it would be reasonable to expect the process to be by the
end of the Project, and to compare this with the actual status. The findings are displayed in Table 18.
More discussion is provided after the Table.
Table 18: Assessment of the Four Dimensions of Sustainability
Dimension of
Where should the process be?
Where is it?
Rating
Sustainability
Financial
Regional: The ratification of the
Regional: Countries have given
MU
TC by all five countries provides
tentative agreement to providing a
a solid legal basis in each country. modest level of funding for the to-
Consequently, at least 3 of the 5
be-formed TC Secretariat.
countries should be giving
minimal funding to the regional
Countries only contributed 15% of
and national Caspian processes.
the anticipated direct contributions
to the CEP-SAP project (although
19 Likely (L): There are no or negligible risks that affect this dimension of sustainability. Moderately Likely (ML): There are
moderate risks that affect this dimension of sustainability. Moderately Unlikely (MU): There are significant risks that affect
this dimension of sustainability. Unlikely (U): There are severe risks that affect this dimension of sustainability.
34
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Dimension of
Where should the process be?
Where is it?
Rating
Sustainability
indirect funding of NCAP-related
activities was high).
National: Countries should be
National: Countries are making
ML
endorsing NCAPs and making
budgetary allocations to NCAP-
budgetary allocations to NCAP
related activities, but the leveraging
activities.
link between NCAP and the actual
funding is not clear.
Socio-political
High-level political support.
High level of political support for
ML
Political leaders recognising
TC process and the general
importance of Caspian
framework of the Convention as
environment;
evidenced by the rapid ratification
of the TC.
Healthy cross-sectoral stakeholder Public in Caspian region interested
involvement in Caspian matters
and taking action, but low level of
with fledgling networks
officially recognized involvement at
operational.
the regional level.
Stakeholder involvement is
hampered by different definitions of
what stakeholder means on the
part of the Governments and the
Project.
During the first COP meeting of the
TC in Spring of 2007, delegates
from the five Caspian Countries
tentatively agreed to contribute a
combined total of $350,000/year to
support the future TC Secretariat.
Institutional/
Laws and institutions are in place
If CEP-SAP project stops tomorrow,
MU
Governance
or under improvement.
the maturation of the Tehran
Convention would likely slow or
At least three of the five national
stop. There would be no central
governments should be paying
coordinating structure to organize
SAPICs to do the job of
meetings, no SAPIC champion for
coordinating country`s Caspian
NCAP in each country, no ongoing
work or some similar position by
coordinated expert attention and
now.
input to key Caspian issues, and no
link with the overall SAP.
At least three of the five countries
should have dedicated, paid
The Project`s systematic approach
advisory experts nominated to
to assessing environmental risk and
work with the regional program
developing solutions for minimizing
on key issues.
it has functioned in nearly every
sub-sector of project work (Seals,
pollution, invasive species, priority
sites). But it is not institutionalized
yet and in the absence of a central
coordinating unit like PCU with
substantial funding, it will falter.
Environmental
Environmental risks that could
Of course there are environmental
ML
jeopardize the ecosystem health
risks that may jeopardize the
of the Caspian should be well
sustenance of project outcomes, but
35
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Dimension of
Where should the process be?
Where is it?
Rating
Sustainability
understood with baseline of
these risks appear for the most part
information and knowledge in
to be under control, with exception
place. Actions to mitigate or
including the mnemiopsis and
remove the causes of these risks
possible oil spills.
should be under implementation.
The main environmental risks are
A systematic approach to
well understood and an information
understanding environmental
baseline of knowledge, while still
problems and priorities and
under development, is in place and
developing solutions to those
does support regional consultations
problems is in place.
and decision-making.
Overall Rating
ML
Financial: At the national level, each of the five countries exhibits significant funding potential for
SAP/NCAP-related activities within their own Caspian territories, and in many cases important
investment programmes are underway. Although there are risks at the national level should the oil
price collapse or should priorities be completely adjusted the overall picture is rather healthy.
There are more weaknesses at the regional level, with countries not yet adequately committed to
financially supporting regional coordination and planning, and little interest in regional or inter-country
investments. At this point in time, it is unlikely that sufficient financial and economic resources will be
available for regional activities once the GEF assistance ends.
Socio-political: Anyone who follows international news and current events knows that there are
significant political risks in the Caspian region and that these may jeopardize the sustainability of a
regional project`s outcomes. However, to date, such political risks have not ruled the day and in fact the
five Caspian countries have demonstrated notable political support for the Tehran Convention and its
evolving protocols as well as for NCAP-relevant national level activities.
The Project has done a good job of working with a core group of high-level stakeholders in government
and technical institutions. Among this core group of insiders, there appears to be sufficient awareness
and support for CEP-SAPs long-term objectives, to keep the process moving. In these circles,
stakeholders see that it is in their interest to maintain the flow of project benefits. However, the situation
is weaker beyond the core group and at operational levels, where many important stakeholders receive
few benefits from the regional collaboration as of yet. At the operational level, there is a risk that the
level of stakeholder ownership will be insufficient to allow for the Project outcomes/ benefits to be
sustained.
In the long-term, public support will be essential. The Project has not measured public awareness and
presently there is no way to assess whether there is sufficient public awareness.
Institutional framework and governance: Unlike strong political commitment to the concept of
regional collaboration on ecological issues, the operational support of or commitment to country-funded
and driven coordination at regional level is weak. The supporting institutional frameworks are not yet
fully operational in any of the countries.
The following points help to explain why this is the case. First, to develop such support requires a
process focused on achieving budget allocations to support such operations.
Second, there is a certain amount of inertia that must be overcome in prying open the door to country
funding for regional initiatives such as this. Up until recently, there were no legally binding
36
CEP-SAP Final Evaluation
Final Draft
30/04/2010
commitments and no legal status associated with the transboundary coordination on Caspian Sea
ecosystem health. This significantly hindered the ability of national Ministries to fund such initiatives.
Third, now that the Convention has been ratified, it has legal status and carries with it legal obligations
for the countries. This, along with the finalization and endorsement of the protocols, should pave the
way for more national support for regional activities. At present, the commitment is to the Convention`s
general platform, there is not yet strong political commitment to specific actions.
And fourth, which country will go first? There is a certain hesitancy to be the first country to actually
fund a regional capability.
Environmental: There is, and always will be, environmental risks that may jeopardize the sustenance
of project outcomes, but these risks appear for the most part to be well managed. One exception is the
invasion of mnemiopsis this may worsen and have unforeseen consequences for the health of flagship
species like the Caspian Seal and for the ecological balance in the Caspian. A second exception relates
to the large and growing exploitation of hydrocarbon resources in and around the Caspian. While
exploitation appears to be following international environmental standards, the chance of a hazardous
material spill is still present and response plans are not yet in place or tested. In general, the main
environmental risks are well understood and an information baseline of knowledge, while still under
development, is in place and does support regional consultations and decision-making.
3.3 Conclusion
In conclusion, some aspects of sustainability have been achieved; others have not. Indeed when directly
considering the Project Outcomes, the overall rating for sustainability is Moderately Unlikely.
Overall, it was unfair and over-ambitious to expect the Project to create fully sustainable Outcomes in
the given period.
On the positive side, when one considers the four dimensions of sustainability, the overall rating for
Sustainability is Moderately Likely. The Evaluators put more weight on this rating because there is a
sound basis and positive momentum towards a sustainable outcome with respect to the TC itself. The
ratification of the TC and the recent COP-1 decision to support a TC Secretariat are evidence that a
reasonable level of sustainability can be achieved with the right level and kind of additional support.
4. Monitoring and Evaluation (M&E) System
4.1 M&E Planning and Design
The Project was designed and developed during 2003. At that time, GEF guidelines on M&E had not
been finalised. However, the idea of results-based management was taking hold in the GEF world.
Beyond GEF, results-based management, adaptive management and effective use of monitoring had
been accepted and important concepts for some time.
The main text of the Project Document provides an adequate but unstructured description of the
baseline situation, using the information generated from the earlier stage CEP-SAP project. The Project
Document also provides a brief overview of the Project approach to monitoring, covering the main
issues. The Project Document suggests a rather ambitious approach to monitoring, including: regular
PSC oversight; regular Tri-Partite Review (TPR) meetings; CEP inter-agency meetings; Regional
Advisory Groups (RAG) providing technical oversight; extensive use of process, stress reduction and
environmental status indicators as determined by GEF IW, and; two evaluations at mid-term and at
37
CEP-SAP Final Evaluation
Final Draft
30/04/2010
project end. However, the Project Document provides too few details as to how the M&E will be
implemented, or the main milestones and products. The entire M&E system is described in less than
one page. In addition, the document did not specify responsibilities for M&E. Moreover, the Project
budget allocated only US$60,000 to monitoring and evaluation (i.e. approximately 1% of total budget)
at the time that was clearly far too small, particularly for a regional project.
The logical framework in Annex B to the Project Document provides Objectives, Outputs (referred to
as Outcomes in the main text), Activities, Verifiable Indicators and Sources of Verification20. These
indicators, which should be a key tool for monitoring and evaluation, suffer the following weaknesses;
There are far too many of them - over one hundred! They provide a detailed list of proposed project
activities and outputs, rather than a manageable number of indicators;
There is no hierarchy. All indicators are at the Activity/Output level, there are no indicators for
Outcomes, for Objectives or for the overall project Objective;
They do not capture the quality or the impact of activities, the focus is almost entirely on whether
an activity has been completed;
They are not sufficiently SMART.
Rating M&E Planning and Design as can be seen from the above discussion, the planned M&E system
was weak. The Evaluators considered two additional factors when giving the rating (i) most GEF
projects approved at that time had similar M&E systems, hence the judgement should not be too harsh;
(ii) as this is a second stage project, one would expect a more complete M&E system. Notably, the
evaluation of the CEP-SAP first stage had identified the weakness of indicators in the first stage and
had made a recommendation to improve the indicator system. Taking into account the above, the
Evaluators rate the M&E Planning and Design as Moderately Unsatisfactory.
4.2 M&E Implementation
The principle tools used for monitoring activities were: RSC, TPR, RAG, PCU, the project
implementation review (PIR) and independent evaluations. Their respective inputs to M&E are
discussed in the following sections.
The focus of the Regional Project Steering Committee (RSC) is the overall CEP programme and so its
remit goes beyond the CEP-SAP. The RSC involves the five participating governments, UNDP/GEF
and other key stakeholder such as the EU, WB and private sector representatives. These all participated
actively in the RSC. This broad membership is an achievement in itself and one that is recognized and
appreciated by many CEP-SAP stakeholders interviewed for this evaluation. A review of the RSC
minutes indicates that the meetings were well prepared and well organised. The key issues to be
discussed were described in a preparatory note that included draft recommendations, the majority of
which were fully adopted. Hence, the RSC was effective at identifying issues21 and overall the RSC was
a well-organised, professional CEP monitoring tool. However, it did not focus on project management`
or project monitoring.
The RAGs appear to have played a more mixed role in monitoring. The Project financed two RAGs,
EU-TACIS financed two RAGs, and the Governments financed one. The RAG meetings focussed on
CEP substantive and technical issues, including reviewing the Outputs from the CEP-SAP. In many
cases this proved useful but was not project monitoring. It should also be noted that international
finance covered only participation in regional meetings22. In between meeting the RAG members were
20 The logical nature of this framework is discussed in Chapter 6 below this Chapter is limited to a discussion on
M&E related issues.
21 The evaluation does note that the RSC struggled to influence issues and resolve problems. Most of the issues
identified as problematic in the first meeting were still problematic at project end (e.g. lack of country ownership,
inability to cooperate with ABC). This is partly a result of the difficult nature of many of the issues.
22 Typically one or two times per year.
38
CEP-SAP Final Evaluation
Final Draft
30/04/2010
not active and there was little continuity. In addition the RAG members changes too often, and
members were not always the most pertinent to the issues discussed.
No evidence was available to the Evaluators with regards to the TPR. The TPR would have been
responsible for project monitoring (as opposed to CEP monitoring which was the focus of the RSC
and RAG), including approving progress reports, approving annual workplans, approving key
appointments and key ToR. One role of the TPR would have been to approve major revisions to the
project budget. No evidence was seen of this process. Notably, the process to prepare and process the
project`s annual workplan is unclear to the Evaluators.
The PCU consisted of an appropriate number of effective professional staff whose main role was
providing technical support and backstopping in their respective domains. The staff seemed to have
fulfilled this technical role. The role the PCU members played in project management and monitoring is
less clear. The role they played in developing annual workplans for their domain, in recommending new
strategic project approaches, or in developing indicators and monitoring progress is not clear. The PCU
members seemed to have mostly focussed on individual activities, rather than on the overall process in
their domain.
The project implementation review (PIR) is the key monitoring tool for GEF and if applied in the right
way it is an excellent learning and adaptive management tool for a project. The use of the PIR should be
consistent and any changes made in the indicators and other elements of the PIR should be well
documented. The CEP-SAP`s use of the PIR was inconsistent presumably in response to well-
intentioned requests from GEF or UNDP - and inadequately documented. For example, in 2005, the
CEP-SAP PIR attempted to create indicators at the Objective level by using (rephrased) Outcomes as
indicators. At the Outcome level, the PIR 2005 used the activities from the Project Document as
indicators. No use was made of the indicators in the Project document. Hence, indicators were very
confused in the 2005 PIR. Then, all changed the following year. The PIR 2006 did not attempt to have
Objective level indicators. At the Outcome level, the 2006 PIR returned to the Project Document for its
indicators although some indicators disappeared and it is not clear why. Finally, the PIR 2006
included an important Annex that was not included in the PIR 2005. PIR 2007 used the same format
and contents as PIR 2006.
In general, the indicators in the PIR focus on completing a task or finalising a product. They do not
adequately attempt to capture either the quality or the impact of the activity/product. Given that most
activities/products were completed, but impact was limited, this led to a certain degree of rating
inflation in the PIR. For example, in the PIR 2006, most ratings were Highly Satisfactory, and the
lowest rating was Satisfactory.
In terms of process, the PIR was mostly completed by the PCU and UNDP/GEF. Experience from other
projects shows that, in order to be an effective monitoring and lesson-learning tool, consultation with
key stakeholders (e.g. NFP, SAPICs, PCU staff) on the PIR is useful. The PIR 2006 was discussed by
members of the RSC in December 2006 - several months after it was finalised and due to be submitted
to GEF.
In effect, the PIR was considered a reporting` tool rather than a monitoring tool. There does not seem
to have been an effective annual review or monitoring mechanism. Such a mechanism would have
identified project weaknesses and indicated ways to address them.
A mid-term evaluation was planned initially but this was not undertaken. This seems to have been a
good decision - given that the project was only due to last three years, and full-blown evaluations take a
lot of time and effort. However, the Evaluators saw no documentation of this decision. Moreover, some
kind of formal, facilitated mid-term reflection would have been useful. This would have identified
project weaknesses and pointed to solutions. In the absence of this and an annual review mechanism,
the project was unable to correct problems.
39
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The project`s monitoring of activities and financial monitoring was fully adequate. Notably, the GEF
quarterly reports were produced as required.
The end-of-project evaluation was planned in a timely manner and designed to cover the main issues. It
received the support of all key project stakeholders. Several limitations (see Section 1.5) meant the
evaluation could not be an effective lesson-learning process.
The use of indicators for project monitoring and management appears unclear. There was no common
set of indicators used through the project, and almost none of the indicators are sufficiently SMART.
The effectiveness of the project`s M&E work was hampered significantly by the absence of SMART
indicators to guide project implementation and to serve as an objective basis for ongoing M&E
discussions throughout project implementation. No attempt was made during implementation to address
this - experience from other projects shows that, once a project is underway, it is often necessary to
rationalise the logframe or to develop a clear set of indicators23..
Finally, there was no clear allocation of M&E roles. No PCU staff member was given the task of M&E.
Experience from other projects shows the most effective way to address M&E is to have one member of
the project team be clearly responsible for monitoring, and be given adequate resources to do this. This
need not be a full-time position.
To summarise, at the outset the project`s M&E framework was inadequate, it was under-resourced, and
few attempts were made to improve it during implementation. Experience from other projects shows
that if the project sponsors do not provide incentives, the project team is unlikely to take the initiative
and revise its M&E framework. In this sense, this project lies somewhere between old generation and
new generation GEF projects.
Accordingly, the procedures for adaptive management24 in the CEP-SAP Project were unclear. No
evidence was seen of the team reflecting strategically upon: overall aim, successes, challenges,
opportunities and then making management decisions. Indicators were not used to support management
decisions. However, it is noted that there were managerial decisions and changes for example the
decision to undertake the survey on algal blooms was taken in response to external changes, and there
were several important budget revisions (as seen in Annex 7) in response to opportunities and needs.
Given the significant shortcomings in the Project`s M&E system, it is rated as Moderately
Unsatisfactory.
4.3 M&E Funding
It is difficult to ascertain the amount of project funds allocated to M&E, given that many activities (e.g.
RSC meetings, RAG meetings) served several purposes, including M&E. The most costly M&E item
was the end-of project evaluation. Overall costs are likely to be in the range 2-3%. This is below the
recommended 5-10% for GEF projects. Hence, funding of the Project M&E system did have significant
shortcomings and accordingly it is rated as Moderately Unsatisfactory.
4.4 Long Term Monitoring
23 It is noted that many GEF project have undergone such a revision after start-up.
24 Adaptive management can be described as the ability to constantly update project approaches in response to
changes, problems, opportunities, successes and failures. Hence, with adaptive management, the Project
Objective, Outputs, Activities and Budget may change. The factors leading to change may be positive or negative.
The factors leading to change may be internal to the project (e.g. training was undertaken as planned but the
capacity was not built) or external (e.g. the overall budget provided by a external co-financer was increased).
40
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The establishment of a long-term mechanism to monitor process, stress reduction and environmental
status is important in the Caspian region. At the Project outset, the lack of data and lack of cooperation
protocols made this infeasible.
The Project has collected data from a range of sources and has become possibly the most
comprehensive environmental database for the region. This database tends to be trustworthy. The
database is accessible through the project website. UNDP/GEF is considered one of the only
stakeholders able to prepare such a regional, trusted database. This database should be considered a
success and major result of the project.
This is a good start and a good basis for the future. There are some weaknesses with this database. First,
it is not fully sustainable. It is not clear where the information will be stored after the project. The
countries are not doing a lot of monitoring. There is no ongoing, systematic process to collect, store and
share updated information on a regular basis. Second, the database is not widely used as of yet, notably
by the participating countries, and this is partly due to the weak internet accessibility in parts of the
region. Finally, the data storage is not adequately structured it is not always easy to find information
on the site.
5. Processes that Affected Project Results
5.1 Preparation and readiness
A) Were the project's objectives and components clear, practicable and feasible within its
timeframe?
Clarity and preparedness were provided by the TDA-SAP, which was developed during the first phase
project, as well as the continued involvement of most key stakeholders from the first phase. This
provided a solid basis for implementation. As a result, the Project`s outputs and strategies remained
largely relevant, even though the Project`s conceptual clarity was assumed` and inherited` from the
first phase, rather than given through the project document itself.
Although the project team appear clear on the Project objectives and approach, in the Project
Document, there are some problems with the clarity of the Project`s objectives and components.
1) Clarity:
The project design leaves room for doubt as to whether the Project was designed simply to coordinate
and facilitate cooperation, or whether it was expected to act on-the-ground and/or catalyze action. This
greatly influences how the project would be implemented and also evaluated.
Considered separately from the TDA-SAP, the Project Document does not clearly evince an overall
strategy, which is directly related to the lack of clarity in the Project`s logical framework (logframe).
Consider as an example, the inconsistency in the use of the terms Outcome and Output in the
logframe. Many of the stated Outcomes are actually Outputs in their wording. For example, in the main
body of the project document there are four Objectives and nine Outcomes (A through I); and in the
Annex B (Log frame) there are four Objectives and nine Outputs, A-I. This hints at some confusion
during project design. Outcomes are best considered in terms of changed situations people/institutions
doing something differently. Outputs are things needed to help generate the desired outcome.
The Project`s logframe does not serve its primary purpose for both project implementers and Evaluators
to bring clarity and specificity to the document and facilitate objective monitoring and evaluation of
41
CEP-SAP Final Evaluation
Final Draft
30/04/2010
project implementation. In recent years GEF and UNDP`s thinking on the definition and use of
indicators has changed significantly. Indicators are much more robust and refined in the new`
generation of GEF projects. Good Indicators of Success force the project designers to be more specific
in their language and in their design, by forcing answers to important questions such as: What does it
mean to strengthen capacity, or to begin implementation of the SAP, or to have tangible
environmental benefits for Matched Small Grants?
A weak logical framework has important ramifications for project preparation and readiness. The
confusion in a log frame ripples down through the design of the project. If an outcome is really an
output, then the project team tends to focus on checking off the output boxes (e.g. holding the training
exercise and producing the report) rather than focusing on generating impact (the trained people
change institutional behavior or the report influences action, planning or budgets). The project
document is written in such a way that caused the project team to focus on the production of things
rather than on how to ensure that stakeholders are actually utilizing the Project`s work to modify and
improve their practices (results).
2) Practicability and feasibility of the project
Overall, the practicability and feasibility of most of the Project`s Objectives was high. But given the
difficulties associated with the project`s logframe, the Evaluators` assessment of practicability and
feasibility is more subjective than desired. Consider as an example, Objective #1: To Commence
Implementation of the SAP in three priority areas... On its face, this objective is entirely practicable
and feasible. But a discussion of it`s practicability and feasibility is hampered by uncertainty over what
is intended specifically by the word commence in terms of who, when, what and where.
The most practicable and feasible elements of the Project relate to the regional-level work catalyzed and
managed by the PCU and its main partners. The Project recognized the high level of support required to
facilitate this regional level work. Indeed, one half of the budget goes towards the Project Coordination
Unit and procuring international expertise from individuals and organizations. Annex 9 illustrates this
point. It also sheds light upon the main strengths and weaknesses in the project`s strategic approach and
its practicability and feasibility:
1) The Project tried to do too many different things. Hence, the energy and efforts were dispersed across
too many issues (mnemiopsis, seals, PoPs, biodiversity, websites, matched small and micro grants,
public participation and education, NGO strengthening, maps). Possibly, this resulted in few of these
being done thoroughly.
2) There is a supply driven nature to the Project. The Project document seems to justify a long list of
interventions that the IW focal area can support. Most GEF Project Documents analyse environmental
problems, threats, root causes and then determine strategic actions. For this project, the Project
Document provided a long list of activities, products and inputs, while not clearly linking these with the
stakeholder needs. This is likely due to the fact that it is a second stage project, and the activities drew
closely from the TDA-SAP. However this should not preclude the need for a self-contained analysis
and justification in the Project Document.
This may have led to some project studies being ill timed. As one stakeholder commented, Studies
need to be at the right time and for the right purpose. For example, the Project supported the IMO in
conducting a ballast water study for the Caspian in order to shed light on the invasive species problem.
Russian stakeholders informed the Evaluator that, while they considered it important work and were
carefully reviewing it, they had some serious concerns with the report and whether the relevant marine
conventions applied in this circumstance.
This situation illustrates the complexity of working in the region and how a technical aspect of the
Project can be caught up in an un-resolved regional political question about whether the Caspian is a
lake or a sea.
42
CEP-SAP Final Evaluation
Final Draft
30/04/2010
3) The project design included some laudable good intentions such as strengthening NGO and civil
society capacity. However, the idea to focus on NGO and civil society created a lot of un-focussed work
for the Project with little return in terms of results. The Project struggled to find effective ways to work
with NGOs in the Caspian region or of achieving the goal of enhanced transparency and stakeholder
participation, and they may require resources well beyond those available to this project.
4) The Project emphasized regional coordination and communication, mapping, field surveys, and the
drafting of strategic plans and programs, and so de-emphasized helping countries achieve impact with
specific measurable activities. The understanding was that countries would take this on, however,
because the countries were not able to co-fund national-level work to the level envisioned in the Project
document, the feasibility and practicability of this approach suffered.
B) Were the capacities of executing institutions and counterparts properly considered when the
project was designed?
Overall, the capacities of executing institutions and counterparts were considered properly during
project design. A review of the Project document and other supporting materials shows that many
counterpart capacities were known to the project team based upon the first phase of GEF involvement
and support for the CEP. This is impressive in itself, given the size of the region and the number of
countries involved.
For example, a wise decision during project design was to keep the SAPIC as a project-funded position.
This gave the Project much needed stability and continuity in most countries where there is an
extremely high rate of staff turnover in Government Ministries.
There were some exceptions, however, which are worth noting for purposes of lessons learned.
For example, the regional Programme Coordinator (PC) position was to be funded by the host
government, and indeed it was. The Government of IR-Iran appointed a qualified, senior level person to
fill the position. However, the Ministry was unable to provide this person with any additional budget for
travel within the region an essential requirement. This meant that the position was unable to fulfill its
role in the Project.
In another example, the Regional Advisory Groups were originally supposed to be supported by the
countries. Although this was agreed to prior to the Project, the commitment and funding has not yet
materialized sufficiently, showing that the capacity and funding limitations of the host institutions in
each country was not fully understood during project development.
C) Were counterpart resources (funding, staff, and facilities), enabling legislation, and adequate
project management arrangements in place at project entry?
In some important areas, the answer to this question is yes and the Project deserves much credit for
this. Staff and facilities and adequate project management arrangements were in place at project entry.
The National Focal Points and SAPICs were identified and adequate project management arrangements
(office space, staff) were in place. The Government of Iran provided excellent PCU office facilities,
enabling the PCU to make the transition from Baku to Tehran.
The enabling legislation was sufficient for the scope of the Project, although as the Project continued to
do its survey and field work, stakeholders identified additional enabling legislation needs. A focus on
enabling national legislation is a key next step following the ratification of the Convention and it should
be a priority of any possible future investments in the Caspian region. Legislation has changed over the
course of the Project`s lifespan and stakeholders, in their work with the Project these past nearly four
43
CEP-SAP Final Evaluation
Final Draft
30/04/2010
years, have gained important insights into the strengths and shortcomings of their own national
institutions and policies. A clear assessment of these and the key pressure points where international
investment could be targeted most effectively would strengthen the baseline for any future investment.
D) Were lessons from other relevant projects properly incorporated in the project design?
Lessons from other relevant projects and processes were incorporated into the project design. The
Project document incorporated knowledge and experience from the Black Sea region and from other
transboundary waters initiatives around the world. For example, the project design, based on lessons
learnt in a GEF supported Black Sea IW project, ensured that the countries would be more directly
engaged in developing protocols for the Convention and in developing national action plans for
implementing the SAP.
Table 19: Showing Examples of Lessons Learnt from International Experience
Other relevant experience
Lesson incorporated into project's work
Espo Convention:
Don`t impose protocols on the Parties to a Convention;
rather work collaboratively with the Parties to develop them.
Convention on the Protection of the Rhine
Transboundary conventions take time to negotiate. Allow
sufficient time to do this and focus on building trust among
the negotiators.
Black Sea Environmental
To the extent possible, combine development of the
Program/Bucharest Convention
protocols with the Convention process, as there may be
difficulty merging the two if done separately.
E) Were the partnership arrangements properly identified and the roles and responsibilities
negotiated prior to project approval?
The partnership arrangements and roles and responsibilities were negotiated prior to project approval in
that most of them carried over from the first phase. Partnership arrangements among UNDP, WB and
UNEP were particularly well worked out and this is a noteworthy aspect to the Project. The
collaboration between UNDP and UNEP in this phase has been exemplary and this is due largely to the
personalities involved and their abilities to focus on results and win-win aspects of their collaboration.
Other roles and responsibilities changed significantly after the first phase. For example, GEF funding
for the CRTC was terminated and co-funded RAGs put in their place. Although this change is clearly
identified in the Project document and was based on long negotiations with the country partners,
evidently the countries were not able to provide the inputs as required. This transformation from GEF-
supported thematic centers to co-financed advisory groups was a difficult one for the individual national
experts involved.
Feedback from stakeholders tells the Evaluators that the effectiveness of the RAGs suffered because
RAG work was above and beyond the normal workload for national experts and as a result was not a
first priority in many cases. This change meant the SAPICs had no management leverage over the RAG
members because they were in essence volunteered by their own organizations. In the words of one
SAPIC, I could not push them to deliver. All I could do was ask them. Finally, this also led to a high
level of turnover among the RAG members.
44
CEP-SAP Final Evaluation
Final Draft
30/04/2010
5.2 Country ownership/driven-ness
Overall, country ownership of this project was adequate as evidenced by the rapid ratification of the
Tehran Convention and subsequent steps taken at the first CoP to the Convention.
Engendering country ownership and drivenness was a top priority of the Project. The introduction of the
Project document states that extended national commitment will be evidenced by increased
responsibilities in the littoral countries for programme management activities and lists six indicators25
of increased national commitment. Table 20 lists the indicators and rates their current status.
Table 20: Indicators of National Commitment
#
Indicator
Comment
Status
1 National support for National Coordination
Support for NCS never fully
Not
Structures (NCS) and the PCU
materialized. Minimal national support
achieved
for PCU (except Iran).
2 National support of Steering Committee Meetings
Countries did host SC meetings and
Achieved.
and Activities
other activities, but level of financial
support unclear.
3 National endorsement of NCAPs and Ministerial
No national endorsement of NCAPs;
Partially
agreement of the SAP.
Ministerial agreement on SAP and some
achieved
NCAP.
4 Ratification of the Framework Convention
Signed and ratified.
Achieved.
5 National Support for Caspian Regional Thematic
No national support in terms of
Not
Centres (CRTCs) and support to the Regional
financing to CRTCs or RAGs
achieved.
Advisory Groups (RAG).
6 Encouragement of stakeholder participation in
The record points more to
Partially
project activities.
discouragement rather than
achieved.
encouragement.
The Evaluators acknowledge that the indicators are demanding, and not many GEF projects in the
world could have truly delivered these. Never-the-less, they do provide a useful basis for discussion and
analysis of country ownership and driven-ness. Overall, two of the six indicators of country ownership
were achieved (33%); two were not achieved (33%) and two were partially achieved (33%). This score
highlights the Project`s mixed record on country ownership and country driven-ness. There are some
positive and constructively critical lessons to be drawn from this showing:
1) Observation #1: There is a strong base of
country ownership of the Project and its goal of
It is very important for us that this project is
maintaining and improving the ecosystem health
working in our country. It helped us and the other
of the Caspian.
countries to be more systematic in our approach to
solving the ecological problems of the Caspian.
The relatively solid foundation of ownership is
reflected in the rapid ratification of the
This would be impossible without this project
because there would be no mechanism. The project
Convention by all five countries and the
gives us the possibility to improve work in all
relatively swift development of draft protocols
countries.
to the Convention (Indicator #4 above). It is
reflected by the delegates at the first COP to the
Stakeholder comment
Convention and their pledge to fund a fledgling
Secretariat. It is reflected in the actions taken by countries in response to project supported work and
recommendations. It is reflected in the comments of stakeholders to the Evaluators, many of whom
praised this project as being extremely productive and helpful.
25 See page 5 of Project Document.
45
CEP-SAP Final Evaluation
Final Draft
30/04/2010
It is also reflected in the Project`s work with the NCAPs. To be sure, the NCAP`s impact and level of
governmental support varies among the five Caspian countries, but overall, the process of developing
the NCAPs engendered a new sense of
Caspian-ness in each of the countries (or in
Russia`s case, the three regions bordering the
Not all Caspian countries are interested to solve
Caspian). The NCAPs helped to overcome the
the Caspian`s legal problems BUT they have to
inertia caused by the complexity and scale of
solve ecological problems these are a shared
priority.
issues facing the region and demystifying just
what the countries could do at the national level
- Stakeholder comment
to help improve the Caspian`s environmental
health. Several stakeholders interviewed
mentioned this.
Other signs of country ownership can be seen with respect to governance reform, budgetary allocations,
and institutional reform. They include:
Governance: NGOs are slowly increasing their level of legitimacy in some of the Caspian countries.
In Kazakhstan, the government is looking to sub-contract reputable NGOs to do work in their areas
of expertise. On the other hand, in Turkmenistan, NGOs are still strongly discouraged;
Budgetary allocations: Each country allocated more to NCAP-related activities than originally
anticipated, but the Project is not yet to the point where it can claim to have catalyzed significant
budgetary allocations by Governments in support of SAP and NCAP;
Institutional reform: Survey work and other field studies are providing new insights and causing
stakeholders to review the apportionment of responsibilities among their own institutions. Russian
stakeholders` work with the Project in measuring and monitoring sediment pollution has caused
them to review their respective institutional responsibilities and clarify them. These discussions
have also highlighted law and policy changes necessary to support longer-term reform. In
Turkmenistan, the Government recently formed an Inter-ministerial Committee responsible for
Caspian issues - although this is not yet operational.
How did the Project generate this ownership? A more thorough analysis of this would be needed to
answer this question appropriately. However, some contributing factors may be:
The ecological health of the Caspian Sea is an important priority for all five countries and this
project was able to tap into that support.
The thorough and tireless work of the PCU, UNDP/UNOPS and UNEP;
The strong network, built up over the years, extending into each country`s experts and
administrators;
The relative novelty for some countries of participating in international, UN supported
processes;
Cleverly developed collaboration across international partners, driven to some extent by PCU;
The growing prosperity in the region, allowing national governments to take on new issues with
environment increasingly a priority.
2) Observation #2: The fact that four of the six indicators were not achieved or partially achieved
points to some lessons that are worth exploring:
First, perhaps it was unrealistic to expect a high level of country ownership at this stage. The Project`s
focus on commencing implementation of the SAP was slightly ahead of its time. At project inception,
the countries had not yet ratified the Tehran Convention and so it did not carry the force of law. This
hampered stakeholders` efforts to secure official endorsement and financial support for PCU, NCAP
and SAP-related activities. It takes time and only now do the legal obligations inherent in the TC
appear to be having a positive effect on increasing country support for related initiatives.
Second, country ownership remained within the relatively small traditional environmental circle of
stakeholders, and did not encompass new and non-traditional environmental stakeholders, such as
46
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Finance or Foreign Affairs in a substantial way. National ownership should have gone beyond the NFP
and the Ministries of Environment in this second phase.
Third, some stakeholders commented that the process was not driven as much by their needs as by the
needs of the international program. All GEF projects must balance global benefits with national
benefits, ownership and country-drivenness. In this
case, it is possible that the country ownership was
You will really own something that you
partly eroded by the supply-driven nature of many
truly need. If you don`t really need it, sure
activities. Objective 3 is an exception to this.
you will still own it, but you will put it in
the garage and forget about it.
Fourth, the fortunate complementarity of sustainable
development needs and environmental benefits may be
-- Stakeholder comment
a double-edged sword. A sewage system project in
coastal Azerbaijan is justifiable on human health and
economic development grounds and it also reduces pollution flowing into the Caspian, fitting perfectly
into the NCAP and SAP priorities. This is complementarity is responsible in large part for the almost
US$ 100 million in parallel co-funding from the five countries for NCAP-related activities. However,
the down side of this may be that it partially erodes the Project`s justification for existing. What
additional is the Project offering? Some stakeholders expressed the desire for measurable results and
more tangible links to their priorities.
Fifth, the conceptual integrity of the Project itself has become a bit muddled. What is the Project
versus the regional strategy, the national action plan, other co-funded activities and the umbrella
Caspian Environment Programme, for which the PCU is also the main coordinating unit? These blurred
boundaries quickly become confusing for people, especially when one adds the EQO of the SAP and
the Process Outcomes and Indicators of the IW Results Template to the Project`s logical framework and
workplans. Indeed, it was somewhat confusing for the Evaluators initially. This no doubt has
contributed to a feeling of fogginess and lack of control on the part of some stakeholders.
Sixth, national sensitivities are important factor underlying country ownership and the Project did a
good job of trying to balance these. The PCU also did a good job of trying to have all five countries
represented among the PCU staff. In hindsight however, a regional project such as this may want to
institute a policy requiring that the Manager of the PCU not be a national from the country in which the
PCU is located.
And finally, GEF project documents not written in a way that is not easily accessible to stakeholders in
the region for whom English is a 3rd or a 4th language. Many stakeholders are under the impression that
the GEF is really not interested in country and stakeholder buy-in because few people can understand
their documents! Perhaps in the future some kind of summary document of 3-4 pages could be written
and translated into all national languages.
5.3 Stakeholder involvement
a) Did the project involve relevant stakeholders through information-sharing, consultation and by
seeking their participation in the project's design, implementation, monitoring and evaluation?
The Project`s website is an important mechanism to share information with the widest range of
stakeholders possible. It is among the most extensive project website these Evaluators have ever
encountered. On it can be found nearly all project-generated documents. Indeed, a web-based
mechanism for sharing information across this huge five-country Caspian region does make sense.
However, the website is visited primarily by international stakeholders. Some thought should be given
as to why more local stakeholders are not visiting it and what if anything the Project could do to
increase visitation and use of the website from the five country region.
47
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The Project did involve many relevant stakeholders and sought to do even more than it has been able to
do. Apart from working closely with the Ministries of Environment from all five countries, perhaps the
Project`s biggest stakeholder success was with the private sector in the form of its partnership with the
international oil companies and consortia operating in the Caspian Sea. This was quite productive and
useful for both sides and there are discussions ongoing to secure longer-term cooperation with this
group of stakeholders. This should ultimately strengthen the long-term sustainability of regional
environmental work in the Caspian. Beyond these big international companies, private sector`
involvement was limited and the Project had a more difficult time engaging the national-level and local-
level private sectors in each country.
The Project did work diligently to implement public awareness activities and outreach campaign. It is
difficult for the Evaluators to asses the effectiveness of this work as there were no target groups or
impact indicators used by the Project to measure success. Overall, the Project`s awareness and
education work suffered from the lack of a strategic focus. The first phase GEF CEP project
commissioned a Caspian Regional Stakeholder Analysis in 2001. But the Project document`s
education and awareness activities do not seem to have benefited from this analysis and the same can be
said regarding the 2004 stakeholder analysis. Again, the emphasis was placed on outputs rather than
outcomes: outreach materials were prepared for school children, but were they used and did they
increase knowledge?
The Project sought to consult with and involve NGOs and civil society, but struggled to find the most
effective way to do it in this Caspian region, where NGOs and civil society are relatively new and
somewhat controversial concepts. In hindsight, the Project`s broad emphasis on strengthening NGOs,
was probably a distraction and may have benefited from a more realistic, targeted approach to this
important, yet difficult issue. There were no local or regional NGOs involved in this final evaluation.
A related important challenge was to involve representatives of influential sectors and sectors using the
Caspian resources such as fishery management agencies, fisher associations, water management
agencies, transport agencies and financial agencies. The role of these agencies is crucial. The
Evaluation was able to gather little evidence as to the active involvement of these stakeholders. This is
possibly an important gap to be filled.
At the regional level notably missing from the Project`s list of actively involved stakeholders is the
Aquatic Bioresources Commission of the Caspian Sea (ABC). The PCU manager attempted to meet
with the ABC and/or attend one of their meetings on multiple occasions, but to no avail. Their role in
the regional resource management framework is not described in the Project document baseline.
b) Did the project consult and make use of the skills and knowledge of the appropriate
government entities, NGOs, community groups, private sector, local governments and academic
institutions in the design, implementation and evaluation of project activities?
At the international level, the Project excelled at consulting with and involving knowledgeable
stakeholders from a broad spectrum of expert agencies, from UNEP to the IAEA to the FAO to the
World Bank. There was good coordination among UNDP, UNEP, EU-TACIS and the World Bank.
This is rare.
At the regional and national levels, the Project worked closely with government entities, academic
institutions and private sector companies in the design and implementation of project activities. In some
cases, this work was made much more difficult by high turnover of individuals in government positions.
For example, in Kazakhstan, there were at least three different Ministers of Environment during the life
of the Project.
The project did an exemplary job of involving some of the region`s most prominent oil companies,
including Total, BP and KazMunaiGaz. As a result, the Project benefited from the perspectives,
48
CEP-SAP Final Evaluation
Final Draft
30/04/2010
information, logistical support, and other resources that these stakeholders were able to provide.
The Project had difficulty consulting local communities, community groups, and NGOs from the region
during implementation. The Project did hold a meeting of mayors from the Caspian region in mid 2007,
but this was too late to be of much use in informing project implementation. Originally, the Project
planned NGO networks and local community networks, but implementing these proved to be more
controversial and difficult than originally anticipated.
5.4 Financial Planning
The Project exercised due diligence and followed best UNOPS practice in the management of funds and
the auditing of financial records. The Project has appropriate financial controls, including reporting and
annual financial planning. The PCU developed annual workplans and corresponding budgets, which
were reviewed by the RSC and the UNOPS/UNDP and approved in advance.
For the most part, this allowed management to make informed decisions regarding the budget and
allowed for timely flow of funds. Efficiency was slightly hampered at times by several factors related to
project institutional structure and standardised budgeting for the expert fee levels across all five
countries.
The rapid change in the economies of Kazakhstan, Azerbaijan and Russia during the life of this project
made some of the Project`s financial planning assumptions invalid or obsolete. For example, in
Kazakhstan, private sector wages for qualified people climbed rapidly and the Project`s pay structure
did not keep pace, hindering the Project`s ability to recruit top experts. Financial planning was also
complicated somewhat by fluctuations in the value of US Dollar. The Euro US$ exchange rate
fluctuated between 1:1.17 and 1:1.36 during the Project lifetime. Given that the project budget was in
US$, but many project expenses were linked to Euros, the overall decline in the value of the US$ led to
complications.
Some of the Project`s work presented real challenges in disbursing funds in a timely manner and
overseeing their use. The MSG program seems to have presented the biggest challenge in this regard,
with most grantees being located in the more remote areas of the five countries. The establishment of an
imprest account by UNOPS, UNDP and the PCU facilitated project implementation.
5.5 Implementing and Executing Agency (IA/EA) Supervision and Backstopping
a) Did Implementing/Executing Agency staff provide quality support and advice to the project,
approved modifications in time and restructure the project when needed?
Implementing Agency Support: Implementing Agency (UNDP) Staff were regularly involved in
providing support and advice to the Project. The standard UNDP supervision procedures were followed,
as discussed in the M&E section of this evaluation.
All told, IA, EA and project staff (including international experts) met frequently and were in email
communication regularly. This support was provided at the annual project steering committee meeting,
it was provided at an annual meeting in Brussels with UNEP and EU partners, and it was provided at
the annual global IW lessons learned workshop. It is difficult to judge the quality of this advice and
support, apart from saying that this project appears had extensive support and advice right on through
its implementation period.
Indeed, with the close cooperation between UNDP and UNEP, and UNEP`s active involvement in and
support of the Convention and protocol development process, the Project benefited from the support of
two IAs.
49
CEP-SAP Final Evaluation
Final Draft
30/04/2010
One shortcoming the Evaluators identified in IA project support was that UNDP never improved the
Project`s M&E and adaptive management approach, and in so doing fine-tune the logframe and bolster
the project`s effectiveness and efficiency. To be sure, the Project was implemented as approved by
GEF, but it is increasingly common for projects to update their logframe indicators post-approval in the
first year or year and a half of project implementation.
Executing Agency (UNOPS) support. Overall, based on the evidence reviewed, UNOPS execution of
the Project was highly satisfactory. This is an impressive accomplishment given the logistical
challenges inherent in this 5-country region. There is a productive rapport between the PCU and
UNOPS and it seems both sides have strived to minimize administrative delays through the use of tools
such as imprest accounts. Of course there were some problems. For example, there were delays in
disbursing financing to MSG projects and other activities. Perhaps these experiences will be
incorporated by OPS into their own internal lessons learned processes.
b) Was the right staffing level, continuity and skill mix provided for the project?
The Project`s staffing level, continuity and skill mix was highly satisfactory. The Project staff
interviewed by the Evaluators were professional and committed. The same PCU manager remained on
the job throughout the Project, and there were few PCU staff changes. This excellent level of continuity
was a big advantage for this project. There were some difficulties in finding and keeping good staff in
the field however. Overall, the effective management of the PCU is an impressive accomplishment
given the complexity of the Project.
5.6 Co-financing
There are two categories of co-funders: a) the Governments, and; b) Other comprised of multi-lateral
organizations, NGOs, and the private sector. There are two types of co-funding discussed: parallel and
leveraged.
Parallel co-funding is co-funding that is directly related to the Project`s goals and objectives but was
not leveraged by project activities.
Leveraged co-funding is funding that is directly related to the Project`s goals and objectives and that the
Project`s work actually catalyzed. This funding would not have been spent on the relevant activities had
it not been for the Project`s efforts.
a) Level of actual co-funding versus planned co-funding and reasons for changes:
The details of co-financing are provided in Annex 8 and summarised in Table 21 below.
Overall, actual co-funding levels for this project are impressive: final co-funding was 422% of the
planned levels.
The additional parallel co-funding for implementation of NCAP-relevant activities in each country was
more than five times the figure anticipated in the Project document, increasing from an estimated US$
19 million to over US$ 96 million.
Overall, co-funding leveraged from other sources by the Project totalled nearly 3 times the planned
amount, increasing from US$ 4.6 million to over US$ 13 million. This is a particularly impressive
achievement for the Project and the CEP. The increased figures reflect the Project`s success in bringing
on board separate work from EC and TACIS so that it is a well coordinated and somewhat integrated
part of the work supported by the GEF. In addition, the Project has been able to draw in the oil
50
CEP-SAP Final Evaluation
Final Draft
30/04/2010
companies in meaningful collaboration and co-financing partnerships. Indeed, the Project hopes to sign
an agreement with BP to co-finance future monitoring work.
Table 21: Co-funding Summary
Type of Co-funder
Planned Co-
Actual Co-funding
Difference
funding (US$'000)
(US$'000)
(% change)
1. Other (bilaterals, multi-laterals,
4,660
13,200
+ 283%
private sector) leveraged co-funding.
2. Governments -
19,200
96,770
+ 519%
Parallel funding for NCAP-related
activities.
3. Governments leveraged co-funding
2,300
350
- 85%
of project implementation.
Total (US$'000)
26,160
110,320
+ 421%
Not all co-funding news was positive, however. Some of the leveraged co-funding most relevant to the
Project`s financial planning did not materialize as hoped. Only 15% of the planned co-funding from
Governments actually materialised for project activities in support of the PCU, of regional coordination,
regional advisory groups, and joint regional activities (see row 3 in Table 21). The Project was overly
optimistic in its original estimations of this direct co-funding. At project inception, the TC had not yet
been ratified and so was not a legal entity. This fact, combined with the novelty to the five countries of
donating money directly to a regional initiative hampered the realization of this co-funding.
Reasons for increased levels of co-funding:
The reasons for the variance between planned and actual co-funding are the following:
First, regarding co-funding from other sources, the project team and its partners were able to leverage
more funding for follow-on activities (e.g the Seal Conservation work funded by the Darwin Fund),
resulting in almost 3 times the planned amount in co-funding.
Second, regarding the increased level of co-funding from Governments for SAP-related work, this 5-
fold increase can be attributed in part to increased tax revenues generated by the boom in the oil-based
economies around the Caspian Sea. Some of that increased revenue was allocated to wastewater
treatment, sustainable agriculture, fisheries and other programs being implemented in the Caspian
watersheds of each of the five countries. These relevant investments are counted as parallel co-funding
for SAP and NCAP implementation. The Project certainly played a role in leveraging, or coordinating,
some of the $97 million in related investments an impressive accomplishment. However, the Project
is not responsible for leveraging most of the $97 million. This distinction is important for purposes of
this co-financing discussion. However the bottom line is that these investments are being made, they do
benefit the ecological health of the Caspian Sea.
Reasons provided by the project for decreased level of country co-funding for the PCU and project:
The original estimate was overly optimistic.
Countries perceived CEP to lack a formal legal basis.` This made it difficult for the Project
Directors in each Country to convince their Finance Ministries to contribute to the project.
Countries did not agree on the hosting arrangements for the Regional Advisory groups (RAGs)
fearing that any agreement might have implications for the hosting arrangements of the Convention.
RAGs hosting would have required the countries to pay for their operations.
National environmental agencies are still amongst the weakest and poorest of the national
institutions.
Environment is still not amongst the top priority issues in the Caspian countries and as such it is
difficult for the national environmental agencies to obtain funding for environmental operations.
51
CEP-SAP Final Evaluation
Final Draft
30/04/2010
The Evaluators feel that these were all known or considered at the time of project development and
approval. In fact, the Project document states that the countries committed themselves to the financial
sustainability of the CEP structure at the March 2003 extraordinary Steering Committee Meeting and
that this support was a pre-condition of the project.
b) Did the extent of materialization of co-financing affect the project's outcomes and/or
sustainability?
Increased co-funding: It is difficult to assess whether the additional co-funding described above affected
the sustainability or outcomes of the Project. The Evaluators` opinion is that the additional co-funding
had little effect on the Project`s sustainability, but did affect some Outcomes positively. For example,
the additional co-financing from international partners affected the Project`s outcomes work under
Immediate Objectives 1-3, especially in terms of the Project`s work on the Tehran Convention, the TC
protocols, biodiversity (such as the Caspian Seals), and pollution monitoring.
The dramatic increase in parallel co-funding for implementation of NCAP-relevant activities in each
country certainly shines a positive light on the Project`s Immediate Objective #1: To commence
implementation of the SAP in priority areas. However, a recent Project report26 concluded that the
specific link between the SAP and the national investments is difficult to ascertain, as the national
funding is not clearly linked to the SAP indicators. But it is reasonable to link the increased level of
funding at the national level for SAP-relevant activities to improved prospects for sustainability of
SAP-relevant interventions at the national level. Of course it is only really sustainable as long as there is
an institution or mechanism that maintains and updates a strategic vision like that of the SAP process.
The sustainability of the SAP vision and process has not yet been achieved.
Decreased co-funding: To date, only 15% of the US$2.3 million in co-funding from countries for
project implementation has materialized. One specific area where planned co-funding never
materialized was with respect to the Programme Coordinator (PC). This position was nominated by the
RSC. Its main job is to coordinate contacts with the NFP and monitor SAP implementation across the
five countries. The Government of IR-Iran did appoint a senior-level competent individual to fill this
position, but was unable to provide this person with sufficient travel funds, a pre-requisite for the
position to function adequately. As it turns out countries have rules and regulations that prevent them
from supporting an international position. In the case of the PC position, the reduced delivery of co-
funding for this position may have affected the production of the NCAPs and as well as the networking
of the National Focal Points, which could have strengthened the level of collaboration among them.
In most cases, it is doubtful whether the reduced delivery of co-funding from the countries to the PCU
affected the project`s outcomes appreciably as the project was able to fill in the gaps. But it did lead to
some lost opportunities with respect to national advisory group work and enhanced regional
coordination. There is also a possibility that the decreased government funding affected the ability of
the SAPIC, the effectiveness of national coordination, and the powers and commitment of the RAG
members. However, given legal obligations that accompany the ratification of the TC and the
commitment made at the COP-1 to fund the Secretariat, this reduced delivery should not affect the long-
term sustainability of the regional mechanism.
5.7 Delays
There have been no major delays in project implementation and completion. The project was
implemented smoothly and rather quickly, within four years. There was a smooth transition of the PCU
from Baku to Tehran for which the PCU manager and UNOPS should be commended. All three
agencies should be commended for the smooth transition from the first phase in which there were three
IA (WB/UNEP/UNDP) to the second phase in which there is one (UNDP).
26 Lewis, T. April 2007. Caspian Strategic Action Program Implementation: A Regional Review and
Assessment. Caspian Environment Programme. Unpublished.
52
CEP-SAP Final Evaluation
Final Draft
30/04/2010
6. Lessons, Recommendations and Examples of Good Practice
6.1 Recommendations and Lessons Pertaining to Future GEF Projects
Lessons and recommendations for those involved in the design and appraisal of future projects
1. It is of paramount importance to have a clear logical framework. This will include:
A clear overall objective and project goal;
A manageable number of SMART indicators, at least at the objective and outcome levels;
A sharp focus on impacts/outcomes or results. This should be in terms of changed practices,
changed behaviour and/or changed situations;
Clarity and simplicity so all stakeholders can understand the project`s goals and aims.
2. Strengthening NGOs and civil society is a major challenge requiring significant resources. It can
also be an overly vague goal in the absence of specific targets. If projects of this nature are to
attempt civil society strengthening, they must have very clear and realistic targets. This should
not be an add-on or a secondary aim.
3. Awareness raising is an important aspect of many projects' work. A challenging, resource-
intensive task, it is not an objective that can be added-on easily. It is important to clarify the
specific targets in terms of stakeholder groups and improved knowledge or changed behaviour,
not in terms of documents produced. It is also important to ensure the right expertise is mobilised.
4. Project design should be adapted to the dynamic environment where government staff turnover is
high. Indeed, this project may have some good practices to share in this regard.
5. GEF regional projects cannot make major region-wide investments, not even in data collection.
They can only be catalytic. Hence, do not expect regional projects to directly undertake
significant groundwork, but expect them to influence, catalyse and leverage.
6. Small grants programmes should be designed to clearly contribute to overall project objectives
and the question should be asked, "Are small grants projects are the most efficient way to achieve
objectives?"
7. Individual small grant projects require significant managerial support, technical support and
oversight. They also require clear objectives and mini logical frameworks. Ensure this is all
included in the project design.
8. Inter-governmental coordination and cooperation and convention processes are complex.
Expectations should be appropriate. Project should not be burdened with too many ambitious
tasks. They should focus on clear and catalytic roles.
9. To achieve sustainability of the project-inspired outcomes, the project design needs to include a
focus on sustainability. Care and attention should be given to defining just what sustainability
means and what specific activities and functions are targeting sustainability.
Recommendations to those responsible for project monitoring and evaluation
10. Projects should upgrade a logical framework that is not working well. This is best done at
inception or in the first year, through a well-documented, transparent process.
53
CEP-SAP Final Evaluation
Final Draft
30/04/2010
11. Ensure there are regular (annual and mid-term) learning and reflection management exercises
for the project staff and the Steering Committee. These reflections should discuss project strengths
and challenges. For example, the PIR process should be adapted into an annual monitoring and
management exercise that links workplans to indicators. The mid-term review should be an integral
part of project learning and planning. Outside facilitation to these would be beneficial.
12. Keep the contents of the PIR constant and/or document any changes clearly.
13. Maintain a documented record of the impact and changes brought about by the project. Projects
like this should, as a matter of priority, document the actions taken by stakeholders in response to
project work or as a result of the project`s work.
Specific recommendations pertaining to `Second Phase' projects
The CEP-SAP constituted a second phase of GEF support to the CEP process. Other second phase
projects in other regions/countries may have similar opportunities and face similar challenges to CEP-
SAP. Notably:
14. Typically, during the first phase, national ownership will have been generated. This must be
considered a starting point, and must be consolidated during the second phase. Steps must be in
place to further build ownership during the second phase with specific targets that define what
"ownership" means and to monitor this consolidation.
15. The balance between continuing the actions of the first phase` and rejuvenating and transforming
the project approach` is an important one. Phase 2 must maintain the gains from Phase 1, but not
simply be a continuation of Phase 1.
16. Take care to avoid a supply-driven nature to the project design and implementation. At the end of
the first phase, the required activities may appear obvious, and the required inputs standing by`.
However, it is necessary to strategize, go back to basic planning, and ensure the activities are
geared towards the needs and objectives, or to addressing the threats and root causes.
Other lessons and recommendations
17. In regional projects it is important to pay attention to specific economic trends and factors in each
country that could affect staff and expert recruitment. In this case, one size does not fit all when it
comes to salary requirements of qualified experts across the five countries.
18. Health and Safety Considerations. One MSG project involved incinerating POPs in a small-scale,
local incinerator. These kinds of on-the-ground funding programs should include consideration of
health and safety issues among their criteria.
6.2 Recommendations and Lessons Pertaining to CEP-SAP Project Stakeholders
Urgent lessons and recommendations, to be addressed preferably during the remainder months
of the CEP-SAP project
For the Project Team
19. The design of any future intervention should be carefully adapted to present context, conditions and
needs. Hold a strategic reflection on how best to utilise available international support in the
future. What are the aims of CEP/emerging TC Secretariat, how to get there, who should do what?
20. Consolidate the documented record of project impact, including:
54
CEP-SAP Final Evaluation
Final Draft
30/04/2010
What changes the project has catalyzed in each country in terms of laws, policies, budgetary
allocations, institutional and individual knowledge, institutional arrangements, and research
and environmental management practices. This will serve as a valuable baseline for any
additional efforts in the region;
The main lessons learned from this project. Utilize sufficient project resources to capture
these. What has worked well and not so well and why;
Encourage each country unit of the project (National Focal Point, SAPIC, RAG leaders) to
document the country-specific policy and institutional barriers (and as a result,
opportunities) to improved environmental management and monitoring. Each country has
learned a great deal about their own institutions and policies as a result of their participation in
the Project. This needs to be considered and captured on paper to enable effective next steps.
Conduct an assessment of the effectiveness of the MSGP. This would give the project a much
better idea of the kinds of tangible impacts the program produced.
21. It is noted that it proved far easier to register large investments into NCAP related activities than to
mobilise small funding to regional cooperation and coordination actions. The reasons for this are
not fully clear, and further analysis is recommended.
22. Look to the ABC for lessons learned (positive and negative) regarding sustainability of a
mechanism for regional cooperation.
For the Participating Country Governments
23. Resolve the location of the TC Secretariat one possibility would be to have a TC Secretariat with
several centres of expertise established for key areas of focus in each of the other countries. The
most obvious and practical place for the Secretariat is Baku, which is located in the middle north-
to-south and the only capital on the Caspian, with air travel connections to each of the other four
countries. PCU can facilitate discussions.
24. There is a likelihood that any future GEF support will include a strong focus on the Fisheries
sector. Fisheries is a major issue in the Caspian and it is one of the most difficult issues. Any
fisheries effort must be driven by the fisheries sector, and must have the full support and backing of
all Caspian countries, particularly Russia, with the largest fishing fleet in the Caspian. Undertake
stakeholder analysis and generate stakeholder buy-in. For example, in order to facilitate
cooperation on fisheries, the Aquatic Bioresources Commission of the Caspian Sea (ABC) has a
clear mandate and is a key stakeholder. Any CEP-SAP action in fisheries should be preceded by a
political effort to link the CEP-SAP with the ABC work programme.
25. Conduct an evaluation of the overall Caspian Environment Programme (CEP). This project is
but one component within the overall CEP. Country governments and partner international
institutions should consider an evaluation of the larger CEP, particularly since the insights and
lessons generated from such an evaluation could be very beneficial to the nascent Tehran
Convention process.
Recommendations to ensure the sustainability of the existing CEP-SAP project outcomes
The CEP-SAP has contributed to several outcomes, some of which are not yet fully sustainable. There
is a possibility that CEP-SAP will benefit from additional support in the future, either from GEF or
other international financers. The following recommendations aim to ensure that the existing benefits
are sustained and sustainable through the future cooperation.
55
CEP-SAP Final Evaluation
Final Draft
30/04/2010
26. It is unrealistic to expect a regional process to reach sustainability in such a short time. The process
has taken important steps forward, but has not yet reached sustainability. International support to
this process is still critical. For example, this would facilitate the determination of the Secretariat
location, mobilize resources to the Secretariat`s mid-term budget and support the process to sign
and ratify protocols. However, a proper exit strategy for international support is essential, with
good indicators and milestones of progress towards sustainability.
27. Be careful to maintain conceptual integrity individual investments/projects, when part of a larger
programme. At times, the lines between the CEP-SAP project and the CEP (programme) seem to
have been blurred. This is helpful and unhelpful: helpful because it means the stakeholders do not
distinguish between CEP-I and CEP-II (CEP-SAP) and see only a continuous programme. It is
unhelpful because it has perhaps contributed to the CEP-SAP losing some project-specific focus.
28. Subsequent to the ratification of the TC, the Convention now has the force of legal obligation in
each country. This is an excellent basis for strengthening sustainability. Future international
support should focus on empowering countries to take responsibility for specific actions in line
with the TC.
29. To increase country driven-ness, future investment should consider giving more thought to
balancing the regional role of international funders with creating strong incentives for robust
national action and national support for transboundary coordination.
6.3 Examples of Best Practice.
1. The project did an excellent job in obtaining ratification of an international treaty by all five
participating countries, and in advancing the related protocols. The exact nature as to how this was
achieved should be documented.
2. The project`s ongoing successful engagement of the region`s oil sector is relatively unique. The
key elements of the approach and how this was done should be documented and disseminated.
3. The key elements that created and sustain the exemplary cooperation among UNDP, UNEP and
UNOPS should be documented and disseminated.
4. The project demonstrated a thorough approach to problem diagnosis and planning, and to
environmental monitoring and measurement, at regional and national levels.
5. Although many project technical activities were completed at a regional level, the project
influenced domestic planning and development processes. This is mostly likely through the NCAP
(prepared by national teams, under the guidance of the regional TDA-SAP) and through the SAPIC
(financed by the project, but well anchored in national institutional frameworks). This was achieved
without the NCAP being fully approved (i.e. no budget allocation).
56
CEP-SAP Final Evaluation
Final Draft
30/04/2010
Annexes:
Annex 1 TOR for the Evaluation
Annex 2 Methodology
Annex 3 Mission Itinerary and List of Persons Interviewed
Annex 4 List of Documentation Reviewed
Annex 5 Summary of Field Visits
Annex 6 Evaluators Ratings of Indicators
Annex 7 Summary Information on Expenditures
Annex 8 Table of Co-Financing
Annex 9 -- Project Implementation Structure Diagram
57