Terminal Evaluation of the UNEP GEF project
"Determination of priority actions for the further elaboration and
implementation of the Strategic Action Programme
for the Mediterranean Sea"
Project Number GF/ME/6030-00-08
Delmar Blasco
Evaluation and Oversight Unit
June 2006
INDEX
Acronyms
and
abbreviations
4
I. INTRODUCTION
Main
goal
and
scope
of
the
project
5
Evaluation
approach
5
II. CONCLUSIONS
Main
general
conclusions
6
Ratings
9
III.
SPECIFIC ASPECTS OF THE PROJECT
Attainment of objectives and planned results
11
Summary level of attainment and general observations
16
Achievement
of
outputs
and
activities 22
Output 1: Updated Transboundary Diagnostic Analysis (TDA) 22
Output 2: Pre-investment studies for selected pollution
hot spots
22
Output 3: Analysis of pollution sensitive areas
23
Output 4: A set of regional guidelines and regional plans
24
Output 5: Capacity building (training courses at regional and
national
levels) 27
Output 6: Regional programme for public participation
28
Output 7: National Action Plans (NAPs) to address pollution
from
land-based
activities
28
Output 8: Economic instruments for the sustainable
implementation of SAP MED
28
Output 9: Strategic Action Plan for the Conservation of
Biological Diversity in the Mediterranean Region
30
Methodologies 31
Cost-effectiveness
31
Financial
planning
32
Impact
33
Sustainability
35
Stakeholder participation / public awareness
39
Country
ownership
/
driveness 41
Implementation approach
41
Replicability
46
Monitoring
and
evaluation
46
2
IV.
FOLLOW-UP
Recommendations
46
Lessons learned
47
Annex 1 Questionnaire addressed to eligible countries
49
Annex 2 Terms of Reference for the Evaluation
58
3
Acronyms and abbreviations
Barcelona Convention
Convention for the protection of the marine environment and the
coastal region of the Mediterranean (Barcelona, 1995)
CC
Coordinating Committee (of the GEF project)
CP/RAC
MAP´s Regional Activity Centre for Cleaner Production
(Barcelona, Spain)
EI Economic
instrument
EQO
Environmental quality objective
GEF Global
Environment
Facility
IASC
Inter-Agency Steering Committee (of this GEF project)
IMC
Inter-Ministerial Committee (for this GEF project in each
country)
LBS Protocol
Protocol for the protection of the Mediterranean Sea against
pollution from land-based sources and activities (under the
Barcelona Convention)
MAP Mediterranean
Action
Plan
MED POL
Programme for the assessment and control of pollution in the
Mediterranean region (within MAP)
NAP
National Action Plan for the protection of the marine
environment from land-based sources (prepared under this GEF
project)
PAP/RAC
Priority Actions Programme (MAP's Regional Activity Centre,
Split, Croatia)
SAP BIO
Strategic Action Programme for the conservation of biological
diversity in the Mediterranean region (prepared under this GEF
project)
SAP MED
Strategic action programme to address pollution of the
Mediterranean Sea from land-based activities
SPA/RAC
Specially Protected Areas (MAP's Regional Activity Centre,
Tunis, Tunisia)
4
I. INTRODUCTION
Main goal and scope of the project
1. The main goal of the project was "to improve the quality of the marine environment in the
Mediterranean Region by better shared-management of land-based pollution through
improved international cooperation in the management of land-based pollution of
transboundary and regional significance."
2. To achieve this goal, the project was designed to support the activities of the
Mediterranean Action Plan (MAP)1 that are related to the implementation of the Strategic
action programme to address pollution of the Mediterranean Sea from land-based
activities (SAP MED), which was adopted by the Contracting Parties to the Barcelona
Convention in 1997 (see footnote 1).
3. The project was implemented in 12 eligible countries which are Parties to the Barcelona
Convention: Albania, Algeria, Bosnia & Herzegovina, Croatia, Egypt, Lebanon, Libyan
Arab Jamahiriya, Morocco, Slovenia, Syrian Arab Republic, Tunisia and Turkey (Serbia
and Montenegro did not receive funding from the GEF project). The non-eligible Parties
to the Convention the European Community, Cyprus, France, Greece, Israel, Italy,
Malta, Monaco, Serbia and Montenegro, and Spain were also involved in some of the
region-wide activities of the project.
Evaluation approach
4. One tool for the evaluation was a questionnaire sent to the eligible countries in English
and French, covering the main elements to be considered in the evaluation. The
questionnaire is attached in Annex 1. The questionnaire was also used as guide to solicit
inputs from project partners.
5. Out of 12 eligible countries, eight (67%) submitted their responses to the questionnaire.
These were: Albania, Algeria, Bosnia & Herzegovina, Croatia, Egypt, Morocco, Syrian
Arab Republic and Turkey. The following four eligible countries did not respond: Libyan
Arab Jamahiriya, Lebanon, Slovenia and Tunisia. In one case, the lack of response may
have been due to problems of communications via internet and telephone: receipt of the
questionnaire by the appropriate government official was never confirmed. In the other
three cases the reasons for not responding (despite of repeated attempts to prompt it) are
not clear. In any case, the experience of the evaluator with getting the responses to the
questionnaire could be a reflection of the difficulties that the project personnel
encountered with a number of countries (though not necessary the same ones that did
1 MAP is a UNEP initiative adopted in 1975 by the Mediterranean countries. In 1976, these countries adopted
the Convention for the protection of the marine environment and the coastal region of the Mediterranean (the
Barcelona Convention, amended in 1995 with this name). The Convention provides the legal framework to
MAP. The Convention has developed six Protocols covering in more detail some of the issues addressed by the
treaty. The GEF project under review related in particular to the Protocol for the protection of the Mediterranean
Sea against pollution from land-based sources and activities (LBS Protocol, Athens, 1983 and amended in
Syracuse, 1996). A Mediterranean Commission on Sustainable Development has also been established under the
aegis of the Convention. 21 Mediterranean countries and the European Community are Parties to the
Convention. The Convention Secretariat also known as MAP Coordinating Unit (MEDU) is provided by
UNEP and it is based in Athens, Greece.
5
respond to the questionnaire) to ensure timely and up- to- the- minimum- required-
standard outputs.
6. Interviews were held with the Project Manager and his Administrative Assistant; the
Coordinator of the Mediterranean Action Plan (MAP); the technical personnel of the
Programme for the assessment and control of pollution in the Mediterranean region (MED
POL); and the staff of the Regional Office for Europe of the World Health Organization
(WHO/EURO) based in Athens. Inputs were also received from the professionals in the
MAP Regional Activity Centres most directly involved with the implementation of the
project, namely: the Priority Actions Programme (PAP/RAC, Croatia); Specially
Protected Areas (SPA/RAC, Tunisia); and the Regional Activity Centre for Cleaner
Production (CP/RAC, Spain). The World Bank and the former MAP Coordinator also
provided inputs.
7. The Consultant reviewed all available documentation concerning the project, including
minutes of meetings, progress and terminal reports, and the available project outputs such
as project-related documents and publications.
II. CONCLUSIONS
Main general conclusions
8. The main general conclusions derived from the responses provided by the eligible
countries to the evaluation questionnaire are summarized here. In principle, these should
be considered as significant conclusions, since they represent the opinion of government
officials directly involved with the project in the recipient countries:
a) respondents were unanimous in considering that the project represented a very
useful start in addressing a number of key issues;
b) the project has assisted the central and local authorities to increase their capacities
for sustainable management of natural resources and for environmental protection
in the coastal zone;
c) the project has demonstrated the usefulness of the integrated coastal zone
management approach;
d) it has contributed to the involvement of a number of stakeholders in dealing with
the issues addressed by the project and to raising public awareness on the
importance of environmental protection in the Mediterranean region;
e) a number of basic documents were prepared for the first time, which have been
used, at least in some eligible countries, for the preparation of other strategic
documents;
f) at least in some countries, the project has served, on the one hand, to bring further
to the attention of the highest level of government the problems of marine
pollution, and, on the other hand, to allow local experts to deal with these issues in
other locations or to use the same methodology to solve other problems;
g) it was the first project to be prepared using a systematic approach and a common
methodology in all participating countries; and
h) there is a need for sustained technical and financial assistance by UNEP-GEF and
UNEP-MAP, as well as for political and financial support by decision-makers at
the national level to facilitate the effective use of a number of key project outputs.
6
In particular, a new follow-up GEF project building on the positive results of this
one would be of the utmost importance.
9. Other important conclusions of this Terminal evaluation are as follows:
a) the overall project design was not appropriate: it was too ambitious, with too many
activities for the planned time frame, and it may have generated unrealistic
expectations;
b) some aspects of the project document were weak: it mixed `objectives' with
`outputs' and was not precise in the way that objectives and activities were
described;
c) another important weakness in the project design was that it did not seek in a
systematic and clearly-stated manner the active participation of the Parties to the
Barcelona Convention that are non-eligible countries. A number of them did
participate in some of the key activities, thanks to the efforts of UNEP/MAP, but
this was left to the goodwill of the countries concerned and, as a result, non-
eligible countries participated at random in the project activities;
d) in its conception, the project did not demonstrate a good understanding of and was
not sensitive enough regarding the `real-world situation' in the eligible countries
concerning issues such as the proposed "regional programme for public
participation" and the way it was handled;
e) the main successes of the project were: i) the up-dating of the Transboundary
Diagnostic Analysis (TDA) (originally prepared in 1997); ii) the preparation of the
National Action Plans (NPAs) to address pollution from land-based sources; and
iii) the preparation and adoption by the Conference of the Parties to the Barcelona
Convention of the Strategic Action Plan for the Conservation of Biological
Diversity in the Mediterranean Region (SAP/BIO);
f) the TDA and SAP/BIO provide very solid basis for undertaking the required
actions towards the long term conservation of the Mediterranean and the
sustainable use of its resources. The NPAs all meet the minimum required quality
standard, but a number of them would require more work to make them really
effective;
g) training was another useful output of the project;
h) the involvement of the national counterparts, project beneficiaries and local
consultants has been significant. However, there was no specific mechanism in
place to ensure the involvement of key stakeholders in all countries, even though
in some countries the involvement of stakeholders represented a real break-
through. It should also be noted that the definition of "stakeholder" differed from
country to country;
i) the process to produce and introduce in the MAP system some of the outputs (e.g.
some of the `regional guidelines' and `regional plans') was blurred, possibly due to
the stress of producing too many outputs at once, leaving some of these outputs in
a certain `limbo' as far as the peer-review system used and the governments'
endorsement;
j) the issues related to the protection of the open sea were not clearly addressed by
the project;
k) participating countries have high expectations about the implementation of the
recommendations, projects, NAPs, and SAP/BIO elaborated as part of the project.
To date there has been minimal implementation, and there is a risk of continuing
with a `paper producing process', without clear concrete implementation;
7
l) the project was conducted mainly in collaboration with the ministries of the
environment which are not among the key decision makers in the concerned
countries. It was demand driven by those ministries and offered as grants to
conduct studies and produce plans. It does not seem that these studies and plans
have been endorsed or approved by ministries of finance and ministries of
development/national planning. The project appears to have increased the
awareness of the decision-makers but this may certainly not lead to `ownership',
funding or operations/activities to jointly decrease the threats to the Mediterranean
Sea;
m) nonetheless, the project has the potential to achieve positive impacts, if the
ministries of environment are able to positively influence national and local
governments to follow up on the recommendations, studies and plans developed
through the project;
n) there is a great and urgent need for follow-up to ensure that the good results are
not wasted, but the approach should be more country-tailored with a stronger and
more effective coordination of all those involved. In particular, it will be important
to rapidly start a process to generate investments from the national budgets and by
the international financing institutions, which unfortunately were not kept
sufficiently abreast of the project development and implementation;
o) the GEF project being developed in UNEP/MAP does not seem to constitute a true
and useful follow-up to this project. In addition, UNEP-MAP, with its present
skills and structure may not be able to ensure implementation of the type of
actions/projects/programmes/plans to be derived from the outputs of this GEF
project. If UNEP-MAP is to be responsible for this, it should revise its mandate
and bring in staff with different skills, including a track record of project design
and implementation as well as project operations;
p) the project may present evidence, once more, of a certain `country fatigue' with
`traditional' projects. There may be a need to consider more innovative and
imaginative projects;
q) the project seems to have operated quite well at the `super-structural' level (project
management and executing agencies) but not so well at the country level, where it
should have really mattered; in a few countries in fact it operated poorly. Reasons
for recipient countries not to perform well with project execution are well known,
but this `truism' should be taken much more seriously into account in project
design, so as to try to ensure a more effective project execution, which in turn
would have a positive impact on the global environment; and
r) the overall conclusion is that project has helped to create a number of key strategic
tools of very significant value, in particular those of a regional value, such as the
TDA and SAP/BIO. The challenge is now to ensure that the NAPs are developed
further, when so required, and that they are fully integrated in the national
development machinery, including assigning and/or seeking the required financial
and technical resources.
10. Concerning the administrative and managerial aspects of the project, the main
conclusions are as follows:
a) the project had a slow start (the project manager came on board six months after it
began). Termination was delayed two years, in spite of the good management of
the project, something that may also reflect a poor project design. Nonetheless, the
formal revisions of the project were prepared on time and approved expeditiously.
The manager and other project leaders in the different subprojects also
8
demonstrated considerable ability to adjust and to innovate in order to obtain the
best possible results;
b) the project manager played a very positive role in helping the project to succeed
but he did not have sufficient authority to exert control on all aspects of the project
and to make decisions. A more proactive management approach would have
benefited the project;
c) recruitment of local experts suffered, in a number of cases, from undue
political/`jobs for the boys' interference, which indicates that the project should
have sought means to avoid this negative (yet well known) impact on project
implementation;
d) the initiatives taken by the project management and the executing agencies to
respond to unforeseen institutional and technical weaknesses in a number of
participating countries for producing expected outputs (e.g. hiring regional
consultants), indicate that there was an effective `adaptive management' approach,
which helped to address some of those weaknesses;
e) nevertheless, given the resistance to change in many countries and other
weaknesses concerning institutional and technical capacities, the project would
have benefited from a more country-tailored approach, including more project staff
devoted to work closely with those countries with less implementation capacity;
f) there was a sound management of funds, in spite of the complexities of the project
administration: 202 Miscellaneous Obligating Documents, of which 43 were
MOUs with financial and administrative implications; 211 Special Services
Agreements; 23 meetings in 11 different locations; and 342 travel authorizations.
Savings and innovation permitted more work, at least in some of the sub-projects,
in particular in relation to training and publications in local languages. The
production of local materials can be rated as very good in seven countries and
good in five countries; and
g) the different accounting systems used in the two UN offices involved meant that
the financial reports from Athens had to be manually updated in Nairobi, which
was a time consuming exercise.. On the other hand, the fact that the project
administration was located in a UN office facilitated the use of UN rules
concerning financial and personnel administration.
Ratings
11.
Section II of this Terminal evaluation analyzes the level of achievement of each of the
objectives stated in the project document and the delivery of each of the expected
outputs. Table 1 below provides the summary of the ratings attributed, as a result of
that analysis, to different categories of the project. The overall rating for the project is
`Satisfactory'.
Table 1 - Summary ratings
Category
Overall summary comments
Rating
Attainment of
Most of the objectives were attained. Some
Satisfactory
objectives and
of them were not completed due to a weak
planned results
project design (e.g. being too ambitious or
unrealistic).
Achievement of
Most of the outputs were of high quality and
Satisfactory
9
outputs and
should serve as a strong basis for action in
activities
the short and medium term future.
Cost
I has been difficult to assess this aspect of the Satisfactory
effectiveness
project but taking into account all the other
aspects assessed it would seem that the
project was cost effective.
Impact
The project did not manage to ensure the
Moderately
commitment of a number of key stakeholders Unsatisfactory
so as to guarantee that the useful outputs of
the project will be put to good use. This is a
pending issue.
Sustainability
The project was not well designed in order to Moderately
ensure its financial and political
Unsatisfactory
sustainability. All will depend on how the
outputs are used in the short term future.
Stakeholders
In a few participating countries, the
Moderately
participation
participation of stakeholders constituted a
Satisfactory
break-through but in general there was no
systematic and solid involvement of key
stakeholders.
Country
Ownership at the level of environmental
Moderately
ownership
ministries/agencies was satisfactory but this
Satisfactory
ownership did not include other key actors in
the governmental and non-governmental
sectors.
Implementation
The project manager and the executing
Moderately
approach
agencies demonstrated a high degree of
Satisfactory
proficiency and dedication in the
implementation of the project but the
implementation approach in a number of
countries was weak because their capacities
were not well assessed in advance. Thus, the
project had to conjure strategies to cope with
this reality.
Financial
Projects funds were used and administered in Highly
planning
a very efficient manner.
Satisfactory
Replicability With
all due adaptations, this project could
Satisfactory
be replicated in other regional seas processes.
Monitoring and
The project implementation was effectively
Highly
evaluation
monitored and evaluations were, in general,
Satisfactory
balanced and impartial.
Overall rating
Satisfactory
III.
SPECIFIC ASPECTS OF THE PROJECT
Attainment of objectives and planned results
12.
The following is an analysis of the attainment of the project objectives stated in
Section 2 of the original project document. It should be noted, though, that in the way
that they were worded, the majority of these `objectives' are in fact the `outputs' then
10
listed in Section 3 of the project document. Consequently, it would be impossible to
make a clear cut difference between this subsection of the Terminal evaluation and the
next one, on Achievements of outputs and activities.
13.
Objective
1: Complete an analysis of the transboundary importance of the 103 hot
spots identified in the Transboundary Diagnostic Analysis of the Mediterranean Sea
(TDA MED) and the SAP MED and finalize the priority list for intervention and
investments ("investments portfolio") (pre-investment studies will be conducted only
in GEF eligible countries).
14.
Concerning the TDA, which was under the responsibility of MED POL:
a)
a 200-page full-coloured publication: Transboundary Diagnistic Analysis
(TDA) for the Mediterranean Sea (UNEP/MAP/MED POL, Athens, 2005)
contains the up-dated TDA (originally produced in 1997), which is the result of
work carried out as part of this GEF project;
b)
the TDA provides the technical basis for refinement of the National Action
Plans (NAPs) for the protection of the marine environment from land-based
sources and the SAP MED in the area of international waters of the GEF;
c)
it also provides the expert opinion on the state of the environment and priority
problems, and a list of actions that are recommended for consideration. This
list is designed to address the major transboundary issues in the GEF focal area
related to international waters;
d)
the TDA process identified three Environmental Quality Objectives (EQOs) as
major goals for the Mediterranean environment and made used of the Strategic
action programme for the conservation of biological diversity in the
Mediterranean region (SAP BIO) and of the Code of Conduct for Fisheries,
which led to specific targets and interventions that can be considered in the
NAPs; and
e)
the TDA now constitutes a significantly stronger tool, both from a scientific as
well as strategic point of view, at the disposal of MAP and each of the Parties
to the Barcelona Convention (including those which are not GEF-eligible) to
undertake the required actions to protect the Mediterranean Sea.
15.
Thus, the objective concerning the TDA has been fully met.
16.
The SAP MED was completed, with targets and needed activities at regional and
national level structured in two areas (urban and industrial) and a number of categories
within each area. Thus, this part of the objective was fully met.
17.
The hot-spots pre-investment studies were carried out in 10 countries, as a pilot
project, with one hot spot in each country. According to the progress reports, the
process of preparation of pre-investment studies was much more complex than
expected and it required more time than originally assumed. The reports also reflect
that this component of the project faced a considerable number of difficulties because
most participating countries were ill-prepared to generate this output of the project.
The lack of capacity was covered, in part, by a very active involvement of the project
manager.
11
18.
Considering that the process of preparing this output was riddled with difficulties and
that they results were not of the magnitude and significance foreseen in the project,
this aspect of Objective 1 should be considered as poorly met. The poor results
concerning this component of Objective 1 may also reflect a weak design of the
project, since participating countries were expected to undertake a task that seems to
have been beyond their capacities and/or current vision.
19.
Objective 2: To formulate and adopt principles, approaches, measures, timetables
and priority actions, that address each major land-based source of pollution and assist
countries in the implementation of such actions.
20.
This objective is expressed in more concrete and clear terms as Objective 5, assessed
below.
21.
The next subsection of this Terminal evaluation report, on Achievement of outputs and
activities, clearly indicates that the project has helped to develop a considerable body
of "principles, approaches, measures, timetables and priority actions" that should
provide the participating countries with very useful tools to address pollution issues.
Thus, this component of Objective 2 should be considered as fully met.
22.
However, the second component of Objective 2: "[to] assist countries in the
implementation of such actions" does not seem to have been met. While this Terminal
evaluation did not have the means to assess actions on the ground in the participating
countries, all indications lead to the conclusion that countries have not entered this
phase on a degree that would make a difference. (See more under the assessment of
Objective 5 below.)
23.
Objective 3: To conduct pre-investment analysis of expected baseline and additional
actions needed to address the selected hot spots, and secure recipient country
agreement to baseline investment.
24. The activities to address this objective included the identification of economic
instruments that would assist in developing a sustainable financial platform for the
continued implementation of SAP MED in the longer term. In total, 13 activities were
envisaged in the project document, of which nine were completed. Other activities
were reviewed and new activities were incorporated in the project revisions, which
were completed.
25.
The GEF project represented a start in the majority of participating countries in the
area of economic instruments (EI). In this sense, the objective was fully met. Yet, the
advantages and need to use EI are not sufficiently recognized by decision-makers in
the majority of participating countries. Thus, the second part of the objective ("to
secure recipient country agreement to baseline investment") has not been met.
26.
Objective 4: Prepare and adopt at the regional level, detailed, operational guidelines
for the formulation of National Actions Plans (NAPs) for the protection of the marine
environment from land based activities.
27.
The project produced a set of the `regional guidelines' and `regional plans' for
preparing the NAPs, including a National Diagnostic Analysis (NDA) and National
12
Baseline Budget (NBB). It should be noted that the NBB was an `add on', not
contemplated in the original project document, and financed to a large extent with
MED POL resources. Training courses were organized for their application. It should
be noted, though, that the guidelines were not `adopted at the regional level', and
possibly there was no need for that.
28.
The successful application of the guidelines in the preparation of a NAP in each of the
12 participating countries indicates that this objective was fully met.
29.
Objective 5: Assist countries to prepare, adopt at the highest level, and implement,
country specific National Actions Plans based on the regionally prepared and adopted
guidelines.
30.
The project clearly assisted the participating countries in preparing their respective
NAPs. The NAPs are considered to be the most significant achievement under the
project. In this sense, the objective was fully met, but with a caveat: as long as the
NAP in each country has not been adopted at the required level on the basis of an
appropriate legal and/or regulatory instrument, the NAP should be considered as a
`proposal', as a `draft', but not as an actual `action plan'.
31.
It should be noted that the elaboration of the NAP was a complex undertaking for
some countries. To quote one of them: « Plan d'action national (PAN) et Plans
sectoriels : l'élaboration de ces plans a connu un grand retard à cause de la non
disponibilité de données et du retard de certains départements pour fournir aux experts
les données nécessaires pour l'élaboration de ces plans. Pour cela, on n'a pas pu
organiser la réunion finale du comité interministériel pour valider le projet du PAN car
l'achèvement de ce travail a coïncidé avec la clôture du Projet FEM/PASMED. »
32.
Another positive aspect concerning the NAPs is the fact that they have also been
prepared by Cyprus, France, Greece, Israel, Italy, Malta, Monaco, Serbia and
Montenegro, and the Palestinian Authority (Gaza Strip). This is a crucial aspect since
reducing pollution loads in a regional sea required the involvement of all countries in
the basin.
33.
Concerning the quality of the NAPs, all of them are considered to be sound planning
documents, since the MED POL staff took care to ensure that the basic quality of each
one of the NAPs was in line with accepted standards, requesting countries to revise the
drafts when so required. However, not all countries were able, at the end, to go into the
same level of detail, in particular in relation to the estimated cost of the application of
the NAP.
34. In relation to the level of completeness, the view of this Terminal evaluation
concerning the NAPs prepared by the 12 eligible countries is that: 1 is excellent; 1 is
very good; 7 are good; and 3 are weak. Of the five NAPs produced by other Parties to
the Barcelona Convention (not GEF-eligible), 4 are good and 4 are weak.
35.
In relation to the other component of the objective under analysis, the project was not
able to secure "adoption of the NAPs at the highest level of government" in the
participating countries and "the implementation of the NAPs". In these two significant
aspects, the objective was not met.
13
36.
Concerning the adoption `at the highest level', the project does not seem to have been
designed for this purpose. To achieve this objective, the project should have involved
from the start other key stakeholders in each country and should have included the
required activities, such as high level interventions, national seminars, prompting the
adoption of the required legal, policy and/or administrative instruments, etc..
37.
As far as implementation is concerned, the project may have been unrealistic in
expecting that countries would be in a position to start implementing the required
actions within the timeframe of the project: preparing, firstly, the technical guidance
and undertaking the complex process of preparing a NAP; and, secondly, changing
policies and legislation, creating the institutional capacity and obtaining the significant
resources that are required to implement a NAP, could not be done in three years
(original duration of the project) or even in five years (final duration of the project).
38.
In addition, in order to initiate real action to address the complex issues of pollution
from land-based activities, the project should have envisioned to actively involve other
key government agencies and partners (such as the private sector) in its
implementation, and not limit itself to working with ministries of environment. These
ministries demonstrated their capacity to effectively participate in assessment studies
and in generating the proposals for NAPs; but in the majority of countries they do not
have the means and the political clout to influence the national development agenda.
39.
Thus, three serious concern remains: a) the NAPs have not yet been formally and
legally adopted by the government in the majority of countries; b) in most cases, the
required national and outside resources to implement the NAP have not been identified
and, even less so, secured; and c) since a number of countries would not have the
capacity to implement their NAP, if there is no quick follow-up to assist them, this
significant and successful effort could be wasted, generating frustration and a lack of
belief in the GEF mechanism.
40.
All said, it should be noted that COP14 of the Barcelona Convention (November
2005) recommended2 to the Parties "to endorse, support and mobilize the necessary
resources for the implementation of National Action Plans (NAPs) ... and to ensure
their integration into existing national strategies and pollution control plans", and
requested the Secretariat "to catalyze and facilitate the mobilization of external funds
to assist Mediterranean countries in the implementation of National Actions Plans
(NAPs)".
41.
Thus, the COP has been prompted to deal with resource mobilization for NAPs before
they have been actually formally adopted in each country. In this and in other issues
related to the GEF project, such as the process to produce, review and approve
regional guidelines and action plans the complex MAP system, which is managed by
a dispersedly-located and loosely coordinated Secretariat stretched to its limits, has
been performing in an untidy manner.
2 The COPs to the Barcelona Convention use the practice of adopting "Recommendations" addressed to the
Parties, the Secretariat and others, instead of `decisions' or `resolutions'. A recent External Evaluation of MAP
has recommended that the COP should change this practice and to start using a system to record decisions
similar to the other conventions.
14
42.
Objective 6: Identify roles for, and ensure effective participation of non-governmental
organizations in the implementation of components of the SAP MED, and where
appropriate incorporate these into the National Action Plans; and to address other
transboundary issues as follows:
a) finalize and adopt a comprehensive and holistic Transboundary Diagnostic
Analysis; and
b) develop and adopt a Strategic Action Plan for Biodiversity in the Mediterranean in
conformity with the provisions of the Protocol on specially protected areas and
biological diversity in the Mediterranean.
43.
This objective mixes issues of completely different nature: national NGOs'
participation and two transboundary matters of high significance in their own right,
one of which was already addressed in Objective 1.
44.
Concerning NGO participation, first it should be observed that the project was going
to be implemented in a number of countries where the NGO movement is still weak,
not playing a significant role in influencing public policies, and even less so in
implementing projects on the ground. Second, in these countries, the ways and means
for decision-making and exercising social control have their own expressions, which
are not always well understood by the mainstream `experts' from the outside. Thus,
devoting an objective in the project to involving a sector that is not very relevant in the
countries concerned, and in some cases practically non-existent, does not seem to have
been a very sensible decision.
45.
In addition, and in order to fulfil the objective of public participation, the project
produced a document entitled Public participation in National Action Plans (NAPs)
for the Strategic Action Programme (SAP MED) to address pollution form land-based
activities in the Mediterranean Common methodology, which was distributed to all
Mediterranean countries.
46.
The document is based, primarily, on the principles of the Aarhus Convention (so far
ratified by only three of the 12 countries participating in the project). Given the very
different situation encountered in Mediterranean countries concerning civil society
participation, it is not sure that a `common methodology' could be applied in these
countries. In addition, the principles of the Aarhus Convention represents the most
advanced and sophisticated development, from a given vision of the human being and
society, concerning access to information, public participation in decision-making and
access to justice on environmental issues. It is not sure that these principles could be
easily adapted and used in a number of participating countries with a very different
approach to these matters. It seems that two very important components were lacking
in this approach: a `reality check' in order to be effective and a sense of cultural and
political `relativism'.
47.
All NAPs contain a Section E on `The public participation foreseen to implement the
actions'. With the exception of one NAP where the issue is elaborated with some level
of detail, including a list of convincing steps and actions to be undertaken, all the other
NAPs have contented themselves with one or two short paragraphs about the
importance of public participation. To quote one of them: "The NAP ...was prepared
with active participation of a multitude of stakeholders (national and local authorities,
15
academia, NGOs, industry) during several meetings. The same approach is expected to
be used during the implementation phase of these actions, although no specific
mention is made in the NAP." (Emphasis added.)
48.
In most cases, it seems that `participation' of stakeholders has been equated with
information and consultation meetings. Surely they have been very valuable and their
input to the process must have been very positive. Nonetheless, stakeholders do not
seem to have been actively engaged in the process with their own experts and
institutional and academic capacity.
49.
Having said all this, in a few countries (not more than three) indications are that the
stakeholders' involvement in the preparation of the NAP (not only NGOs) represented
a break-through.
50.
In relation to the TDA component of Objective 6, the issue has been dealt with in
relation to Objective 1 above.
51.
Concerning the development and adoption of a Strategic Action Plan for Biodiversity
in the Mediterranean (SAP BIO) in conformity with the provisions of the Protocol on
specially protected areas and biological diversity in the Mediterranean, the SAP BIO
was successfully developed and formally adopted by the Conference of the Parties to
the Barcelona Convention in 2003. It is important to note that this has meant the
endorsement by the 22 Parties to the Convention, all of which considered that the
SAP/BIO represented a commendable effort in terms of technical and scientific rigour
and policy relevance.
52.
It should be noted that the preparation of the SAP BIO generated a number of analysis,
studies, and action plans. These included:
a) 58 National Action Plans for specific priority issues, including an investment
portfolio;
b) 168 National Priority Actions, other than those included in NAPs; and
c) 30 Regional Actions, including an estimated budget.
53.
This Terminal evaluation could not verify the quality of the outputs mentioned in the
previous paragraph, nor their implementation status. Prima facie, it may have been an
excessive proliferation of plans, knowing the limited institutional and financial
capacity of a number of the participating countries. It has not been possible to
ascertain that there was a real need for such proliferation in order to produce a good
SAP BIO.
54.
As far as the stated objective is concerned developing and adopting a SAP BIO the
objective was fully met.
Summary level of attainment of the objectives and general observations
55.
Table 2 below summarises the level of attainment, in the view of this Terminal
evaluation, of the project objectives.
Table 2. Level of attainment of project objectives
16
Objectives
Level of
Observations
attainment
Objective 1:
a) To complete Transboundary Diagnostic
Fully met
Analysis of the Mediterranean Sea (TDA
MED).
b) To complete the SAP MED.
Fully met
The SAP MED has been
completed but it is not legally
binding.
c) To finalize the priority list for
Poorly met The project seems to have been
intervention and investments (`investments
poorly designed to meet this
portfolio') in GEF eligible countries.
objective.
Objective 2
a) To formulate and adopt principles,
Partially
Technical tools and policy
approaches, measures, timetables and
met
guidance have been developed
priority actions, that address each major
but in general they have not been
land-based source of pollution.
formally adopted.
b) To assist countries in the
Not met
The design and timeframe of the
implementation of such actions.
project could not provide such
assistance.
17
Objective 3
a) To conduct pre-investment
Fully
This objective has been interpreted as
analysis of expected baseline and
met
relating to the economic instruments (EI),
additional actions needed to
the subproject implemented by PAP/RAC.
address the selected host spots
b) To secure the recipient country
Not met
The project was unrealistic in trying to
agreement to baseline investment.
achieve this objective within the project
time frame.
Objective 4
To prepare and adopt at the
Fully
The level of achievement is considered as
regional level, detailed, operational met
`Fully met' even if the guidelines (except
guidelines for the formulation and
one) were not formally `adopted' at
National Actions Plans (NAPs) for
regional level. The timing of the
the protection of the marine
finalization of at least some of the
environment from land based
guidelines is an issue, since they were
activities.
developed at the same time as the NAPs.
Objective 5
To assist countries to prepare,
Partially While the preparation of the NAPs is
adopt at the highest level, and
met
considered the major achievement of the
implement, country specific
project, the objective is rated as `Partially
National Actions Plans based on
met' because the NAPs have not been
the regionally prepared and adopted
adopted at the highest level and
guidelines.
implemented. More than a failure in the
project implementation, this would indicate
a poor project design.
Objective 6
The objective has been poorly established,
mixing issues of very different nature.
a) To identify roles for, and ensure
Partially The project was poorly designed in relation
effective participation of non-
met
of the situation concerning civil society
governmental organizations in the
involvement in public affairs issues in most
implementation of components of
of the participating countries.
the SAP MED, and where
appropriate incorporate these into
the National Action Plans.
b) To finalize and adopt a
N/A
This matter was already covered under
comprehensive and holistic
Objective 1.
Transboundary Diagnostic
Analysis.
c) To adopt a Strategic Action Plan Fully
The SAP BIO was formally adopted by the
for Biodiversity in the
met
Parties to the Barcelona Convention at their
Mediterranean in conformity with
COP in November 2003.
the provisions of the Protocol on
specially protected areas and
biological diversity in the
Mediterranean.
Summary concerning the level of achievement of the objectives:
5 objectives Fully met (100%)
18
2 objectives Partially met (66%)
2 objectives Poorly met (33%)
2 objectives Not met (0%)
Using these percentages, the project would have a level of achievement of objectives of 63%.
It has to be noted though, that this percentage would be fully relevant if the project objectives
were all equal in importance, which in principle they are not.
56.
There remains no doubt that the project:
a)
has developed and/or updated a number of key instruments to assist in the
implementation of the SAP MED at national and regional level;
b)
in spite of important shortcomings, has broken new ground in the area of
stakeholders participation and on the need to work with economic
instruments; and
c)
has developed a key biodiversity instrument, the SAP BIO, a component for
the conservation and sustainable use of the natural resources of the
Mediterranean that constituted a glaring gap.
57.
Forming a joint management programme and strategy involving all countries in the
Mediterranean that will eventually lead to reducing threats to the marine and coastal
environment continues to be a pending issue.
58.
To start with, the key instruments developed under the project, such as the National
Action Plans, have not been formally approved at a high level in the participating
countries, and the issue of investments, both national and external, has not been
addressed. This fact, together with the weak capacity of most of the countries to make
an active use of the project results without external assistance, means that the GEF
investment in this project could be wasted if there is not a quick and well thought-
through follow-up. The new GEF project being prepared in UNEP/MAP does not
seem to go in this direction.
59.
Through a number of project outputs, such as the NAPs including the NDAs and
NBBs , the work on EIs, the SAP BIO, and the criteria for prioritization of
transboundary pollution hot spots, as well as the numerous training activities, the
project has helped to build a considerable degree of consensus among some
stakeholders on transboundary issues and the management measures to address them.
This is reflected mainly in the agreed-upon Environmental Quality Objectives (EQOs)
for the Mediterranean Sea, related to the SAP MED, fisheries (a new significant
development in UNEP/MAP) and the SAP BIO, with specific targets for making
progress towards these EQOs.
60.
However, it should be noted with concern that the consensus referred to in the
previous paragraph appears to remain largely among officials in the ministries of
environment and other stakeholders with a pre-established interest in environmental
and sustainability issues. The challenge is now to expand this consensus to include in
each country and at the regional level other key governmental and non governmental
actors, including the private sector and the academia and expert groups in fields other
than the environmental sciences. A considerable number of these actors do not have
environmental protection as their main concern or area of activity, but they do have an
impact some times decisive on ecosystem functions and services.
19
61.
Through the project, the technical and analytical capacities of an important number of
stakeholders in each country in identifying and setting up management frameworks for
transboundary marine/coastal issues have been considerably enhanced. Now these
capacities have to find their ways into the more illusive realm of policy making,
regulation, legal frameworks, national investment, and national plans to attract foreign
investment and development and technical assistance. MAP/UNEP/GEF should be
prepared to assist in a timely and effective manner in order to ensure that this will
happen, through their direct intervention and/or by catalyzing the involvement of other
key multilateral institutions and processes, such as the World Bank, the European
Commission, and the New Alliance for Africa Development (NEPAD). Bilateral
assistance mechanisms should also be given due and serious consideration.
62.
The questionnaire used for this Terminal evaluation included the following question:
"To what extent do you consider that the results of this project will assist your country
in forming a joint management programme and strategy that will eventually lead to
reducing threats to the marine and coastal environment in the Mediterranean?" Five
countries responded `Very mucho so'; and three countries `To some extent' (four
countries did not respond). These responses could be construed as representing a level
of satisfaction of 87%.
63.
There were two comments connected to this question, from two different countries,
which are worth quoting:
1: "This is the first time that MAP is working with contracting Parties in a systematic
approach, starting from a specific base-line, aiming at reaching a specific target within
a specific period of time, with a thorough preparation of an action plan to achieve the
objectives."
2: "The GEF project will be successful if it can be implemented by sustainable
financial resources at the national and regional level."
64.
To the question: "Has the project helped build consensus among stakeholders on
transboundary issues and management measures to address them?", one country
responded `Very much so'; five countries `To some extent'; and one country `Not al
all' (five countries did not respond).
65.
Some of the comments made by countries in relation to this question were:
a) the project activities had a positive impact on the stakeholders at the central and
local level, which is demonstrated in an enhancement of commitments from
government and private sectors in the infrastructure investment (water supply,
treatment of waste water, urban solid waste management, hot spots clean up, road
construction, designation of new protected areas, etc.);
b) the project built up consensus on the issues addressed but still the stakeholders
need concentration on activities that will lead to build up consensus regarding the
management level;
c) the project helped build consensus among stakeholders because it used a
participatory approach through many workshops and brainstorm discussions; and
d) transboundary issues have not been discussed yet within the scope of the GEF
project.
20
66.
To the question: "Has the project enhanced the capacity of your country and the
stakeholders to identify and set up a management framework for transboundary
marine/coastal issues?" two countries responded `Very much so'; four countries `To
some extent'; and one country `Not al all' (five countries did not respond).
67.
One of the comments related to this question said: "We have a vision concerning the
framework, but we have not yet embarked in the preparation of this framework."
68. The project's outcomes are fully consistent with the GEF Focal Area 3 on
International waters and fully in line with the long-term objective of the GEF
Operational Programme 8: "to undertake a series of projects that involve helping
groups of countries to work collaboratively with the support of implementing agencies
in achieving changes in sectoral policies and activities so that transboundary
environmental concerns degrading specific waterbodies can be resolved".
69.
More specifically, the project outcomes are fully in line with the short-term objective
(e) of OP 8: to "fully develop a GEF strategic approach to a specific, damaged Large
Marine Ecosystem (LME) so that significant investments are leveraged and regular
programs of implementing agencies are harnessed to address priority transboundary
environmental concerns in the highly damaged large marine ecosystem".
70.
In addition, the project also generated outputs that are in line with other OPs in the
GEF focal area of international waters, such as Integrated land and water multiple
focal area Operational Programme (OP 9) and Contaminant-based Operational
Program (OP 10).
71.
The project outcomes include a component (the SAP BIO) which is fully consistent
with the GEF Focal Area 1 on Biodiversity, and in particular with OP 2 on Coastal,
marine and freshwater ecosystems.
72.
The GEF Project represented the necessary step towards the long term solution of
some key environmental problems in the Mediterranean Sea by generating a number
of concrete tools and by beginning to install the required `working approach' and
`modus operandi' at national and regional level to effectively deal with the complex
issues at hand.
73.
In summary, the GEF project has not solved the problems addressed (it could not do so
with the resources allocated and its limited time frame) but it has established a
considerable part of a solid base, which was missing, in order to move forward
towards the stated goal of improving the quality of the marine environment in the
Mediterranean Region (see paragraph 1).
74.
Thus, judging from its results, the project was successful in contributing to its overall
goal. But from a more formalistic view point, when comparing the original project
document against the outputs, the project suffered from a series of weaknesses in its
design, such as: a) creating false expectations concerning what the project would do;
b) not making a realistic reading of the institutional and technical capacity of the
participating countries and of the `operational system' of their societies (not enough
21
`reality check'); and c) not including all the activities that were required in order to
achieve some of the key objectives and generate some of the expected outcomes.
75.
It is fair to say also that one `positive weakness' of the project design was revealed by
the fact that the project was able to generate, without additional resources, a number of
outputs that were not envisioned. In this sense, the project manager and the leaders of
the different sub-projects deserve commendation.
Achievement of outputs and activities
76.
This section of the Terminal evaluation assesses the project's success in producing
each of the programmed outputs, both in quantity and quality as well as usefulness and
timeliness. The following ratings are used: highly satisfactory; satisfactory;
moderately satisfactory; moderately unsatisfactory; unsatisfactory; and highly
unsatisfactory.
Output 1: Updated Transboundary Diagnostic Analysis (TDA)
77.
The effectiveness of this output has been dealt with in paragraphs 14-15 above. As an
output, it is rated as highly satisfactory.
78.
Nonetheless, the eight eligible countries that responded to the evaluation questionnaire
rated this output of the project as follows:
1 country as highly satisfactory
2 countries as satisfactory
3 countries as moderately satisfactory
2 countries did not rate this output (Does this reflect a lack of familiarity with this key
output?)
79. These ratings from the eligible countries raise some concern: while the output
reflected in the publication mentioned in paragraph 12 a) above reflects an output of
very high quality, the participating countries do not seem to have the same
appreciation of this result.
Output 2: Pre-investment studies for selected pollution hot spots
80.
The effectiveness of this output has been dealt with in paragraphs 17-18 above. In
addition, it should be noted that this activity was initiated in Algeria, Morocco,
Lebanon and Tunisia only in 2005, where it is being financed by the Fond Français
pour l'Environnement Mondiale (FFEM), due to the fact that the funds were late to
come.
81.
The results are rated as moderately unsatisfactory, in particular taking into account that
the pre-investment studies would have been a key tool for actual implementation of the
technical outputs of the project.
22
82.
The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
2 countries as highly satisfactory
3 countries as satisfactory
1 country as moderately satisfactory
2 countries did not rate this output (Does this reflect a lack of familiarity with this key
output?)
83.
Once more, there is a discrepancy between the rating of this Terminal evaluation and
the opinion of the eligible countries.
84.
One comment made in relation to this question is worth quoting: « L'étude de pré
investissement est en cours de lancement. Il serait souhaitable de prolonger la durée du
projet car les procédures administrative prennent du temps et il est difficile de
respecter les échéances d'un pays à un autre. Surtout que le PAS s'inscrit dans la
durabilité, ce projet est spécifique et sa mise en oeuvre s'intégrer dans les stratégies à
long terme de la préservation du littoral et du milieu marine. »
Output 3: Analysis of pollution sensitive areas
85. A
List of regional pollution sensitive areas was included as Annex IV (page 186) in
the publication Transboundary Diagnostic Analysis (TDA) for the Mediterranean Sea
(UNEP/MAP, 2005). The List includes 37 areas in 10 countries participating in this
GEF project, ranging from only one area in Egypt to up to 18 areas in Turkey (Bosnia
& Herzegovina and Libya are missing from the List). In addition, the List includes 22
areas in five EU-member countries in the Mediterranean. In summary, the List
includes areas in 15 countries, out of the 21 states that are Parties to the Barcelona
Convention. For all countries the List includes information on the origin of pollution
and on the remedial action (except for two countries).
86.
The project generated the Scientific criteria for the evaluation of pollution sensitive
areas in the Mediterranean countries, published in 2003, "in order to establish a series
of criteria and to develop a ranking system for the evaluation of the pollution sensitive
areas in the Mediterranean Sea. The ranking system should be capable of describing
the pollution effects and their severity on the sensitive areas, taking into consideration
both the environment and human health." The original draft criteria prepared by
experts were discussed at a meeting that took place in November 2002 with the
attendance of 12 government-designated experts from eight countries participating in
the GEF project and from four other Mediterranean countries. This means that only
57% of the Parties to the Barcelona Convention were in attendance.
87.
The published 2002 List was an update of the original list produced in 1997. No newly
updated list has been made public after the adoption of the criteria. Thus, on the one
hand, the project has contributed to advances in this matter. On the other hand, the
facts that: a) the published List does not include all the Parties to the Barcelona
Convention; and b) in some cases it may not include all the sites that are `pollution
sensitive areas', lead to the conclusion that there is work still to be done to complete
the List, and even more so to act on the areas in order to ensure their long term
conservation.
23
88.
The results of this outcome should be considered as moderately satisfactory.
89.
The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
2 countries as highly satisfactory
2 countries as satisfactory
3 countries as moderately satisfactory
1 country as moderately unsatisfactory
Output 4: A set of regional guidelines and regional plans
Regional guidelines
90.
In total, nine regional guidelines were prepared under the project with the main
purpose of assisting countries in the preparation of NAPs. The guidelines were
published as part of the MAP Technical Reports Series.
91.
Four regional guidelines where prepared using the following process:
a)
individual experts were hired, with contracts finalized by March 2002 (e.g.
14 months after the formal initiation of the project on 1 January 2001);
b)
the first draft of the guidelines was ready by July 2002;
c)
the drafts where reviewed by correspondence by a small group of experts
established for each subject. The review was completed in December 2002;
and
d)
one of the revised drafts (on industrial wastewater) was reviewed at a special
expert meeting held in March 2003. The other three revised drafts were
reviewed at a meeting held in April 2003. The meeting was attended by
government-designated experts from seven of the 12 countries participating
in the GEF project and seven non-eligible countries (64% of the Parties to
the Barcelona Convention). In addition the meeting included four temporary
experts.
92.
This set of guidelines includes:
- Guidelines on management of coastal litter for the Mediterranean region.
MAP Technical Reports Series No. 148, UNEP/MAP, Athens, 2004
- Guidelines on environmental inspection systems for the Mediterranean
region. MAP Technical Reports Series No. 149, UNEP/MAP, Athens, 2004
-
Guidelines on sewage treatment and disposal for the Mediterranean region.
MAP Technical Reports Series No. 152, UNEP/MAP, Athens, 2004
-
Guidelines for the management of industrial wastewater for the Mediterranean
region. MAP Technical Reports Series No. 153, UNEP/MAP, Athens, 2004.
93.
MED POL was also responsible for preparing the Guidelines for river (including
estuaries) pollution monitoring programme for the Mediterranean region. MAP
Technical Reports Series No. 151, UNEP/MAP, Athens, 2004.
24
94.
In addition, three `regional guidelines" where commissioned to the MAP Regional
Activity Centre for Cleaner Production (CP/RAC), based in Barcelona, Spain:
-
Guidelines for the application of Best Available Techniques (BATs) and Best
Environmental Practices (BEPs) in industrial sources of BOD, nutrients and
suspended solids for the Mediterranean region. MAP Technical Reports Series
No. 142, UNEP/MAP, Athens, 2004
-
Guidelines for the application of Best Environmental Practices (BEPs) for the
rational use of fertilisers and the reduction of nutrient loss from agriculture for
the Mediterranean region. MAP Technical Reports Series No. 143,
UNEP/MAP, Athens, 2004.
-
Guidelines for the application of Best Available Techniques (BATs), Best
Environmental Practices (BEPs) and Cleaner Technologies (CTs) in industries
of the Mediterranean countries. MAP Technical Reports Series No. 146,
UNEP/MAP, Athens, 2004.
95.
In turn, CP/RAC contracted out the preparation of the guidelines. In its Terminal
Report for external projects, CP/RAC acknowledges that "it was difficult to identify
and recruit appropriate consultants that could carry out the guidelines...", that "the
scope of the regional guidelines had not been clearly defined and the title was too
ambiguous" and that "the foreseen timing was inappropriate". These guidelines were
reviewed by the National Focal Points of CP/RAC.
96.
Finally, the group of `regional guidelines' includes a publication of a different nature:
the Reference handbook on environmental compliance and enforcement in the
Mediterranean region. MAP Technical Reports Series No. 150, UNEP/MAP, Athens,
2004. Produced by a four-expert team, "the purpose of the Handbook is to raise the
level of performance of the environmental inspectors and support the above mentioned
guidelines [recommended guidelines on compliance and enforcement] by providing
details on assessing, developing, implementing and sustaining a viable inspection
programme."
97.
It is regrettable that the documents published in the Technical Report Series do not
provide information on the expert review process used for each of them and on their
formal status vis-à-vis the Barcelona Convention and its Protocols.
98.
COP14 of the Barcelona Convention recommended "to take note of the Guidelines for
municipal water re-use in Mediterranean countries..." (apparently a different name
was used here for the Guidelines published as MAP Technical Reports Series No. 152
mentioned in paragraph 85 above).
Regional plans
99.
Four `regional plans' were produced:
-
Plan on reduction of input of BOD by 50% by 2010 from industrial sources for
the Mediterranean region. MAP Technical Reports Series No. 144,
UNEP/MAP, Athens, 2004.
25
-
Plan for the reduction by 20% by 2010 of the generation of hazardous wastes
from industrial installations for the Mediterranean region. MAP Technical
Reports Series No. 145 UNEP/MAP, Athens, 2004
-
Plan for the management of hazardous waste, including inventory of hazardous
waste in the Mediterranean region. MAP Technical Reports Series No. 147,
UNEP/MAP, Athens, 2004
-
Plan for the management PCBS waste and nine pesticides for the
Mediterranean region. MAP Technical Reports Series No.155, UNEP/MAP,
Athens, 2004.
100. COP13 of the Barcelona Convention recommended "to adopt the approach proposed
by the Secretariat in the Regional Plan for a 50 per cent reduction in BOD from
industrial sources concerning the target date of 2010 to be reviewed in 2007..."
(Recommendation II.A.1.2). COP13 also recommended "to adopt the approach
proposed in the regional plan for the reduction of the generation of hazardous waste
from industrial installations by 20 per cent by the year 2010 to be reviewed in 2007..."
and requested the Secretariat to assist the countries in the implementation of the
regional plan (Recommendation II.A.3). (Underlining has been added.)
101. Prima facie, the technical quality of the guidelines and regional plans is high and in
this sense the project made a good contribution to the goal of achieving a clean
Mediterranean Sea. Thus, technically, this output can be rated as highly satisfactory.
102. In relation to process, the input from governments and other national institutions and
stakeholders appears to having been weak, reduced to participation at a few review
meetings not always well attended, and consultations with some experts and/or the
National Focal Points of one or two RACs. The so called `regional plans' in particular
should in fact be considered more as technical proposals than true plans adopted in the
region.
103. Concerning ownership, COP13 and COP14 have "adopted the approach proposed by
the Secretariat" in two of the regional plans and have "taken note" of one of the
guidelines. This reflects a poor introduction of the output into the more political and
formal process of the Convention.
104. Consequently, the process followed to prepare the `regional guidelines' and `regional
plans' appears to not have been rigorous enough and should be rated as unsatisfactory.
105. After combining the two ratings assigned to different aspects of the output, the overall
rating could be established as moderately satisfactory.
106. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
1 country as highly satisfactory
5 countries as satisfactory
1 county as moderately satisfactory
1 country did not provide a rating
Output 5: Capacity building (training courses at regional and national levels)
26
107. At least 650 trainees have participated in eight regional and 17 sub-regional / national
training courses on one of the following subjects:
· environmental inspection;
· environmental impact assessment;
· river pollution monitoring;
· management and operation of wastewater treatment plants;
· water reuse;
· economic instruments; and
· cleaner production: techniques and practices
108. The regional training courses were meant to train at least one trainer from each eligible
country for a subsequent preparation of a national training course. Sub-regional /
national training courses were conducted in national languages.
109. Modern training techniques were used and most training materials were prepared `in-
house' (instead of commissioning them out). The training packages delivered to the
trainers at the end of regional courses included transparencies and explanatory notes in
hard copy and software form, prepared in a way that facilitates translation and desk top
publishing in any of the Mediterranean languages. Some training materials produced
under the project are now being used in universities in some participating countries.
110. It should be noted that more training was offered than originally contemplated in the
project, which was made possible thanks to the sound management of funds allowing
the savings necessary to cover these additional costs. This movement of funds within
the project was duly authorized.
111. A considerable effort went into assisting the eligible countries in producing their own
training materials. As a result, the materials produced in five countries can be rated as
`very good' and in seven countries as `good'.
112. This output should be rated as highly satisfactory.
113. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
1 country as highly satisfactory
5 countries as satisfactory
2 countries as moderately satisfactory
Output 6: Regional programme for public participation
114. This output has been dealt with in the section related to Attainment of objectives and
planned results (see paragraphs 44-49 above). As an output, it should be rated as
moderately unsatisfactory.
115. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
27
6 countries as satisfactory
1 county as unsatisfactory
1 country did not provide a rating
Output 7: National Action Plans (NAPs) to address pollution from land-based
activities
116. This output has been dealt with in the section related to Attainment of objectives and
planned results (see paragraphs 30-41 above). While the NAPs should be considered
as one of the most significant achievements of the project, this output, because of its
shortcomings in relation to completeness of a number of the NAPs, the lack of true
involvement of some key stakeholders, and the lack of formal adoption at the
appropriate level of government, should be rated only as moderately satisfactory.
117. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
4 countries as highly satisfactory
3 countries as satisfactory
1 country as moderately satisfactory
Output 8: Economic instruments for the sustainable implementation of
SAP MED
118. This output was the responsibility of the MAP Priority Actions Programme Regional
Activity Centre (PA/RAC), based in Split, Croatia. The main components of this
output included:
a) preparation of a methodology to evaluate economic instruments (EIs) to assist in
the sustainability of SAP MED, including an expert meeting to validate the
methodology;
b) proposing means for incorporating the use of EIs in the NAPs;
c) training of national experts on the application of EIs;
d) analysis of the application of EIs in selected countries;
e) a seminar on the state- of-the-art in the application of EIs;
f) an expert meeting to prepare proposals for introducing the use of EIs at national
level;
g) promotional materials on advantages of using economic instruments and on
tourism and the environment;
h) nine power point presentations on different issues related to economic instruments
and the environment;
i) a database on economic instruments (still under construction); and
j) a series of 22 pilot projects using EIs in 10 countries of the 12 participating
countries (3 projects in Albania, 3 in Algeria, 4 in Bosnia & Herzegovina, 6 in
Croatia, 1 in Lebanon, 2 in Morocco (plus one case study), 1 in Slovenia, 1 in
Syria, and 1 in Turkey).
119. Generally, more outputs were produced than envisaged in the original project
document and its revisions, and information dissemination activities and public
awareness-raising were more substantial than outlined in the project document.
28
120. An analysis of the list of participants in the meetings and training activities related to
EIs would indicate that: a) participants came from a relatively reduced number of
eligible countries and even fewer from other Parties to the Convention; and b) that
most government participants came from environment agencies and very few from
other key sectors (such as ministries of economy and national planning, or from
parliaments). In this sense, this component of the project appears to also suffer from
one important weakness of the project in general: lack of involvement of key
stakeholders that have a stronger impact on SAP MED issues than the environmental
authorities.
121. In spite of the usefulness and impact of these outputs, as stated in the Terminal Report
produced by PAP/RAC, the "creation of a sustainable financial platform for the
continued implementation of the SAP MED is a long term goal depending on the
government will and on the relevant decisions that are going to be taken." The project
has made a contribution in this direction but a lot still remains pending.
122. The conclusions and recommendations contained in the Terminal Report of PAP/RAC
are worth considering in any follow up in the area of EIs.
123. All considered, this output of the project should be rated as satisfactory.
124. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
2 countries as highly satisfactory
3 countries as satisfactory
1 county as moderately satisfactory
1 country as moderately unsatisfactory
1 country did not provide a rating
Output 9: Strategic Action Plan for the Conservation of Biological Diversity in
the Mediterranean Region
125. This output has been dealt with in the section related to Attainment of objectives and
planned results (see paragraphs 51-54 above).
126. As an output of the project, this should be rated as highly satisfactory, in view of the
quality of the SAP BIO and its formal adoption by COP13 (November 2003) of the
Barcelona Convention.
127. Nevertheless, more than two years after its formal adoption, indications are that the
implementation of the SAP BIO is not progressing at the required pace. COP14 of the
Barcelona Convention (November 2005) passed an almost generic recommendation
asking the Parties "to consider the SAP BIO as an essential tool for the conservation of
Mediterranean biodiversity and for the implementation of the SPA [Specially
Protected Areas] Protocol and to ensure the involvement and coordination of
competent national and regional institutions for its implementation".
128. COP14 was more specific in requesting the SPA/RAC, as part of the Secretariat:
29
"1.
To further integrate the SAP BIO into its programme of activities;
2.
To prepare project proposals and seek funding for the implementation of the
SAP BIO, also exploring possibilities with external donors;
3.
To seek financial support from GEF for the same purpose;
4.
To continue to cooperate with all relevant organizations, agencies and
institutions for the implementation of SAP BIO; [and]
5.
To strengthen cooperation among the various Regional Activity Centres,
particularly PAP/RAC, SPA/RAC and REMPEC, in order to carry out
integrated projects for the protection of ecosystems against pollution and to
promote marine biodiversity."
129. One key actor who participated in the development of the SAP BIO, interviewed in the
course of the preparation of this Terminal evaluation, has said: "Unfortunately, the
GEF (through its officers and consultants) imposed its views and opinions concerning
the next steps, by-passing therefore the priorities identified at country level. The
option of launching a PDF B as follow-up to the project is not in-line with the
outcomes of the project. [Taking] the example of the SAP BIO: the project allowed
building a strategic programme based on national and regional bottom-up
consultations. The Strategic programme was endorsed by the Mediterranean Ministers
in 2003. It included a set of more than one hundred clearly defined NAPs. As follow-
up, instead of starting a process of developing regional and especially national projects
built on the basis of these NAPs, the GEF pushed toward a PDF B phase oriented, as
far as biodiversity is concerned, to protected areas and fishing, without taking into
account the NAPs. PDF B means new preparation phase, and implementation would
be considered in 2008, while the countries finalised the elaboration of their NAPS in
2003."
130. The eight eligible countries that responded to the evaluation questionnaire rated this
output of the project as follows:
2 countries as highly satisfactory
3 countries as satisfactory
1 county as moderately satisfactory
1 country as moderately unsatisfactory
1 country did not provide a rating
Methodologies
131. An analysis of the project documents and outputs reveals that sound scientific and
technical information and knowledge have been incorporated in all project activities.
The project manager and the leaders of project activities are all experienced
professionals with a track record of a rigorous scientific and technical approach in
their work.
132. To the question: "Do you consider that the methodologies used for the updating of the
TDA, the identification of hot spots and sensitive areas, and the development of NAPs
were sound and effective?", five countries responded `Very much so'; and three
countries, `To some extent'.
30
133. Some of the comments provided in relation to this question are as follows:
a)
Les PANs sont le résultat d'un processus, qui à son tour nécessite plus de temps
pour l'instauration du concept et de l'approche participative et de l'intégration.
b)
For development of documents within the SAP/GEF project, a lot of
information and data were required which was not always possible to find and
some data were not available.
c)
The methodology was sound because it was based on a systematic approach.
And
d)
The methodology for updating the TDA was effective to some extent. Because
the identification of hot spots and sensitive areas had been prepared before
updating the TDA but the NAP has been developed after updating of TDA.
134. To the question: "To what extent the project outputs produced have the weight of
scientific authority/credibility, necessary to influence policy makers in your country?",
two countries responded `Very much so"; five countries, `To some extent'; and one
country, `Not at all', because "policy makers have not been influenced yet because of
lack of information".
135. One of the countries indicated that their response is `to some extent' "because of the
contradiction between the concept of calculation the base-line budget of pollution
release in the year 2003, while the estimation of pollutant release in 2010 will depend
on a field analysis approach. So, there is a kind of gap which makes the comparison
between 2003 and 2010 impossible from a scientific viewpoint, but it could be
achieved only from a logical viewpoint."
Cost-effectiveness
136. Concerning the efficiency of the project, the amount of outputs, the quality of most of
them, the number of participating countries, and the number of institutions and
individuals involved, indicate that the project has been cost effective.
137. The financial management has been very sound in spite of the fact that there was only
one financial officer assisting the project manger and that the number of transactions
was very high, involving a large and diverse number of partners and individuals.
138. The project had to undergo three revisions but these were prepared in a timely and
efficient manner and approved expeditiously. The original project time schedule of
three years had to be extended to five years but this delay was not related to poor
management. It has to do with: a) the fact that the project implementation started late
(the project manager came on board six months after the start of the project); b) a
weak original design of the project, which for some outputs meant a lack of good step
by step planning and realistic time schedules; and c) the fact that unexpected
bottlenecks were found, in particular in relation to local capacities to generate project
outputs.
139. The following table shows co-financing in cash and in-kind as it appeared in the latest
project revision document. It does not seem that the project has leveraged additional
resources directly related to the activities paid for by the project. Surely it has
leveraged additional resources in the participating countries both eligible and non-
31
eligible countries mostly in-kind (e.g. the salaries of government officers and other
stakeholders involved in the project activities), but this Terminal evaluation was not in
a position to assess the amounts involved.
32
Table 3 - Co-financing of the Project Expressed in US$
CONTRIBUTION IN CASH
6030-00-08
6030-00-71
6030-00-72
6030-00-73
6030-00-74
TOTAL
UMBRELLA
WHO/EURO
PAP/RAC
RAC CP
SPA/RAC
MED Trust Fund
570,000
120,000
60,000
45,000
40,000
835,000
FFEM
239,105
239,105
CONTRIBUTION IN KIND
Counterpart
contr.
3,000,000
3,000,000
FFEM
1,914,016
1,914,016
METAP
450,000
450,000
WHO/EURO
75,000
75,000
PAP/RAC
10,000
10,000
RAC/CP
55,000
55,000
In-kind including 150,000
150,000
UNEP
Financial planning
140. Financial controls, including reporting, and planning were effective and allowed the
project management to make informed decisions regarding the budget and ensured a
proper and timely flow of funds for the payment of satisfactory project deliverables,
all within the limits of a complex administrative system such as the UN's.
141. There was a sound management of funds, in spite of the complexities of the project
administration: 202 Miscellaneous Obligating Documents, of which 43 were MOUs
with financial and administrative implications; 211 Special Services Agreements; 23
meetings in 11 different locations; and 342 travel authorizations.
142. Savings and innovation permitted more work, at least in some of the sub-projects, in
particular in relation to training and publications in local languages. In this regard, the
project manager and the sub-project leaders are to be commended for their
commitment and dedication to ensure the best possible results.
143. The fact that the project administration was located in a UN office facilitated the use
of UN rules and regulations. On the other hand, the different system of accounting in
use at that time in the two UN offices involved with the financial administration of the
project (Athens and Nairobi) required considerable effort to ensure that the figures
matched in both locations.
144. The Fond Français pour l'Environnement Mondial (FFEM) and the Mediterranean
Environmental Technical Assistance Program (METAP) covered the costs of some of
the in-country project activities directly. The FFEM had committed an important
contribution in cash but finally only a small fraction of it was paid; instead, the FFEM
paid directly for project activities in four French-speaking countries. This created
delays in the implementation of these activities, some of which are still being carried
out.
Impact
145. The questionnaire used for this Terminal evaluation included three questions in
relation to the impact of the project.
33
146. Question 1: Please evaluate the immediate impact of the project on the
implementation of agreed actions by your country, either as part of the regional or sub-
regional efforts or as part of the national efforts. Two countries indicated `High
impact'; five countries, `Some impact'; and one country, `No impact'.
147. Some of the comments made in relation to the above question were:
a)
The immediate actions are the decision of government to construct for the first
time at the country level, the sewage water treatment plant in the main coastal
towns and measures for improvement of the management of solid waste and
clean up of contaminated sites. But there is still a lot to be done for law
enforcement for protection and wise use of natural resources in the coastal area.
b)
Le projet trace et reprend les grands soucis et problèmes et ne va pas dans les
détails. Pour la prise en charge des actions découlant d'un projet , il est
nécessaire de prendre des mesures pour inscrire l'après projet car un suivi et
une mobilisation de fonds s'imposent. Les projets ont besoins d'être
accompagnés et soutenu jusqu'à leurs réalisations.
c)
The implementation of proposed measures require some time, therefore the
impact would be not immediate.
d)
Because after the approval on our NAP to address Mediterranean pollution by
the Higher Council for Environmental Protection, it means that all concerned
ministries will take the conclusion of the NAP in their consideration during the
preparation of each ministry action plan. And
e)
There is no immediate effect of the GEF project.
148. Question 2: Has the project the potential of having longer term impacts in setting
priorities and helping in the implementation of agreed actions in your country? Three
countries responded `Very much so'; and five countries, `To some extent'.
149. Question 3: If your response to the previous question was `very much so' or `to some
extent', which would be the major `channels' in your country for longer term impact?
The responses included:
a) The improvement of environmental legislation, capacity building and financial
support from donors in the investment for sustainable use of natural resources.
b) Le projet doit s'inscrire dans la continuité et la conformité par rapport à la stratégie
nationale du pays et le plan d'action national élaboré par les pays dans le cadre de
leurs propre priorité et règlements. Il est souhaitable de ne pas limiter la période du
projet ou laisser la flexibilité de la période du projet spécifique à chaque pays.
c) Water and environment authority (on cantonal, entity and state level). Legislative
authority (parliaments on entity and state level).
d) Implementation of all measures contained in the NAP.
e) The Cabinet of Ministers. This is what we are aiming at.
f) Higher Council for Environment Protection. And
g) The GEF project has the potential of having longer term impacts in setting
priorities and helping in the implementation of agreed actions in our country if it
can be implemented by sustainable financial resources at the national and regional
level.
150. This project should be considered as a `ground-setting' activity. Practically all the
outputs were necessary requirements for an effective implementation of the SAP
34
MED. In other words, the project created or helped to improve some of the necessary
instruments, without which action would have been impossible or inefficient. Now, it
will be important to ensure that these instruments are put to work, at the national and
regional level. This is why this Terminal evaluation is putting such strong emphasis on
the follow-up to the project.
151. If the immediate follow-up is ensured, thus not losing the already gained momentum,
the project has the potential of having longer-term impacts for setting priorities and
presenting agreed actions for implementation.
152. For this to happen, it will be critical:
a)
to take the necessary steps in each country to introduce the NAPs at the right
levels of the government machinery and to promote their formal adoption
thought appropriate legal and/or regulatory instruments that clearly make
provisions for the policy, planning and financial provisions needed for the
effective implementation of the NAP; and
b)
to work towards the adoption of a legally binding SAP MED, also with the
necessary financial and execution provisions to ensure its effective
implementation by all Parties to the Barcelona Convention.
Sustainability
153. The questionnaire used for the preparation of this Terminal evaluation included eight
questions regarding sustainability.
154. Question 1. Financial sustainability: What is the likelihood that financial and
economic resources will be available in the sense that the project outcomes/benefits
will be sustained once the GEF assistance ends (resources can be from multiple
sources, such as the public and private sectors, income generating activities, and
market trends that support the project's objectives)? One country responded `Very
high'; four countries, `Some possibilities exist'; and one country, `None'.
155. The comments made in relation to this question included:
a)
The approval of the NAP by the Higher Council for Environmental Protection
means that there is a kind of commitment by all concerned ministries to
allocate a special budget for the achievement of the results of the project. And
b)
Implementation of NAP which is the major outcome/benefit of the GEF project
will not be sustainable without GEF assistance.
156. Question 2. Was the project successful in identifying and leveraging co-financing for
activities in your country or that are benefiting your country? Two countries responded
`Yes'; and five countries, `To some extent'.
157. The comments made in relation to this question included:
a)
Most of projects for the environmental protection are financed/co-financed by
foreign sources.
b)
In a developing country like ours, things keep changing.
c)
Some donors asked us to provide them with the projects in the investment
portfolio in order to make their assessment for financing. And
35
d)
The project was successful to some extent in identifying co-financing for
activities addressed by NAP in general.
158. Question 3. Socio-political sustainability: What is the likelihood that the level of
stakeholder ownership will allow for the project outcomes/benefits to be sustained?
Two countries responded `Very high'; and five countries, `Some possibilities exist'.
159. The comments made in relation to this question included:
a)
Beside the work done so far, the support for capacity building remains a
priority for sustainable management.
b)
`Very high' once convinced and given the proper justification and tools.
c)
Because the outcomes of the project indeed represent some priorities of all
concerned stakeholders. And
d)
One stakeholder and two public participation meetings were organised in the
NAP preparation phase. They were moderately successful in terms of quality
and quantity of the participation.
160. Question 4. Is there sufficient public / stakeholder awareness in support of the long
term objectives of the project? Two countries responded `Yes'; and five countries,
`Some'.
161. The comments made in relation to this question included:
a)
Both the stakeholders and the public at large need continuous work for
awareness-raising.
b)
Le public ou la société civile nécessite plus de temps pour leurs intégrations.
L'information et la sensibilisation est un paramètre important dans la réussite
du projet.
c)
Environmental issues still need to strongly be addressed to the public.
d)
All public stakeholders need more awareness in this regard.
162. Question 5. What is the likelihood that institutional and technical achievements, legal
frameworks, policies and governance structures and processes will allow for the
project outcomes/benefits to be sustained? Two countries responded `High'; and five
countries, `Some possibilities exist'.
163. The comments made in relation to this question included:
a)
The investment underway and implementation of Strategies and Action Plans
on tourism and agriculture development, biodiversity protection, environmental
action plan and steps in progress for stabilization-association will create
conditions for the sustainable continuation of the projects outcomes/benefits.
b)
It could be high if the right tools and resources were provided.
c)
The government is willing to achieve its action plan in a perfect way, since this
plan is representing one of its priorities towards sustainable development. And
d)
Investments are decided by the State Planning Organisation. The NAP was sent
to this governmental body. If this body approves the NAP and the Ministry of
36
Finance and Undersecretary of Treasury support it, investments will be able to
be applicable and sustainable.
164. Question 6. What is the relevance and applicability of the project's recommendations
to central and local authorities? Four countries responded `High'; and three countries,
`Some relevance'.
165. The comments made in relation to this question included:
a)
Everyone acknowledges the work done, but the lack of resources blocks the
wide distribution of the results to build up a national consensus on the outputs.
b)
The projects recommendations are relevant to local and central authorities
because they have been prepared with a participatory approach between all
concerned authorities. And
c)
There may be some relevance and applicability of the GEF project's
recommendations to central and local authorities as far as they are aware of
them.
166. Question 7. Ecological sustainability: What is the likelihood that project achievements
will lead to sustained ecological benefits? Three countries responded `High'; and four
countries, `Some possibilities exist'.
167. The comments made in relation to this question included:
a)
The projects have identified problems, found the root causes and define the
solutions.
b)
The project achievements should reflect a kind of ecological benefits if the
implementation of the project recommendation and outcomes were sustained.
And
c)
Some possibilities exist that the project achievements will lead to sustained
ecological benefits, if adequate and sustainable financial resources can be
found for the implementation of the GEF project.
168. Question 8. Replication and catalysis: Is the GEF project having a positive impact (or
has it the potential of having a positive impact) in the design and implementation of
other projects in your country? Three countries responded `Very much so'; and four
countries, `To some extent'.
169. The comments made in relation to this question included:
a)
Some components of the GEF project have had a positive impact in generating
other projects and also for increasing local capacities for environmental
management and improving environmental legislation.
b)
The experience and knowledge of the GEF project personnel will surely play a
crucial role in designing and implementing other projects.
c)
Some other projects may depend on the conclusion of our NAP. And
d)
Some research and development projects are being prepared in line with the
GEF project. Therefore, the GEF project is having a positive impact and it has
the potential of having a positive impact in the design and implementation of
other related projects in our country.
37
170. The continued long-term project-derived outcomes and impacts after the GEF project
funding ends is not assured. The key factor that is likely to contribute to the
persistence of benefits after the end of the project will be the ability of the MAP
system3 to:
a) provide and/or catalyze the political, technical and financial support needed in 11
of the eligible countries to make an effective use of the tools generated by the
project (Slovenia, as an EU-member country is now in a different situation). As
stated before, the new GEF project being prepared by MAP does not seem to
contribute to this type of follow-up. In addition, at least in the immediate future,
MAP does not have the technical and financial capacity do this. MAP may be
working in the direction of developing such capacity, but in the immediate future it
will be up to the UNEP/GEF to pay attention to this matter, if the Facility does not
want to see its resources (and reputation) wasted. To this end, it could be useful if
the GEF establishes a `Projects follow-up service / Project watch-dog service /';
and
b) ensure that the transboundary nature of most of the issues dealt with by the GEF
project is fully addressed, ensuring that all the Parties to the Barcelona Convention
are on board.
171. It is quite likely that financial resources, at least to a limited extent, could be found in
each of the eligible countries, and with much stronger probability in the other Parties
to the Barcelona Convention. But this will require bringing the tools generated by the
project, in particular those related to economic instruments, to the attention of the
appropriate decision-makers. It is not sure that those involved in the project will have
the necessary political clout and/or the sufficient scientific and technical prestige to do
this. Thus, a key component of the follow-up to the project will be to pay attention to
this important issue.
172. The original project included holding two donors meetings: the first one to inform
potential donors and involve them in the projects right from the planning stage, and
the second one to be held once all the pre-investment studies would have been
completed. At its meeting in November 2003, the project's Inter-Agency Steering
Committee (IASC) decided that the conditions were not suitable to call a donors
meeting, which could be considered for 2005. At its meeting in June 2004 the IACS
agreed that "a stock-taking meeting should be organized in September 2004 as an
occasion for involving donors and other partners in the process of preparation of NAPs
and pre-investment studies" and that "the donor meeting could be held subsequently as
part of the PPDF process". These meetings have never been held, thus missing the
opportunity of involving the donor community in the project development and in its
follow-up.
173. Concerning
external
investments,
including private investments, the GEF project now
being developed by MAP includes, at least in part, this issue. Nevertheless,
UNEP/GEF, working with the MAP system, should ensure that this question is
3 By the `Map system' is meant the Barcelona Convention and its Protocols, including all the Parties to these
instruments (among which is the European Community), MED POL, the Mediterranean Commission on
Sustainable Development, and the six MAP Regional Activity Centres.
38
addressed and as much as possible is well coordinated, otherwise it may happen that
either nobody deals with it in an effective manner or that a myriad of uncoordinated
initiatives emerge, since the tools generated by the project could be a tempting element
to be used by recipient countries and potential donors alike.
174. Stakeholder ownership of the outcomes of the project does not seem to be widespread
beyond the environmental agencies. This sense of ownership has to be maintained in
these agencies by starting to generate concrete results on the ground, but at the same
time this will not happen if the sense of ownership does not expand to other sectors of
government and society.
Stakeholder participation / public awareness
175. These issues were an explicit component of the project and have been dealt with in the
sections above dealing with objectives and outputs.
176. The analysis and conclusions above indicate that there is a need to continue working
on the issues related to stakeholders' participation by:
a) keeping on board those which have been involved by making room for their
participation in the application of the project outputs;
b) doing more work on the methodologies and incentives for and cost-benefit of
stakeholders' involvement, particularly in those countries with limited experience
in this field; and
c) using the outputs of the project to involve the stakeholders that so far have not
participated fully in the project activities, which should include the stakeholders
that for different reasons may not have any interest in and/or sympathy for
conservation issues.
177. Concerning public awareness, as stated by countries in their responses to the
questionnaire, this is a `life-long' process that never ends. The key question will be to
ensure that a serious (not lip-service) public awareness component is included in the
activities that should be undertaken using the project outputs. In this sense, the section
in the NAPs devoted to public participation is in great need of further development in
practically all of them.
178. The questionnaire used for the preparation of this Terminal evaluation included three
questions regarding stakeholder participation and public awareness.
179. Question 1. Do you consider that the mechanism(s) put in place by the project for
identification and engagement of stakeholders was successful? Four countries
responded `Very much so'; and three countries, `To some extent'.
180. The comments made in relation to this question included:
a) I can not elaborate. The conceptual design is full of strengths, but the real
implementation is full of weakness. Whose fault is this?
b) There is no mechanism which would `force' all stakeholders to be involved.
And
39
c) There was a kind of participatory approach, but it should also be done at the
decision-makers level.
181. Question 2. Do you consider that there was an adequate degree and effectiveness of
collaboration/interactions between the various project partners and institutions in your
country during the course of implementation of the project? Five countries responded
`Very much so'; and two countries, `To some extent'.
182. The comments made in relation to this question included:
a)
In this regard the collaboration of different actors in the project has been
effective.
b)
Yes if properly practiced and monitored by the controlling authority.
c)
There was very effective interaction, since all concerned partners were
interested in the project. And
d)
Some investments have been started for the implementation of the GEF project.
183. Question 3. Was there sufficient public awareness activities undertaken during the
course of implementation of the project in your country? Two countries responded
`Very much so'; four countries, `To some extent'; and one country, `Not al all'.
184. The comments made in relation to this question included:
a)
Most activities of such kind were organized during the project and there is no
continuous work for public awareness in particular.
b)
Yes and no. The design of the project is mentioning this, but the allocated
budget does not permit a proper addressing of any issue to the public. And
c)
We will achieve some awareness activities, but it has not yet been done.
Country ownership / driveness
185. Even with the questionnaire specifically prepared for this Terminal evaluation, it has
not been easy (and it may never be) to determine the degree of country ownership of
the project. However, all indications are that in most of the participating countries the
project managed to develop a senses of ownership among those directly involved with
the project implementation. There is no evidence that this sense of ownership was
transmitted to a wider circle of government agencies and stakeholders.
186. The project outputs at the national level also indicate that the project was effective in
catalyzing action; and that the sense of a `shared sea' that the MAP process has been
building up for 30 years has been reinforced by the opportunity afforded by the project
to the 12 countries eligible countries in particular, and to the Parties to the Barcelona
Convention in general, to advance together in the common endeavour of protecting the
Mediterranean.
187. The questionnaire used for the preparation of this Terminal evaluation included the
following question: "Do you consider that your country developed a sufficient sense of
`ownership' of the project?" Two countries responded `Very much so'; and five
countries, `To some extent'. The comments offered did not add useful information.
40
Implementation approach
188. The project implementation involved six executing offices/agencies (including the
project management team) operating on a common timetable and work plan. The main
executing agency was the Secretariat of the Barcelona Convention, usually known as
the MAP Coordinating Unit (MEDU). At MEDU, the project was managed by a three-
member team: the Project Manager, an Administrative Assistant and a secretary, under
the supervision of the MAP Coordinator. Within MEDU, MED POL made a
substantial contribution to the implementation of the project. Sub-projects were
executed by the WHO Regional Office for Europe (WHO/EURO) with personnel
based at MEDU, and three MAP Regional Activity Centres (RACs): the Priority
Actions Programme (PAP/RAC), Croatia; Specially Protected Areas (SPA/RAC),
Tunisia; and Cleaner Production (CP/RAC, Spain).
189. In spite of involving six executing offices/agencies, which in turn contracted a large
number of other agencies and individual consultants to generate project outputs, there
were no particular cases of serious conflict, and deadlines were in general met, except
when exceptional circumstances entered into play. All indications are that the project
team was able to maintain very positive working relations. The cooperation and
working relations with UNEP/DGEF were also very positive.
190. The Project Manager played a key role in the smooth running of the project, but his
Terms of Reference should have included a higher level of autonomy to take decisions
and to supervise activities and project staff, including at the country level. This should
have allowed him to play a more proactive role in all aspects of the project execution.
191. The capacity of some eligible countries to deliver represented a problem. This was the
main reason that the project had to be revised three times and that its finalization was
delayed two years.
192. To deal with this reality in a more efficient manner, the project contracted regional
consultants, but ideally the project should have contemplated in its design a more
country-tailored approach, instead of assuming that all participating countries would
have the same capacity to participate in the project. The project team should have
included two professionals whose main responsibility should have been to constantly
tour the 12 participating countries in order to provide technical and administrative
advice, supervise project activities and play a catalytic role to ensure better
involvement of key stakeholders.
193. In spite of the problems encountered, the project management was able to adapt to the
circumstances, maintain positive working relations with the participating countries,
and take the appropriate decisions at the right moment, so that at the end, and in spite
of the delay in finalizing the project, the prevailing feeling is that the project was well
run. Once more, the weak points of the project have to do more with its design than
with its implementation.
194. The project established three committees to assist in its implementation: the Inter-
Agency Steering Committee (IASC); the Coordinating Committee (CC); and the Ad
41
Hoc Technical Committee. In addition, SPA/RAC established an Advisory Committee
for the preparation of the SAP BIO.
195. The IASC was composed of the project implementing and donor agencies. It was
chaired by UNEP/MAP, as the lead-executing agency, and included representatives
from UNEP/GEF Coordination Unit, MED POL, WHO/EURO, GEF, UNDP, the
World Bank, METAP, FFEM and ICS/UNIDO. This Committee was in charge of
overseeing the general execution of the project, making decisions on how the project
had to be carried out and coordinated. The Committee met five times: in March 2001,
March 2002, January 2003, November 2003, and June 2004.
196. Attendance at meetings of the IASC was as follows: UNEP/GEF Coordination Unit,
MED POL, METAP, WHO/EURO, and ICS/UNIDO: all 5 meetings; GEF: 2
meetings; FFEM and World Bank: 1 meeting; and UNDP: none.
197. The minutes of the meetings prove that the IASC played an important role during the
implementation of the project. The minutes are informative and well written, but lack
clarity concerning the decisions taken. It would be useful if the GEF would prepare a
model for minutes of this type of committee meetings, with clear instructions to record
the decisions under each agenda item and with a clear wording (as decisions).
Decisions should also be numbered with three components: the body adopting it, the
meeting at which the decision is adopted, and the decision number (e.g. IASC/I/10, or
IASC/II/15). This would facilitate the follow-up and referencing of the decisions.
198. The Coordinating Committee (CC) was composed of the members of the Inter-Agency
Steering Committee and of GEF coordinators in the participating countries, who were
often also the MED POL National Coordinator. He/she maintained the country contact
with the GEF Project Manager and had the responsibility of organizing all the
project's activities taking place at the country level. The Coordination Committee met
three times back-to-back with the IASC: in March 2001, March 2002 and November
2003. They all were a one-day meeting.
199. Attendance at the CC meetings by eligible countries and other Parties to the Barcelona
Convention is shown in Table 4.
Table 4 Attendance at meetings of the Coordination Committee
Parties to the
Meetings
Convention
GEF-eligible
March 01 March 02 Nov. 03
Albania
X X X
Algeria
X
B
&
H
X X X
Croatia
X
X
Egypt
X X X
Lebanon X
Libya X
Morocco X X X
Slovenia X
Syria
X X X
42
Tunisia
X
X
Turkey
X X X
Non GEF-eligible
Cyprus X
EC X
X
France X
Greece X
Monaco X
Spain
X
200. The table reveals that six eligible countries (50%) attended all the three meetings; two
countries (17%) attended two meetings; and four countries (33%) only one meeting.
Five non-eligible countries which are Parties to the Convention attended one meeting
and the EC attended two meetings. In general, this can not be considered a good
attendance, especially when the CC meetings were the only occasion when all
participating countries in particular, and all Parties to the Convention more generally,
could come together to share experiences and, most importantly, to develop the sense
of the transboundary nature of many of the issues being addressed by the project.
201. It should also be pointed out that no other stakeholders (e.g. NGOs) attended any of
the CC meetings.
202. The minutes of the first meeting show almost no participation from the eligible
countries and little participation from other Parties. Ninety per cent of the minutes
record information provided by the Secretariat and other agencies.
203. The minutes of the second meeting record a more active participation by eligible
countries, but still most of the text records information provided by the project
management. The minutes of the third meeting reflect considerably more participation
from the eligible countries.
204. Judging from the minutes, it does not seem that the Coordinating Committee played a
significant role in the project. To start with, not much can be achieved in a one-day
meeting; and secondly the meetings served mainly to pass information from the
project management to the participants and to discuss administrative matters
concerning the implementation of the project. It would seem that the Coordinating
Committee should have served more to entertain substantive discussions on the issues
addressed by the project, and in particular on their transboundary dimensions.
205. The
Ad hoc Technical Committee was composed of representatives from MED POL,
WHO/EURO, METAP and its partners, GEF, FFEM and ICS/UNIDO. It was involved in the
coordination of the pre-investment studies, offering guidance to the governments. METAP
also offered technical follow-up to the execution of the studies. The Ad hoc Technical
Committee met three times in the course of the project. They were two-day meetings.
206.
The minutes of these meetings reveal that the Committee played a useful purpose.
43
207.
The SAP BIO Advisory Committee was composed of the Interim Secretariat of the Agreement
on the Conservation of Cetaceans of the Black Sea, the Mediterranean Sea and the Contiguous
Atlantic Area (ACCOBAMS); the Arab League Educational, Cultural and Scientific
Organization (ALECSO); the UN Food and Agriculture Organization (FAO); the Centre for
Mediterranean Cooperation (IUCN-The World Conservation Union); the Mediterranean
Wetlands Initiative (MedWet-Ramsar Convention on Wetlands); the Bern Convention
Secretariat; and the World Wide Fund for Nature (WWF- Mediterranean Programme). The
Committee met four times during the duration of the project.
208. It has not been possible to review the minutes of the SAP BIO Advisory Committee to
ascertain the role played by this body.
209. The project required that an Inter-Ministerial Committee (IMC) should be established
in each eligible country. The requirement was that the IMC should be composed of
high level government official authorized to make important decisions about major
national issues linked to the SAP MED. In principle, the function of the IMC was to
coordinate the work of national agencies, institutions and experts engaged in the
national SAP activities and to oversee the integration of such activities into the
national planning and decision-making processes and the national development plans.
The IMC should have also held the main responsibility for identifying the national
financial resources that will support the National Action Plans (NAPs) and secure the
sustainability of the SAP MED in the long-term. Through a Memorandum of
Understanding signed with UNEP/MAP, the eligible countries were able to receive
financial support from the project for the IMC activities.
210. Setting up the IMC in a number of countries was not an easy task. The MOUs were
ready for signature only in March 2002 (14 months after the start of the project), and
the minutes of the November 2003 meeting of the IASC indicates that five countries
had not yet set up their IMC by then. This was almost three years after the beginning
of the project, with only little more than one year to close it (by then the duration of
the project had been extended to 31 December 2004).
211. Overall, the IMC arrangement does not seem to have worked well. IMCs were finally
established in 10 countries (out of 12) and operated relatively well in seven of them.
212. Means and ways should be found to establish more effective in-country mechanisms,
which should be established as part of the project design, so that on the one hand the
participating countries can be usefully involved in the preparation of the project and,
on the other hand, have their arrangements ready to operate when the project starts.
This approach may entail advancing financial support for the establishment and
operation of the in-country arrangements, possibly through co-financing arrangements
that would be in a position to `risk' pre-investments before the GEF project is actually
approved.
213. The day-to-day running of the project at the country level was the responsibility of the
GEF National Coordinators. Problems were also encountered in relation to their
capacity to perform. The second meeting of the IASC (March 2002) addressed this
issue in some detail and it was decided that a limited amount of money should be
offered to the GEF National Coordinators to assist them to speed up project
implementation. By the next meeting of the IASC in January 2003, some Coordinators
have made use of this support, but not all.
44
214. The questionnaire used for the preparation of this Terminal evaluation included the
following question: "Do you consider the project management by UNEP-MAP was
effective and efficient and with sufficient adaptability?" Seven countries responded
`Very mucho so'. Some of the comments they offered were:
a)
UNEP-MAP provided the countries with guidelines for the preparation of
documents within the project. Meetings, workshops and consultations have
been organized during the project, so the countries had effective and efficient
assistance from UNEP-MAP.
b)
UNEP/MAP officials provided the project with the maximum help that they
could. The problems are within the countries themselves.
c)
They spared no efforts to support the preparation of the NAPs at national level.
And
d)
The technical and financial assistance given by UNEP-GEF and UNEP-MAP
were adequate in terms of the GEF project management.
Replicability
215. As indicated before, the project needs a quick follow-up to ensure that the momentum
now created is not lost, and that there is a continuation of what is still bubbling in most
countries, thus avoiding having to start anew.
216. The project, with due adaptation and using the lessons learnt in the Mediterranean,
could be quite easily replicated in other Regional Seas.
Monitoring and Evaluation
217. The project monitoring and evaluation was done through the regular meetings of the
different project committees and the progress reports and external Mid-Term Report
foreseen in the project document. All records of meetings have been well kept and are
public, and the progress reports were presented on time and reflected the true picture
of the project implementation.
218. In a project of this duration and complexity, it could have been useful, in addition to
the progress and mid-term reports, to perform external evaluations at special junctures
of the project (e.g. when key outputs were being generated and an external evaluation
could have helped to improve performance).
IV.
FOLLOW-UP
Recommendations
219. There is a great and urgent need for follow-up to ensure that the good results of this
GEF project are not wasted; but the approach should be more country-tailored with a
stronger and more effective coordination of all those involved. In particular, it will be
important to ensure that the project outputs are formally recognised at the appropriate
45
level of government in each country and introduced into the national development
agenda through the required legislation and/or regulations and with the necessary
institutional arrangements.
220. Having done so, it will be necessary to rapidly start a process to generate investments
from the national budgets and by the international financing institutions, which
unfortunately were not kept sufficiently abreast of the project development and
implementation.
221. To do the above, countries can not be left to their own. There is a need for sustained
technical and financial assistance by UNEP-GEF and UNEP-MAP. In particular it is
essential to ensure that the new GEF project being designed at present at UNEP/MAP
constitutes a true follow-up to this one and does not take a different direction, as
currently seems to be the case.
222. Since it would take some time for UNEP/MAP to develop a true capacity to provide
the assistance that the countries require in order to use the output of this GEF project,
UNEP/MAP should play the useful role of catalyzing the involvement of other
agencies with a current capacity to provide such assistance without too much delay.
223. As immediate steps, UNEP/MAP should: a) continue working with all the Parties to
the Barcelona Convention to further refine each NAP, including their review and
acceptance by the key stakeholders in government and outside government that so far
may have not been sufficiently involved in the process; and b) to start the discussions
with the Parties towards converting the SAP MED, on the basis of the new TDA, into
a legally binding instrument in the framework of the Barcelona Convention and its
Protocols.
Lessons learned
224. More care should be taken with the project design. This does not mean that the design
of GEF projects should become even more cumbersome, but rather the design should,
inter alia:
a)
differentiate clearly between `objectives' and `outputs' and be more
rigorous in their description in order to avoid creating false expectations
(e.g. not to state emphatically that an output will be approved at the
highest level of government, when in most cases this is something very
difficult to predict);
b)
establish a clear sequence between the `objectives', the `outputs'
required to achieve each one of the objectives, and all the `activities'
that should be undertaken in order to generate each one of the outputs;
c)
be more attentive to the `real world' situation of the eligible countries,
and not to assume that a GEF project will be able to change some
strongly-rooted social and institutional realities;
d)
in region-wide projects, curry out a more rigorous assessment of the
institutional and technical capacity to implement the project in each
eligible country (e.g. not assume that they are all at the same level). The
findings of this assessment should be incorporated in the project design
concerning the outputs, the activities that should lead to generate them,
46
and the implementation approach (e.g. in the project under review, the
project should have planned for more project personnel to work with
different groups of countries in the region); and
e)
foresee which key stakeholders should be involved in the project
implementation and/or in receiving and accepting the outputs, so as to
incorporate into the project an implementation approach that would
ensure the involvement of those stakeholders.
225. The eligible countries should be more actively involved in the project design, so that:
a) they can assist in making the project more attune to their needs and
capacities to deliver; and
b) they can be prompted to start considering the required and implementable
in-country mechanisms in advance of the start of the project. This would
ensure speedy implementation once the project is approved. (One key
problem of the project under review was that the key in-country
mechanism was established very late in most countries, and in some it was
never established.)
226. In the case of region-wide projects involving transboundary issues, it will be very
important:
a) to formally involve all countries in the region, including those that are not
GEF-eligible, in the design and implementation of the project. Otherwise,
the outputs of the project would be in jeopardy if not all the countries
concerned with the transbounday issues follow the same pace;
b) to ensure that there is a meaningful and effective coordination mechanism
imbedded in the project design to allow all countries to share experiences
and to develop the sense of `shared issues'; and
c) to make arrangements within the project for technical and financial bilateral
assistance within the region.
227. The duration of a project should be such as to ensure that the necessary key results are
achieved. The alternative would be to foresee the immediate follow-up that will be
required to ensure that the project outputs are put to good use. (In the case of the
project under review, the project duration did not allow time for carrying out required
processes to obtain high-level government approval of the key outputs, and no follow-
up was foreseen to do this. As a result, the long term impact of the project is
uncertain.)
228. Implementing agencies should not wait until formal approval of the project to start
looking into the selection of project personnel. If necessary, the rules should be
amended as to allow `pre-recruitment' of project personnel to ensure speedy
implementation once the project is approved. (In the case of the project under review,
the project manager came on board six month after the project had started.) In other
words, it should not be taken for granted that `this is the way that the UN/the GEF
operates...'.
47
48
Annex 1
Terminal Evaluation of the UNEP GEF project GF/ME/6030-00-08
"Determination of priority actions for the further elaboration and
implementation of the Strategic Action Programme
for the Mediterranean"
QUESTIONNAIRE ADDRESSED TO ELIGIBLE COUNTRIES
The completed questionnaire should be transmitted by e-mail to the Consultant in charge of the Terminal
Evaluation no later than 10 April 2006:
Mr.Delmar Blasco
Director
C&N - Community and Nature
c/Escoles 4
17761 Cabanes (Girona)
Spain
Tel. +34 - 972 508 589
Mobile: +34 - 669 811 965
E-mail: dablasco@wanadoo.es
IMPORTANT NOTE: All the responses to the questionnaire will be treated as confidential information by the
consultant as well as any other communication with him on this matter. The report to UNEP-GEF will be
completely anonymous.
Please write your responses or check the appropriate boxes with an "X" in the grey areas of this Word
document. The grey areas will automatically expand to fit your response.
Person responsible for filling out this questionnaire:
Full name:
Position:
Institution:
Full mailing address:
Telephone:
E-mail:
49
1. Attainment of objectives and planned results:
2. Attainment of objectives and planned results:
1.1.
To what extent do you consider that the results of this project will assist your country
in forming a joint management programme and strategy that will eventually lead to
reducing threats to the marine and coastal environment in the Mediterranean?
Very much so
To some extent Unlikely
Please elaborate if you have any comments on this issue:
1.2.
Has the project helped build consensus among stakeholders on transboundary issues
and management measures to address them?
Very much so To some extent Not at all
Please elaborate if you have any comments on this issue:
1.3 Has the project enhanced the capacity of your country and the stakeholders to identify and
set up a management framework for transboundary marine/coastal issues?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
50
3. Achievement of outputs and activities:
2.1 Taking into account the needs and expectations of your country, how successful was the
project in the delivery of the outputs and activities listed below?
Please insert in the grey box one of the following keys to evaluate each output/activity:
HS
= Highly Satisfactory
S
= Satisfactory
MS
=
Moderately
Satisfactory
MU
=
Moderately
Unsatisfactory
U
=
Unsatisfactory
HU
=
Highly
Unsatisfactory
DK
= I do not know
NA
=
Non
applicable
Updated Transboundary Diagnostic Analysis (TDA)
Pre-investment Studies for Selected Pollution Hot Spots
Analysis of Pollution Sensitive Areas
A set of Regional Guidelines and Regional Plans
Capacity building (training courses at regional and national levels)
Regional Programme for Public Participation
National Action Plans (NAPs) to address pollution from land-based
activities)
Economic Instruments for the sustainable implementation of SAP
Strategic Action Plan for the Conservation of Biological Diversity in the
Mediterranean Region
Please elaborate if you have any comments on these issues:
51
2.2 Do you consider that the methodologies used for the updating of the TDA, the
identification of hot spots and sensitive areas, and the development of NAPs were sound
and effective?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
2.3 To what extent the project outputs produced have the weight of scientific
authority/credibility, necessary to influence policy makers in your country?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
4. Impact:
3.1 Please evaluate the immediate impact of the project on the implementation of agreed
actions by your country, either as part of the regional or sub-regional efforts or as part of
the national efforts:
High impact
Some impact Ni impact
Please elaborate if you have any comments on this issue:
3.2
Has the project the potential of having longer term impacts in setting priorities and
helping in the implementation of agreed actions in your country?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
52
3.3 If your response to the previous question was "very much so" or "to some extent", which
would be the major `channels' in your country for longer term impact?
5.
Sustainability: Sustainability is understood as the probability of continued long-term
project-derived outcomes and impacts after the GEF project funding ends.
4.1 Financial sustainability. What is the likelihood that financial and economic resources will
be available in the sense that the project outcomes/benefits will be sustained once the GEF
assistance ends (resources can be from multiple sources, such as the public and private
sectors, income generating activities, and market trends that support the project's
objectives)?
Very high
Some possibilities exist None
Please elaborate if you have any comments on this issue:
4.2 Was the project successful in identifying and leveraging co-financing for activities in your
country or that are benefiting your country?
Yes To some extent Not at all
Please elaborate if you have any comments on this issue:
4.3 Socio-political sustainability: What is the likelihood that the level of stakeholder
ownership will allow for the project outcomes/benefits to be sustained?
Very high
Some possibilities exist None
53
Please elaborate if you have any comments on this issue:
4.4 Is there sufficient public / stakeholder awareness in support of the long term objectives of
the project?
Yes Some None
Please elaborate if you have any comments on this issue:
4.5 What is the likelihood that institutional and technical achievements, legal frameworks,
policies and governance structures and processes will allow for the project
outcomes/benefits to be sustained?
High
Some possibilities exist None
Please elaborate if you have any comments on this issue:
4.6 What is the relevance and applicability of the project's recommendations to central and
local authorities?
High
Some relevance None
Please elaborate if you have any comments on this issue:
4.7 Ecological sustainability. What is the likelihood that project achievements will lead to
sustained ecological benefits?
54
High Some possibilities exist None
Please elaborate if you have any comments on this issue:
4.8 Replication and catalysis. Is the GEF project having a positive impact (or has it the
potential of having a positive impact) in the design and implementation of other projects
in your country?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
5
Stakeholder participation / public awareness. Stakeholders are the individuals, groups,
institutions, or other bodies that have an interest or stake in the outcome of the GEF-
financed project. The term also applies to those potentially adversely affected by a project.
5.1 Do you consider that the mechanism(s) put in place by the project for identification and
engagement of stakeholders was successful?
Very much so
To some extent Not at all
Please elaborate briefly on the strengths and/or weaknesses of the mechanism(s)
in put place for stakeholders' involvment:
5.2 Do you consider that there was an adequate degree and effectiveness of
collaboration/interactions between the various project partners and institutions in your
country during the course of implementation of the project?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
55
5.3 Was there sufficient public awareness activities undertaken during the course of
implementation of the project in your country?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
6
Country ownership
Do you consider that your country developed a sufficient sense of "ownership" of the project?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
7
Implementation approach:
Do you consider the project management by UNEP-MAP was effective and efficient and with
sufficient adaptability?
Very much so
To some extent Not at all
Please elaborate if you have any comments on this issue:
8
Your final comments and recommendations:
56
Please summarize in a few sentences your general evaluation concerning the usefulness of
the project for your country, and if possible make recommendations for follow-up by
UNEP-GEF and or UNEP-MAP:
THANK YOU VERY MUCH FOR YOUR CONTRIBUTION.
57
Annex II: Terms of Reference for the Evaluation
TERMS OF REFERENCE
Terminal Evaluation of the UNEP GEF project
"Determination of priority actions for the further elaboration and implementation of the
Strategic Action Programme for the Mediterranean"
GF/ME/6030-00-08
PROJECT BACKGROUND AND OVERVIEW
Project background
The Contracting Parties to the Barcelona Convention adopted in 1997 a Strategic Action
Programme (SAP MED) to address pollution from land-based activities. The SAP MED
identifies the major pollution problems of the region, indicates the possible control measures,
shows the cost of such measures and establishes a work plan and timetable for their
implementation. The immediate objectives of the SAP MED which have been agreed upon
are: to protect the marine environment and coastal zones through prevention of pollution and
by reduction and, as far as possible, elimination of pollutant inputs, whether chronic or
accidental; and to develop and implement national programmes of action for the protection of
the marine environment from land based sources.
A GEF MED Project "Determination of priority actions for the further elaboration and
implementation of the Strategic Action Programme for the Mediterranean Sea"
(GF/ME/6030-00-08), whose overall objective is to improve the quality of the marine
environment in the Mediterranean Region by better shared-management of land-based
pollution, through improved international cooperation in the management of land-based
pollution of transboundary and regional significance, was approved by the GEF Council in
2000. The GEF project also aims at the preparation and adoption of a Strategic Action Plan
for Biodiversity (SAP BIO) for the Mediterranean, to serve the needs of the Specially
Protected Areas and Biodiversity Protocol.
Overall goal of the GEF MED project is to improve the quality of the marine environment in
the Mediterranean Region by better shared-management of land-based pollution through
improved international cooperation in the management of land-based pollution of
transboundary and regional significance.
Specific objectives of the GEF MED projects are:
· to complete an analysis of the transboundary importance of the 103 hot spots identified
in the TDA MED and SAP MED and finalise the priority list for intervention and
investments (an Investment Portfolio) (pre-investment studies will be conducted only
in GEF eligible countries);
· to formulate and adopt principles, approaches, measures, timetables and priorities for
action, that address each major land-based source of pollution and assist countries in
the implementation of such actions;
· to conduct pre-investment analysis of expected baseline and additional actions needed
to address the selected hot spots, and secure beneficiary country agreement to baseline
investments;
58
· to prepare and adopt at the regional level, detailed, operational guidelines for the
formulation of National Action Plans (NAPs) for the protection of the marine
environment from land-based activities;
· to assist countries to prepare, adopt at the highest level, and implement, country specific
National Action Plans based on the regionally prepared and adopted guidelines;
· to identify roles for, and ensure effective participation of non-governmental
organisations in the implementation of components of the SAP MED, and where
appropriate incorporate these into the National Action Plans;
· to finalise and adopt a comprehensive and holistic Transboundary Diagnostic Analysis;
and
· to develop and adopt a strategic action plan for biodiversity in the Mediterranean in
conformity with the provisions of the protocol on Specially Protected Areas and
Biodiversity.
Relevance to GEF Programmes
The project conforms to the GEF Operational Programme 8 "Waterbody based Operational
Programme. By contributing to this Operational Programme, the project aims at assisting the
countries in meeting their obligations under: the United Nations Convention on the Law of the
Sea (Article 2073); the Convention on Biological Diversity (Annex I of Decision 11/10 of the
Contracting Parties4); Agenda 21 (Chapter 175); the Convention for the Protection of the
Marine environment and the Coastal Region of the Mediterranean (Barcelona Convention);
the Protocol for the Protection of the Mediterranean Sea against Pollution from Land-based
Sources and Activities (LBS Protocol); the Protocol concerning Specially Protected Areas and
Biological Diversity in the Mediterranean (SPA Protocol); and the Action Plan for the
Protection of the Marine Environment and the Sustainable Development of the Coastal Areas
of the Mediterranean (MAP).
Executing Arrangements
The project was executed by the Coordinating Unit for the Mediterranean Action Plan.
The following committees were designed:
(a) Inter-Agency Steering Committee
This committee will be composed of the implementing and donor agencies. It will be
chaired by the lead implementing Agency (UNEP), in accordance with the agreed procedures
for GEF International Waters Projects and will include representatives from the UNEP GEF
Coordination Unit, UNEP/MEDU, GEF, UNDP, the World Bank, METAP and French GEF
(FFEM), as well as other donors which will participate following the donors meeting. This
committee will overview the execution of the project and will decide on the precise modalities
for execution and coordination of the activities. It is expected to meet once a year.
3 Article 207 requires states to adopt laws and regulations to prevent, reduce and control pollution of the marine environment
from land-based sources: to take measures as may be necessary to prevent, reduce and control such pollution: and to
endeavour to harmonise their policies in this connection at the appropriate regional level.
4 Annex I of Decision 11/10 of this Convention specifically relates to land-based activities.
5 Chapter 17 identifies prevention, reduction and control of degradation of the marine environment from land-based activities as
the main management related activity contributing to the goal of sustainable development of coastal and marine areas.
59
(b) Coordination Committee
This committee will be composed of the members of the Inter-Agency Steering
Committee and by nationally nominated coordinators for activities undertaken in the
framework of the project. The Coordination Committee is expected to meet once a year and
right after the meeting of the Inter-Agency Steering Committee. It will be briefed on the
progress of the activities.
(c) Ad-hoc Technical Committee
This committee will be composed of representatives from MAP/MED POL, WHO/MAP,
METAP and its partners, GEF and FFEM. It will be responsible for the coordination of the
implementation of the pre-investment studies and METAP will follow up technically the
implementation of these studies. However, the pre-investment studies will be undertaken
under the responsibility of the Governments themselves with the guidance of the Ad-hoc
Technical Committee.
MAP will have the overall responsibility of the management of the project. However, in
the execution it will be assisted by MED POL and its Regional Activity Centres. More
specifically, MED POL in cooperation with WHO/MAP will be the main actor in the
implementation of the project having the responsibility for pollution hot spots and sensitive
areas, pre-investment studies, preparation of regional action plans and guidelines and most of
the training programmes. PAP/RAC will have the responsibility of the component on the
sustainability of SAP MED and the organisation of the training programme on integrated
coastal zone management. SPA/RAC will be responsible for the preparation of the SAP on
biodiversity (see below for the details of the mechanisms for the implementation of this
activity). CP/RAC will assist in the preparation and adoption of three regional guidelines, one
regional plan and the organisation of one training programme.
METAP as a member of the Ad-hoc Technical Committee will be responsible for
following the pre-investment studies and advising the Government accordingly. The World
Bank is expected to cooperate in the preparation of the economic instruments.
Project Outputs
The initial project duration was 36 months starting July 1999, which was later revised and
extended to 76 months ending October 2005.
The GEF MED project consists of numerous activities, which are directed towards the
elimination of pollution hot spots, the preparation of regional guidelines and plans for the
elimination and pollution control, the capacity building, the creation of a sustainable financial
platform for the continued implementation of the SAP MED, the preparation of a regional
programme for public participation, the preparation, adoption and implementation of National
Action Plans (NAP) specifically related to the elimination of pollution from land-based
activities, and the preparation of SAP BIO.
The main outputs of the project are:
· Updated Transboundary Diagnostic Analysis
· Pre-investment Studies for Selected Pollution Hot Spots
60
· Analysis of Pollution Sensitive Areas
· A set of Regional Guidelines and Regional Plans
· Capacity building (training courses at regional and national levels)
· Regional Programme for Public Participation
· National Action Plans (NAPs) to address pollution from land-based activities
· Economic Instruments for the sustainable implementation of SAP
· Strategic Action Plan for the Conservation of Biological Diversity in the Mediterranean
Region
Budget
The total budget was US$ 11,994,000, with US$ 6,069,000 funded by the GEF Trust Fund
and co-funding from; Mediterranean Trust Fund US$ 875,000, Other counterpart contribution
US$ 3,000,000, FFEM US$ 1,600,000, and METAP US$ 450,000.
TERMS OF REFERENCE FOR THE EVALUATION
1. Objective and Scope of the Evaluation
The objective of this terminal evaluation is to examine the extent and magnitude of any
project impacts to date and determine the likelihood of future impacts. The evaluation will
also assess project performance and the implementation of planned project activities and
planned outputs against actual results. The evaluation will focus on the following main
questions:
1) What is the extent of the policy implications for the developed Strategic Action
Programmes and National Action Plans in assisting the countries in the region to
establish the necessary policy framework and to implement the agreed action of
transboundary concerns? To what extent have the specific needs of the target
groups of stakeholders been considered in the process?
2) To what extent has the project successfully updated the Transboundary Diagnostic
Analysis in order to identify and distinguish transboundary issues, develop clearly
prioritised `hot spot' lists and to use the TDA information to develop the SAPs?
3) What is the extent of the effect of the project in enhancing the capacity of the
participating countries in identifying and determining priorities for action through
the development of SAPs and NAPs.
2. Project Ratings
The success of project implementation will be rated on a scale from `highly unsatisfactory' to
`highly satisfactory'. In particular the evaluation shall assess and rate the project with respect
to the eleven categories defined below:4
1. Attainment of objectives and planned results:
The evaluation should assess the extent to which the project's major relevant
objectives were effectively and efficiently achieved or are expected to be
achieved and their relevance.
· Effectiveness: Evaluate how, and to what extent, the stated project
objectives have been met, taking into account the "achievement
4 However, the views and comments expressed by the evaluator need not be restricted to these items.
61
indicators" in the project logframe / project document. In particular,
evaluate whether and to what extent the results of this project will assist
the countries in forming a joint management programme and strategy
that will eventually lead to reducing threats to the marine and coastal
environment. The analysis of impact and outcomes achieved should
include, inter alia, an assessment of the extent to which the project has
(1) built consensus among stakeholders on transboundary issues and
management measures to address them and; (2) enhanced the capacity
of the countries and stakeholders in identifying and setting up
management framework for, transboundary marine/coastal issues.
· Relevance: In retrospect, were the project's outcomes consistent with
the focal areas/operational program strategies? The evaluation should
ascertain the nature and significance of the contribution of the project
outcomes to the wider portfolio of marine/coastal projects in GEF
Operational Programme 8.
· Efficiency: Include an assessment of outcomes in relation to inputs,
costs, and implementation times based on the following questions: Was
the project costeffective? How does the cost-time vs. outcomes
compare to other similar projects? Was the project implementation
delayed?
2. Achievement of outputs and activities:
· Delivered outputs: Assessment of the project's success in producing
each of the programmed outputs, both in quantity and quality as well as
usefulness and timeliness.
· Assess the soundness and effectiveness of the methodologies used for
the updating of the TDA, identification of hot spots and sensitive areas,
development of SAPs and NAPs, identification of objectives, targets
and costs for agreed action.
· Assess to what extent the project outputs produced have the weight of
scientific authority / credibility, necessary to influence policy makers.
3. Cost-effectiveness:
Cost-effectiveness assesses the achievement of the environmental and
developmental objectives as well as the project's outputs in relation to the
inputs, costs, and implementing time. It also examines the project's compliance
with the application of the incremental cost concept. The evaluation will:
· Assess the cost-effectiveness of the activities of the project which were
funded by GEF and whether these activities achieved the goals and
objectives within planned and/or reasonable time and budget. How did
the costs compare to the costs of similar projects in similar contexts?
· Assess the contribution of cash and in-kind co-financing to project
implementation and to what extent the project leveraged additional
resources.
· Determine the extent to which scientific and technical information and
knowledge have been incorporated within, and have influenced the
execution of, the project activities.
·
4. Financial Planning
62
Evaluation of financial planning requires assessment of the quality and
effectiveness of financial planning and control of financial resources
throughout the project's lifetime. Evaluation includes actual project costs by
activities compared to budget (variances), financial management (including
disbursement issues), and co- financing. The evaluation should:
· Assess the strength and utility of financial controls, including reporting,
and planning to allow the project management to make informed
decisions regarding the budget and allow for a proper and timely flow
of funds for the payment of satisfactory project deliverables.
· Present the major findings from the financial audit if one has been
conducted.
· Identify and verify the sources of co- financing as well as leveraged and
associated financing (in co-operation with the IA and EA).
· Assess whether the project has applied appropriate standards of due
diligence in the management of funds and financial audits.
· The evaluation should also include a breakdown of final actual costs
and co-financing for the project prepared in consultation with the
relevant UNON/DGEF Fund Management Officer of the project (table
attached in Annex 1 Co-financing and leveraged resources).
5. Impact:
· Evaluate the immediate impact of the project on the implementation of
agreed actions by the countries in the region, either collectively on a
regional or sub-regional scale or individually.
· As far as possible, also assess the potential longer-term impacts of
setting priorities and presenting agreed actions for implementation,
considering that the evaluation is taking place upon completion of the
project and that longer term impact is expected to be seen in a few
years time. Frame recommendations to enhance future project impact in
this context. Which will be the major `channels' for longer term
impact? The evaluation should formulate recommendations that outline
possible approaches and necessary actions to facilitate an impact
assessment study in a few years time.
6. Sustainability:
Sustainability is understood as the probability of continued long-term project-
derived outcomes and impacts after the GEF project funding ends. The
evaluation will identify and assess the key conditions or factors that are likely
to contribute or undermine the persistence of benefits after the project ends.
Some of these factors might be outcomes of the project, i.e. stronger
institutional capacities, legal frameworks, socio-economic incentives / or
public awareness. Other factors will include contextual circumstances or
developments that are not outcomes of the project but that are relevant to the
sustainability of outcomes. The evaluation should ascertain to what extent
follow-up work has been initiated and how project outcomes will be sustained
and enhanced over time.
Five aspects of sustainability should be addressed: financial, socio-political,
institutional frameworks and governance, ecological (if applicable), and
63
replication5. The following questions provide guidance on the assessment of
these aspects:
· Financial resources. What is the likelihood that financial and economic
resources will be available such as the project outcomes/benefits will
be sustained once the GEF assistance ends (resources can be from
multiple sources, such as the public and private sectors, income
generating activities, and market trends that support the project's
objectives)? Was the project was successful in identifying and
leveraging co-financing.
· Socio-political: What is the likelihood that the level of stakeholder
ownership will allow for the project outcomes/benefits to be sustained?
Is there sufficient public / stakeholder awareness in support of the long
term objectives of the project
· Institutional framework and governance. What is the likelihood that
institutional and technical achievements, legal frameworks, policies and
governance structures and processes will allow for the project
outcomes/benefits to be sustained? What is the relevance and
applicability of the project's recommendations to federal and local
authorities? While responding to these questions consider if the
required systems for accountability and transparency and the required
technical know how are in place.
· Ecological. The analysis of ecological sustainability may prove
challenging. What is the likelihood that project achievements will lead
to sustained ecological benefits?
· Replication and catalysis. What examples are there of replication and
catalytic outcomes that suggest increased likelihood of sustainability?
Replication approach, in the context of GEF projects, is defined as
lessons and experiences coming out of the project that are replicated or
scaled up in the design and implementation of other projects.
Replication can have two aspects, replication proper (lessons and
experiences are replicated in different geographic area) or scaling up
(lessons and experiences are replicated within the same geographic area
but funded by other sources).
7. Stakeholder participation / public awareness:
This consists of three related and often overlapping processes: information
dissemination, consultation, and "stakeholder" participation. Stakeholders are
the individuals, groups, institutions, or other bodies that have an interest or
stake in the outcome of the GEF- financed project. The term also applies to
those potentially adversely affected by a project. The evaluation will
specifically:
· Assess the mechanisms put in place by the project for identification and
engagement of stakeholders and establish, in consultation with the
stakeholders, whether this mechanism was successful, and identify its
strengths and weaknesses.
5 Replication refers to repeatability of the project under quite similar contexts based on lessons and experience
gained. Actions to foster replication include dissemination of results, seminars, training workshops, field visits to
project sites, etc. GEF Project Cycle, GEF/C.16/Inf.7, October 5, 2000
64
· Assess the degree and effectiveness of collaboration/interactions
between the various project partners and institutions during the course
of implementation of the project.
· Assess the degree and effectiveness of any various public awareness
activities that were undertaken during the course of implementation of
the project.
8. Country ownership / driveness:
This is the relevance of the project to national development and environmental
agendas, recipient country commitment, and regional and international
agreements. The evaluation will:
· Assess the level of country ownership. Specifically, the evaluator
should assess whether the project was effective in catalyzing action
taken by the countries in the Mediterranean.
9. Implementation approach:
This includes an analysis of the project's management framework, adaptation
to changing conditions (adaptive management), partnerships in implementation
arrangements, changes in project design, and overall project management. The
evaluation will:
· Ascertain to what extent the project implementation mechanisms
outlined in the project document have been closely followed. In
particular, assess the role of the various committees established and
whether the project document was clear and realistic to enable effective
and efficient implementation, whether the project was executed
according to the plan and how well the management was able to adapt
to changes during the life of the project to enable the implementation of
the project.
· Evaluate the effectiveness and efficiency and adaptability of project
management and the supervision of project activities / project execution
arrangements at all levels.
· Assess the effectiveness of supervision and administrative and financial
support provided by UNEP/DGEF.
· Identify administrative, operational and/or technical problems and
constraints that influenced the effective implementation of the project.
· Assess whether the logical framework was used during implementation
as a management tool and whether feedback from M&E activities more
broadly was used for adaptive management.
10. Replicability:
· Assess whether the project has potential to be replicated, either in terms of
expansion, extension or replication in other countries and/or regions and
whether any steps have been taken by the project to do so and the
relevance and feasibility of these steps.
11. Monitoring and Evaluation:
· The evaluation shall include an assessment of the quality, application and
effectiveness of project monitoring and evaluation plans and tools,
including an assessment of risk management based on the assumptions and
risks identified in the project document. The evaluation shall comment on
how the monitoring mechanisms were employed throughout the project's
lifetime and whether this allowed for tracking of progress towards project
65
objectives and how the project responded to the challenges identified
through these mechanisms. The tools used might include a baseline, clear
and practical indicators and data analysis systems, or studies to assess
results that were planned and carried out at specific times in the project.
The ratings will be presented in the form of a table. Each of the eleven categories should be
rated separately with brief justifications based on the findings of the main analysis. An
overall rating for the project should also be given. The following rating system is to be
applied:
HS
=
Highly
Satisfactory
S
= Satisfactory
MS
=
Moderately
Satisfactory
MU
=
Moderately
Unsatisfactory
U
=
Unsatisfactory
HU
=
Highly
Unsatisfactory
3. Methods
This terminal evaluation will be conducted as an in-depth evaluation using a participatory
approach whereby the UNEP/DGEF Task Manager, key representatives of the executing
agencies and other relevant staff are kept informed and regularly consulted throughout the
evaluation. The consultant will liaise with the UNEP/EOU and the UNEP/DGEF Task
Manager on any logistic and/or methodological issues to properly conduct the review in as
independent a way as possible, given the circumstances and resources offered. The draft
report will be circulated to UNEP/DGEF Task Manager, key representatives of the executing
agencies and the UNEP/EOU. Any comments or responses to the draft report will be sent to
UNEP / EOU for collation and the consultant will be advised of any necessary revisions.
The findings of the evaluation will be based on the following:
1. A desk review of project documents including, but not limited to:
(a) The project documents, outputs, monitoring reports (such as progress and
financial reports to UNEP and GEF annual Project Implementation Review
reports) and relevant correspondence.
(b) Review of specific products including laboratory reports and assessments,
reports and publications, targeted information products and deliberations of the
meetings of the Contracting Parties to the Barcelon Convention and
recommendations related to wider adoption of the results of the project.
(c) Notes from the Steering Committee and other meetings.
Relevant material published on web-sites maintained by the MAP and its Regional Activity
Centres and other partners. As appropriate, these interviews could be combined with an email
questionnaire.
2. Interviews with project management (such as the Project Coordinator, the Executive
Secretary of MAP, selected Director of Regional Activity Centres, representatives of
involved countries and members of the Steering Committee).
66
3. Interviews and telephone interviews with other stakeholders, including NGOs which
participated in the project. As appropriate, these interviews could be combined with an
email questionnaire.
4. The Consultant shall determine whether to seek additional information and opinions
from representatives of donor agencies and other organisations (e.g. FFEM, METAP)
by e-mail or through telephone communication.
5. Interviews with the UNEP/DGEF project task manager and Fund Management Officer,
and other relevant staff in UNEP dealing with International Waters related activities as
necessary. The Consultant shall also gain broader perspectives from discussions with
relevant GEF Secretariat staff.
4. Evaluation report format and review procedures
The report should be brief, to the point and easy to understand. It must explain; the purpose of
the evaluation, exactly what was evaluated and the methods used. The report must highlight
any methodological limitations, identify key concerns and present evidence-based findings,
consequent conclusions, recommendations and lessons. The report should be presented in a
way that makes the information accessible and comprehensible and include an executive
summary that encapsulates the essence of the information contained in the report to facilitate
dissemination and distillation of lessons.
Evidence, findings, conclusions and recommendations should be presented in a complete and
balanced manner. Dissident views in response to evaluation findings may be appended in an
annex. The evaluation report shall be written in English, be of no more than 50 pages
(excluding annexes), use numbered paragraphs and include:
i)
An executive summary (no more than 3 pages) providing a brief overview of
the main conclusions and recommendations of the evaluation;
ii)
Introduction and background giving a brief overview of the evaluated
project, for example, the objective and status of activities;
iii)
Scope, objective and methods presenting the evaluation's purpose, the
evaluation criteria used and questions to be addressed;
iv)
Project Performance and Impact providing factual evidence relevant to the
questions asked by the evaluator and interpretations of such evidence;
v)
Conclusions and rating of project implementation success giving the
evaluator's concluding assessments and ratings of the project against given
evaluation criteria and standards of performance. The conclusions should
provide answers to questions about whether the project is considered good or
bad, and whether the results are considered positive or negative;
vi)
Lessons learned presenting general conclusions from the standpoint of the
design and implementation of the project, based on established good and bad
practices. Lessons must have the potential for wider application and use, and
the context in which lessons may be applied should be specified; Lessons
learned, should be explored mainly beyond project design and management
issues and also incorporate possible technical aspects such as effectiveness of
technical methodologies, scope and buy-in of stakeholder participation.
vii)
Recommendations suggesting actionable proposals regarding improvements
of current or future projects. The evaluator shall make recommendations that
may (1) enhance the likelihood of further project impacts beyond the life of the
67
project, (2) contribute to the assessment and development of GEF's portfolio
of marine/coastal environment related projects and (3) ensure linkages and
synergies between on-going and new marine and coastal environment related
projects implemented by UNEP. Recommendations should always be specific
in terms of who would do what and provide a suggested timeframe;
viii) Annexes include a breakdown of final actual costs and co-financing for the
project prepared in consultation with the relevant UNON/DGEF Fund
Management Officer of the project (table attached in Annex 1 Co-financing
and leveraged resources); terms of reference, list of interviewees, and so on.
The scope of the evaluation is as specified in the "Global Environment Facility Guidelines for
Implementing Agencies to conduct Terminal Evaluations, May 2003" 6 to evaluate the
activities supported by GEF through this project. The "achievement" indicators provided in
the log frame of the project document should be used together with the evaluation parameters
described.
Examples of UNEP GEF Terminal Evaluation Reports are available at www.unep.org/eou
Review of the Draft Evaluation Report
Draft reports submitted to UNEP EOU are shared with the corresponding Programme or
Project Officer and his or her supervisor for initial review and consultation. The DGEF staff
and senior Executing Agency staff are allowed to comment on the draft evaluation report.
They may provide feedback on any errors of fact and may highlight the significance of such
errors in any conclusions. The consultation also seeks agreement on the findings and
recommendations. UNEP EOU collates the review comments and provides them to the
evaluators for their consideration in preparing the final version of the report.
Quality Assessment of the Evaluation Report
All UNEP GEF Terminal Evaluation Reports are, themselves, subject to quality assessments
by the GEF independent Office of Evaluation (GEF OE). UNEP EOU therefore applies these
GEF OE quality assessment criteria and the GEF Minimum Requirements for Terminal
Evaluations to the draft Terminal Report as a tool for providing structured feedback.
The quality of the draft evaluation report will be assessed and rated against the following
criteria:
Report Quality Criteria
UNEP EOU Assessment notes Rating
A. Did the report present an assessment of relevant
outcomes and achievement of project objectives in the
context of the focal area program indicators if applicable?
B. Was the report consistent and the evidence complete
and convincing and were the ratings substantiated when
used?
C. Did the report present a sound assessment of
sustainability of outcomes?
D. Were the lessons and recommendations supported by
the evidence presented?
E. Did the report include the actual project costs (total
and per activity) and actual co-financing used?
F. Did the report include an assessment of the quality of
the project M&E system and its use for project
6 http://www.gefweb.org/MonitoringandEvaluation/MEPoliciesProcedures/MEPTools/IA_Guidelines_for_TE.pdf
68
management?
Rating system for quality of terminal evaluation reports
A number rating 1-6 is used for each criterion: Highly Satisfactory = 6, Satisfactory = 5, Moderately
Satisfactory = 4, Moderately Unsatisfactory = 3, Unsatisfactory = 2, Highly Unsatisfactory = 1, and unable to
assess = 0.
A score for the quality of the terminal evaluation report is calculated by applying the GEF OE formula as
follows:
Quality of the TE report = 0.3*(A + B) + 0.1*(C+D+E+F)
The total is rounded and converted to the scale of HS to HU
Quality of the TE report = Moderately Unsatisfactory
General comments on the draft report with respect to compliance with these TOR will also be
compiled and shared with the evaluation team.
5. Submission of Final Terminal Evaluation Reports.
The final report shall be submitted in electronic form in MS Word format and should be sent
to the following persons:
Segbedzi Norgbey, Chief, Evaluation and Oversight Unit
UNEP, P.O. Box 30552
Nairobi,
Kenya
Tel.:
(254-20)
624181
Fax: (254-20) 623158
Email: segbedzi.norgbey@unep.org
With a copy to:
Olivier Deleuze, Officer in Charge
UNEP/Division of GEF Coordination
P.O. Box 30552
Nairobi,
Kenya
Tel: + 254-20-624166
Fax: + 254-20-624041/4042
Email:
olivier.deleuze@unep.org
Takehiro Nakamura
UNEP/GEF Task Manager
United Nations Environment Programme (UNEP)
Division of GEF Coordination (DGEF)
PO Box 30552
Nairobi, Kenya
Tel: 254 20 623886
Fax: 254 20 624041
Email: takehiro.nakamura@unep.org
The final evaluation report will be printed in hard copy and published on the Evaluation and
Oversight Unit's web-site www.unep.org/eou. Subsequently, the report will be sent to the
GEF OME for their review, appraisal and inclusion on the GEF website.
69
5. RESOURCES AND SCHEDULE OF THE EVALUATION
This final evaluation will be undertaken by an international evaluator contracted by the
Evaluation and Oversight Unit, UNEP. The contract for the evaluator will begin on 1st March
2006 and end on 16th May 2006 and will entail 22 days spread over 11 weeks (4 days of
travel, to Athens, and 18 days desk study). The evaluator will submit a draft report on 30th
April to UNEP/EOU, the UNEP/DGEF Task Manager, and key representatives of the
executing agencies. Any comments or responses to the draft report will be sent to UNEP /
EOU for collation and the consultant will be advised of any necessary revisions. Comments to
the final draft report will be sent to the consultant by 8th May 2006 after which, the consultant
will submit the final report no later than 16th May 2006.
The evaluator will, after an initial telephone briefing with EOU and UNEP/GEF, travel to the
MAP in Athens and meet with project staff at the beginning of the evaluation.
In accordance with UNEP/GEF policy, all GEF projects are evaluated by independent
evaluators contracted as consultants by the EOU. The evaluators should have the following
qualifications:
The evaluator should not have been associated with the design and implementation of the
project. The evaluator will work under the overall supervision of the Chief, Evaluation and
Oversight Unit, UNEP. The evaluator should be an international expert in Oceanography or
coastal management and have the following minimum qualifications: (i) experience in
transboundary marine environmental management; (ii) experience with management and
implementation of projects and in particular with policy framework development; (iii)
experience with project evaluation. Knowledge of UNEP programmes and GEF activities is
desirable. Field experience in the Mediterranean and knowledge of marine and coastal issues
in the region an advantage. Fluency in oral and written English is a must, and knowledge of
other Mediterranean languages an asset.
6. SCHEDULE OF PAYMENT
The evaluator will receive an initial payment of 40% of the total amount due upon signature
of the contract. Final payment of 60% will be made upon satisfactory completion of work.
The fee is payable under the individual SSAs of the evaluator and is NOT inclusive of all
expenses such as travel, accommodation and incidental expenses. Ticket and DSA will be
paid separately.
In case, the evaluator cannot provide the products in accordance with the TORs, the
timeframe agreed, or his products are substandard, the payment to the evaluator could be
withheld, until such a time the products are modified to meet UNEP's standard. In case the
evaluator fails to submit a satisfactory final product to UNEP, the product prepared by the
evaluator may not constitute the evaluation report.
70
Annex 1. Co-financing and Leveraged Resources
Co-financing
IA own
Government
Other*
Total
Total
Financing
Disbursement
Co financing
(mill US$)
(mill US$)
(mill US$)
(mill US$)
(mill US$)
(Type/Source)
Planne
Actual
Planned
Actual
Planned
Actual
Planned Actual
Planned
Actual
d
-
Grants
-
Loans/Concessional
(compared to market
rate)
-
Credits
-
Equity investments
-
In-kind support
-
Other (*)
-
-
-
-
-
Totals
* Other is referred to contributions mobilized for the project from other multilateral agencies, bilateral development cooperation agencies, NGOs, the private sector
and beneficiaries.
71
Annex 2
Evaluation Ethics (from the UN Evaluation Group Norms and Standards for evaluation)
Selected Norms
Evaluators must have personal and professional integrity.
Evaluators must respect the right of institutions and individuals to provide information in confidence and
ensure that sensitive data cannot be traced to its source. Evaluators must take care that those involved in
evaluations have a chance to examine the statements attributed to them.
Evaluators must be sensitive to beliefs, manners and customs of the social and cultural environments in
which they work.
In light of the United Nations Universal Declaration of Human Rights, evaluators must be sensitive to and
address issues of discrimination and gender inequality.
Evaluations sometimes uncover evidence of wrongdoing. Such cases must be reported discreetly to the
appropriate investigative body. Also, the evaluators are not expected to evaluate the personal
performance of individuals and must balance an evaluation of management functions with due
consideration for this principle.
Selected Standards
· Evaluations should be carried out in a participatory and ethical manner and the welfare of the
stakeholders should be given due respect and consideration (human rights, dignity and fairness).
Evaluations must be gender and culturally sensitive and respect the confidentiality, protection of
source and dignity of those interviewed.
· Evaluation procedures should be conducted in a realistic, diplomatic, cost-conscious and cost-
effective manner.
· Evaluations must be accurate and well-documented and deploy transparent methods that provide
valid and reliable information. Evaluation team members should have an opportunity to disassociate
themselves from particular judgments and recommendations. Any unresolved differences of opinion
within the team should be acknowledged in the report.
· Evaluations should be conducted in a complete and balanced manner so that the different
perspectives are addressed and analysed. Key findings must be substantiated through triangulation.
Any conflict of interest should be addressed openly and honestly so that it does not undermine the
evaluation outcome.
Evaluators should discuss, in a contextually appropriate way, those values, assumptions, theories,
methods, results, and analyses that significantly affect the interpretation of the evaluative findings.
These statements apply to all aspects of the evaluation, from its initial conceptualization to the
eventual use of findings.
· The rights and well-being of individuals should not be affected negatively in planning and carrying
out an evaluation. This needs to be communicated to all persons involved in an evaluation, and its
foreseeable consequences for the evaluation discussed.
Full details from:
UNEG Norms and Standards. http://www.unep.org/eou/Pdfs/Norms.doc
72