Program Performance Indicators for

GEF International Waters Programs














02/02/04






GEF Monitoring & Evaluation Unit


This paper was prepared by the GEF Monitoring & Evaluation Unit with important contributions from Gareth Porter, Aaron Zazueta, Jarle Harstad, Andrew Hudzon, Marea Hatziolos, Alfred Duda, Juha Uitto, Andrea Merla, John Pernetta, Laurence Mee,Alexis Maimov and Vladimir Mamaev.

Table of Contents



Abstract……………………………………………………………………………...4


Glossary……………………………………………………………………………...5


List of Acronyms……………………………………………………………….........7


I. The Conceptual Framework………………………………………………….8


II. Coverage Indicators………………………………………………………...13


III. Process Outcome Indicators……………………………………………….16


IV. Indicators for Stress Reduction……..……………………………………31


V. Indicators for Environmental Status……………………………….…….34


  1. Indicators for Results of Scientific Assessments……………………….35


VII. Presentation of Process and Stress Reduction Indicator Data……..….36


Annex 1…………………………………………………………………………. 38


Annex 2………………………………………………………………………….. 40





List of Tables


Table 1: Process Outcome Indicators for TDAs………………………………..21


Table 2: Process Outcome Indicators for SAPs…………………………………23


Table 3: Process Outcome Indicators for Joint Institutional Arrangements….25


Table 4: Process Outcome Indicators for Inter-ministry Committees………….27


Table 5: Indicators for Obtaining Financing for Investment Needs…………….28


Table 6: Indicators for Demonstration Activities..……………………………….30


Table 7: Indicators for Monitoring of Stress Reduction ………………………...33


Table 8: Indicators of Success in Stress Reduction……………………………….33

..

Table 9: Indicators of Monitoring and Reporting on Changes in Environmental Status…………………………………………………………………………………35


Table 10: Indicators for Results of Scientific Assessments……………………….36




Abstract



This paper presents a proposed system of program performance indicators for the three GEF Operational Programs in the International Waters focal area—OPs 8, 9 and 10. This scheme will be applied for a trial period, in order to test its usefulness and to make the changes necessary to best serve the needs of the GEF International Waters programs.

The proposed system includes coverage indicators, process indicators, stress reduction indicators and environmental indicators, as well as indicators of results of scientific assessments and knowledge management activities.


Coverage indicators measure how much of the worldwide problem is covered by GEF projects and how well they are distributed geographically and functionally. Coverage indicators for OP 8 and OP 9 are the same, but they are different from the coverage indicators for OP 10. Process indicators measure the quality and effectiveness of the multilateral and single-country actions supported by the GEF. Stress reduction indicators measure the degree of change, associated with the GEF project, in human activities or their proxies that cause environmental stress. Environmental status indicators measure changes in actual environmental quality. All three OPs will use the same set of process indicators. However, the indicators system recognizes that each of the three Operational Programs involve different circumstances and therefore should not be compared directly with one another in terms of project results. No sets of specific stress reduction and environmental status indicators are proposed in this system. Instead the system proposes to measure progress toward implementing a system of monitoring and reporting regarding stress reduction and environmental status and reporting on how well projects that have adopted stress reduction and environmental status targets have done in achieving the targets. Indicators on GEF international waters scientific assessments and knowledge management activities focus on the quality, accessibility and usefulness of the knowledge generated by these activities to policymakers and relevant stakeholders.

The data gathered on the basis of the sets of indicators provided for various purposes can displayed based on three distinctions: (a) the Operational Program; (b), whether the data is aggregated or compared; and (c) type of ecosystem.


Glossary


baseline: The situation that existed at the beginning of a project, defined in terms of inter-governmental institutional arrangements, human activities that degrade the environment or environmental status.


coverage indicator: A measure of the adequacy of the scope and distribution of projects in the Operational Programs of a GEF focal area.


geographically-based approach: A strategy for a given region involving a series of international waters projects over time aimed at building capacity and political commitment for intergovernmental cooperation and implementation of agreed policy and institutional reforms; also known as programmatic approach.


inter-ministry committee: A mechanism representing all relevant government agencies formed to decide on government policy and regulatory reforms in response to a Transboundary Diagnostic Analysis and/or Strategic Action Programme.


joint institutional arrangement : Intergovernmental mechanism established to coordinate or harmonize government actions in regard to a waterbody.


joint process indicator: Measure of effectiveness of intergovernmental actions.


process indicator: Measure of effectiveness of a multilateral or single-country actions.


Large Marine Ecosystem: A large region of ocean encompassing coastal areas from river basis and estuaries to the seaward boundary of continental shelves and the seaward margins of coastal current systems.


Operational Program: A conceptual and planning framework for the design, implementation and coordination of a set of projects to achieve global environmental objectives in a particular focal area.


programmatic approach: A strategy for a given region involving a series of international waters projects over time aimed at building capacity and political commitment for intergovernmental cooperation and implementation of agreed policy and institutional reforms; also known as geographically-based approach.


project impact indicator: Measure of results of a project.


project implementation indicator: Measure of progress in carrying out planned tasks in a project.


replication: Conscious application in project design of an approach proven to be successful in another project.


indicator: Tool for measurement of process quality or results

stress reduction: Reduction in the level of human activity that negatively affects environmental quality.


Strategic Action Programme: An agreement among participating countries on actions needed to resolve priority threats to international waters, including actions for the national benefit of each country, actions addressing trans-boundary issues and institutional mechanisms at regional and national level for implementation of those actions.


Trans-boundary Diagnostic Analysis: A scientific-technical assessment by which environmental issues affecting international waters in a region are identified and quantified, their causes analyzed and their impacts assessed, and the main actions needed to improve the problem are identified.


List of Acronyms


GEF: Global Environment Facility


IMC: Inter-ministry committee


IW: International Waters


JIA: Joint Institutional Arrangements


LME: Large Marine Ecosystem


SAP: Strategic Action Programme


TDA: Trans-boundary Diagnostic Analysis


UNEP: United Nations Environment Programme


I. The Conceptual Framework



      A. Why Performance Indicators are Needed

    1. A system of performance indicators is needed at the program level to assess as accurately as possible what has been accomplished in the aggregate by the International Waters projects in each Operational Program. Such indicators should also serve as a valuable tool for strategic planning by revealing patterns of results that had not previously been evident. The creation of such a system of program performance indicators implies a degree of uniformity in the selection of indicators at the project level, insofar as projects in different waterbodies have common characteristics, whether in regard to the process assisted by the GEF or the nature of the threats to the waterbodies.

    2. Program level indicators provide the basis for collecting data at the project level, which can then be both aggregated and compared across all relevant projects. They should eventually allow the GEF to view snapshots of the entire portfolio or of relevant subgroups of the portfolio of IW projects showing the relative performance by waterbody in regard to a particular indicator.


B. Performance Indicators for Operational Programs


    1. The conceptual framework for the system of program performance indicators in the International Waters focal area distinguishes among the three Operational Programs in the focal area -- OPs 8, 9 and 10 -- in regard to both, indicator selection methods of displaying the data collected on the indicators. Although OPs 8 and 9 have both aspects in common, they have and important points of difference in regard to performance indicators, which are reflected in the scheme.

    2. Both OP 8 (Waterbody-Based Operational Program) and OP 9 (Integrated Land and Water Multiple Focal Area Operational Program) aim at assisting groups of countries having access to a waterbody, whether freshwater or marine, to work collaboratively in making changes in sectoral policies and activities leading to reduced threats to the waterbody from pollution, overfishing, habitat degradation and non-indigenous species. Under both of these Operational Programs the GEF carries out a range of interventions, including assisting groups of countries in conducting trans-boundary diagnostic analyses (TDAs), in the formulation of strategic action programmes (SAPs), and in the adoption of approaches to the management of aquatic resources, such as integrated coastal management. In doing so, the GEF assists individual countries in carrying out follow-up activities aimed at policy and regulatory reforms. Program performance indicators are not clearly differentiated, therefore, between these two Operational Programs.

    3. However, these two Operational Programs differ in important ways in regard to the nature of the threats with which each deals. OP 8 focuses on the most seriously threatened waterbodies, in which the trans-boundary threats to each country involved are relatively well-recognized. The emphasis in OP 8 is on remedial measures to address those threats. Therefore participating countries have a relatively strong incentive to commit political capital as well as financial resources to the processes of analysis, prioritization and policy/regulatory reform that the GEF supports.

    4. OP 9, on the other hand, includes projects in which the trans-boundary threats have not yet become so serious. It puts more stress on prevention of damage than on remediation. The focus of OP 9 is less on the waterbody itself than on integrated approaches to the use of land and water resources and management practices. Therefore, countries participating in OP 9 projects have less incentives for strong commitments than those countries participating in OP 8 projects. Projects in OP9 often provide benefits in more than one focal area.

    5. Reflecting these differences between OP 8 and OP 9, data that is collected on the indicators for processes in OP 8 projects would not be compared or aggregated with data collected for OP 9 projects.

    6. OP 10, the contaminant-based Operational Program, supports projects aimed at demonstrating ways of overcoming barriers to adoption of best practices for limiting contamination of international waters. OP10 has four distinct components: the Land-Based Activities Demonstration, the Global Contaminants, the Ship-Related Contaminants, and the Regional and Global Technical Support component.

    7. The Land-Based Activities Demonstration component was designed to include a series of demonstration projects consisting of basins or areas draining to coastal/marine waters. Demonstrations involving the use of economic instruments are a high priority. OP 10 sets the target of having at least one demonstration project in each development region of the world.

    8. The Global Contaminants component of OP10 is designed to support activities that help characterize the nature, extent and significance of “global contaminants” and support activities that demonstrate prevention or reduction of releases in recipient countries.

    9. The Ship-Related Contaminants component addresses issues related to releases of oil and garbage from ships, transfer of non-indigenous species in ship ballast water, and ship collisions in busy corridors.

    10. The fourth component, Regional and Global Technical Support, focuses on regional or global capacity-building projects that can help groups of countries share experiences and lessons on other areas. This component also includes global projects of a strategic nature that assess the contribution of contaminants to the environmental status of international waters or that develop longer-range approaches.

    11. OP 10 projects represent interventions that are generally very different from those of OP 8 and OP 9. Therefore the program level indicators selected for OP 8 and 9, cannot be applied to all projects in OP 10. Furthermore, projects within OP 10 vary so widely from one another that common indicators cannot be applied across most of its projects. Some OP 10 projects, however, share characteristics that are common to OPs 8 and 9, in that they involve the development of national plans of action, including sectoral policy and regulatory reforms. In those cases, the same indicators for inter-ministry committee processes that are used for OP 8 and 9 projects would be used for OP 10 projects.

      C. Different Levels of Indicators

    12. The conceptual framework for the system of program performance indicators builds on the system proposed by the International Waters Task Force for the project level, which includes process indicators, stress reduction indicators and environmental status indicators. This framework adds program coverage indicators to these three levels, redefines the role of stress reduction indicators in relation to program performance indicators and recognizes the limitations on environmental status indicators in international waters projects.

    13. Coverage indicators provide measures of the scope and distribution of GEF projects according to various criteria, whether geographic, ecosystem type or others. The indicators chosen are based on program coverage issues raised in the Operational Programs themselves as well as in other GEF documents.

    14. Process indicators are those which measure the effectiveness of a selected set of actions taken jointly or singly by states participating in projects supported by the GEF. They are based on the types of GEF interventions identified in the Operational Strategy – the elaboration of a trans-boundary diagnostic analysis (TDA) or a Strategic Action Programme (SAP), the establishment or strengthening of joint institutional arrangements (JIAs) or the operation of an inter-ministry committee to plan policy and regulatory reforms in various sectors. These processes indicators are the same for all waterbodies in which one of these four processes is carried out.

    15. Stress reduction indicators are measures of the degree of success achieved by projects in reducing the human behaviors that are known to contribute to the degradation of the international waters addressed by the GEF project. However, the specific measures of stress reduction chosen for one project are unlikely to be the same as those chosen for any other. Consequently, this system of program performance indicators does not propose common stress reduction indicators. It does propose a measure of how far a project has gone toward actually monitoring and reporting on the change that has occurred in the stresses it has chosen to monitor, compared with the baseline.

    16. Environmental status indicators are measures of change in the state of the environment. Although they are included in the framework, their applicability is limited primarily to basins that have been the object of a “programmatic approach” or “geographically-based approach” by the GEF. Examples of basins to which they would apply are the Black Sea-Danube Basin and the Yellow Sea. Status indicators might also apply to other basins in which GEF projects are directly related to the reduction of a particular stress or set of stresses. This could be case, for example, in activities designed to reduce over-fishing.

    17. Environmental status indicators for each basin would not be comparable because each country with a stake in that basin may choose a different set of environmental status indicators related to its specific targets. Therefore, this system does not prescribe common environmental status indicators. Instead, as in the case of stress reduction indicators, this system proposes to measure how far a project has gone toward actually monitoring and reporting on the change that has occurred in the environmental status it has chosen to monitor, compared with the baseline.

      Two additional sets of indicators are included for the results of GEF international waters scientific assessments and knowledge management activities. In both cases, the criteria for evaluation focus on the quality of the knowledge produced and the accessibility and usefulness of the activities to policymakers and relevant stakeholders.

    18. This framework proposes to measure program performance by aggregating data on project results where such aggregation is possible. For process indicators, the aggregation would be among all projects involving common processes. For projects in which stress reduction indicators are being measured, the results can be aggregated in terms of both progress toward reporting stress reduction and the degree to which the different stresses have been reduced. Results from the projects that monitor environmental status would not be aggregated, however.

      D. Project Impact Indicators vs. Project Implementation Indicators

    19. The difference between indicators of project implementation and indicators of project impact or success is a starting point for understanding the role of indicators in measuring program performance. When they are asked for indicators of implementation progress, project managers sometimes list the expected project outputs or tasks to be carried out, such as setting up a steering committee, holding a ministry conference to endorse a SAP or enhancing public awareness. Although these outputs can be considered measures of progress in implementation of the project, they are not the same as indicators of the project’s impact. The question of interest to the program evaluator is whether a given process has advanced toward outcomes ( such as policy reforms, a better understanding of the problem or institutional capacities) that will be effective in changing the relevant human behaviors that degrade international waters.

    20. Similarly, project impact indicators are sometimes confused with the specific objectives of the project. Thus “institutional strengthening” and “reduced pollution load,” which are actually specific project objectives, may be listed as indicators. But institutional strengthening could be a criterion for project success, whereas an indicator of that criterion might be how many of the governments involved in the project have line items in their budgets to support the institution responsible for the management of the waterbody. An indicator of reduced pollution load would be the percentage change in nitrogen levels at a selected set of monitoring sites in the waterbody. Indicators thus involve either a quantitative or qualitative measure of how successfully a major action has been to achieve its objective or the quality of the process involved.

      E. The Importance of the Baseline

    21. Each project or water body proceeds from a different set of circumstances, and it is important in assessing both project and program performance that the existing level of cooperation—or lack of it—be carefully documented. Thus, for process, stress reduction and environmental status indicators, the establishment of an accurate baseline is of utmost importance in making the indicator meaningful. The baseline is the situation that exists at the beginning of the GEF involvement in a initiative. Collecting data that identify that situation in quantitative or qualitative terms makes it possible to measure the degree of change that takes place as a result of the project or series of projects in a given waterbody or basin. The baseline will be established by applying these indicators before project approval by GEF. This will provide an information base that will permit to track changes in the indicators thought GEF involvement and beyond. For example, if a joint institution arrangement already exists before the project is implemented, the functions of that institution and other characteristics that indicate how effective the institution has been in facilitating joint efforts to reduce environmental threats to transboundary waters should be noted.

    22. Also, for process indicators, the baseline data will take the form of a quantitative or qualitative characterization of the level or degree of cooperation or conflict among the countries, institutions or stakeholders participating in the trans-boundary waterbody project. For stress reduction indicators, the baseline required would measure the extent of the human activity contributing to the environmental degradation at the outset of the project or if the indicator chosen is related to the progress of a government program aimed at reducing a particular stress, the initial status of the human activity involved (annual rate of reforestation, rates of fish catches proportion of farmers using best practices, etc.). For environmental status indicators, the baseline would measure the quality of the environment in regard to a manageable set of indicators with which later data can be compared. Stress reduction and environmental status indicators are only required for SAP implementation and GEF activities with a programmatic approach.

  1. Implementing the Program Performance Indicators System

    1. The GEF Monitoring and Evaluations Unit will be responsible for compilation and presentation of coverage indicators. The implementing agencies will be responsible for carrying out the necessary monitoring and evaluations on other indicators at the project level, in accordance with agreed operational methodologies. Project monitoring staff will ensure that their monitoring plans integrate the collection of data on the program performance indicators into their work.

    2. When the data has been collected, the implementing agency will be responsible for reporting it to the GEF Monitoring and Evaluations Unit. The M & E Unit will then compile and analyze the data from different projects periodically for the purpose of issuing reports presenting the data. (For discussion of the ways in which the data can be presented, see pp. 32-33.)

      II. Coverage Indicators

      Coverage indicators should provide a clear picture of how the program’s portfolio of projects relates to the scope of the problems that it is intended to address on the global scale. They help answer such questions as how much of the problem is encompassed by the portfolio, and how much is left out, and what balance has been struck among different threats, different types of project interventions and different geographic areas. They should also tell us whether all the main threats to international waters are being addressed in the Operational Programs in the International Waters focal area. Thus coverage indicators should help in assessing the overall pattern of programming in specific Operational Programs and in the focal area as a whole.

      A. Coverage Indicator for all OPS

      Only one coverage indicator is proposed that spans the three Operational Programs: the distribution of projects by project intervention type. Ten frequent types of activities supported by the GEF include:

      B. Coverage Indicators for OPs 8 and 9

    1. The following coverage indicators are proposed for OP 8 and OP 9.

      Proportion of the number and area (Km2) of the world’s GEF-eligible Large Marine Ecosystems (LMEs) in which the GEF has projects. The Lists of GEF-eligible LMEs and the LMEs in which the GEF has projects is shown Annex 1. The list is based on the Large Marine Ecosystems of the World website (http://www.edc.uri.edu/lme/clickable-map.htm).

      Proportion of GEF-eligible LMEs in which the GEF has helped catalyze adoption of management frameworks for priority actions agreed by riparian countries (See Annex 1).

      Proportion of numbers and area (Km2) of GEF eligible International River Basins. International river basins will be identified on the basis of international river basins information provided in www.transboudarywaters.orrst.edu.

      Distribution of GEF project components by major category of threat. The distribution of GEF project components according to the threats addressed was analyzed on the basis of a review of project documents1. Annex 2 includes a list of threats addressed by GEF projects.

      Distribution of GEF project components between remedial and preventive activities.

      Distribution of GEF projects by type of environment, including:

      Distribution of GEF projects by geographic region. The regions considered in this monitoring system are:

      C. Coverage Indicators for OP 10

    1. Coverage indicators for OP 10 must reflect the fact that they are not focused on threats to a particular transboundary waterbody but at demonstrating particular approaches to reducing the threat of contamination of waterbodies globally.

    2. The coverage indicators proposed for OP 10 are:

      Distribution of technical support projects by subject.

      III. Process Outcome Indicators

    1. The primary role of the GEF in International Waters is to catalyze and support effective joint and individual state responses to threats to trans-boundary waterbodies. The most important measures of the effectiveness of the portfolio of projects in the focal area, therefore, are process and institutions that GEF activities help establish and strengthen. The framework presented here seeks to provide measures of the quality or effectiveness of processes and institutions that GEF projects help catalyze.

    2. Although GEF Operational Programs 8 and 9 differ in regard to the degree of trans-boundary threat involved and their emphasis on remedial (OP #8) and preventive (OP #9) measures, the basic processes for the countries involved to determine priority causes and the measures required to address them are similar. Therefore the same process outcome indicators apply to both OPs. Some OP 10 projects also involve the same actions at the national level as are found in OPs 8 and 9.

      A. Three Types of Process Indicators

    3. Three types of process outcome indicators may be distinguished: indicators for joint multi-country processes, indicators for single-country processes, and indicators for demonstration projects.

      1. Multi-country Joint Process Indicators.

    4. These indicators measure the results achieved and quality of process in “Initial Strategic Regional Projects” by joint processes involving two or more countries that use and affect the same waterbody. These process outcomes include:

      2. Single-country Process Outcome Indicators

    1. The second type of process outcome indicator relates to the results achieved by and quality of the process associated with actions by a single country. In many cases, these actions may represent implementation of the strategies or agendas for action negotiated by two or more countries. In other cases, they are national plans of action on a multi-sectoral problem or simply policy and regulatory reforms that require the participation of several government agencies. The actions that will be measured by single-country process indicators include the following:

      3. Demonstration Projects

    1. GEF-supported demonstration projects are actions undertaken at one or more sites with the objective of showing that a particular approach to protection of international waters or remediation of damage to them, is both effective and replicable. Demonstration projects may be undertaken by multi-country groups or by a single country. Because these actions are fundamentally different from the other actions that are to be assessed under this indicators scheme, a separate set of indicators is proposed for the evaluation of the process quality and effectiveness of such projects.

      B. Indicators of Results and Process Quality

    2. Some process outcome indicators can be scored on the basis of yes or no answers. The question of ratification of relevant regional treaties obviously lends itself to such a positive or negative score. The question of integration of investment priorities into bilateral and multilateral development assistance agencies can also be translated into such scoring by asking whether the country has integrated priority actions found in the SAP into Country Assistance Strategies agreed upon between the country and the World Bank, or the UNDP’s Country Cooperation Frameworks or into bilateral aid agencies’ country plans.

    3. Process outcome indicators for other multilateral or single-country actions will provide a measure of the success of actions taken by governments on a scale rather than simply recording whether or not an action was taken. A indicator can be used either for a joint inter-government process --a trans-boundary diagnostic analysis (TDA), Strategic Action Programme (SAP), or Joint Institutional Arrangements (JIA)-- or for the single-country process of establishment and operation of an inter-ministry committee (IC). Whether a indicator is applied to one of the three joint processes or to the IC at the single-country level, it should measure one of two things: (1) how effective the process was in contributing to address the problem or (2) the extent to which the process meets criteria for adequate government involvement, stakeholder participation, and availability of varied sources of information.

    4. indicators are based on a suite of descriptions of possible outcomes in ascending or descending order of effectiveness or quality in different dimensions of the process. These different levels of process quality may also represent different stages of development of a project. Thus the indicators of government involvement, stakeholder participation, or access to sound information may be changed at different stages of the project.

    5. Indicators of the results of a process provide scales of effectiveness of the quality and outcome of the process, whether it is a TDA, a SAP, JIA or agreement on policy and regulatory change by the IC. Each of these scales range from an outcome that represents no progress whatever, at the low end of the scale, to major steps toward bringing about necessary changes, at the high end of the scale.

    6. indicators of government involvement measure the degree to which governments have committed to processes or the extent to which relevant ministries engage in single-country follow-up processes. Depending on the type of process, the agreement or endorsement of outputs such as TDAs or SAPs multi-ministry support, staff and financial support, and the assignment of ranking officials are generally indicators of how committed governments have been to the intergovernmental or inter-ministry process.

    7. Based on the “Good Practice Note Addressing Social and Participation Issues in GEF-Financed International Waters Projects,” prepared by the GEF International Waters Task Force, indicators for stakeholder participation focus on three dimensions of participation. The first dimension is stakeholder identification, i.e., the degree to which all relevant stakeholders have been identified. This process would involve addressing a serious of pertinent questions about what social and economic groups depend on the water resources in question. The second dimension is the formulation and implementation of a public plan for stakeholder involvement, involving information dissemination and consultation with relevant stakeholders. The third dimension, which builds on the first two, is the degree of stakeholder satisfaction with their participation in the process. The indicators for stakeholder participation are the same for all the processes to be evaluated.

    8. Indicators for “sound information” in the multilateral and single-country processes outcome measure the extent to which mechanisms to obtain the best available scientific information and other types of information are in place and are operating effectively. Some kind of scientific advisory mechanism should be established and utilized. . The process should have a means of accessing both scientific and other forms of knowledge (for example, local or traditional knowledge), where appropriate.

    9. The indicators of process results and process quality for each of the four processes (i.e., TDAs, SAPs, joint institutional arrangements and inter-ministry committees) have been combined in a single table showing four different values for each dimension (i.e. government involvement, stakeholder participation, sound information, and process results) in descending order of effectiveness or quality. The tables are presented below.

      1. Indicators for the Trans-boundary Diagnostic Analysis Process

    10. The inter-governmental process of producing a TDA does not by itself generate commitments to specific actions by the participating governments. However, it should prioritize major environmental threats to international waters and diagnose their causes, identifying the primary sources of environmental stress and the options for addressing them.

    11. The scale for the results of the TDA process outcome measures how far the TDA goes toward specifying the root causes of trans-boundary environmental threats and towards distinguishing between domestic and trans-boundary contributions to the problem.

    12. The degree of government participation is measured in terms of the extent and level of government endorsement of the TDA and the extent of provision of staff and funding by governments for their activities in support of the TDA.

    13. For TDA processes that have concluded, in the scale for government endorsement the highest value would be assigned to a TDA process that have proof of agreement of all participating governments with the TDA results. The next highest value would be assigned TDA processes that have proof of agreement of TDA results of less than 100 percent or more than 80 percent of the participating governments. The next highest value would be assigned TDA processes that have proof of agreement of TDA results of less than 80 percent but more than 50 percent of the participating governments. The lowest value would be assigned to TDA process that have proof of agreement of TDA results of less than 50 percent of participating governments.

    14. For TDA processes that have not concluded, in the scale for government staff and funding support, the highest value would go to a TDA process in which all governments have provided the resources necessary for their TDA-related activities. The next highest value would be assigned TDA processes in which less than 100 percent but more than 80 percent of participating governments have provided the resources necessary for their TDA-related country activities. The next highest value would be assigned TDA processes in which less than 80 percent but more than 50 percent of participating governments have provided the resources necessary for their TDA-related country activities. The lowest value would be assigned to TDA process that in which les than 50 percent of the participating governments have provided the resources necessary for TDA-related country activities.

    15. Sound information will give special consideration to the extent to which the TDA process has included local scientists and other relevant stakeholders and the extent to which TDAs benefited from the proper technical and methodological guidance.

    16. 2. Indicators for the Strategic Action Programme Process

    17. For the SAP process, the highest value for the process results would be assigned if the SAP presents a manageable programme that includes quantitative targets, timetables for policy or regulatory reform for the critical problems identified in the TDA. If the SAP lacks one or more such targets, timetables or has none at all, progressively lower values would be assigned. The lowest value would be assigned for a SAP that has no specific policy/regulatory reforms for all problems identified in the TDA.

      The measure for government involvement of the SAP process is similar to the one for government involvement in TDAs, with the difference that it focuses of formal government endorsement of completed SAPs. The indicators for stakeholder participation focuses on the extent to which the stakeholders were appropriately identified, a stakeholder participation plan was developed and implemented and the extent to which stakeholders consider they have been adequately included in the SAP elaboration process. Sound Information indicator criteria focus on the extent to which SAP reflect the information and analysis of the TDA and the extent to which advisory groups function as tools to insure the SAP considered information from the relevant sources and perspectives.

      3. Indicators for Joint Institutional Arrangements

    18. Some projects focus on the formation or strengthening of joint institutional arrangements (JIAs) for the waterbody. JIAs inter-government institutions that have the formal participation of two or more riparian governments and have as a mandate to address an issue or set of issues concerning the international water body. In some cases they are an outcome of GEF projects while in other cases they are intergovernmental mechanisms that predate GEF involvement. These arrangements may involve just one institution or several institutions, each having a different functions, such as joint monitoring and policy coordination, harmonization or the implementation of other joint activities.

    19. The criteria for assessing the results of the JIA sponsored processes will focus on the JIAs credibility and actual influence on member governments decisions regarding policies, reforms, investments and other relevant actions to move forward agreed upon plans and programmes.

    20. The criteria for government involvement in the JIA are then umber of governments that undertake each of the three forms of support to the JIA: Allocation of staff resources, assignment of high-level officials and maintaining a line item in the budget in support of it. Each of these criteria would have its own scale in the actual evaluation. The highest value on each of the scales would be assigned if all governments provide the support, whereas the lowest value would be assigned if most governments fail to do so.

    21. The indicators for stakeholder participation focus on the establishment and functioning advisory groups or similar mechanisms as tools to incorporate information, perspectives and interests of stakeholders in the activities carried out by JIAs.

      4. Indicators for the Inter-Ministry Committee Process

    22. An action by countries participating in GEF projects that is central to measuring program success is the establishment and operation of an inter-ministry committee (IMC) or other national mechanism for ensuring that needed sectoral policy and regulatory reforms, investments and other actions to address root causes are formulated and implemented.

    23. The effectiveness of the individual state’s IMC will depend on how far it goes toward adopting binding, quantitative targets, timetables for sectoral policy and regulatory reforms, investments and other actions. Another measure of the effectiveness of IMCs is the extent to which IMCs meet regularly. The highest value on each of the scales would be assigned if all governments have met their commitments or have established clear timetables and quantitative targets whereas the lowest value would be assigned if most governments fail to do so.

    24. The criteria for government involvement in the IMC are similar to those of the JIA: the number of governments that undertake each of the three forms of support to the JIA: Allocation of staff resources, assignment of high-level officials and maintaining a line item in the budget in support of it. One additional measure of government involvement in IMCs is the extent to which IMCs meet regularly. Each of these criteria would have its own scale in the actual evaluation. The highest value on each of the scales would be assigned if all governments provide the support, whereas the lowest value would be assigned if most governments fail to do so.

    25. The indicators for stakeholder participation on the establishment and functioning advisory groups or similar mechanisms as tools to incorporate information, perspectives and interests of stakeholders in the activities coordinated by ICs. The highest value on each of the scales would be assigned if all governments have functioning mechanisms mechanisms for stakeholder participation, information and feedback, whereas the lowest value would be assigned if most governments fail to do so.

      5. Indicators for mobilization of Resources

      Another necessary follow-up action by individual states is to obtain the financing necessary to carry out investment projects that are identified as priorities by the TDA and SAP processes. The measurement of the effectiveness of this follow-up action is based on the proportion of available funds needed for the priority investment projects identified in the SAP. A measure of effectiveness can then be used to measure how much of the financing for the investment projects identified was obtained

      6. Indicators for Demonstration Activities

    26. In addition to GEF’s catalytic role in supporting inter-governmental and national government processes aimed at sectoral policy and regulatory reforms and investments, all three GEF International Waters programs support a large number of demonstration projects and project components. Nearly a third of all International Waters projects involve demonstration activities and the central purposes of a number of them is to demonstrate that a particular approach can be successful in addressing an environmental threat to international waters. Program performance indicators are needed to measure what has been achieved in these demonstration activities.

    27. The purpose of demonstration activities is to interest local, regional authorities with similar conditions and problems in replicating techniques and approaches that are shown to bring the desired results in terms of pollution reduction or habitat preservation. Criteria to assess success of such projects, therefore, regardless of the specific problem being addressed by the project, are: in the case of concluded activities, the extent to which third parties are using demonstrated approaches and in the case of ongoing activities (1)extent to which the demonstration projects have used appropriate indicators for judging the effectiveness of the demonstration of the approach in providing a solution to the environmental problem, (2) the degree to which they have implemented a strategy for monitoring and evaluation of the success of the project, and (3) the degree to which they have implemented a strategy for ensuring replication of the demonstration , including the involvement of those who are in a position to replicate project activities in design and implementation of the pilot. The program performance indicators should also measure actual results in terms of activities aimed specifically at bringing the technology or approach to the attention of those who are in a position to replicate them.

IV. Indicators for Stress Reduction



A. Stress Reduction Indicators at the Project Level

    1. Stress reduction indicators measure the degree to which project activities have contributed to changes in the sectoral activities (or some proxy for those activities) that threaten the environmental quality of international waters, degrade or destroy habitats, or deplete marine resources. Specific stress reduction indicators for each project will differ according to the waterbody type and project objectives.

    2. The stresses addressed by projects in OP 8 and OP 9 can be grouped under six distinct categories of threats to international waters, one of which has been divided into three subcategories. These categories are shown in Box A Examples of stress reduction indicators that could be used at the project level, grouped under each of the six major threats addressed by international waters projects, are shown in Annex 2. These stress reduction indicators are intended to be merely indicative.

    3. Some projects are far more likely than others to be able to document a reduction in specific stresses as a result of the project itself. Waterbodies that benefit from a “programmatic approach” can be expected to have a measurable impact on sectoral activities, whereas projects that support the formulation of TDAs and SAPs are not likely to have such an impact during the project implementation period. Projects for actual implementation of SAPs should have relatively short-term impacts on stresses in most cases.

    4. Because of the wide range of stresses that may be measured in different international waters projects, it is not possible to identify a selected group of stress reduction indicators that could be compared across projects and serve as program performance indicators. Therefore, this system of program performance indicators does not propose a set of specific stress reduction indicators that would relate to the same stresses across a group of water bodies.

      Box B: Stress Reduction Categories

              1. Trans-boundary pollution

              2. Over-fishing

              3. Habitat loss

              4. Water availability and water withdrawals

              5. Land degradation

              6. Vulnerability of human populations

      B. Program Performance Indicators Related to Stress Reduction

    5. Instead of identifying specific stress reduction indicators at the program level, this system would measure project success on two dimensions: (1) the extent to which projects have properly identified, monitored and reported on stresses that are most relevant to the project and (2) the extent to which the project has succeeded in meeting its own objectives in regard to stress reduction, where such objectives have been adopted. These two indicators would be monitored and reported only by those projects where actual stress reduction is a project objective.

      1. Progress in Monitoring and Reporting on Selected Environmental Stresses

    6. The indicators of monitoring and reporting are shown in Table 7. The most desirable outcome of a monitoring and reporting system at the project level is that the monitoring plan has been implemented and has produced reporting of data on the change in the level of environmental stress or stresses from the baseline data, if any. The least desirable outcome would be that no monitoring plan for stress reduction is established. These ascending levels of achievement may represent stages of development of the project.

      2. Progress in Achieving Stress Reduction Objectives

    7. Although this indicators system does not adopt a set of specific stress reduction indicators to be used by different groups of projects, it proposes to aggregate and compare the data from projects on the degree to which they have achieved their stress reduction objectives where such objectives have been adopted. Stress reduction objectives may involve proxies for stress reduction indicators, such as concrete measures of progress in programs to change behaviors that stress the environment.

    8. The indicator for the realization of stress reduction objectives is simple: either the project has not come close to realizing its stress reduction objective, progress in target, has come close to achieving project’s stress reduction the objective or has gone beyond it.

    9. It may be that proxy indicators can be improved much more easily and more rapidly than indicators of human activities that put stress on the environment. Should that prove to be the case, the aggregation and comparison of data would have to be based on separation of proxy indicators from more direct indicators of stress reduction.

      VI. Indicators for Environmental Status

  1. Limited Application and Non-Comparability of Environmental Indicators

    1. This system of program performance indicators does not prescribe the environmental status indicators to be used at the project level. The reason is that, as it is the case with stress reduction, with few exceptions, only those waterbodies that benefit from a “programmatic approach” and some demonstration projects can be expected to have a measurable impact on environmental status during the time frame of a GEF project. And those few waterbodies that might have such a programmatic approach cannot be expected to focus on the same or similar environmental status measures. In this sense they will not be comparable. Moreover, the choice of environmental status targets will in each case be a political choice made jointly or singly by the participating states.

    2. Because the International Waters focal area has the resources only for catalytic interventions that are not expected in themselves to result in changes in environmental status during the period of the project, environmental status indicators generally will not be relevant to its Operational Programs. The exception to this generalization thus far is the Black Sea-Danube Basin, which is the one basin that has been the beneficiary of a “programmatic approach” by the GEF. A programmatic approach, also referred to as a “geographically-based approach,” means that several related projects have been supported by the GEF for the same waterbody or basin, and that the projects are implemented in stages that extend over a sufficient period of time that states have the possibility of adopting strategic targets for environmental quality.

    3. The 17 Danube/Black Sea Basin countries have already committed formally through the Istanbul Commission and the International Committee for the Protection of the Danube River to the target of reducing nutrient levels and other hazardous substances to levels that would permit the Black Sea ecosystem to attain the water quality it had in the 1960s. However, this is a long-term target, and it is unclear how long it will take before any change in environmental status could be registered.

    4. Other basins enclosed seas/LMEs and their associated drainage basins now covered by multiple GEF projects (for example, Ginea Current LME, South China Sea, and Plata Basin) could also become foci of the programmatic or geographically-based approach in the future, if the GEF decides to support further projects in those basins. If so, the states participating in projects would be expected to agree on certain targets for environmental quality in the transboundary waterbodies in question. However, it is unlikely that these would be the same issues chosen for targets by a GEF International Waters project in another waterbody.

    5. Because the countries belonging to each such basin are likely to decide on different targets for environmental changes in the basin, it would be fruitless to propose a common set of environmental status indicators. Thus, each basin subject to a programmatic approach will be treated as having its own set of program-level environmental status indicators.

      B. Indicators for Monitoring and Reporting on Environmental Status

    6. Instead of proposing common environmental quality indicators for waterbodies in which a programmatic approach is being applied, this scheme for program level indicators provides a set of process indicators for monitoring and reporting on environmental status in those projects where such monitoring is appropriate. It does not assess the appropriateness of the environmental status indicator chosen but only the progress on monitoring and reporting itself.

    7. The indicators for monitoring and reporting on changes in environmental status are similar as those for monitoring and reporting on stress reduction and reflect stages of development of the project. They include (1) establishment of a baseline, (2) development of practical indicators (3) establishment of a monitoring system Table 9 shows the set of indicators for this purpose.

      Indicators for Results of Scientific Assessments

    8. GEF finances some International Waters projects to mount scientific assessments at various levels. These are project that are targeted to do research and are generally focused on issues of global or regional importance such as understanding global environmental threats, treats in a region or in a specific ecosystem. These projects may address some fundamental scientific issues. What they have in common, however, is that unlike TDAs, the scientific assessments referred to in this section are not meant in resulting in a SAP or a specific action programme, they are meant to generate information and to improve understanding of issues of regional or global importance.

    9. Because these projects are unique in their procedures and methods, their effectiveness and value cannot be reasonably evaluated on the basis of the process indicators applicable to their types of other project activities. The sets of indicators proposed for these projects aretherefore base on the quality of the results used rather than on process. The criteria for desirable results are accessibility to stakeholders (including those who would be using them), usefulness to policy or programming decisions and quality of outcome.

      IX. Presentation of Data on Program Performance Indicators

    10. The presentation of data on program performance indicators will be based on three distinctions: (a) the Operational Program; (b) whether the data is aggregated or compared; and (c) whether the data pertains to all waterbodies or to waterbodies within each type of ecosystem. Therefore, for each of the OPs (8, 9 and 10), the data on process indicators and stress reduction-related indicators can be presented in three possible ways:

      1. The first mode of presentation is by OP. The data would show how many projects in each of the OPs achieved each of the four levels measured for each of the processes assessed. If, for example, 18 projects have undertaken a SAP, of which nine projects are in OP 8 and nine in OP 9, the presentation of data would add up how many of the nine in OP 8 have achieved the highest level of achievement on the indicators for Government Involvement in the SAP, how many have achieved the next highest level of effectiveness, etc. The data on the same nine projects would then aggregated for each of the four dimensions of process quality and process results for SAP. The same procedure would apply to each of the other three processes and for obtaining financing for investment needs.

      2. Another form of data presentation would aggregate the data by both OPs and by type of water body (LME, coastal zone, freshwater body, aquifer) showing how many waterbodies of each ecosystem type fell into each of the cells of the indicators for each project.

      3. Data could also be presented showing all of the projects in the same OP and type of ecosystem to be compared in regard to process results.

      ANNEX 1: COVERAGE INDICATORS

Table A GEF-eligible Large Marine Ecosystems (LMEs) *


LATIN AMERICA & THE CARIBBEAN GEF project

California Current

Gulf of California

Gulf of Mexico

                    Pacific Central America Coast

Caribean Sea X

Humboldt Current

                  Patagonia Shelf X

South Brazil Shelf

East Brazil Shelf

North Brazil Shelf X


AFRICA


Canary Current X

Guinea Current X

Benguela Current X

Aqulhas Current

Somali Costal Current


ASIA


                    Bay of Bengal X

                    Gulf of Thailand X

                    South China Sea X

                    Sulu-Celebes Sea

                    Indonesia Sea

                    East China Sea X

                    Yellow Sea X

Sea of Japan


MIDDLE EAST AND NORTH AFRICA


Mediterranean Sea X

Arabian Sea

Red Sea X


EASTERN EUROPE

Baltic Sea X

Black Sea X


      * Source : Large Marine Ecosystems of the World Http://www.edc.edu/lme/clickable-map.htm and GEF project files.

    1. ANNEX 2: Examples of Stress Reduction Indicators

      (BY CATEGORY OF THREAT)

      1. Trans-boundary Pollution

      1.A. Excessive Nutrient Load

      1.B. Persistent Organic Substances

      1.C. Other Contaminants

      2. Over fishing

      3. Habitat Loss/Destruction

      4. Water availability/ Water Withdrawals

      5. Land Degradation/Sedimentation

      6 Vulnerability of human populations

      Population with access to clean water and sanitation

      Reduction of floods, inundation of costal territories and droughts


wb151381

Z:\websites\cls\INWEH\content\CoursesAndGuidelines\TDA-SAP Training Course\Module 5\Supporting material\3. Monitoring and Evaluation (2).doc

1 The leading category of threat is transboundary pollution involving more than one type of transboundary pollutant, with 26 projects. The second largest category is excessive nutrient load, with 20 projects. Over fishing is addressed in 17 projects and land degradation or sedimentation in 10.

34