Mission and Principles

1. The newly appointed Director of Monitoring and Evaluation is required by his terms of reference to prepare and submit draft M & E policies, guidelines and procedures for GEF Council decisions, after requesting inputs from relevant GEF entities. At the Council meeting in November 2004 the Director will ask for approval to set in motion a process leading up to a new GEF monitoring and evaluation policy in preparation for GEF4. This new policy will also take into account the findings and recommendations of the Third Overall Performance Study. Consequently, the draft GEF M & E policy will be presented to Council in the second half of 2005. This note presents principles for and elements of the new GEF M & E policy for approval by the Council, building on the terms of reference of the independent M & E unit and existing initiatives to improve M & E in the GEF family. Separately, a first draft for a four year work plan and budget will be presented to the Council, which will built on the principles and guidelines presented here and translate them into concrete proposals for monitoring and evaluation.

2. To reflect the independence of M & E, the M & E unit has been renamed the Office of Monitoring and Evaluation.

3. The Office of Monitoring and Evaluation proposes the following mission statement:

Enhancing global environmental benefits through excellence, independence and partnership in monitoring and evaluation

4. This mission statement reflects the crucial roles of independence and professionalism, but also highlights that evaluation results need to be widely shared in order to enable the GEF partners to improve and enhance their activities. Furthermore, the Office of Monitoring and Evaluation will continue to work in close partnerships with other entities in the GEF and extend this collaboration to the global evaluation community in order to remain on the cutting edge of methodologies and derive maximum benefits from collaboration.

5. The independence of the Office has been established to guarantee that data gathering and analysis, judgments on criteria, findings and recommendations will not be influenced through conflicts of interest or undue interference by management at any level. This independent assessment of activities and results in which others have high stakes calls for the highest possible level of professionalism and full transparency in how any evaluation will be carried out and on what grounds judgments will be reached. The Office proposes to adhere to the following three principles:

Impartiality
Professionalism
Transparency

6. These three principles will be applied in the way that the Office of M & E conducts its business. The Office will be impartial in the sense that it will conduct evaluations and arrive at the findings in a balanced and unbiased way. Professionalism will be achieved by applying the latest evaluation knowledge and skills with integrity, accountability, and respect. Transparency will ensure full disclosure and active communication with stakeholders on M & E priorities, terms of reference of evaluations, methodology and the formulation of findings and recommendations, at appropriate times.

7. These three principles, which will support and enable independence, accountability, added value and feedback, will be applied by the Office of M & E in carrying out its tasks as set out in the terms of reference:

· Independently monitor and evaluate, on a continuing basis, the effectiveness of GEF programs and resource allocations on project, country, portfolio and institutional bases.

Provide a basis for decision-making on amendments and improvements of policies, strategies, program management, procedures and projects.

Promote accountability through the assessment of processes, results, impacts and the performance of the parties involved in GEF activities.

Provide feedback to subsequent activities and promote knowledge management on results, performance and lessons learned.

8. The four year rolling work plans and annual work program and budget will contain indicators and output targets for the Office of M & E.

Interaction with the Council

9. The key principles in the relationship between the Office of M & E and the GEF Council have been laid out in the Terms of Reference for an independent M & E unit approved on July 28, 2003:

The GEF Council appoints the M & E Director, upon nomination by the CEO/Chair. The M & E Director can be removed only by decision of the Council.

The Council will review, discuss and approve M & E policies and procedures, four year rolling M & E work plans, the annual work program and budget.

It will receive all M & E reports directly from the Office of M & E.

In addition to interaction under the agenda item on monitoring and evaluation at every regular Council meeting, the M & E Director will, when needed, in consultation with the Council arrange for special meetings or informal briefings between the Council and the Office of M & E.
On behalf of the Council, the M & E Director will also seek to strengthen the M & E relationships with GEF’s member countries, the relevant international environment convention secretariats and other key stakeholders regarding issues of M & E.

10. A basic pillar of the independence of the M & E office is its direct reporting to the GEF Council. The M & E Director has the authority to issue final evaluation reports directly and simultaneously to the GEF Council and its CEO, without prior clearance from anyone outside the Office of M & E. The Secretariat, Implementing Agencies and Executing Agencies, the Scientific and Technical Advisory Panel (STAP) and other affected parties will receive and will be requested to comment and to respond to the draft and final reports, but will not have the right to approve, hold back, request changes to or otherwise modify such reports.

Follow Up of M & E Reports

11. The Office of M & E will cooperate closely with the GEF Secretariat, the Implementing and Executing Agencies and STAP in the planning, execution and follow-up of its studies and their reports. Satisfactory follow up of M & E reports requires adherence to specific rules and practices by all GEF entities. GEF M & E reports will distinguish clearly between the major recommendations, which are expected to be addressed by the GEF Council and subsidiary recommendations which are prepared for consideration at a lower organizational and/or procedural level. All major recommendations should be of overriding importance, formulated clearly and few in number (although each recommendation could include sub-points for purposes of illustration).

12. A management response will be required for all reports presented to the GEF Council by the Office of M & E, in particular the Annual Project Performance Report (to be renamed the Annual GEF Performance Report), all GEF focal area program evaluations, cross-cutting and thematic reviews or any other study specially requested by or presented to the Council. Management responses will normally be presented to Council by the CEO at the same time as the M & E report is presented by the Director of M & E.

13. Management responses on project level reviews (e.g., TERs and SMPRs) may be required, but will be conducted under separate procedures (to be prepared).

14. Before the management response process starts, the draft evaluation report will be sent to the relevant GEF entities for their review and comment on factual and substantive issues. Comments should as a rule be received by the Office of M & E within two weeks. If necessary, a meeting will be organized to discuss the comments provided and how these comments will be incorporated in the final version of the report. The Office of M & E will incorporate comments as it finds appropriate and prepare the final draft report which will be sent to all relevant GEF entities to initiate the process of preparing the management response. In parallel, the process of copyediting to prepare the final report will take place.

15. A first draft of the management response should be prepared within two weeks after receiving the final report of the study and sent to the M & E Director who will corroborate that the response addresses the findings and recommendations in the evaluation report (as a rule within three days).

16. The GEF CEO is responsible for the coordination of the preparation of the management response for GEF Council consideration. Given that each evaluation report involves different parts of the GEF partnership, the GEF CEO will delegate the responsibility of the management response and decide on the appropriate procedure to be followed in each case, and which entities of the GEF family should contribute inputs to the response.

17. The management response should be directed to the GEF Council and include the following items:

A brief overall reaction to the M & E findings;
Reaction to each of the major recommendations;
Proposed decision(s) for Council consideration, including plan of action, where required.

18. The management response may, with appropriate explanation, accept or not accept, fully or in part, the M & E recommendations. The Director reserves the right to communicate to the GEF Council on a separate note any disagreement he may have with the content of the management response.

19. The Council is expected to discuss and review the evaluation report and the corresponding management response, take any necessary decisions and give guidance to GEF on policies or an appropriate plan of action within specific timeframes. The decisions will be recorded on a special sheet, called the Management Action Record (MAR), which will be kept by the Office of M & E.

20. The GEF Secretariat will report annually to the Council on the follow-up of the Council decisions included in the Management Action Record. Recommendations which are not approved by the GEF Council, those which have been fulfilled, and/or have been incorporated into other decisions/policies will be noted and taken off the Management Action Record sheet. The Office of M & E will comment upon the actions taken by management, which comments will also be noted in the sheet. In some cases, the Office for M & E may see the need to verify the reported follow-up actions through subsequent or additional reviews. The Director of M & E will report to Council on M & E’s verification of the follow-up of Council decisions as noted in the Management Action Report.

21. The published version of M & E reports will contain the management response to the Council as an annex. Furthermore, the GEF Secretariat will have the right to put a footnote in the main body of the report if it or another GEF entity objects strongly to a finding. The footnote will contain a short description of the disagreement. The Office of M & E has the right to rebuttal in a footnote on the footnote, if it feels there is a need to do so.

22. For the purpose of preparing its discussions of M & E reports and management follow-up, the Council may consider advancing the start of Council meetings and devoting an additional half day to address M & E issues.

Interaction with GEF Entities and Partners

23. In addition to the responsibility of the GEF Secretariat and other GEF entities to provide a formal management response to final M & E documents, the Office will seek their full interaction on its evaluations in all stages of the process.

24. GEF evaluations are developed and managed by the Office of M & E, which is responsible for their technical quality and independence of analysis and judgment. However, such studies can only be effectively implemented in full collaboration and coordination with the GEF Secretariat and the Implementing and/or Executing Agencies involved. For major studies, this may involve the appropriate GEF entities in study development and planning meetings and in reviewing and commenting upon the study methodology and approach, as developed in its inception report. Where appropriate and possible, specialists of the GEF Secretariat and other GEF entities may also participate in fieldwork studies, led by the representative of the Office of M & E. It is also valuable for the GEF entities to comment on draft versions of study report chapters. The Office of M & E will decide for each evaluation whether limitations in time and/or resources permit a high level of interaction and participation, and invite other GEF partners accordingly.

25. In all cases, the Office of M & E will take responsibility for the independence and quality of the final product, with acknowledgement of inputs and responses from study partners. Where other GEF entities have substantive differences of interpretation from those of the Office of M & E, these will be appropriately referenced in report texts or in a separate annex, or in a footnote to the text.

26. The Office of M & E is tasked to set “minimum M & E standards” within the GEF family, in order to ensure improved and consistent measurement of GEF results. In line with finished and on-going work on assessing the M & E systems of the GEF partners, the Office of M & E will start up a collaborative effort with the evaluation units, offices and departments of the GEF partners to benchmark (i.e. identify best practices in) environmental evaluations and identify best practices in monitoring the specific issues that the GEF is concerned about. The challenge is to ensure that best practices in results measurement are adopted throughout the GEF family. The possibility of a benchmarking exercise will be explored in the coming months, leading to a proposal to the June 2005 Council session.

27. The Office of M & E will undertake to dialogue, consult and collaborate with all relevant partners in the GEF family in a spirit of partnership, so that an (informal) network of M & E professionals can be established which will enhance possibilities for feedback and learning, and added value in applying lessons learned.

Towards a New Division of Labor on M & E in the GEF

28. Since its initiation in 1996 the M & E unit has been responsible for coordinating monitoring and evaluation activities in the GEF. Two reasons are emerging which require a rethinking of the role of the various partners in the GEF as regards monitoring and evaluation. Firstly, monitoring if used for the purposes of keeping activities on track is first and foremost a management instrument, which should be undertaken by management rather than by an independent Office of M & E. Secondly, recent trends to move monitoring to the level of results will allow for a closer interaction with independent evaluation. Furthermore, efforts to improve basic census and other data in partner countries, as researched and managed by (inter)national bureau of statistics and similar organizations, may lead to an increased use of these data in evaluations.

29. The current monitoring system in the GEF concentrates on implementation issues rather than on progress towards achieving objectives, outcomes and impacts. It provides little or unconfirmed information on results, which is the information the Council is most interested in, although some promising efforts are underway. There has also been a very rapid growth in projects under implementation which has made it increasingly difficult for the Office of M & E to properly track and assess the project implementation reports submitted annually by the IAs.

30. As regards the performance of the implementing and executing agencies, the Secretariat has been asked by the Council to set up a system of performance indicators. After consultation between the Secretariat and the Office of M & E, it was decided to propose to the Council to transfer this responsibility to the Office of M & E. This would allow this system of performance indicators to be extended to the Secretariat as well. If the Council accepts this suggestion, the Office of M & E will present proposals to the next Council session, building on existing systems of oversight and performance audit at the IAs and EAs and including proposals to measure the performance of the Secretariat.

31. The Office of M & E proposes to explore the following distinction in monitoring roles across the GEF, to better use the monitoring capacities and resources across the GEF system.

Roles of GEF Secretariat, Implementing Agencies and Executing Agencies

32. Portfolio and project monitoring. This is essentially a management instrument. Project monitoring should remain the responsibilities of IAs and portfolio monitoring should become the GEF Secretariat responsibility. Instead of reporting to the Office of M & E, each IA and EA would report to the GEF secretariat on project implementation on an annual basis. The GEF Secretariat would be responsible for consolidating reports of IAs and presenting them to Council with the appropriate proposals and action plans to address issues that emerge during the monitoring process. This new division of labor between the Office of M & E and the GEF Secretariat should be informed also by a discussion with other GEF partners on a rationalization of M & E tools, so as to prevent duplication of efforts and to ensure that the burden of M & E on the implementation of projects and programs is reduced where appropriate.

33. Project evaluation. IAs and EAs will continue to be responsible for project evaluation and will continue to be required to submit terminal evaluations and midterm reviews to the Office of M & E.

Roles of the Office of M & E

34. Oversight and validation. The Office of M & E will play an oversight and validation role of monitoring and evaluation systems put in place by the GEF Sec, IAs and EAs to insure that GEF concerns and policies are properly addressed. The Office of M & E will expand and/or initiate collaboration in its oversight role with the evaluation departments of each IA and EA. Building on existing systems of M & E discussions will be initiated to rationalize existing instruments and tools to prevent duplication of efforts, to strengthen and support mainstreaming in the IAs and EAs and to support harmonization efforts. As part of its oversight and validation function the Office of M & E will conduct spot checks of projects and reviews of M & E systems in place, for example through participation in mid-term and terminal evaluations of the IAs and EAs. This validation and oversight role will include, among others, review of and/or the harmonization among IAs, EAs and partner countries of:

Program indicator systems.

Methodology to calculate GEF achievements.

Internal review and feedback systems in monitoring reports, midterm and terminal evaluations, including rating criteria and practices across agencies.

Review of midterm and terminal evaluations for project results and quality of reporting.

“Projects at risk” system and other systems to asses the health of the portfolio.

Criteria and practices for the development of TORs, selection of evaluators, and to insure transparency and stakeholder input in evaluations.

Procedures to ensure lessons learned are properly incorporated in policies, strategies, programs and project design or implementation.

35. Overall Performance, Crosscutting and Program Studies. The Office of M & E will submit to the Council draft four year rolling work plans. These will include GEF Overall Performance Studies (OPS) and program studies every three to four years, connected to the replenishment and Assembly cycles. It will also include proposals for crosscutting, thematic and portfolio evaluations and reviews that seek to assess progress towards GEF objectives as well as compliance with GEF policies, criteria and procedures.

36. Reporting to Council. The Office of M & E proposes to transform its current Project Performance Report into an annual GEF Performance Report to the GEF Council on issues related to the quality of M & E systems, results, and follow-up, including:

· The condition of M & E systems across the GEF family. This will address the oversight and validation role of the Office. It will include an overall assessment of the extent to which GEF policies, objectives and concerns are incorporated in the operations of the GEF Secretariat, IAs and EAs. Furthermore, it will include an annual assessment of the consistency of midterm and terminal evaluations carried out by IAs and EAs, and results of spot checks. It will also include results of selected quick reviews on specific issues that will vary from year to year.

· Accomplishments and shortcomings reported by Terminal Evaluations. A summary and an assessment of the results reported in terminal evaluations in a given year will be presented to Council.

Progress of implementation of recommendations approved by the Council (through the Management Action Record).

International Collaboration

37. The Office of M & E is required to conform to the highest internationally accepted principles, standards and practices. This means that international trends in evaluation will be reviewed closely and incorporated in the guidelines on M & E where and when appropriate. Any proposals for major changes in M & E policy or guidelines will be presented to the Council. The Office of M & E will maintain contact with the most important evaluation groups such as the OECD/DAC Network on Development Evaluation, the Evaluation Coordination Group of the international financial institutions, the UN Evaluation Group and the various professional organizations such as IDEAS (International Development Evaluators Association).

38. GEF activities are being implemented through various partnerships of international organizations (IAs and EAs) and national or non-governmental entities. Bilateral donors may be involved through co-financing. Furthermore, GEF funded activities take place in a wider context which may be guided by sector policies, country assistance strategies, national policies and/or PRSPs. In recent years, the OECD/DAC has formulated the principle that where donors finance together, they should evaluate together. The GEF Office of M & E will explore the possibility of joining international evaluations which would provide the GEF with insights and feedback which might not be reached through a stand-alone evaluation.

39. Potential benefits of joint international evaluation for the GEF are: access to data sets, activities and contextual information that is not directly related to the GEF funded activity; deeper scope of the evaluation through co-financing; enhancing quality through broad-based evaluation management; wider acceptance of findings and recommendations; greater scope for feedback. Possible drawbacks of joint international evaluations are: the longer time horizon which may endanger the timely presentation of evaluations to the GEF Council; loss of focus on GEF-related issues; reduced feedback within GEF due to less specific findings and recommendations. The OECD/DAC Evaluation Network is currently undertaking a study of lessons learned and best practices in joint international evaluation, which will be taken into account by the Office of M & E upon further deliberation of this issue.

40. In light of the increased emphasis on ownership as a prerequisite for sustainable impact, many donor organizations are transferring support activities to partner countries. This trend is reinforced by the need to establish better partnerships. This has lead to the formulation of two other principles in the OECD/DAC Network on Development Evaluation: if activities are undertaken in partnership, the evaluation should be preferably be done by the partners together; if ownership is transferred, ideally the evaluation should be transferred as well. These principles cannot at the moment be applied in more than a few countries. In order to proceed along these lines, evaluation capacity must be build in partner countries.

41. Capacity building in partner countries needs to proceed on three levels: that of individual skills, of organizational competence and of the national system which allows for independent evaluations to take place. In recent years the emphasis has been on increasing and enhancing individual skills through for example the International Program for Development Evaluation Training in Ottawa, Canada and through the professional associations. Some institutions, especially with regional mandates, have received support for increasing evaluation competence. Some partner countries have recently built independent systems of evaluation, such as Mexico, Chile and the Philippines. Only rarely has attention been given to environmental evaluation concerns. The Operations Evaluation Department of the World Bank is currently working on an overview of evaluation systems in partner countries. The Office of M & E will take the outcome of the study into account when making proposals for further steps.

42. The Office of M & E will according to its terms of reference (July 29, 2003) “develop a knowledge management strategy based on user needs and priorities and the latest technologies and approaches, subject to budget resource availability”. This strategy will need to be firmly placed in an overall knowledge management strategy of the GEF family. It must be pointed out that M & E is on the supply side of knowledge. Any knowledge management strategy from the supply side will fail if there is no structure or strategy on the demand side. Hence the Office of M & E will seek full interaction with the relevant other actors in knowledge management to further explore how its products should be made available in and outside of the GEF family.

43. Collaboration with the research community will be sought in three areas. Firstly, evaluation methodologies must be up-to-date and scientifically valid. There are many areas of concern that the Office of M & E needs to be aware of, such as research into the linkage of scarce environmental resources and human conflict; livelihood strategies in environmentally endangered areas; developments in green accounting, and so on. The Office of M & E has established contacts with a range of leading academic, NGO and private sector institutions in the environmental arena, such as Stockholm Environment Institute, Wildlife Conservation Society of Washington, Capetown University, India Institute of Management, WCMC Cambridge, Fauna and Flora International and IUCN. Initial contacts have been made to broaden this collaborative network through the inclusion of the International Food Policy Research Institute (IFPRI) in Washington, DC and the Institute of Development Studies in West Sussex, UK.

44. Secondly, the latest scientific developments must be available in the work on indicators and monitoring of trends. The Office of M & E will follow the on-going work in the Paris21 initiative and the World Bank-led initiative to build independent data gathering and analysis capacity in partner countries on the millennium development goals. STAP could potentially play an important role.

45. Thirdly, in order to establish impact, impact assessment must be an integral part of project preparation. Most impact assessments suffer from a lack of baseline data, which can rarely be reconstructed from the past in a precise or cost-effective manner. A collaboration with the Research Department of the World Bank will be explored to see whether it is possible to include impact assessment concerns in some pilot projects. Furthermore, impact assessments will become part of the evaluation tools of the Office of M & E, which will be further explored in the draft four year work plan and budget.

46. While following trends in science, technology and the (inter)national statistical community, care will be taken that new evaluation methodologies will be introduced in collaboration with GEF partners and in a timely and proper fashion – without disrupting well established practices and existing guidelines.

Recommended Council Decision