ROMANIA
AGRICULTURAL POLLUTION CONTROL PROJECT
WORKING PAPER 14
PROJECT MONITORING AND EVALUATION SYSTEM
Project Preparation Unit
Calarasi
ROMANIA
AGRICULTURAL POLLUTION CONTROL PROJECT
Working Paper 14
Project Monitoring and Evaluation System
Table of Contents
A. Monitoring Physical Execution of Project
B. Environmental Monitoring
C. Social and Economic Assessments
Annexes:
Annex 1: Project Design Summary
Annex 2: Project Performance Indicators
Annex 3: Terms of Reference for Beneficiary Survey
2
A. MONITORING PHYSICAL EXECUTION OF PROJECT
Monitoring/Evaluation System
A well-designed monitoring and evaluation system will be critical for ensuring the project's timely
and successful implementation, and enhancing its impact by a systematic analysis of lessons learned
and their effective dissemination.
Project monitoring and evaluation would be the responsibility of the PMU, specifically the Technical
Specialist to be appointed once the project becomes effective.
The M&E system proposed will use the following three approaches.
·
Regular Monitoring through quarterly and annual progress reports of the physical
implementation of project activities;
·
Concurrent Evaluation of physical and financial impact of improved practices at
the farm and village level;
·
Ex-post Evaluation of the financial and economic impact of project activities
Project Performance Monitoring:
Monitoring of project performance will be based on the baseline survey undertaken during the
preparation phase of the project. Extensive data by comunas and villages have been collected and the
Public Health Directorate and the EPI-Calarasi have provided baseline data for soil and water quality
levels. The PMU would annually monitor and evaluate project performance through conducting
beneficiary surveys. The results of M&E activities will be fed back into the implementation process
as improved practices.
An internationally recruited expert would assist the PMU to design a simple Management
Information System (MIS) for M&E, reporting formats for each component, including targeted
annual performance objectives and monitoring indicators using the Project Design Summary, Annex
1, and the proposed indicators in Annex 2, as a basis.
Reporting Arrangements:
Quarterly reports will cover progress in physical implementation, the use of project funds and project
impact. The Quarterly reports will be consolidated by the PMU into half-yearly progress reports to
be submitted through MWEP to the Bank within two months of the end of each six-month reporting
period. These half-yearly progress reports will also include an implementation plan and work
program for the next six months following the reporting period. The format of reports will be agreed
with the Government and the World Bank.
The PMU would submit the monitoring reports to the GEF/World Bank and Government.
Mid-term Review:
A mid-term review will be carried out to assess overall progress. Lessons learned, with
recommendations for any improvements, would be used in restructuring the project, if necessary.
3
B. ENVIRONMENTAL MONITORING
Agricultural Practices to Reduce Environmental Pollution:
A comprehensive monitoring system will be put in place to monitor the environmental impact of
project actions as described in Working Paper 12.
C. SOCIAL AND ECONOMIC ASSESSMENTS
Social Assessment:
The social assessment (SA) is a crucial mechanism for establishing how effectively the Project meets
its goals. The farmers' attitude to environmental management will affect project results and potential
replication elsewhere.
The baseline survey completed during project preparation has ascertained current attitudes to
agriculture and environmental practices, potential farm demand for support such as that offered by
the Project, access to information, and attitudes to institutions. The SA team will repeat the baseline
survey at two-year intervals so as to track changes resulting from project activities.
The second instrument will be beneficiary surveys with a sample of participating households.
Householders will be interviewed shortly after joining the Project, and one or two years after
completing the investment. The interviews will focus on the level of interest and expectation,
satisfaction or dissatisfaction, and on how benefits from the manure management system are
perceived by the beneficiaries. The interviews should also focus on eligible households not in the
Project to help managers understand the reasons why they did not participate or withdrew.
The social assessment process will evaluate Project impacts on the farmer and the household, the
roles of various cooperating parties in supporting the beneficiaries, and the effects of public
awareness and training programs from the perspective of both the beneficiaries and the broader
community. It will continue throughout implementation, to provide feedback on Project design,
management, and implementation. The procedures and practices in this Operational Manual) should
be changed in response to the lessons learned from this social assessment. Project managers,
consultants, contractors and stakeholder groups will all use this information.
The PMU will contract an independent organization to conduct the social assessment under terms of
reference to be agreed with the Bank.
Economic Assessment:
If this Project is to be replicated nationally or internationally, it needs to evaluate the financial and
economic impact of its activities as compared to the effectiveness of other intervention methods
under similar programmes in Romania and in other countries. It should also study the impact of its
interventions on farm profitability. The Economic Assessment will also ascertain farmers'
willingness and ability to finance future interventions, and the economic justification of such
interventions on a farm/regional/national level.
The PMU will contract an economic assessment to address these issues with terms of reference to be
agreed with the Bank.
4
Annex 1
ROMANIA: AGRICULTURAL POLLUTION CONTROL PROJECT
Project Design Summary
Hierarchy of Objectives
Key Performance
Indicators
Monitoring & Evaluation
Critical Assumptions
Sector-related CAS Goal:
Sector Indicators:
Sector/ country reports:
(from Goal to Bank Mission)
Protection and sustainable
Gradual Improvements in soil Agricultural statistics
Improved agricultural
management of natural
and water quality
Periodic data collection on soil practices contribute to national
resource and the environment
and water quality of major
economy through an increase
water bodies, by EPI
in average incomes, and
environmental enhancement
Assist Romania in
Capacity to address
National reports
EU membership is also likely
implementing the National
environmental degradation of
to increase average incomes
Environment Action Plan
the Black Sea.
(NEAP)
Assist process of integration
Progress towards meeting
Periodic EU assessments
Policy standards adopted meet
with the European Union
environmental compliance
EU requirements
targets with EU legislation
Strengthen the capacity of
Environmental Protection
Inspectorate (EPI) and Public
Health Directorate (PHD) in
Calarasi
GEF Operational Program:
The Project's objective of
Increased awareness of threats Agricultural Statistics
Government's ability to
reducing nutrient discharge to to pollution of trans-boundary Regional Surveys
mobilize resources to reduce
Danube river and Black Sea is water bodies from nutrients
Periodic collection of data on threats to water bodies and
consistent with OP No. 8,
from animal waste and
water quality from major water build institutional capacity for
Water body-based Operational agricultural chemicals
bodies in the project area by
future environmental
Program which focuses mainly
EPI & PHD
challenges
on threatened water bodies and
the most important trans-
boundary threats to their
ecosystems. Project goals are
also consistent with OP No. 9,
Integrated Land and Water
Multiple Focal Area
Regional
Surveys
Sustained effort to raise the
public awareness and demand
for protection and
improvement to environmental
factors
5
Global Objective:
Outcome / Impact Indicators: Project reports:
(from Objective to Goal)
To increase significantly the
Increased awareness of
Agricultural statistics
Project-developed
prevalence of environment-
environmental issues in
Water quality data sets
interventions are replicated on
friendly agricultural practices agriculture among
Social Assessment
a wide scale.
among farmers' associations,
farmers/households within and
family farms and other eligible outside project area.
Economic and Financial
Adoption of improved
farmers in target project areas.
Assessment
environmental policies by
The ultimate goal is to reduce
Increased area of adoption of
government to address non-
the discharge of nutrients and
production and resource
Annual regional and national point agricultural pollution
other agricultural pollutants to conservation technologies.
reports
control.
the Danube River and Black
Sea through integrated land
High satisfaction rate among
Interviews with farmer
and water management.
participating farmers.
groups and local
governments
High percentage of
participating farmers
implementing environment-
friendly agricultural practices
at least two years after joining
the project.
Output from each
Output Indicators:
Project reports:
(from Outputs to Objective)
Component:
1. Calarasi Judet
ASSP, CGS, sub-projects
Promotion of new
Quarterly reports from APCP Technologies respond to
with environment-friendly
environment-friendly
and CGS (ASSP)
farmers' needs.
focus in the Judet.
agricultural prtactices
Packages developed for
High level of participation (all Quarterly reports
manure management
communas, all villages and
65% of individual farmers) in
target areas that have built
manure stores etc.
A well documented pilot
High level of participation (all Quarterly reports
completed and evaluated for
comunas, all villages and 65 %
Markets and prices provide
replication.
of individual farmers) in target
sufficient incentives to
areas where nutrient
producers and processors.
management plans have been
developed and other
environment-friendly practices
evaluated/demonstrated.
Sustainable management
Use of environment-friendly
Quarterly reports
Continued land use based on
adopted in Boianu-Sticleanu
agricultural practices.
EPI monitoring reports and
plans developed. Other
polder.
Area planted to agro-forestry. periodic collection of water
government programs do not
Management plan adopted for quality data.
conflict with project goals.
Iezer-Calarasi reserve.
Improved water quality in
drainage canals.
Good monitoring system for
Better soil and water quality
Annual monitoring reports
Continued adequate support
water and soil quality.
monitoring.
from EPI and Public Health
from local and national
Directorate, Calarasi
government continues for
carrying out the components
6
2. National Policy and
Regulatory Capacity
Improved policy framework
Policy framework adopted
Government legislation
Continued support and
applied for non-source
for non-source pollution
enforcement of policy
pollution control.
meets EU criteria.
Code of Good Agricultural Adoption of code monitored Quarterly reports
Provide resources to monitor
Practices adopted
by EPI
and regulate standards.
Strengthening of institution for Information system and
Agricultural statistics
Organic farming.
legislation ion place.
3. Public Awareness &
Replication:
Increased knowledge &
Public awareness
Social assessment sample
awareness of ways to reduce
Adoption of environment-
surveys
nutrient pollution of water
friendly agricultural practices Quarterly reports
bodies in Calarasi Judet.
Increased awareness of ways
Public and farmers aware of
Social assessment sample
to reduce nutrient discharge
the potential to improve
surveys
Allocation of resources
from agriculture in other
income while protecting the
Judets.
environment.
Quarterly reports
Demands from other local
governments for replication
of project investments.
Increased Awareness and
Visits and awareness of
Quarterly reports
Farmers and leaders in other
demand for replication in the
farmers, NGOs, and officials
countries become interested in
region.
of other countries of the
reducing non-point source
project in the Calarasi Judet
pollution from agriculture and
allocate resources to replicate
project activities.
4. Project Management
Well-managed project.
Continued support from the
Supervision Reports
Adequate availability of
Project Steering Committee
necessary institutional
and Project Co-ordination
support by government
Committee.
agencies.
7
Project Components / Sub-
Inputs: (budget for each
Project reports:
(from Components to
components:
component)
Outputs)
1. Calarasi Judet
US$ 9.22 million
Matching grant for manure
US$ 5.20 million
Progress Reports (quarterly)
Local government support
management practices
the pilot initiative by
contributing resources.
Testing and demonstrating
US$2.47 million
Progress Reports (quarterly)
Project incentives are
environment-friendly
sufficient to motivate farmers
agricultural Practices
to participate in the project
Integrated management of
US$1.09 million
Progress Reports (quarterly)
Enforcement of land-use plan
Boianu-Sticleanu Polder and
ecological restoration of the
Calarasi-Raul Polder.
Monitor Soil and Water
US$0.46 million
EPI and PHD annual reports of Implementing agencies able to
Quality and Environmental
soil and water quality.
attract and retain qualified
Requirements.
Sample survey
staff.
2. National Level
US$ 0.27 million
Develop policy framework for US$0.09 million
Draft of appropriate policies
Continued support and will for
non-point source pollution.
enforcing policies
Develop Code of Good
US$0.12 million
Draft of Code
Agricultural Practices.
Promotion of organic
US$0.06 million
Status Institutional frame-work
farming.
3. Public Awareness &
Replication Strategy(US$
0.45 million)
Public awareness in Calarsi
US$0.21 million
Annual social assessment
Timely availability of
Judet
sample survey
counterpart funds
Public awareness, and
US$ 0.17 million
Sample Survey
Continued support for
replication national level
implementing agency
Regional cooperation for
US$0.07 million
Progress Reports (quarterly)
Ability to interact with each
replication
other for mutual benefit.
4.Project Management Unit
US$0.86 million
Project Administration
US$0.65 million
Progress Reports (quarterly)
Ability to maintain staff,
offices and support from
local government and
communities
Project
Progress Reports (quarterly)
Monitoring/Evaluation
US$0.21 million
8
Annex 2
Table 18: Baseline and Targets for Key Performance Indicators*
(Figures for indicators marked with * are cumulative)
KEY PERFORMANCE INDICATORS
Baseline
2002
2003
2004
2005
2006
2010
2001
PROJECT IMPACT INDICATORS:
a) Percentage of households with livestock in
project area using village manure storage,
-
5%
15%
25%
35%
45%
65%
household bunkers and segregating waste
materials.*
b) Percentage of area under nutrient
management systems including crop rotation,
crop nutrient management with soil testing,
-
<1%
2%
10%
20%
30%
65%
and use of organic manures.*
c) Percentage of area under environment-
friendly agricultural practices.*
<1%
2%
10%
20%
30%
65%
d) Trend in water quality indicators for
designated sites.
TBD
TBD
TBD
TBD
TBD
TBD
PROJECT OUTPUT INDICATORS:
1.1 Calarasi Judet
1.1.1
Number of environment-
friendly practices promoted
2
5
7
10
using CGS.*
1.2 Project Area of Seven Comunas
1.2.1
Percentage participation by
households in areas that have built
village-level manure stores and
-
5%
15%
25%
40%
65%
associated household bunkers.*
1.2.2
Percentage of farmers participation
in areas where nutrient
management plans have been
-
<1%
10%
20%
40%
65%
introduced.*
1.2.3
Percentage of farmers adopting one
or more environment-friendly
practices in areas where
-
<1%
10%
20%
40%
65%
demonstrated.*
1.2.4
Percentage of vulnerable terrace
areas planted to trees.
10%
26%
45%
65%
100%
1.3.1 Conservation management
plan operating in the Iezer-
X
X
X
X
X
Calarasi Nature Reserve.
1.3.2
Percentage of vulnerable areas in
Boianu-Sticleanu polder planted to
-
2%
10%
30%
60%
100%
100%
trees.*
1.3.3
Percentage of arable area in
Boianu-Sticleanu polder where
principles of Code of Good
-
<1%
10%
25%
55%
80%
100%
Agricultural Practices adopted.*
1.3.4
Percentage of Calarasi-Raul
Polder restored to wetland.*
-
-
-
5%
20%
20%
9
KEY PERFORMANCE INDICATORS
Baseline
2002
2003
2004
2005
2006
2010
2001
1.4.1
Water quality indicators for terrace
area.
TBD
TBD
TBD
TBD
TBD
1.4.2
Water quality indicators for
drinking wells..
TBD
TBD
TBD
TBD
TBD
1.4.3
Water quality indicators for
drainage/irrigation canals in the B-
TBD
TBD
TBD
TBD
TBD
S Polder.
2. Strengthening National Policy & Regulatory
Capacity
2.1 Policy framework for
non-point source pollution meets
X
EU criteria.
2.2 Code of Good Agricultural
Practices adopted
X
2.3 Institutional framework to
promote Organic Farming
X
strengthened.
3. Public Awareness & Replication
3.1
Increased knowledge & awareness
of ways to reduce nutrient
X
pollution of water bodies in
Calarasi Judet.
3.2
Increased awareness of ways to
reduce nutrient discharge from
X
agriculture in other Judets.
3.3
Increased Awareness and demand
for replication in the region.
X
4. Project Management Unit
4.1 PMU established and dealing
Effectively with all project
X
X
X
X
X
matters.
4.2 Monitoring and Evaluation
system in place and reporting.
X
X
X
X
X
PROJECT COMPONENT INPUT
INDICATORS:
1.1 Calarasi Judet
1.1.1
Number CGS sub-projects for
environment-friendly practices
-
3
4
4
4
4
awarded in the Calarasi Judet.
1.2 Project Area of Seven Comunas
1.2.1
Number of village-level manure
storage facilities constructed.*
2
6
10
14
14
1.2.2
Number of household bunkers
constructed.*
500
1500
2750
4000
4000
1.2.2
Number of villages with manure
collection, handling and spreading
2
6
10
14
14
systems.*
1.2.3
Number of demonstration sites for
environment-friendly practices
5
10
15
25
30
established.*
1.2.4
Area of trees planted in vulnerable
terrace areas.*
45
113
196
297
432
1.2.5
Number of training workshops
held.
3
3
5
4
2
1.2.6
Number DGAIA/OJCA staff
trained.*
5
10
20
25
25
10
KEY PERFORMANCE INDICATORS
Baseline
2002
2003
2004
2005
2006
2010
2001
1.3.1
Conservation management plan for
the Iezer-Calarasi Nature Reserve
X
prepared.
1.3.2
Conservation management plan for
the Iezer-Calarasi Nature Reserve
X
30%
60%
80%
100%
implemented.
1.3.3
Area trees planted in vulnerable
areas of Boianu-Sticleanu polder.*
30
110
320
635
1090
1.3.4
Plan for ecological restoration of
Calarasi-Raul polder prepared.
X
1.4.1
Ecological restoration of Calarasi-
Raul polder implemented.
X
X
X
1.4.2
Number of piezometers
established.*
28
44
60
76
76
1.4.3
Number of piezometers established
for monitoring manure stores.*
4
8
24
40
56
1.4.4
Number of drinking wells
monitored.
62
62
62
62
62
1.4.5
Number of irrigation/drainage
canals monitored.
3
3
3
3
3
2. Strengthening National Policy & Regulatory
Capacity
2.1 MWEP unit equipped
X
2.1 MAFF/MWEP group established
to work on Code of Good
X
Agricultural Practices.
2.2 MAFF Organic Farming Unit
using TA
X
X
3. Public Awareness & Replication
3.1 Multi-media program developed
for Calarasi Judet.
X
3.2 Nationwide public information
Program developed.
X
3.3 Regional workshop program and
Exchange of views at regional
-
-
1
1
1
Level.
4. Project Management Unit
4.1 Manager and staff of PMU
Appointed.
X
4.2 Staff trained in administrative,
financial management and
X
M/E procedures, and systems
established.
11
WP 14
Annex 3
Romania
AGRICULTURAL POLLUTION CONTROL PROJECT
TERMS OF REFERENCE
Beneficiary (Household) Survey
OBJECTIVE:
The general objective of the assignment is to repeat the baseline household survey at regular intervals
to determine the impact of the project. The data obtained will be centralized an analyzed according
with the WB Social Assessment Unit, the conclusions representing an important input in the
monitoring of the project components.
TASKS:
The principal tasks will be as follows:
(i) Update the questionnaire to be distributed in the area with the purpose to reflect the current
situation from farmers perspective. The data collected will represent part the baseline data for the
preparation of the project.
(ii) Translate the final version of the questionnaire in Romanian and prepare the sampling method.
(iii) Train people from Calarasi (OJCA, DGAIA, EPA ) in how to interview the farmers and how to
fill in the questionnaire. The consultant will also go at the beginning in the field with each one the
trainees in order to supervise their activity.
(iv) The consultant will collect the questionnaires and screen them (taking out the ones which are not
relevant).
(v) Data analysis and interpretation will be done with the support of
(vi) The conclusions of the survey will be communicated to all the other bodies and local
administration representatives (DGAIA, EPA, OJCA, Public Health Department, County Council,
Prefecture)
OUTPUTS (DELIVERABLES):
(i) Updated Questionnaire and sampling method
(ii) Trained staff in interviewing farmers (applying the questionnaire)
(iii) Indicators of social impact t against baseline
(iv) Final report (data analysis and conclusions)
12
REPORTING AND TIMING:
The consultant will report to the Administrator, PMU, who will agree the work plan with the
consultant at the start of the assignment, as well as the timing for, and contents of the draft report.
The Draft Report will be reviewed by the Administrator and Technical Specialist , PMU, and their
comments will be incorporated in the Final Report
The following documents will be submitted to the PMU:
· The original set of questionnaires filled in during the survey
· The results of the first screening of the questionnaires
· The final report, which will include the data analysis and the conclusions reached and the
indicators measuring the social impact of the project.
The report should be presented in English and Romanian and should be submitted in two copies to
the PMU.
13