Monitoring and Evaluation

Planning the Monitoring System

AGI Lessons Learned

Monitoring is necessary at every stage of project implementation. Monitoring is needed during outreach activities, the application and shortlisting processes, skills training, and importantly, when graduates are in jobs/internships/businesses after the training ends. Oftentimes skills training projects may pay attention to what happens during the classroom phase, but they fail to monitor what happens before the classroom training starts or after the training ends.

Individual attendence monitoring (not just at the classroom level) is critical for improving the quality of skills training projects. AGI training providers were rarely required to digitize and submit attendance records at the individual level. As a result, projects are mostly unable to examine whether more vulnerable participants (for example, younger girls, poorer girls, girls with children) dropped out of the program at higher rates or whether trainees with lower attendance had worse labor outcomes. For those projects that did successfully monitor individual attendence, capacity building to support trainers to systematically collect this information was essential.

M&E plans should be detailed, but they should also be simple and realistic. While project monitoring plans need to be comprehensive and support monitoring at multiple levels with appropriate feedback loops, they also need to be straightforward and understandable. Roles and reponsibilities and accountability mechanisms for actors at all levels should be clearly articulated. M&E plans should also support the development of practical guides and monitoring tools for important project elements so that monitors have the tools they need to support their tasks.

M&E plans should be developed and implemented in a participatory manner. Traditional policing methodologies have created a sense of fear toward M&E. Skills training providers may be unaccustumed to rigorous monitoring and may need to be encouraged to comply and buy into the process. M&E plans should be built on principles of mutual respect and trust (for instance, the monitor is not a boss or supervisor and should serve with integrity) and transparency and equity (for example, all service providers should be appraised on a set of common agreed standards). Involve monitors and service providers in the definition of indicators and the development of monitoring tools. Involve service providers in the training for monitors.

Integrate a capacity-building strategy for the monitoring team into the M&E plan. Provide monitors and service providers with technical assistance, problem-solving support and encouragement. Particularly in low-resource settings, the team may need training in basic computer literacy and electronic data maintenance. Ensure that the M&E team is fully familiar with the project objectives and that they understand the valuable contribution they are making. In addition to capacity-building efforts, involve the M&E team in other project activities and keep them invested in the project's success, for instance, by involving them in routine project meetings, the project launch, and other social events.

Monitoring is an ongoing data collection process of the program outputs. Particularly in the project pilot phase, it is important to gather detailed monitoring data to help identify and correct unforeseen weaknesses in the project design and to replicate successful features during scale-up. The best projects are those that can flexibly adapt their design in response to emerging realities. The M&E system is the main tool to help in the adaptation process.

Each AGI pilot was guided by an M&E plan that outlined the objectives and components of the project’s M&E system. An M&E plan is critical to developing an effective M&E system. The plan should be developed during the project design phase and should include the following elements (adapted from Gertler et al. 2011):

M&E Plan Components	Component Description
Project Overview	Project objective, components, timeline, and target beneficiaries
Results Framework	Expected results (outcomes and outputs) and indicators with baseline and target values
Data sources	Such as a survey, a review, stakeholder meetings, and so on
Responsibilities	Team members responsible for collecting or providing the data (for example, independent evaluation team, project staff, and so on)
Monitoring Schedule	Frequency of data collection, analysis, and reporting
Budget	Explanation of resources required for carrying out planned M&E activities
End Use	A plan for the dissemination and utilization of the information gained
Risks	Assumptions and risks associated with the planned M&E activities

Monitoring plans should be both inclusive (engage stakeholders at all levels of project implementation) and comprehensive (establish a systematic approach to project monitoring that tracks all project activities from beginning to end of the project cycle, tracking sex-disaggregated indicators), as detailed, below:

Strategy	Resources
Monitoring plans should engage all project stakeholders in monitoring responsibilities. For example, in the Liberia AGI, the project implementation unit provides oversight and capacity building to the training agencies; the training agencies provide internal monitoring of the training sites and the implementation milestones; the trainers monitor attendance and trainee performance; and a team of quality monitors provide feedback to the project implementation unit based on random and unannounced spot checks and a set of monitoring checklists.	Liberia EPAG Monitoring Plan Liberia EPAG M&E System Framework (PPT) Liberia EPAG Monitoring from the Point of View of the Service Providers (PPT)
Monitoring systems should have accountability mechanisms to ensure that all project activities are monitored during the project cycle, from start to finish. Skills training projects often neglect to monitor key steps in the implementation, such as recruitment, attendance during training, training completion, training content, and posttraining activities.	Key Processes that Skills Training Projects Should Monitor
Coeducational skills projects should collect monitoring data at the individual level and disaggregate the analysis by sex to ensure that young women and men are benefitting equally and to course correct when necessary. For example, the Nepal Adolescent Girls Employment Initiative (AGEI) and the Lao PDR pilot were unique among the AGI pilots because they were each part of a larger co-ed skills training program.	Nepal Employment Fund Monitoring Guidelines

Monitoring plans should include tools to guide the project team in its monitoring activities. The following table presents key AGI monitoring tools:

Monitoring Element	Resources and Tools
Individual attendance monitoring is needed to examine whether more vulnerable participants (for example, younger girls, poorer girls, girls with children) drop out of the project at higher rates or whether trainees with lower attendance had worse outcomes in the labor market. In Liberia, for instance, the project devoted resources to analyze attendance data and work with training providers to improve attendance and remobilize dropouts (and their families). This oversight was conducted with a spirit of collegiality rather than policing, which encouraged training providers to report attendance rates honestly without fear of penalty. Attendance records should be digitized—capacity building of training providers was sometimes required to keep attendance accurately and enter it into a spreadsheet.	Liberia EPAG Trainees Attendance Tracking and Trainees Directory Rwanda Attendance and Stipend Monitoring Tool
Key quality monitoring tips: Develop a monitoring schedule that accommodates the availability of the monitors Appoint a Monitors’ Supervisor to coordinate fieldwork (maintain and distribute working tools and equipment, collect scored checklists, write monthly activities report, and so on) Conduct monitoring visits frequently. In Liberia, for example, the monitoring plan allowed for two days of monitoring per week. The Liberia EPAG project conducts routine, random, unannounced quality monitoring visits to all classroom trainings to assess the training venue and the quality of training. Quality monitors submit paper forms to their monitoring supervisor, who is responsible for digitizing the records and sharing findings with the training providers. Monitor in pairs. In Liberia, monitor pairs both observe training delivery for about 15 minutes; one person scores the indicators while the other interviews two trainees selected on a random basis. Assign analysis and reporting duties. In Liberia, both monitors tally scores and report filled forms to supervisors who submits data to the M&E Officer. Provide timely feedback. In Liberia, feedback is provided to training providers within one week. For serious or urgent concerns, senior management follows up with the training provider the same day.	Liberia EPAG Quality Monitoring Guideline Liberia EPAG Training Venue Assessment Checklist Liberia EPAG Classroom Observation and Interview Forms
EPAG promotes positive competition among service providers by aggregating quality monitoring scores across providers and recognizing the best performers. Sharing the consolidated quality monitoring report keeps training providers “on their toes” and enhances performance.	Liberia EPAG Quality Contests Tool
Monitoring does not stop after the classroom training ends (see Verifying Employment). It is important to follow-up with trainees to monitor their performance in their businesses or jobs. Monitoring can alert the service providers to participants who may be in need of additional support or follow-up services to help get back on track. Monitoring may also improve participants' performance in their jobs if they know that someone is checking in and monitoring their performance.	Liberia EPAG Business Performance Monitoring Tool Liberia EPAG Job Performance Monitoring Tool