Skip to 0 minutes and 13 seconds In the final two steps of the planning cycle, planners consider the implementation and impact of a project. Step five is the monitoring phase. This is planned in detail at the beginning of the project, and then managed throughout the project lifespan. Let’s consider the basics of monitoring using a simple example. Imagine that you want to travel from point A to point Z. First you need to acquire the necessary resources. For example, a car or money to buy fuel. These are called the inputs. Now you can plan the journey, the route you will travel, how many people will be traveling, and how much fuel is needed. You can also estimate the time for traveling.
Skip to 0 minutes and 53 seconds Once the journey starts– the implementation of the plan– every few hours, you look at the fuel gauge, check the distance traveled, and perhaps ask how the passengers are doing. This is monitoring. You may have planned to assess this information at several points along the way. Here, you decide if the car needs to be refueled, or if a break is necessary to rest the passengers. Information on fuel, distance, and how the passengers are doing are referred to as indicators. Indicators tell you how the journey is going. Monitoring activities check if a project is proceeding according to the plan. Are you doing what you said you will do? Monitoring is important, because what gets monitored is more likely to get done.
Skip to 1 minute and 38 seconds If you don’t monitor performance, you can’t tell success from failure. If you can’t see success, you can’t reward it. And if you can’t recognise failure, you can’t correct it. Finally, if you can’t demonstrate results, you can’t sustain support for your actions.
Skip to 1 minute and 56 seconds In any plan or programme, achievement is aligned with completion of objectives. There are usually several activities that need to be carried out for an objective to be achieved. Each activity requires inputs, such as finances, resources, and so on, and for specific tasks to be completed. This is known as the process.
Skip to 2 minutes and 20 seconds Inputs in a diabetic retinopathy screening programme include correctly equipping service providers to enable them to undertake their required tasks.
Skip to 2 minutes and 32 seconds As a result of following the process, the planned activities and objectives can be achieved. This in turn leads to an outcome and an impact. Indicators measure what or how much has been done. Process indicators provide information on tasks done and inputs consumed as part of activities to achieve objectives. Process indicators are collected regularly on a weekly, monthly, or quarterly basis. Outcome indicators are used during a project to assess if the path taken is working well, and if changes need to be made to the plan’s objectives. These are collected over longer intervals, once or twice a year. An impact indicator is an indication of change that has resulted from a plan.
Skip to 3 minutes and 17 seconds Impact indicators are collected and reported on at the end of a project, after the objectives have been completed. In a programme, it is important to decide what information should be collected and when. Planners must decide on what could be monitored, what should be monitored, and what must be monitored.
Skip to 3 minutes and 41 seconds Here are some examples of process indicators. Number of persons with diabetes screened per month, number of persons with sight-threatening diabetic retinopathy identified and referred per month, number of patients treated for sight-threatening diabetic retinopathy per month.
Skip to 4 minutes and 6 seconds Here are some examples of outcome indicators. The percentage of people with diabetes who are provided with screening each year. This is known as the coverage.
Skip to 4 minutes and 18 seconds The percentage of people with diabetes and sight-threatening diabetic retinopathy who are referred and accept treatment per year.
Skip to 4 minutes and 30 seconds The key impact indicator that the programme will aim for is the ongoing coverage of the DR screening service. That is, the percentage of the population with diabetes being provided with annual DR screening compared to the baseline.
Skip to 4 minutes and 48 seconds Golden rules for monitoring, - do not collect too many monetary indicators, or collect indicators too often. - Use all the monitoring indicators you collect and discard indicators that aren’t used. - Use the monitoring indicators at the level that they are collected at, as process, outcome, or impact indicators. - Educate staff about the need to collect monitoring indicators. - And don’t make things worse. Don’t destroy a monitoring system that works.
Skip to 5 minutes and 19 seconds Managing monitoring. Planners need to decide who collects the indicators at each level. Once the data has been collected, where are the reports sent, and who will review them? And how will the programme act on the feedback from the review? These are key details that must be managed by the programme or project manager.
Skip to 5 minutes and 40 seconds Appropriate selection and training of key people to carry out the monitoring is essential.
Skip to 5 minutes and 49 seconds Monitoring reports are collected and reflected on by the programme or project manager on a regular basis. It’s important that they also give feedback to staff about how the plan is doing. The manager reports to the national coordinator. At the end of the programme period, it is the final responsibility of the national coordinator to identify resources and address the impact of the programme or project. Collecting the right data is important to guide the project or programme and its success. There is a saying that if you put rubbish in, you get rubbish out. So planners must remember to select indicators properly.
Skip to 6 minutes and 28 seconds In summary, - Monitoring is important because it improves accountability for the use of funds and resources, - improves performance to achieve outcomes, - and provides a system for recording lessons learned. These can then be shared to improve strategies in the future.
Monitoring diabetic retinopathy programmes
Monitoring & evaluation are terminologies that are often spoken about together but they are not the same process or done at the same time.
Monitoring is concerned with the ongoing oversight of a programme as it is implemented. It helps you to verify whether the programme is running as planned and reaching the people it is meant for. Monitoring is used to guide implementation and must allow for revisions and changes to a programme as required.
Evaluation is an objective assessment of an ongoing or, more commonly, a completed programme. It can be done at the mid-point (mid-term evaluation) or at the end (terminal evaluation) of the programme. An evaluation looks at a whole programme and establishes if it has delivered its objectives and outcomes in the planned time frame. It also examines whether a programme is sustainable, or can be scaled up, in the future.
The key to an effective monitoring system is to collect and use the ESSENTIAL and MINIMAL core data needed to chart overall progress and NOT to simply re-use the same data as collected for day-to-day project management.
Data can be collected manually or digitally and from many different sources for monitoring purposes. The source will be dependent on the specific objectives of the programme.
A health information management system (HMIS) is used in many places. An HMIS enables the collection, storage, reporting, processing, analysis and dissemination of health-related data, based on a programme’s monitoring needs. Programmes may need to design their own data formats to collect specialised information, for example to assess:
- Which geographical areas are using services better than others
- Patient satisfaction with services
- The overall quality of life of individuals receiving a service.
All data formats should be pre-tested before being finalised for use in an HMIS.
Monitoring indicators are categorised based on the component of the programme they are being used to assess and are measured at different times. Generally, four types of indicators are used:
- Input indicators assess whether the required inputs (e.g human resources, equipment, finances) are in place to achieve the programme’s objective and targets
- Process indicators are directly linked to activities that are expected to have been completed in a given time. For example, number of patients screened per month
- Outcome indicators measure how closely what is happening in the programme compares to what was originally planned (the set targets). Data from process indicators is collated together to track and draw a conclusion on the outcome. For example, the total number of people with diabetes who have been screened at the end of the first year
- Impact indicators measure how successful the programme has been or the percentage change that it has brought about. For example, the prevalence of blindness from DR at the end of 5 years.
As you watch the video, consider why it’s not possible to have a universal monitoring protocol that can be applied across all DR screening programmes?
© London School of Hygiene & Tropical Medicine CC BY-NC-SA 4.0