Understanding ‘The Monitoring and Evaluation Plan’

October 4, 2021

A monitoring and evaluation (M&E) plan is a document that helps to track and assess the results of the interventions throughout the life of a program. It is a living document that should be referred to and updated on a regular basis. While the specifics of each program’s M&E plan will look different, they should all follow the same basic structure and include the same key elements.

An M&E plan will include some documents that may have been created during the program planning process, and some that will need to be created new. For example, elements such as the logic model/logical framework, theory of change, and monitoring indicators may have already been developed with input from key stakeholders and/or the program donor. The M&E plan takes those documents and develops a further plan for their implementation.

 

Why develop a Monitoring and Evaluation Plan?

It is important to develop an M&E plan before beginning any monitoring activities so that there is a clear plan for what questions about the program need to be answered. It will help program staff decide how they are going to collect data to track indicators, how monitoring data will be analyzed, and how the results of data collection will be disseminated both to the donor and internally among staff members for program improvement. Remember, M&E data alone is not useful until someone puts it to use! An M&E plan will help make sure data is being used efficiently to make programs as effective as possible and to be able to report on results at the end of the program.

 

Who should develop a Monitoring and Evaluation Plan?

An M&E plan should be developed by the research team or staff with research experience, with inputs from program staff involved in designing and implementing the program.

 

When should a Monitoring and Evaluation Plan be developed?

Monitoring and Evaluation plan should be developed at the beginning of the program when the interventions are being designed. This will ensure there is a system in place to monitor the program and evaluate success.

Who is this guide for?

This guide is designed primarily for program managers or personnel who are not trained researchers themselves but who need to understand the rationale and process of conducting research. This guide can help managers to support the need for research and ensure that research staff have adequate resources to conduct the research that is needed to be certain that the program is evidence based and that results can be tracked over time and measured at the end of the program.

Steps

Step 1: Identify Program Goals and Objectives

The first step to creating an M&E plan is to identify the program goals and objectives. If the program already has a logic model or theory of change, then the program goals are most likely already defined. However, if not, the M&E plan is a great place to start. Identify the program goals and objectives.

Defining program goals starts with answering three questions:

  1. What problem is the program trying to solve?
  2. What steps are being taken to solve that problem?
  3. How will program staff know when the program has been successful in solving the problem?

​Answering these questions will help identify what the program is expected to do, and how staff will know whether or not it worked.

Example: If the program is starting educational skill uplift for girls, the answers might look like this:

Problem High rates of unskilled Girls
Solution Skill enhancement of girls in respective community centers
Success Lowered rates of unskilled Girls in locality

From these answers, it can be seen that the overall program goal is to reduce the rates of unskilled and unaware girls in the community.

It is also necessary to develop intermediate outputs and objectives for the program to help track successful steps on the way to the overall program goal.

Step 2: Define Indicators

Once the program’s goals and objectives are defined, it is time to define indicators for tracking progress towards achieving those goals. Program indicators should be a mix of those that measure process, or what is being done in the program, and those that measure outcomes.

Process indicators track the progress of the program. They help to answer the question, “Are activities being implemented as planned?” Some examples of process indicators are:

  • Number of training held with Girls
  • Number of outreach activities conducted at girls-friendly locations
  • Number of courses taught at girls-friendly locations
  • Percent of girls reached with program awareness messages through the media or WOM

Outcome indicators track how successful program activities have been at achieving program objectives. They help to answer the question, “Have program activities made a difference?” Some examples of outcome indicators are:

  • Percent of girls turning up for course in first round
  • Number and percent of trained peer leaders from that from local community  providing services to girls
  • Number and percent of new girls enrolling in course and also, those turning up for jobs with latest skills acquired

These are just a few examples of indicators that can be created to track a program’s success.

Evaluating the performance over time:

Usually a Quasi-experimental design of pre and post comparison of the same group is applied i.e. the treatment group. The individual score will determine the status of girls’ like in our example; score on a specific time i.e. the score at pre time or post time. The difference between the two will determine the progress of girls from the pre to the post situation. This difference will be calculated through single difference and will be considered as impact of the program.

Step 3: Define Data Collection Methods and Timeline

After creating monitoring indicators, it is time to decide on methods for gathering data and how often various data will be recorded to track indicators. This should be a conversation between program staff, stakeholders, and donors. These methods will have important implications for what data collection methods will be used and how the results will be reported.

The source of monitoring data depends largely on what each indicator is trying to measure. The program will likely need multiple data sources to answer all of the programming questions. Below is a table that represents some examples of what data can be collected and how.

Information to be collected Data source(s)
Implementation process and progress Program-specific M&E tools
Service statistics Facility logs, referral cards
Reach and success of the program intervention within audience subgroups or communities Small surveys with primary audience(s), such as provider interviews or client exit interviews
The reach of media interventions involved in the program Media ratings data, broadcaster logs, Google analytics, omnibus surveys
Reach and success of the program intervention at the population level Nationally-representative surveys, Omnibus surveys, community data
Qualitative data about the outcomes of the intervention Focus groups, in-depth interviews, listener/viewer group discussions, individual media diaries, case studies

Once it is determined how data will be collected, it is also necessary to decide how often it will be collected. This will be affected by donor requirements, available resources, and the timeline of the intervention. Some data will be continuously gathered by the program (such as the number of training), but these will be recorded every six months or once a year, depending on the M&E plan.

After all of these questions have been answered, a table like the one below can be made to include in the M&E plan. This table can be printed out and all staff working on the program can refer to it so that everyone knows what data is needed and when.

Indicator Data source(s) Timing
Number of training held with Community Girls Training attendance sheets Every 6 months
Number of outreach activities conducted at Girls-friendly locations Activity sheet Every 6 months
Number of course taught at youth-friendly locations Subject Sheet / teachers manual Every 6 months
Percent of girls receiving program messages through the media or WOM Population-based surveys Annually
Percent of prospective girls willing to take skill enhancement program population-based survey Annually
Number and percent of peer leaders providing trainings to Girls Facility logs Every 6 months
Number and percent of new enrollments population-based survey Annually

Step 4: Identify M&E Roles and Responsibilities

The next element of the M&E plan is a section on roles and responsibilities. It is important to decide from the early planning stages who is responsible for collecting the data for each indicator. This will probably be a mix of M&E staff, research staff, and program staff. Everyone will need to work together to get data collected accurately and in a timely fashion.

Data management roles should be decided with input from all team members so everyone is on the same page and knows which indicators they are assigned. This way when it is time for reporting there are no surprises.

An easy way to put this into the Monitoring and Evaluation plan is to expand the indicators table with additional columns for who is responsible for each indicator, as shown below.

Indicator Data source(s) Timing Data manager
Number of training held with Community Girls Training attendance sheets Every 6 months Activity manager
Number of outreach activities conducted at Girls-friendly locations Activity sheet Every 6 months Activity manager
Number of course taught at youth-friendly locations Course sheet Every 6 months Activity manager
Percent of girls receiving program messages through the media or WOM Population-based survey Annually Research assistant
Percent of prospective girls willing to take skill enhancement program population-based survey Annually Research assistant
Number and percent of peer leaders providing training to Girls Facility logs Every 6 months Field M&E officer
Number and percent of new enrollments population-based survey Annually Research assistant

Step 5: Create an Analysis Plan and Reporting Templates

Once all of the data have been collected, someone will need to compile and analyze it to fill in a results table for internal review and external reporting. This is likely to be an in-house M&E manager or research assistant for the program.

The Monitoring and Evaluation plan should include a section with details about what data will be analyzed and how the results will be presented. Do research staff need to perform any statistical tests to get the needed answers? If so, what tests are they and what data will be used in them? What software program will be used to analyze data and make reporting tables? Excel? SPSS? These are important considerations.

Another good thing to include in the plan is a blank table for indicator reporting. These tables should outline the indicators, data, and time period of reporting. They can also include things like the indicator target, and how far the program has progressed towards that target.

Step 6: Plan for Dissemination and Donor Reporting

The last element of the M&E plan describes how and to whom data will be disseminated. Data for data’s sake should not be the ultimate goal of M&E efforts.  Data should always be collected for particular purposes.

Consider the following:

  • How will M&E data be used to inform staff and stakeholders about the success and progress of the program?
  • How will it be used to help staff make modifications and course corrections, as necessary?
  • How will the data be used to move the field forward and make program practices more effective?

The Monitoring and Evaluation plan should include plans for internal dissemination among the program team, as well as wider dissemination among stakeholders and donors. For example, a program team may want to review data on a monthly basis to make programmatic decisions and develop future work plans, while meetings with the donor to review data and program progress might occur quarterly or annually. Dissemination of printed or digital materials might occur at more frequent intervals. These options should be discussed with stakeholders and your team to determine reasonable expectations for data review and to develop plans for dissemination early in the program. If these plans are in place from the beginning and become routine for the project, meetings and other kinds of periodic review have a much better chance of being productive ones that everyone looks forward to.

Conducting Online In-depth Interviews (IDIs)

April 22, 2020

A Focus Group Discussion (FGD) is a group discussion used mainly for assessing the gaps identified by companies/ a company to streamline activities with their overall objectives. Assessment derived from FGDs let companies plan future actions. The group’s composition and the group discussion are carefully planned where people are free to talk openly. Participants of this discussion are actively encouraged to express their opinions.

In Pakistan, Ace Research has conducted many Focus Group Discussions in a variety of projects and the main purpose was to extract information from respondents in length. FGD is a handy approach for assessment of clients, consumers and end users of products and services. Development as well as the Corporate sector can seek benefits of FGDs. Retailers, Banking sector, Telco’s, FMCGs can utilize this tool to get an insight about the expectation, perception and value derived by their end users from products and services.

Designing Objective Questionnaire

The research is always based on some objectives and these objectives are in line with problem statements developed after assessing underlying gaps in activities performed by any organization. Research Objective will determine the structure and pattern of the Questionnaire. Therefore, the first and foremost important factor prior to conducting any research activity is assessment of problems followed by preparation of research instrument i.e. Questionnaire. This instrument plays a very vital role in probing factors lurking behind problems identified.

Questionnaire preparation must be carried out with keen eye for events and arguments associated with the problem. Also, previous instruments used for similar researches can be consulted and with little or no modification if it serves the purpose can be utilized

Designing Manual & Script

Once the Questionnaire has been prepared and consent of all research stakeholders have been obtained, the next step is to design a manual for moderators. This manual serves as the guideline for conducting any Discussion in light of the designed questionnaire. This manual helps moderators to assess the required information effectively.

Script is the detailed version of manual which supplements moderator with minute details such as, probing point for open-end questions, outlining theme for each section with brief description. This allows the moderator to grip hands on whole the questionnaire and during discussion he/she will not face any problem at all. Also, the manual and script are so effective that if they are supplied to the moderator with no prior experience and he/she studies it thoroughly they can conduct any session with ease without any difficulty to understand anything. With the Aid of Manual and Script FGD is initiated with respondents and now another important need of FGD is Good Moderator.
Your moderator will determine the success of your group. Moderators should have some experience of conducting FGDs in the past. Engaging participants in the discussion is very important otherwise precious time can be lost in lengthy unnecessary discussions.

Views of the participants have to be respected and moderator has to remain neutral throughout and must only steer the discussion as per guidelines. Moderators can come across a variety of participants or maybe participants have a lot to share from various perspectives, enlightening the moderator with a large pool of information. For instance one of the FGDs conducted by Ace Research for IT board attracted respondents from diverse fields that included politicians, Expert IT technicians, IT investors, CEO of IT companies etc. the discussion was very fruitful and many questions were sufficiently responded by participants but steering this type of diverse FGD has to be managed very effectively. You have to listen to everyone and at the same time everyone must understand and stick to the agenda of discussion.

Arranging Target Audience

FGD can be arranged in a simple yet organized way. First Step toward arranging any FGD is to identify your participants. The Projects often clearly indicate the characteristics of their participants otherwise with consultation from client this can be sorted out.
Respondents of Projects seeking B2B assessment via FGD’s can be arranged by referrals from client or snowball sampling. Whereas respondents of B2C can be arranged by referrals or Random sampling.
You can find participants by acquiring information from clients or available online data bases such as companies offering IT services to be involved in FGD related to IT companies have been located via secondary research. In other cases, when you have figured out some companies with reference from a client or online database you may seek their reference to approach other IT companies respectively.
Participants initially get a telephonic call where you brief them about your intention to carry out FGD. Once you get their consent you can send them an email with all necessary details. In your first email or the next you can share details about venue, timings, agenda, moderator of interview and any incentive accompanying this discussion. Although, the respondents might confirm their availability but due to unwanted reasons maybe some of them won’t join you in discussion. So it is very important that suppose if you want to hold FGD of 8 members you must approach 20-25 respondents. Among them many might not be available for discussion or may share some kind of reservation. You might get confirmation from 12-15 and again all of them may not be available on days of discussion past experiences suggest, it is therefore encouraged to approach more people for any FGD group.

Conducting Focus Group Discussion

Step 1: Making Venue arrangements (online. Zoom, team viewer, Skype)
If the FGD is taking place offline, it is important to note that the venue must be arranged somewhere convenient and accessible to all respondents. Prior to finalizing your venue you can drop an email to respondents about desired venue location and if the majority have a problem of accessibility it can be changed.
Similarly, Online FGD is easy and accessible for all respondents. All that is required is good internet connectivity. Group Discussion can be arranged on a variety of online platforms such as Zoom Meeting Room, TeamViewer, Skype etc.

Step 2: Inviting Audience
Audience consent to join FGD will be followed by invitation to session. Offline session invitations/ an offline invitation shall include venue location, description about FGD, moderator, refreshments included, and etc.
Online Invitation will also be sent via email, it will contain links/ a link to any of the desirable online discussion platforms along with the description of FGD.

Step 3: Recording Arrangements online
In offline FGD, recorder can be set with the permission of respondents while, in online session recording can be turned on again after consent from Respondents.

Step 4: Actual Discussion
Once the Respondents are all at the venue or have joined an online session, the FGD is good to begin. Moderator will introduce himself and agenda of discussion, after this manual and script will be followed to conduct the session. Assistant moderator can assist moderator in conducting this session.

Step 5: Concluding refreshment
In offline sessions, refreshment can be served to respondents after completion of FGD. Whereas, during online discussions a letter of appreciation can be sent to respondents for sparing their precious time for this session.

Reporting

Step 1: Keeping summary notes during the discussion
It is important to take notes of key points during discussion. For this Assistant moderator can be tasked to carry out this activity of taking notes.

Step 2: Writing Transcripts
Transcripts of FGDs are very crucial and important for extracting relevant information. All the discussions have to be written down either during session or later (if recordings taken) for Reporting and Analysis.

Step 3: Make Summary of each discussion
Summary of all responses by each participant has to be extracted from transcripts and a comprehensive report will be created for highlighting key areas which can include improvement areas, core problem areas, opportunities, threats, strength and weaknesses.

Step 4: Topline Report
Based on the summary topline report will be prepared, it contains all the important points identified by respondents which are directly related with Objectives of FGD.

Step 5: Analysis
Based on overall discussion and topline report an analysis is drawn to conclude the outcomes of the discussion in line with Objectives of carrying out this discussion. Analysis is shared with Client and thereupon necessary actions plan can be prepared to address the problem.

Digital Research ‘Lifeline for The Survival of Industries”

April 22, 2020

Effect of coronavirus on Businesses

The novel coronavirus has disrupted nearly all operations across the globe. Industries in general and business, in particular, are the victim of this virus. This virus which affects the respiratory system of the victim has to date killed many innocents globally, with highest deaths recorded in Italy i.e. around 10k.

Traveling has been put to halt and as a matter of fact all human to human interaction has been minimized as this virus is contagious. Social distancing is the only approach effective so far to mitigate the far-reaching effects of coronavirus.

Impact on businesses

As per Moody’s assessment industries that will bear the biggest brunt of this pandemic are; Textile, automotive, consumer durables, gaming, lodging/leisure and tourism, airlines, retail (non-food) and shipping. Those getting moderately affected include, beverages, chemicals, manufacturing, media, metal and mining, oil and gas, property developers, agriculture, services, steel and technology hardware. Industries that will bear the lowest impact are construction, defenses, equipment and transportation, rental, packaging, pharmaceutical, food, telecom and waste management.

Potential positive impact will be upon, internet services companies, retail online and gold mining.

Why conventional research is not an option now?

Conventional research involves field surveys and human interaction is involved but given the circumstance this cannot take place now but instead online means can be adopted effectively to conduct research. In recent stats it is being revealed that internet usage has increase significantly and people are spending more time online.

Digital research has seen a shift from conventional paper and pencil research in recent times and with ever increasing use of online platforms companies are using digital means to assess needs and preferences of their customers. It has been observed that FMCGs and Telecom are affected with less by coronavirus and this is good time or such industries to analyze the shift in demand of their customers and be prepared to meet the with maximum productivity.

How to conduct Digital Research

Online Market Research is a research method in which the data collection process is carried out over the Internet.

Online Market Research can be either Qualitative or Quantitative.  Qualitative Online Tools include Video Ethnography and Market Research Online Communities (MROCs).  Quantitative Online Methods include mobile and app surveys.

This research can evaluate the performance of a product or service and may allow companies to glean insight into consumer purchasing behavior. With the rising use of the Internet, digital research has become a popular tool among market research firms.

Digital research can provide additional information about a buyer, such as her prior purchasing history. Digital research projects can be carried out by a company itself or by a hired research firm. There are several ways that may be effective for carrying out digital research. Quantitative research can be carried out via online questionnaires and web-based experiments.  Qualitative research can be carried out via online in-depth interviews, online focus groups and participant observation, in which a researcher acts as a part of a community to observe behaviors.

Online questionnaires and online polls are some of the most popular digital research tools. Online questionnaires may need to be carefully designed in terms of format and length. Some of the key digital research models Ace Research offers include;

1. Brand Heath tracking

Brand Analysis: Brand research has similar profiling features (“Who uses this brand?”) and also aims at identifying the reasons for brand loyalty or fickleness.

Scanner Research: Scanner research uses checkout counter scans of transactions to develop patterns for all manner of end uses, including stocking, of course. From a marketing point of view, scans can also help users track the success of coupons and to establish linkages between products.

2. Consumer research

Audience Research: Audience research is aimed at discovering who is listening, watching, or reading radio, TV, and print media respectively. Such studies in part profile the audience and in part determine the popularity of the medium or portions of it.

Product Research: Product tests, of course, directly relate to use of the product. Good examples taste tests used to pick the most popular flavors-; and consumer tests of vehicle or device prototypes to uncover problematical features or designs.

Psychological Profiling: Psychological profiling aims at construction profiles of customers by temperament, lifestyle, income, and other factors and tying such types to consumption patterns and media patronage.

Database Research: Also known as database “mining,” this form of research attempts to exploit all kinds of data on hand on customers-; which frequently have other revealing aspects. Purchase records, for example, can reveal the buying habits of different income groups-;the income classification of accounts taking place by census tract matching. Data on average income by census tract can be obtained from the Bureau of the Census.

3. Pre/post campaign evaluation

Post-sale or Consumer Satisfaction Research: Post-consumer surveys are familiar to many consumers from telephone calls that follow having a car serviced or calling help-lines for computer- or Internet-related problems. In part such surveys are intended to determine if the customer was satisfied. In part this additional attention is intended also to build good will and word-of-mouth advertising for the service provider.

4. Monitoring and evaluation

M&E can be conducted using a wide array of tools, methods and approaches. These include, for example: performance monitoring indicators; the logical framework; theory-based evaluation; formal surveys such as service delivery surveys, citizen report cards, living standards measurement surveys (LSMS) and core welfare indicators questionnaires (CWIQ); rapid appraisal methods such as key informant interviews, focus group discussions and facilitated brainstorming by staff and officials; participatory methods such as participatory M&E; public expenditure tracking surveys; rigorous impact evaluation; and cost-benefit and cost-effectiveness analysis. With the aid of digital research all these facets of M&E can be conducted online.

5. Online Discussions (Focus Group and In-depth)

Another common practice for online surveys is the use of online panels. An online panel is a group of selected individuals that have agreed to participate in digital research projects for a particular company at specific intervals over a period of time. These participants are selected through a screening process according to their demographics, lifestyles and habits, and are usually rewarded for their efforts by the research company regularly. Online panels may allow companies to glean insight into creating long-term relationships with their customers.  These panels may also allow customers to give direct feedback about products and services without the potential reluctance that may occur in face-to-face interactions.  Online panels may also mitigate bias caused by peer pressure to agree on a certain viewpoint, a phenomena that may occur in face-to-face panels.

Benefits of Digital Research

Amid the outbreak of this virus, businesses are widely affected and we all hope for better results with least damage in coming days to people and businesses alike. Nonetheless, once things get settle down the businesses have already borne the wrath of this virus and revenues are dwindling. So in these circumstances it is crucial for companies to closely monitor their clients and consumers so after this virus settles down, a strategy based on factual research can be devised immediately to recover the losses effectively.

To sum up, we have following advantages of conducting digital research.

  • Cost advantages
  • Speed advantages
  • Data collection in real-time
  • Advanced analytics
  • Efficient global and multi-country survey management

Conducting digital research can be a complex procedure and may require considerable expertise on the part of researchers in obtaining accurate data.  It may be challenging to recruit participants in digital research for several reasons.  Recipients may be reluctant to participate in digital research because they may be afraid that the privacy and confidentiality of their personal information may be violated.  Since the identity of the researcher cannot be verified completely, people may find it difficult to trust such research methods.  Researchers often present participants with some monetary or non-monetary rewards for their participation