M&E

Ten Common Monitoring and Evaluation Mistakes

By Alanna Shaikh | | 25 Comments |

As you may have noticed, monitoring and evaluation is a topic near to my heart. One thing I’ve noticed is that we repeat the same errors, over and over and over. I’ll elaborate on these in my next three posts, but for now, I will tease you with some lists:

The Top Three Monitoring and Evaluation Mistakes Experienced NGOs Make

  1. Using the same indicators they’ve always used, even as projects change
  2. Too much evaluation, not enough monitoring
  3. Leaving M&E up to the M&E team

The Top Three M&E Mistakes New NGOs make

  1. Choosing really great indicators that are nearly impossible to measure
  2. Confusing a program with an RCT
  3. Focusing on the donor’s data needs when choosing indicators

The Top Four M&E Mistakes Everyone Makes

  1. Too many indicators
  2. Not focusing on data use
  3. Too many process indicators, not enough impact indicators
  4. The IKEA effect

 

(photo credit: tgkohn)

Snail’s Faith: on M&E and the pace of change

By Alanna Shaikh | | 21 Comments |

 

Things get better slowly. Achingly, painfully, snails-race-past slowly. They got better slowly in the US. As Chris Blattman recently pointed out, Andrew Jackson was a child soldier and genocidaire. My parents were married in 1969. If my mom hadn’t insisted on finishing her graduate work first, they would have needed to choose their wedding location with care. Anti-miscegenation laws weren’t eliminated in the US until 1967.

Things get better slowly in development. Sometimes at a speed invisible to the naked eye. Often at a speed invisible to our careers. That’s why we do monitoring and evaluation; it’s meant to catch the accretion of tiny changes that will eventually add up to something that matters. You don’t need to be able to see it when you have numbers to track it.

Of course, not all projects have M&E that works. Sometimes you’re tracking the wrong stuff, and you don’t capture slow change. Sometimes you’re not doing a very good job of tracking anything. That doesn’t mean the intervention isn’t working; it just means your M&E isn’t working. Sometimes it’s not working because it’s not well designed, and sometimes it’s not working because the trade-offs aren’t worth it.

Sometimes you stand there, throwing rocks down a well with no visible change. Just plop, plop, plop. Then, all at once, the water overflows in a big splash. You can’t see that with your naked eye, and the wrong kind of M&E will only capture how many rocks you’ve thrown, not the infinitesimal increases in the water level. But sometimes paying the money to measure the water level means you can’t afford enough rocks to throw.

Look at gay equality in the US. Years and years of discrimination and abuse. Court cases that went nowhere. Change that didn’t come. And then, apparently out of the blue, equality started winning. Now we’ve got gay marriage in Iowa. What do you measure to capture that slow change? The number of failed legislative efforts is a meaningless number; it’s not like you get to 139 and you’re done. Incredibly detailed public opinion polling would catch it, but that’s serious money. Money you could spend on strategic litigation instead.

You need to believe in your theory of change if you can’t measure what you need to.

That’s an act of faith. Faith is what we’re left with when M&E can’t answer our questions. Faith is a tricky thing. It can get you Freedom Riders or the Mayan apocalypse. You should save your faith for when it’s deserved. But it is, occasionally, deserved. Sometimes we just hang have to in there.

 

(PS: this one’s for you, Danielle)

photo credit

Field Notes from the Development Industry: 1/13/2013

By Alanna Shaikh | | 1 Comments |

1. Someone asked me if the new State Department Global Health Diplomacy office is likely to be hiring. Unfortunately, I’d guess no. All the major positions will most likely be staffed with career foreign service officers. They may be hiring support staff. They could pull that staff from existing government administrative employees, or they could hire using a contractor that places people at the State Department. They’ll probably do a mix of both. I don’t know which contractors the State Department is using for personnel right now, but spending some time on clearancejobs.com would probably help you figure it out.

2. After a chance meeting with someone working for a faith-based organization (one of the little ones) I started thinking about their funding model. As far as I can tell, people fundraise in their home countries from their own churches, and use the funding to pay for their overseas time. While overseas, they donate their time to faith-based projects developed by other expats. This means that the projects themselves have a great looking balance sheet. All their (theoretically) expensive labor is volunteer.  This is a similar model to secular projects that require you to pay them if you want to volunteer overseas.

3. As I’ve mentioned before, monitoring your project implementation is useless if you can’t actually change your intervention based on the results. That’s what matters in a project designed to improve people’s lives. (A research project is a different animal, and changing the intervention would defeat the purpose.)  Bottom Up Thinking recently published a nicely nuanced take on the issue.

4. We’re posting three times a week over on Tomorrow Global. If you’re not interested in the exploration of my deepest fears for the future, you might like Danielle Parsons’ in depth looks at HIV issues or Lorea Russell’s explorations in social entrepreneurship, emergency relief, and other international development topics. Please come by and leave a comment.  The commenters make Blood and Milk a thousand times better than it would be with just me. I’d love to see that at Tomorrow Global.

5. I am learning to use video as part of my job, which means I spend a lot of evenings watching YouTube videos about global health. (You envy me my exciting life, I know you do.) I’ve been putting them up with brief comments at Global Health Videos. The site is really just meant for me (thus the total lack of design) but I offer it in case it’s useful to someone else.

6. My husband just left for a two month guest worker stint in another country. My mom does all our cooking and has a totally different view of healthy food than I do. Also, she and I are sharing a winter coat so we can only go out one at a time. Apparently I’ve lived in the developing world for so long I no longer have First World Problems. (note: I’m not that broke. I have a winter coat coming soon in our shipment from Dushanbe and I’m too cheap to buy one just for the next two weeks.)

 

(Photo credit, me, from my new daily photoblog)

UN Week Notes: Thursday

By Alanna Shaikh | | 1 Comments |

I spent the whole day today camped out at the Digital Media Lounge, because I think I have strep throat and schlepping all over town for press events was beyond my capacities. I had originally planned on attending a cookstove event, a global warming press conference, and the CGI keynote. Luckily, the Digital Media Lounge had some interesting stuff, so it wasn’t all that disappointing.

The highlights were the live broadcast of Obama’s Israel/Palestine speech, Rajiv Shah’s talk, and the absolute awesome president of Greenpeace. There was other stuff but those were the biggies.

Obama’s speech – it was a beautiful speech, beautifully delivered. But there was nothing new in it that’s going to move the needle on the situation. And this line “And efforts to threaten or kill Israelis will do nothing to help the Palestinian people,” is, unfortunately, demonstrably untrue. No one cared about the Palestinians until the intifada. Pretending otherwise isn’t going to win any supporters.

Rajiv Shah – we had to submit questions in writing and he picked which ones to answer. Not surprisingly, he didn’t pick any of mine. He stressed the expansion of the “core development community,” specifically mentioning Coca-Cola and young people who problem-solve. Rajiv Shah seems a little obsessed with innovation. It makes sense, since he comes from Gates, but I am not convinced it’s what USAID needs to focus on.

It also sounded to me like Shah is fully on board with the idea of development as a pillar of American foreign policy. I’m not opposed to that, but I also believe in development as a moral imperative. I wish I knew if he did too.

I live-tweeted Shah – you can see the tweets starting here.

Then we had the presentation from the unmovement people. It’s a social network that connect small NGOs and lets people learn about them and donate if they want. It’s supposed to help groups that don’t get social media tap its power anyway. I was fine with the whole thing until they mentioned that they don’t need to monitor or evaluate the NGOs because beneficiaries can just use the website to rate the projects and that’s how you’ll know if they are working.

Uh, NO. M&E is about way more than whether the beneficiaries like it. They communities you work with don’t necessarily know all your goals – they have better things to do than memorize your project objectives – and even really bad work can be popular with people, even if it’s wasting money and not doing much good. We need really M&E to make sure that we’re having the impact we want to have, and to show us how to adjust midstream if we don’t.

The, after a long, long wait in the auditorium, we had a presentation from Kumi Naidoo, the Executive Director of Greenpeace International. It was supposed to be a talk about how to meet the MDGs and protect the environment. It turned out to be a barn-burner of a rant that made it vividly clear how plastic most of this week has been.

He came out swinging – his first comment was that while he’s wholly in favor of reusing, this week had been nothing but recycled comments and baby steps. It was a great talk, and it left me feeling angry and vaguely inspired. My live tweets start here.

 

Disclosure: I attended UN Week as an Oxfam VOICE, which funded my trip as part of an effort to increase awareness of the MDGs.

+++++++++++

Kumi Naidoo at some other event. photo credit: wwf France

How I’m Judging You

By Alanna Shaikh | | 14 Comments |

Statue of Justice

These are my (arbitrary, personal, non-evidence-based) rules of thumb for identifying good development work:

Bad signs

  1. Starting out by buying cars.
  2. Claiming to work in “Africa” without specifying a location.
  3. More than four partners in your implementation coalition.
  4. A local to expat ration of less than 5:1 (10 or 15 to 1 – or more – is far better).
  5. Planning/budgeting for more than 4 visits from HQ a year.
  6. Extensive use of international interns.
  7. Using program staff as translators and interpreters.

Good signs

  1. National staff in management positions over expats.
  2. Terrifying, highly experienced financial staff and a rigorous financial reporting system.
  3. Close collaboration with government on its lowest level – with city, town, and village authorities.
  4. Sharing of monitoring and evaluation data with the communities the projects works with, and training those communities on how to review the data.
  5. A clear vision of what the target area (group, community…) will look like once the project is over and what will have changed. Approval from the target area/group/community of the vision, and support for it.
  6. Extensive use of paid local interns.
  7. Specific rather than standardized indicators for monitoring and evaluation.
  8. Translators on staff.

PS – Thanks to Brendan for reminding me why I write.

———-
Photo credit: Citizensheep
Chosen because, you know, judging, justice…look, it’s not easy choosing images.

Things I don’t believe in #3 – most kinds of evaluation (July 2008)

By Alanna Shaikh | | 1 Comments |

Indecipherable graph

Note: August is looking like a crazy and stressful month for me, with no time to blog here. To make sure no one gets bored and abandons me, I am going to re-run some of my favorite posts from the past.

Most forms of monitoring and evaluation annoy me. Instead of serving their true – and vital – functions, they are pro forma decorations created externally and staple-gunned onto a project once it’s already been designed. Usually a clean-looking table featuring a timeline and a list of indicators they plan to measure. I loathe those tables, for a lot of reasons.

Monitoring and Evaluation are not the same thing. The purpose of monitoring is to observe your program as you do it, and make sure you’re on the right track. The purpose of evaluation is to determine whether you are meeting your goals. These should not be confused.

Let’s use a hypothetical project. Say you’re trying to reduce infant mortality rates among young mothers in rural Bangladesh. That’s your goal. You need to start by defining your terms. What’s a mother? Just women with children, or pregnant women too? And exactly how old is young? So, decide you want to work with pregnant women and women with young children, and they must be under the age of 25. How do you want to keep these children alive? You decide to teach young mothers how to take care of sick children, and how to prepare nutritious food.

Your monitoring should make sure you’re reaching as many young mothers as possible. It should make sure that your educational efforts are well-done include accurate information. It should make sure you’re reaching young mothers, and not grandparents or childless women. Are you actually doing the stuff you said you would? Are you doing it well? That’s monitoring.

Evaluation is about whether you’re reaching your goal. You could be doing great education on children’s health and nutrition. Your young mothers could love your trainings, and lots and lots and lots of them could attend them. Your trainings could be amazing. But improving mothers’ knowledge may not actually decrease infant deaths. That’s what your evaluation will tell you – if your program actually achieving your goal.

What do these questions have to do with the neat little table on page 17 of your proposal? Very little. Monitoring, to be useful, needs to be constant. It can be based on very simple numbers. How many teachers/doctors/lawyers/mothers have you trained? Are the trainings still attracting participants? When your master trainers observe trainings, do they still like them?

Once you start getting answers to these questions, you need to use them. That’s why it’s better if managers collect monitoring data themselves. If participants don’t like your trainings, find out why, and fix it. If you’re not training enough people, maybe you’re not scheduling enough trainings, or maybe you’re not attracting enough participants. Monitoring is like biofeedback. Observe. Measure. Make your changes.

Evaluation happens less often. You’re not going to see impact in a month, maybe not in a year. Annually is usually often enough for evaluation, and you can get an outsider to do it. The important thing about evaluation is that your team needs to believe in it. If you get to the second year of your project, the project your team loves and you’ve given your blood and sweat to it, and the evaluation says it is not having any impact – your heart breaks into a million pieces. It is tempting and easy to simply decide the evaluation is wrong and keep wasting money on a project which just doesn’t work. You need a rock-solid evaluation you can trust so that if it tells you to change everything, you actually will.

(photo credit: leo.prie.to, chosen because I have no idea what it means)

Making Mistakes

By Alanna Shaikh | | 8 Comments |
overturned SUV
overturned SUV

(photo credit: Kim Scarborough)

 

1. In Tajikistan, where I currently live, and in Central Asia in general, married women wear scarves on their heads. So do unmarried women older than about 25. It’s not a religious thing at all. It’s just what women do. Visitors often come to Tajikistan for a week and leave thinking that it’s a deeply religious country because of all the women wearing hijab. If you either a) asked someone or b) knew enough about Islam to know what a hijab has to cover, you wouldn’t make that mistake. But people don’t know, and they don’t ask. They walk around, they make assumptions, they go home and share their misinformation.

2. In order to graduate from my alma mater, Georgetown’s School of Foreign Service, you have to be proficient in a foreign language. My roommate and I both chose French. In the weeks leading up to our proficiency exams, we spoke French to each other at all times to practice. Once, I heard someone comment as we walked by, “That’s why I love Georgetown – the constant exposure to other cultures.”

That’s my convoluted way of saying we get things wrong all the time. Sometimes our science is bad, sometimes we haven’t bridged the culture gap as securely as we’d like, sometimes we’ve made so many compromises that we ended up somewhere we don’t belong. Some of that we can prevent. Both of my examples above could be prevented through spending more time and doing more research.

We can’t prevent all of it. As long as our programs are designed and run by human beings rather than infallible robots, mistakes will happen.

We do, however, need a resilient system to catch our mistakes and a corporate culture that lets us make changes when we realize we’ve screwed up. We can catch our mistakes through monitoring and evaluation. That means not just collecting data, but looking at it, thinking about what it means, and using that meaning to guide program decisions. And we can keep our errors to a minimum by cultivating an atmosphere where people are encouraged to admit their mistakes. If you maternal and child health director realizes that the patient education classes aren’t doing anything, she needs to be free to re-design the curriculum or cancel the activity and spend the money on childbirth kits.