Evaluation: does it have to be all or nothing?
I’ll come clean: some of what my company does now as evaluation is a waste of money. Taking really very modest projects and programmes and applying to them the full panoply of the Government’s Impact Evaluation Framework is simply daft. We do it because our clients’ funders require us to do it.
I cherish the moment when a former colleague gave a public sector client a little lecture on how to do a perfect evaluation, only to be told: “we all know about the Gold Standard: I’m interested in what’s realistic”.
The risk is that funders will stop paying for evaluation because the money isn’t there to pay for the Gold Standard.
Yet we have Ministers taking every opportunity to say how they want “evidence-based decision-making”, and conscientious managers as keen as ever to know whether what they’re doing is working or not. The problem is that good evidence doesn’t just appear from thin air. And credible evidence requires the teller to have a degree of objectivity.
Babies and bathwater come to mind, but it doesn’t need to be like this. We’ve supported clients before with slim-line – ie cheap – external support to their evaluation, helping them to design a robust process, and the main research tools, and critiquing their draft report, but leaving the research in the middle (which is the expensive bit) to them. In one case, I designed a process which had project leaders pairing up to critique each others’ projects, using a structured design, and tools, which I’d helped shape with the overall programme manager.
Was it as good as a classic, wholly-external, evaluation? No. Was it a good deal better than nothing? Certainly.
If you’re determined to write a report saying that everything is wonderful, this approach will give you lots of scope to cheat – but you’d be wasting your time, and cheating those whom you work for, because no one will believe your report.
But if you are conscientious and want to get informed outsiders to help you to review what you’re doing so you can make it work better, this approach is a good half-way house. No one gains if we simply stop evaluating because we can no longer afford the Gold Standard.
Posted on Monday, 26th July 2010 | This entry has 2 comments
Matthew Green | Monday, 26th July 2010 | 04:25 PM
I think you can take heart from the fact that even if Ministers are “taking every opportunity to say how they want evidence-based decision-making” there is precious little evidence to suggest they are actually interested in evidence-based decision making - especially if the evidence happens to cut against the ideological grain.
Kevin Mckenzie | Friday, 30th July 2010 | 10:06 AM
For an example of this see the evaluation framework for Generations Together which is a national demonstrator programme run through the former DCSF.
A very complex cohort evaluation was designed to measure the impact of the programme which aimed to recruit 12000 youth and older volunteers. The outcomes were to be measured using a lengthy survey and local partners were charged with getting their volunteer participants to sign up. The basic idea is that they will contact participants by phone and administer the questionnaire twice once to establish a baseline and then to measure the impact.
Where it totally falls down is with the assumption that they will be able to keep a young volunteer on the phone long enough to administer a 12 page questionnaire.
I often find that external evaluators come with a loaded agenda that places too much emphasis on robust research methods at the expense of getting anything meaningful at all.
For the money paid to the consultants we could have designed a local evaluation that would have delivered plenty of robust evidence. We would not however have used a survey approach, instead we would have trained programme participants to use structured interview techniques.
The survey approach only makes sense because the evaluators are based 300 miles away from our project.