I mentioned in my review of Made to Stick (incidentally, they have a website about it here) the other day about statistics giving credibility, but often inducing glazed eyes and heavy eyelids. But, in the broader area of evaluation and measurement, they are incredibly important in this sector.

I had a great conversation with Steve Lawrence (founder of Work Ventures in Australia) at the SSE residential recently about the SSE evaluation, and how we can seek to extend and improve upon that work. You can find and download a full 100+ page copy of the report in the Outcomes and Impact section of the main website…and an exec summary version as well. One challenge we talked about was how much of the nuance and detail you lose when boiling this stuff down to headlines. A few times, Steve mentioned different things he was interested in, only for me to say that some of them were in the 100+ page version, but not the shorter exec summary; or had been in an earlier version and then cut. I'd love everyone to read that full report, as it gives such a rounded view of SSE programmes and their various outcomes. It also has more on the evaluation process, such as how we tried to provide incentives / anonymous responses (in order to get a good cross-section of participants), how we also used the process to empower SSE students and Fellows to use evaluative tools, and so forth. But, ultimately, the highlights are incredibly important organisationally because of people's time, and their need to grasp something swiftly. And simple, concise points do that. Balancing that with the need to give as full and holistic picture as possible is one challenge for those heading up evaluative work. (And, lest we forget, to improve and develop internally as an organisation from that learning).

You can see something of the same debate in the SROI vs. social auditing argument. See this exceprt from the Social Enterprise Magazine article linked to above:

"Where there are impacts which can be easily ‘financialised', like
people moving off benefits into work, by all means use SROI. But there
will always be impacts which are difficult to attribute reasonably in
this way. In these cases how people think and feel is important and
social audit methods will be more appropriate"

Not quite the 'highlights' vs. 'holistic' wrangle, I'm talking about, but it is about that same sense of crunching something down and losing something of the reality and totality of it all. But, as Craig Dearden-Phillips puts it in a typically provocative column, the sector can't rely on anecdotal evidence either:

"For years we have got away with being the sector of great anecdotes.
When asked about the difference we make, we often bang on about our
best-ever success or offer improbable statistics that would do a
Soviet-era government proud ("our two staff provide services for
267,000 people" and so on). In a tougher climate, those who properly
measure and prove impact will thrive while those who bleat that it's
all too difficult will sink."

I agree with this, and make sure we introduce evaluation, impact measurement, outcomes, and all the rest of it to our students while their projects are at an early stage. So much easier to build it in at the start than try and retrospectively collect and analyse….

Having said that, the masters of the twisted statistic remain the retail sector. The current Flora Buttery advert has Gary Rhodes offering crumpets to people around the country: one with Lurpak Spreadable butter on, and one with Flora Buttery on. The whole ad is based around the fact that "more people said they preferred Flora". At the bottom of the screen (as with the classic shampoo ads), the "proof" pops up. In this case, it has 200 people who were asked, of whom 48% preferred Flora…..and 45% preferred Lurpak. With 7% having no opinion. That's 96 people for Flora, 90 for Lurpak and 14 for neither. So, yes, in a one-off exercise conducted by Flora with 200 people, 6 more out of those 200 preferred Flora to another brand. In my humble opinion, that proves absolutely nothing….do another 20 tests with the public, independently, and not conducted by a celebrity chef driving a Flora van with a crumpet on top…and then I might be swayed. (Mis)using statistics like that is why people say things like this:

"Statistics are the triumph of the quantitative method, and the quantitative method is the victory of sterility and death" (Hillaire Belloc)

and, more poetically,

"Like dreams, statistics are a form of wish fulfillment" (Jean Baudrillard)