Stop designing the perfect approach and get something good out

A BLOG WHEN IT WAS JUST ADAM AND PEARSON INSIGHT

“How many responses do we need?”

This is the question I get at the start of pretty much every community research or consultation project I deliver: The short answer is it’s not that simple. Let all 5 feet 6 inches of me try to shed some light on why size isn’t always everything.

Take these 2 examples. Both might involve a survey. Both might be aimed at the same community. But they are very different exercises.

1. You want to get a representative view from your community, so you can use that intelligence to inform how services are shaped and delivered. This, in simple terms, is research.
2. You want to run a robust consultation exercise open to anybody who might be affected by your proposed service changes, so you can use it to assess and mitigate impact.

Yes, you want a ‘good’ response in both scenarios. But there’s much more to it than pure volume.

You could deliver what you think is a piece of research. Capture 1000s of responses. Pat yourself on the back. Then realise it’s never going to be representative. You pushed it out on social media, made it open to everyone and consequently it’s riddled with self-selection bias.

Similarly, you run what you think is a consultation exercise on service changes by sending a survey to a sample of service users. You get a decent response and you know they’re all affected by the consultation. But you’ve not given everyone affected the opportunity to give their views, a must for any consultation.

The pesky response rate

Another measure of success people seem to like when running a survey is the response rate but quite often it’s misunderstood and misused in research. I hate response rates. There, I said it. It’s a fairly limited gauge in my book. We’ve moved on and we can turn to richer data on reach and response through analytics built into online survey software and social media. Personally, I find analysis of drop off rates useful. What proportion of people started but gave up? At what part in the survey were they most likely to lose the will? Did the average response time tally with the expectations you set before they started? I can learn from these and do something about it.

Anyway, back to traditional response rates. Hit 1000 responses and as long as you’ve delivered some best practice market research then you’re likely to have a ‘good’ confidence interval regardless of your population, a generally accepted measure of how statistically representative your findings will be. What you don’t want to fall into the trap of is calculating a response rate against your overall population. Example: 1000 responses from a sample of 4000 addresses in a community of 100,000 people = 25% response rate, not 1%. Unfortunately, the latter tends to be waved about when someone is intent on challenging the quality and reliability of research.

When it comes to consultations where access and open opportunity is key, my measures of reliability would focus on the quality of engagement and reach. You can’t force someone to respond, but you can make every effort to give them the opportunity, provide them with the information they need, engage with them in a meaningful way, the list goes on. I fear tips on delivering an effective consultation will need its own post.

I’m about to contradict myself

Perhaps contrary to what I’ve written so far, sometimes we need to worry less about delivering the gold-plated piece of research and focus on what matters. That’s giving your local community a voice, taking on their views in a range of different ways and then piecing the insight together to shape decisions and bring change to their area.

I like to think of it as community insight. I’m sure others more intelligent and qualified than me have defined community insight, but this is my version. For me it’s about taking the best of research, consultation and engagement practices and making them work together. As long as you know what the different practices can and can’t tell you, go for it. Because on their own I’m pretty sure they won’t tell you enough.

It’s about that deep understanding of your community and what makes them tick. It’s about thinking a little less and doing a little more. Stop designing the perfect approach and get something good out there. Probably most importantly it’s about doing it at the right time, so community insight stimulates rather than validates change.

Learn about good stuff

Continuing with the emphasis on community, it looks like there’s some great work happening in local government. I particularly like the sound of what’s going on in Redbridge where residents are being put at the beating heart of place-shaping. It goes beyond my version of community insight, it’s also about what happens once you’ve started that conversation to make things happen. Check out their blog on the NLGN website to find out more.

In my former life as a local government research bod, we developed our own version of a community research model with a range of benefits (although I would say that). Employing local people to undertake local research out on the streets of Blackpool, interviewing residents and visitors alike, the model not only provides a continuous live feed of intelligence each and every month but at the same time it's giving residents valuable work experience and retains spend within the local economy. People aren’t hard to reach, it’s just sometimes we don’t make enough effort to go out and find them.