« Want to Fundraise More Effectively? Strengthen Your Organizational Culture | Main | Social Technographics »
How to think about testing new creative
When you’re developing new creative work - new branded imagery, new tag lines, new campaign lines, or new messages - how do you find out whether your your creative work resonates with your target audience?
Do you use your gut instinct? Is your hunch enough? If you’re uncertain (or risk-averse), you have a variety of options to choose among to support and inform your decision-making. Bullet-proof marketing management is increasingly all about the quality of decision-making that is taking place.
When it comes to making strategy decisions, there is a continuum that runs from “hunch� to “certain.� There is a lot of territory between those poles at the opposite ends of that continuum. Hunches are useful. Most marketers use their “gut� on a daily basis to make decisions. We can’t test everything and we shouldn’t try. Good quality research isn’t free.
So when should we pull the trigger and hire research professionals? I believe that research becomes increasingly important when we must absolutely trust that we're making the right decisions.
Good decision-making has always been important, but most of us know that the sheer power of a big marketing budget can make up for poor creative strategy. Put enough money behind a mediocre idea for a long enough period and the idea seems to get better. I might get unpopular for suggesting this, but I think the Aflac Duck wouldn’t fly for two seconds with a nonprofit arts marketing budget. (Yes, I know all about how well this campaign has work in raising awareness of Aflac.)
Fundamentally, that’s our problem in the nonprofit arts sector; our ideas have to be amazingly good to make up for our small wallet. We have to stick at low budgets. So, how do we decide whether our idea is good or we’re temporarily delusional?
For decades, focus groups have been used for this purpose and to good effect. But, with focus groups, there’s always that nagging concern in the back of one's mind: findings from focus groups are not definitive because they are not statistically reliable. The sample size is too small and the process - as useful as it is - can (and often does) influence respondents.
In addition to this concern, increasingly I see organizations trying to do their own focus groups, thinking that something is better than nothing when it comes to consumer feedback. I couldn’t disagree more. A reliable and credible process requires a trained professional moderator who knows (and who will tell you) how the information you glean will be useful and what the limitations of the focus group process are. I wouldn’t base organizational strategy on amateur research any more than I would start a medical treatment regimen based on the advice of a snake-handling witch doctor. Forgive the bluntness, but resources are just too scarce.
When focus groups are convened, run, and interpreted by a professional moderator, even then findings are not definitive in statistical terms. In other words, what you have is an enriched, textured, and more vivid set of opinions. In my opinion, groups are most useful in refining or elaborating on creative strategies and product, and 2) figuring out what to test using quantitative methods.
Speaking of quantitative methods, the web-research company Zoomerang has developed a web-based platform for testing creative product. Ann Murphy, creative director of the New York-based firm, EMG, recently suggested using this platform to test new brand tags for our mutual client. This process is reliable because Zoomerang has recruited respondent pools that are robust enough - per market - that they have the ability to mirror the demographics and lifestyles of various consumer target groups. For additional money, you can even use your own database to enrich and compare against Zoomerang’s test group.
Our process was relatively inexpensive, very fast, and generated a response rate that was high enough to suggest statistical confidence. The finding were clear enough (there was a big enough difference between tested approaches) that I believed we had conclusions we could trust.
For those organizations that want to run quick processes and who haven’t queued up a place with their own research firm to do so, this is a reasonable option. Is this approach as good as custom market research? No. Is it reliable enough? Yes. Does it beat focus group testing in reliability? Almost certainly. Is it better than a hunch? You decide.
Changes in strategy disrupt longitudinal comparisons. Basically, when you change strategies, the results from what you were doing before can’t be compared against what you are considering doing. The outcomes can be directly attributed to the strategy.
If your decision rises to the level of “strategic� - and by that I mean that you are building and considering a new strategy - then by all means you should use research. Consider calibrating the intensity of your research program by three dimensions: 1) the degree of certainty that you think you need to move forward; 2) the degree to which you believe you are making a significant change in messaging and/or imaging strategy; and 3) the amount of resources you intend to invest in prosecuting the new strategy.
Posted on April 10, 2007 10:31 AM | Permalink
Bookmark with: 
del.icio.us, 
StumbleUpon, 
digg
Comments
I have watched good work fall victim to poorly assembled, poorly moderated, and poorly interpreted focus groups enough times to agree wholeheartedly with what you're saying in today's post.
On the other hand, I have seen well moderated focus groups tease out the real essence of the interaction between an idea and it's potential audience in a powerful way that proved useful moving forward in the process.
The effectiveness of hunches is directly proportionate to the skill, experience and intelligence of the person acting on the hunch, and even then it's still just that. I work largely in the non-profit sector and in the smaller business sector, too, and your observation about exponential relationship of the strength of ideas to the size of budget is true. Lots of good campaigns start life on hunches and small budgets, but any additional (and affordable) tools that allow one to be a little more sure are worth every penny.
My field -- the design field (particularly graphic/communications design) is traditionally skeptical of research. Many of us want to maintain the illusion that we are brilliant thinkers and purveyors of a particular brand of elusive magical creative power and that research is mundane at best. On the other hand, I have seen clients take research so literally as to try to dictate the smallest of details, totally missing the point of research findings.
In my humble opinion, there is an uneasy alliance, at best, between research and creative in most of the world today, and lots of new and exciting ground to be broken when it comes to making a better alliance. This Zoomerang service that you point out is an excellent example of someone exploring some of that new ground.
What's most exciting, perhaps, is that there may be a shift in power on this issue. The mega agencies and research firms may no longer own this field of study and the innovative solutions may be happening at the base of the ivory towers, at street level. That's where the really good stuff is.