Subscribe to this blog to receive announcements about best practices.

It's A Puzzle! Testing Direct Responses

Were you the kind of kid who wanted to know how things worked?  Who took apart the telephone to figure out what was going on (back in the days when there were actually land line phones that is)?  Did you love jigsaw puzzles?
I love puzzles, I love solving puzzles and figuring out the best way to get things done.  To take a process and refine it and then sit back and figure out ways to even make it better.
So testing direct response projects was a joy for me.  Pinpointing the one thing to test at a time was a challenge though.  But, as most of us that do testing know, you need everything to be the same except the one thing you want to test or your test won’t give you valid results.
For example, if you had different copy, colors and envelope design, how could you pinpoint what might be making a difference?!
What I focused on was things that could potentially save us money while increasing, or at least not decreasing, response and revenue.  One of these things was having a black and white outer envelope versus a color print envelope.  One time it was laser print of the address versus a window.
The first step is figuring out what you want to test.  The next is figuring out how you are going to code the results so you can get to the most important step – analyzing the results.
For coding, I kept it pretty simple – slightly different appeal codes for each version.
For the list pull, I did a random, A/B, selection based on last name or mailing address (not zip code).
And for analyzing after the results were mature, usually 6-8 weeks after the mail date, I would pull an export from Raiser’s Edge including ID, Name, Gift Amount, Total Giving to Appeal (as some gifts are monthly), Appeal ID and if I had any segments or other variables (for example at times I would put gift array in the appeal comments so I could see if they were giving what we asked them to give).
Since we mailed monthly and we repeated the same test for several mailings this enabled me to track any trends in the data.
The testing lead us to increase response from 4% to upwards of 7% and revenue, for some tests, doubled.
Would love to hear about your testing best practices?  What did you learn?  What worked and what didn’t?
Happy Fundraising!!

(Community Manager's Note: This is a blog entry in our Summer School contest. It has not been vetted by the Raiser's Edge team for accuracy.)
Posted by Joanne Felci on Jul 24, 2017 11:40 AM America/New_York

Leave a Comment

Log in to post a comment.

Direct Mail Tests: For cold mail acquisition, we've tested dollar ask strings (we'd had an initial gift of $19 for a long time and wanted to try $25 and the $25 won), circling the dollar amount vs. not (too close to call), blank outer envelope vs. one with a message (blank won), recycled brown outer envelope vs. white (brown won on response, but extra expense made it lose overall), copy A vs. Copy b (A has been our control for over 10 years). In renewals, most recently we tested a larger number of name labels vs. smaller, and smaller won. We always repeat our cold mail tests at least twice before we implement. Because renewals only happen once a year, if the results are strong, we just go with them. We test more extensively in email.
  • Posted Mon 24 Jul 2017 03:42 PM EDT
I always wanted to do testing like this.  But I never had the time or the support of my team (i.e. the ability to control enough pieces of the process to actually do a test).  Really interested to read what everyone has discovered, though!
  • Posted Tue 25 Jul 2017 09:38 AM EDT
We did a spring appeal with two different approaches - one practical (help us keep the lights on...) and one heartstring (help Johnny go to college...). We randomly sorted the list and sat back to analyze the results --- and, drumroll ---  - they had nearly identical response rates. We did decide that our sample was too small to draw any definite conclusions, but it was interesting and we'll try a similar experiment again with a larger mailing.
  • Posted Thu 10 Aug 2017 02:13 PM EDT
I love the idea of expanding our idea of how we can test. Thanks for helping us expand our thoughts!!
  • Posted Fri 15 Jun 2018 04:37 PM EDT
Thanks for the great tip!  We have been talking about doing A/B testing and this helps to put a few things in perspective.  
  • Posted Mon 25 Mar 2019 04:21 PM EDT