How To Set-up And Measure A Simple A/B-Split Direct Mail Test

 A/B-split testing occurs when mail pieces are separated into two distinct versions, with only one element different between them.

1. A constituent group for the test
2. Defined test element
3. An appeal letter and reply method
4. Coding in your CRM
5. A calculator

1. Determine what you want to test. For example: Which solicitation produces a higher average gift amount?
   • Choice A: No suggested gift amount (donor chooses his/her gift amount)
   • Choice B: Suggested ask amount string (gift amounts are suggested to the donor)

2. Determine which constituent records will be used in the test. For example:
   • Example 1 (Fundraising Essentials scores): Records with Donor Type Rating A and Next Ask Amount $1-$250 and with a gift of $1-$25 within the past 24 months
   • Example 2 (ProspectPoint Modeling scores): Records with Annual Giving Likelihood scores between 501 and 1000 and Target Gift Range $1-$50 (TGR 1) and with a gift of $1-$25 within the past 24 months
3. Query your database to gather the appropriate records from item 2 above.
4. Randomly split the records into 2 groups - approximately 50% / 50% (though this does not have to be exact).  You can do this with your CRM if you have a split function or by other criteria such as all A-M Last Names and all N-Z Last Names.
   • You will have Group A and Group B
     • For example Group A has 1,057 records and Group B has 956 records
5. Code each record with an appeal code so that you know which record received which solicitation (Group A is mailed Letter A and Group B is mailed Letter B). Coding and barcode scanning may be available in your CRM or through your mail house vendor. If you do not have a coding system, here’s an easy one:
   • Group A is coded for the Letter A appeal: YearMoDayAppealname-Package# (20161115AFTest1-A)
   • Group B is coded differently for the Letter B appeal: (20161115AFTest1-B)
6. Create/print your appeal letters.  Since our example is testing a blank gift amount against a suggested string of gift amounts, the only difference in the letters will be the ask amount line or the line on the reply card.  The language and layout of the letter should be as identical as possible.  Here’s how the appeal language might be different from letter A to letter B:
   • Letter A: ...your gift, in an amount of your choosing, goes to work immediately providing emergency food and shelter to abandoned animals in our community.
   • Letter B: ...your gift of $30, $40 or $50 goes to work immediately providing emergency food and shelter to abandoned animals in our community.
   • Note, you can also mimic the choice differences on a reply card you’ve inserted:
     • Reply card A:  Here is my gift of $_______________ (write in a gift amount of your choosing) to provide needed food and shelter to abandoned animals
     • Reply card B:  Here is a gift of $30, $40, $50 (circle one) to provide needed food and shelter to abandoned animals
7. After responses have been received and entered into your CRM constituent records, analyze the results, report and determine next steps.

• Response rate (renewal rate)
  • Group A received 589 responses
    • 589 responses divided by 1,057 records sent Letter A = 55.72% response rate
  • Group B received 601 responses
    • 601 responses divided by 956 records sent Letter B = 62.87% response rate
• Average gift amount
  • Group A produced total revenue of $11,520.84
    • $11,520.84 divided by 589 responses = average gift of $19.56
  • Group B produced total revenue of $20,788.59
    • $20,788.59 divided by 601 responses = average gift of $34.59
• Lift in Revenue
  • Group A provided total revenue of $11.520.84 and Group B provided total revenue of $20,788.59
    • Lift in response is calculated: Lift = [Test (Group B) minus Control (Group A)] divided by Control (Group A) times 100
      • ($20,788.59 - $11,520.84) / $11,520.84 x 100 = 80.44335
  • The ask-string test letter provided a lift in response of 80.44% over Group A!

Our example test had a significance of 99%!