The Inbox: Quick Guide to A/B Testing

All of the best practices we’ve established here in the email department have been the result of testing, so we strongly encourage self-service clients to dive in and start testing their campaigns. A/B testing is essential to discovering what works with your audience and getting the best open rates and click through rates possible.

In the email world, an A/B test is simply sending two campaigns that differ in just one way to a small portion of your list, determining the most successful campaign of the two, and sending that campaign to the remainder of your list. It sounds complex, but thankfully our email software makes testing easy.

What can you test?

• Subject lines
• Calls-to-action
• Images
• Email length
• Promotion language
• Number of properties listed
• Time of send
• Day of send

There are so many ways you can use A/B testing. The key is to get creative and test as much as possible. Unsure about a subject line? Test it! Can’t decide if “$100 off” or “10% off” will produce the best results? Test it! Testing is really a cure-all for any dilemma you may have when building your email campaigns. So how do you get started?

1. Choose a variable that you would like to test, such as a subject line or a call-to-action. You should only alter one variable for each test so that you can accurately determine the reason for any improvement in your conversions.

2. Follow these steps to set up your test:

• If you would like to test a subject line, follow these steps:
  1. Select “Create a new campaign” from your homepage
  2. Select the  “A/B test” icon at the top of the page
  3. Select “Subject line” from the list of options
  4. Enter your first subject line into field “A” and enter your second subject line into field “B”
  5. Select “Next” at the bottom of the page and proceed to edit or schedule your campaign as you normally would.
• If you would like to test content within your email, follow these steps:
  1. Select “Create a new campaign” from your homepage
  2. Select the “A/B test” icon at the top of the page
  3. Select “Email content” from the list of options
  4. Enter your campaign name, subject line, from name, and reply-to address. Select “Next”.
  5. Select the template you would like to use and build out the content for Version A. When complete, select “Preview” in the top right corner, then select “Define content for version B” in the top right corner
  6. Select “Copy from Version A” and edit only the variable that you would like to test. When complete, select “Preview” in the top right corner, then select “Define recipients” in the top right corner.
  7. Choose the list you would like to send your campaign to, and select “Define A/B split”.
  8. Adjust the percentage of your list being sent the test versions by moving the scroll bar left and right. We recommend sending the test versions to 40% total, 20% receive version A and 20% receive version B.
  9. Choose which conversion you would like to determine which campaign is sent to the remainder of your list under the “Selecting a Winner” section.
  10. Select the length of your test under the “How long should we run the test” section, then click “Next” and proceed to edit or schedule your campaign as you normally would.

3. At the scheduled time, our email software will send Version A and Version B to the designated percentage of your list and monitor opens and clicks for the amount of time you indicated. After that time, the winning campaign will automatically be sent to the remainder of your list.

4. The discrepancy between Version A and Version B may be small or quite significant. To view your test results, click the title of your campaign from the “Recently sent campaigns” section of your “Overview” homepage. Click the “See A/B Report” link above the graph on the reporting snapshot. From here, you will be able to see how much the conversions from each campaign version varied. This information is important to pay attention to so that you can learn more about your audience and improve your campaigns based on the information collect.

Long story short, A/B testing is the key to improving engagement with your email campaigns and it’s never too soon to start trying a test or two. If you have any questions about A/B testing or want more one-on-one instruction, feel free to reach out to one of our email project managers. We would be happy to help!