All About A/B Variant Testing

Updated on: February 13, 2017 Category: A/B Testing, Features, Miscellaneous

This feature is available to our Pro and Ultimate Users – click My Account > Upgrade Details for rates on upgrading to Pro if you’re on Standard and would like to use A/B Testing!

A/B testing (sometimes called split testing) is sending and comparing two versions of an email campaign to see which one performs better. You compare two email sends by creating two or more variants (let’s call them A and B) and then send each one to a sample portion of your list at the same time. The one that gets a better conversion rate (either clicks or opens), wins!

How to Set Up an A/B Test

Click Create a New Campaign and build your campaign if you haven’t already. You will set up your A/B split on the third step of the campaign builder, Settings. If you already have an email created, click EDIT under the campaign name, then click Next Step until you get to step 3, Settings:

ab

Once you click on Add A/B Variation, you will see this page:

manage-variants

This is where you will control all your A/B test variants.

Variation: The default is two variants. To add another variation, click Add Variation or Actions > Duplicate to make a new copy of a specific variation.

Weight: This percentage will automatically update as you add or remove variants.

List Sample: The slider lets you choose the percentage of your total list that you want the variants to go to. So if your total list size is 100 emails, and the slider is at 20%, the sample send size to determine the winner would be 20 emails total. If you have two variants, 10 would get Variation A, 10 would get Varation B.

Timer: This lets you choose how much time you want the sample to collect information until we pick the winner.

Choosing a Winner: the winning variant can be determined either by unique opens (open rate) or unique clicks.

Edit Variation: Click here to return to the variant email campaign to make changes.

Preview: Preview the email campaign.

Actions: You can Duplicate or Delete a variant using this menu.

Edit Variation: This button lets you edit the Subject Line and/or From Name. You can also click Edit Content from here to go back and edit the design:

variantb

How do I remove the variants and go back to a normal campaign?

Delete variants until you only have one left.

How do I send test emails for an A/B Campaign?

Navigate to the final Test & Send page for the campaign:

test-and-send-ab

You can use the dark grey Send Test Email button to send one test email of each separately, or use the lighter grey Send Test Emails button to send one of each version simultaneously.

What happens after I send the initial test variants?

Once you send your test variants, the system will send to your chosen sample size and begin collecting data over the time period you set before choosing a winner. During this time, you will see your campaign status under Campaigns show as “A/B Testing:”

Once the time period is up, the system will pick your winner based on clicks or opens (whichever you chose) and send out the remaining emails.

How do I pick a winner before the timer is up?

If you decide you don’t want to wait for the timer to finish and want to send your campaign right away, click Options under the campaign name, then Pick Winner:

From here, simply click Pick Winner for the campaign you wish to send out immediately:

You can also contact us here

  • Live Chat Support

    Live chat is open
    between 9am-7pm EST.

  • Email Support

    During business hours,
    we usually answer emails within minutes.

  • FAQ's

    Check out our FAQ
    for our most common issues.