Priya Nain
Author: Priya Nain  |  

Product & Email marketer at Mindvalley. Making stuff less boring to read. Infrequent Trekker. Nice girl, rarely mean.


Read more posts by this author.


In email marketing, A/B testing is the process of sending one variation of your email to a subset of your subscribers and a different variation to another subset of subscribers and then looking at results to see which one performed better. The winning variation is then sent to all of the remaining subscribers in your list.

Source: Testing-email-marketing

A/B testing can be done on all individual elements of an email like subject line, preview text, from-name, or on the whole email such as changing the template, long copy vs short copy.

In this guide, we will look at what you should test, and tips for running an effective a/b test. These frameworks and tips can be used to increase the open rate, engagement, and overall ROI of your emails for any brand and industry.

The one mantra we tell everyone trying to improve their email marketing ROI by just looking at test data published by other brands or industries is - Do your own test.

That's because the results may vary from one brand or industry to another. For example, a short copy might get a better click-to-open rate in eCommerce but might perform poorly with infomercial products. That's why instead of relying on generic research data, test with your audience and product to draw the correct conclusions.

Let's look at the 2 most important tips to follow before you start a/b testing

2 Things To Note Before A/B Testing

Have a clear hypothesis

The definition of hypothesis - explanation made on the basis of limited evidence as a starting point for further investigation.

It basically follows the structure: “If I change this, it will have this effect." It should be aimed at getting some insights.

For example, you saw a CTA that was very effective on you and you now want to a/b test it in your emails.

A bad hypothesis or no hypothesis statement is - ' Change the CTA copy because it works better  '

A good hypothesis would be ' Changing the CTA from Enroll Now to Learn More will reduce confusion and will increase email conversions.'

So formulate your statement like this:

  • Changing _______(element in your email) into ______(new idea you have) will ________ (end effect you want), because ________(your assumption).

Test one thing at a time

Test one thing at a time so that it is clear as to what affected the results. If you test multiple things at once e.g. you changed the CTA placement and also switched copy from long to short, you won't be sure which one of these increased or decreased your conversions.

For example, if you are comparing two different subject lines, you should keep the send time, from name, preview text same.

Now let's look at some of the ideas to help you get started with a/b testing.

What To A/B Test

From name

If you open your inbox and just observe the ' from name' , you can find many variations.

Here is a screenshot from my inbox

Source: email inbox image
Source: deliveroo email

We have an informal 'from name' (Yo, it's Ash), person's name along with company name (Tim from Copyblogger), just person's name (Corey Haines) because they have a strong personal brand, Just brand name (Deliveroo).

Personally, I think that people are naturally more likely to open an email when it has an actual person’s name in it vs the name of a business.

But don't assume anything unless you test. For example, I recently observed that Lunch Money went from using Jen From Lunch Money to just Lunch Money. I am assuming the company name worked better for them.

Source: lunch money email

Metric to measure the result: Open Rate

Subject line

Try personalisations in your subject line by including the subscriber's name. It won't work if you overdo it and use it every time. Test using it for some of the emails that require immediate action or if you are launching something new where you need maximum eyeballs

Source: personalised emails

Use of emojis in the subject line has shown higher open rates but when every other brand starts doing that, the inbox feels cluttered and the novelty of an emoji wears off.

The trick to using emojis correctly is:

  • Using them not too often and not overdoing it with too many emojis that it feels spammy
  • Owning an emoji and using the same one every time (See how Google Maps uses the earth emoji everytime or Morning brew uses the coffee emoji)
  • Sticking to a theme (Calm mostly uses a heart emoji and just vary the colors)
  • Instead of using it at the front of the subject line, try using it at the end (see ClassPass email)
Source: Calm email
Source: Google Map Timeline email
Source: classpas email
Source: email ab test sl
Source: morning brew two emails

You can also test short vs long subject lines. For example, Morning Brew uses 2-3 word subject lines vs Lunch Money has longer subject lines.

Source: morning brew email subject lines
Source: lunch money email

Metric to measure the result: Open Rate

Long copy vs short copy

A lot of marketers who aren’t copywriters are certain that short copy will win over long copy every time.

The problem is that logic isn’t irrefutable, and there are case studies to prove it. Copyhacker’s case study about 3.5xing Wistia’s paid conversions from email is a perfect example. When Joanna first showed the test version to Wistia, they immediately objected with “that copy is longer than we’re used to”.)

And look what happened.

Sometimes people don’t act because they aren’t adequately persuaded. That calls for a longer, more detailed copy.

The question is what should you include in that extra copy? Yep, that’s what you’ll be testing.

Common examples of additional copy include:

  • Customer testimonials
  • A longer list of all the features or benefits associated with your product or service
  • Stories that explain, in detail, how your product or service helped people overcome a problem or achieve their goal
  • Stats and data that reinforce the need for your product or service
  • Frequently asked questions (with answers)

Here’s an example from The Nue Co. This is an eCommerce brand, which is an industry that’s prone to short copy. Check out what this company does instead:

Source: email content

Rather than just having a well-designed image and a brief description of the product, The Nue Co. adds some extra copy to clarify the problem their products are solving.

If you’re not sure what to add with your extra copy, do a quick survey at the end of the nurture campaign you want to optimize. Make it one question, and ask “What’s stopping you from doing x?” (In this context, “x” is whatever action you want people to take.

Use the answers from that survey to uncover what’s holding people up, and then test some additional copy and see if your results improve.

Metric to measure the result: Click-through-Rate

CTA

You can test the copy, placement, frequency, and design of the CTA. Of course, test one thing at a time.

  • For copy, for example, you can test between ‘Join Now’ & ‘Enroll Now’.
  • For placement, you can test above the fold vs below the fold.
  • Test one versus multiple CTAs in your email campaigns and see if there is any difference in the conversion rate.
  • Lastly, for the design, you can vary the color or size of CTA. You can also test a button CTA vs a text CTA.

Metric to measure the result: Click-through-Rate

Image vs No image

The first level of image tests can be checking whether images increase or decrease your click-to-open rate. Then if you discover that images help improve conversion, you can test with the kind of images used and placement of images.

Here is an example of email from M&S where one email has an image and that pushes the download app below the fold. Then there is another email without an image that shows the download app above the fold.

So these can be tested to see which one gets you more clicks on the download app buttons.

Source: email with image ab testing
Source: email wo image ab testing

Metric to measure the result: Click-through-Rate

Conclusion

A/B testing is a constant process and should not be avoided if you have a small list or stopped because now you are a big brand. Online behaviour of people keeps evolving and to keep up with that, you need to constantly keep track of what's working and what's not working.

So, test early and test often for best results.
If you are looking for an email marketing software that can help you set up your tests without any technical difficulty or confusion, give SendX a try. You can sign up using your email ID and get a 14-day free trial that will give you access to all features (testing, segmentation, email creation etc.).