A/B split testing is a marvelous tool for all email marketers but you can’t fall into the trap of just reacting on the numbers alone and modifying the direction of your overall campaign without understanding the various nuances which can surface in testing. These aspects often have a very significant effect on how well the test results reflect the reality of the customer behavior and experience. Here are some of the factors you need to take into consideration in order to assure that you have a firm grasp on what your figures really mean.

  1. Metric result without the reason for it. A/B split testing will usually provide a determination of which variant performed better than the other, but to understand the reason why that performance differed is not as easily garnered. Insights into customer behavior can often be evaluated to a greater degree if user comments are also taken into the equation to help understand the underlying reasons for that specific behavior or those particular choices.
  2. Go short vs. go long. There are as many opinions on the preferred duration of a particular A/B split test on an element as there are testers. Going too short will negate the impact of longer term effects and may skew the entire test. Latent conversions also need to be taken into consideration as there is usually a lag time factor from when the recipient first experiences a new element and the time that they react to it by doing something about it. This newness bias can actually affect your test results both ways: when an element is brand new some users will investigate it in depth and “overuse” it; while others may be confused by the break with tradition and use it less. That’s why it’s usually wise to err on the side of caution and run you’re A/B split testing as long as you possibly can… and a bit longer after that.
  3. Get ratted out. Especially in community, local, or tightly vertical marketing A/B split testing you may find that some of your recipients may notice that they are receiving a different variant than their family, friends, or colleagues. Although this type of email comparison is rather rare, there is a much greater effect at play here. Now that hundreds of millions of people all over the world are using multiple internet access devices, they may notice that they are receiving different emails on their various devices, possibly due to that device’s pre-existing cookies or even different email accounts.
  4. Elements to not test. The conventional wisdom that you can test absolutely everything in an email is not necessarily true as there are some elements which need to go out to your entire segment, or even your entire list. If you’re having a major promotion, a huge premiere, or an important media announcement, you’d be extremely foolish to test it thus denying part of your customer base from having access to that information. Only test the elements in your email which would not create a negative effect to those recipients who don’t see it.
  5. Ramp it up slowly. We’re all human and we make mistakes, so in order to ensure that some inadvertent error doesn’t creep into you’re A/B split testing, don’t start off with a 50/50 split to a massive list. If there is a problem in one of those two sides which would not present itself in your pre-launch testing you could create a powerfully negative effect in half your list. If you are blessed with enough subscribers to be able to gain statistically significant results at much smaller sample sizes, start with a 0.1/99.9 then after a bit go to 0.5/99.5. If everything is still holding steady jump it to 2.5/97.5 then 10/90 before taking a deep breath and plunging into the full 50/50.

All those sheer numbers you’re getting from your A/B split testing are great but if you’re being blinded by them and allowing them to lead you to an incorrect conclusion about your customers’ preferences and behavior they will turn out to be a fate worse than ignorance!