A standard deviation is a critical factor that email marketers engaged in A/B split testing should be intimately aware of. The standard deviation is the determinant of the range of probability of your A/B split testing results and it doesn’t take a chalkboard full of obtuse equations to calculate it accurately, just a fairly simple “fill in the blanks” formula.

Heads or tails

When you flip a coin you either get heads or tails and when you run an A/B split testing you will also have two results, either the A or the B. It’s completely possible and even somewhat likely that you’ll flip a coin three or four times in a row and always get heads, just as it is possible that a small A/B split testing sample will be predominated by As or Bs. When the sample becomes much bigger then the percentage will settle in to greater accuracy, and when you have a huge sampling in the millions, you can be assured that your results are absolutely spot on.

How many trials per conversion?

When we apply this process to something like an email message element’s conversion rate we can see the immediate relevance of this calculation to our overall A/B split testing. In calculating a conversion rate for an element the only possible results are simple: convert or non-convert. In order to estimate the value of that figure we do a number of trials. Once we have derived the data from those trials we can now proceed to determine how many of those trials were necessary to result in a single conversion.

A handy shortcut

So now we repeat the process several times and as is overwhelmingly likely you’ll receive a different convert/non-convert value each time. Once you’ve done enough of these trials, you’ll obtain a range for your conversion rate. Math provides a handy shortcut to endlessly doing trials and that is something called a standard deviation. This factor determines what the standard deviation from the convert/non-convert value can be anticipated if the trial is conducted far more often than it actually is. If your deviation is large then you still have work to do as you haven’t fed in enough raw data yet, but as your deviation becomes smaller you’re zeroing in on the actual value.

Real world example

So let’s take a real world example. Your conversion values in 8 n trials are:

2, 4, 4, 4, 5, 5, 7, 9

Then your standard deviation is the sum of those values divided by how many values you have. So since those are eight numbers and they total 40, then 40 divided by 8 is 5 which is the average conversion value in those 8 trials. Now you have your critical average value for your calculations, as it’s 5.

Now we start determining the standard deviation by figuring out what the difference from each data point is, and squaring each. Don’t worry it will make sense soon:

(2-5)2 = (-3)2 = 9
(4-5)2 = (-1)2 = 1
(4-5)2 = (-1)2 = 1
(4-5)2 = (-1)2 = 1
(5-5)2 = (0)2 = 0
(5-5)2 = (0)2 = 0
(7-5)2 = (2)2 = 4
(9-5)2 = (4)2 = 16

Now we add up all those sums:


And we get 32. We divide that by the number of sums and we know that’s 8. So we get 4. We’re almost there as now we have to take the square root. That’s 2. So now we can confidently state that the standard deviation is 2.

Plug in the numbers

Therefore we know that the average is 5 and the standard deviation is 2. What that means is that even if you keep doing the A/B split testing almost indefinitely your final p values should always fall somewhere between 5-2=3 or 5+2=7. If the standard deviation had been 1 it would have been between 5-1=4 or 5+1=6, and if it had been 1.75 it would have been between 5-1.75=3.25 or 5+1.75=6.75.

It’s extremely significant to be able to calculate an accurate standard deviation as it will save you countless days and weeks of repeated A/B split testing!