Never Stop A/B Testing

What is A/B Testing?

A/B testing or split testing is a quantitative experiment in which we try to compare two values of a variable. For A/B testing, we create a new variation (B) by changing one variable in our product will compare it with the original variation (A). In this process, it is best to keep all other variables exactly the same, to be able to measure the result of our change precisely.


While A/B testing is widely being used in online marketing, it is being used in other areas as well. However, in online marketing, we usually A/B test the followings:

  • Landing pages
  • Banners
  • Advertising Targeting


If you’re running an performance marketing business, you should never stop A/B testing in order to be able improve the performance of your campaigns. There is always room to improve your banners, landing pages or campaigns. Also when testing different parameters, try to avoid letting intuition get to your head to prejudge the results. I have been running advertising campaigns for over 5 years, and I still get surprised by the performance of some new banners or landing pages that I test. Put your ego aside, and test as much as you can.


Building an A/B testing machine

One of my secrets of my success in the business is that I split test as many items and variables that I can think of. However, in my early days in online marketing, I had been doing my A/B test especially banners and landing pages in a random and unstructured way. For instance, I would create a new set of banners based on intuition or what competitors were running, and I was testing it with an old set of banners. The result was still satisfactory in most cases because I was able to find a few good performing banners.


The problem of this method was that first, I was not learning why a new banner is working better than the other, second I didn’t have a system or machine to have a sustainable growth. Sometimes I could be lucky and come up with a very good new banner, and sometimes not and the reason was that I wasn’t leaving my growth to pure and random creativity.


So I have decided to create a method that would split test systematically, which can improve my campaigns in a sustainable way. I call this method my A/B testing machine.


Step 0 – Basics

This step is not an actual part of the A/B testing but it is the basics and infrastructure that you need to have and I had before building your A/B testing system (that’s what I am calling it step 0). So before starting anything you should prepare the followings:


  • Define your campaign’s goal: whether you have an online shop, running affiliate marketing, a lead generation  or any other type, you should define your campaign goal. Is it a product purchase, or add to cart? Is it a lead or an app install? You should know that your A/B testing goal is the same as your campaign goal. For example, I have seen some people are making mistake and set CTR as their goal for banner split testing. This is a terrible mistake. A better CTR does not necessarily provide better conversion. So pay attention to this.
  • Prepare your tracking or reporting tool: a basic reporting or tracking tool is essential for your A/B testing. You basically, need to be able to see your stats on your variations. For instance, if you’re testing banners, you should see the stats of banner A and B in your reports.

Step 1- Plan

Now that you’re all set, you can start building your split test mechanism. No matter if you want to A/B test your banners or landing page here is how you should start:


  1. Create a design template: or if you are  already running traffic take one of your best performing creatives or landing pages.
  2. Identify the variables in your template: You should see which elemets in your template you can change to make new variations. Here are a few examples:
    1. Image
    2. Background color
    3. Call to action buttom color
    4. Call to action text
    5. Content and Hader Text
    6. Signs, symbols, badges
  3. Create 1 to 5 variations: Change one variable at a time and create new variations. Depends on the volume of traffic and the number of conversions you receive, you can create 1 to 5 variation. Don’t go for too many though
  4. Name the varitions based on the variables.


Step 2- Measure

An important step in A/B testing is to measure the performance of each variation properly.


  1. Make sure to send the varition id or name to your tracking / reporting system.
  2. Run traffic to your variations: It is recommended to send equal amount of traffic to each variation. However, if you think it might be too risky, you can send 10% to 20% of your traffic to the new variations and the rest to the original.
  3. Create an excel sheet with the reports of all your variations. Even if you have a good reporting system, having this excel sheet is good to keep the track record as an all-in-one report.
  4. Calculate the confidence level and compare the variations: Before coming to any conclusion or stopping the A/B test, you should make sure that the results are statistically signifiant. In order to that, you should measure the confidence level between variation B and A. Well, if don’t know a lot about complex statistical calculation, you should just know that in order to come to conclusion that a variation is better than the other, it is recommenced that your confidence level would be higher than 95%. However, some people (including me) believe that 90% is also enough to make this decision. Basically it means, according to the test, we are 90% sure that one variation is performing better than the other. Below are the tools that calculate the confindence level for you.
    1. [For Beginners]
    2. [For Advanced users]


Step 3- Learn

Ok, now that you’ve measure the results of your split test properly, and you identified, which are better than the other, it is time to interpret the resutls and the logic behind it. In this step, you want to know why a certain text, image or color is performing better than the other. Of course, this interpretions might not be 100% accurate, but even having a few guesses and research is good enough. Take notes of your thoughts and researches, after some more A/B test rounds you will see the patterns more clearly and it will help you to create better variations and improve your campaigns better and better.

Step 4 – Go to step 1

Congradutions, your first round of A/B testing is over. Now you are in the route to boost your CR and ROI. Go back to step 1, run another round, and never stop your A/B testing machine. You will see it will work like a machine for you!


Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>