The one thing your split testing strategy has been missing

Do you know your ABC’s of marketing?

This is not meant to test you (or your patience) but as modern-day marketers, we must understand the power of A/B testing. Measuring performance is how we understand if we are successful or not.

In science experiments, there is a controlled variable. In medical research, there is a placebo. And in A/B testing, there should always be a control group.

In this article, we cover why all marketers should split test their campaigns including the control group and top it off with successful examples.

Taking the Placebo out of split testing

A control group is a segment of your campaign (for instance 10% of your email list) that won’t experience anything different. Many think that A/B testing is where you split test just two options by definition. That is not true, it is more the idea of multiple groups and versions. But there is a special addition: A control group helps make sure you’re not taking your own placebo pill.

Let’s say you want to test the impact a colored button has on click-through rate (CTR) of your lead-gen pop-up. You A/B test between a blue and red button. After your campaign finishes, the results are in—the red button received a 2% CTR, and the blue 3%. You should move forward with the blue button, right? Well, what you actually should do is test another color in addition to the original red and blue tests.

So why would you want to continue testing the red button color you know is underperforming? Because you must have some benchmark to measure external factors against. The control group lets you look at the data and know whether the results are from button color or something else. If the red button performs better in the second test, you’ll know that the data is skewed.

One way to weed out skewed data is to use a multivariate testing strategy.

Multivariance as the Silver Bullet of Split Testing

Multivariate testing is a fancy way of saying you need to add more variety in your testing strategy. One way to do this is to split test your split test. Very meta, I know.

Here’s how to do multivariate testing on an email campaign example. Take your entire email list and split it into two groups.

Group one is subdivided into A, B, and C, receiving 2 testing variations and a controlled variable. This campaign is sent out at a certain date and time. Group two is also subdivided into A, B, and C, but their campaign is sent out at a different date and time.

split testing time

Testing more than one variable is the key to multivariate testing. Multivariate testing works best in outbound channels, like email and SMS.

Here is A/B/C testing: a A/B Tests with more control

A/B/C testing, or split testing with a control group, should be used in email marketing, landing page optimization, and more. Let’s say you’re looking to optimize your lead-gen page and think you’ve finally found the right combination of copy, color, and call-to-action. You’ve been split testing two variations and one is the clear outperformer.

The problem here is that your outperformer is a local maximum. Yes, the red button looks great compared to the performance of the blue button, but there is no control group to compare. With a black button as a control group, for example, you truly measure the performance of the red versus blue button.

split testing example control group

A/B/C testing is especially helpful in businesses that have become set in their ways. Allegiance to the status quo happens, and the only way to push the limits of performance is to scrutinize those practices.

Let’s say a business has used the same email campaign, site layout, or funnel page for a long time with acceptable results. This is where they fall trap to the common if-it-ain’t-broke-don’t-fix-it mentality. Instead, A/B/C testing new variations helps push the simply acceptable, into the realm of unbelievable.

Real-World Examples of A/B/C Testing

Testing and technology go hand in hand.

Developers test their software to make sure there are no bugs in the code. Marketers must also test campaigns to make sure there are no bugs in their hypotheses and marketing strategies.

Let’s say we still plan to A/B test button color, but now we want a multivariate test to determine the background image’s impact. In this case, we will be able to determine the best possible combination of background and button color:

split testing example control group multivariate

Here are some other examples of the most innovative tech giants using control groups within their A/B testing strategy:

1. Airbnb Uses Control Groups To Optimize Search

Airbnb uses control groups for testing everything about their product and marketing, including search engine optimization. When Airbnb runs a test driving traffic to their website and towards a new landing page type, they have difficulty measuring performance since each city has different levels of demand.

arbnb split testing control

When testing a new feature, comparing cities like San Francisco and New Orleans can be difficult given the sample size of each city.

Instead, Airbnb analyzes performance across all URLs, and determine the results of new products, features, and other A/B/C tests.

2. Netflix uses control group to test promotion channels

The team at Netflix is busy binging on your data.

Netflix is famous for its culture of A/B testing. Along the way, they have encountered interesting problems and challenges testing at scale. One of the things touted is the AI that is able to predict the right thumbnails to screenshot / render and show in the Netflix App. Although I dont think AI will replace marketers soon, because some of the testing also needs that human touch.

Let’s say for example the company is testing promotion on a certain show or movie. They can’t accurately tell if the conversion was from a site feature or external advertising, like billboards and print media.

split testing netflix example

One strategy they’ve rolled out is to use entire regions as control groups. They limit billboards in one region and focus on promotion within the app to test against markets that see the billboards versus those that don’t. They call this quasi testing, which just sounds cool.

Use control groups in your tests

Data-driven everything is the future, but the future is now in marketing.

If you plan to grow your business or team, you must learn to accurately test as much as possible. Test your emails, test your website, test your patience…okay, hopefully we are not contributing to this last one.

A/B/C testing is the future of marketing, and we invite you to test it out in your business today. If you want to learn about more trends that are driving the future of email marketing, check out the prediction of email trends to watch out for this year.

Drew Page

Drew Page

Drew is a content marketing lead from San Diego, where he helps create epic content for companies like CleverTap. He loves learning, writing and playing music. When not surfing the web, you can find him actually surfing, in the kitchen or in a book.