This confession has been a long time coming: I love split testing and conversion rate optimization.
There… I said it!
These two practices have been the main focal point of my entire professional career!
In this post you are going to read a lot of things that might make it look like I hate split testing (I don’t!). It is still one of the best ways to measure what works and what doesn’t.
What we’re really doing today is taking a more critical look at split testing and finally busting some of the biggest myths around this extremely useful practice.
Here are the big 7 you’ve heard time and time again.
(NOTE: If you’re just getting started with split testing – check out this post.)
Myth #1 – Split Testing And Optimization Are The Same Thing
It took several years for split testing to hit the mainstream, but today it is one of the most recognizable optimization tactics.
It’s growth in popularity stems from two seductive things:
- Extremely low barrier to entry (low cost and easy tech implementation),
- Countless case studies depicting 300% lifts in conversions.
Testing has become a recognizable practice at the C-Level, and it’s no wonder why – who wouldn’t want a 300% lift from a split test?!
In fact, TrustRadius found that 48% of companies plan to spend more on split testing this coming year.
Unfortunately it is the increased popularity that has caused a fallacious reduction of terms. Too many marketers associate split testing as conversion rate optimization (CRO), which is a mistake.
Simply put…
Conversion rate optimization is a process that uses data analysis and research to improve the customer experience and squeeze the most conversions out of your website.
(RELATED: Leveraging Conversion Rate Optimization to Drive Growth)
Split testing is a CRO tool used to verify your optimization strategies.
Myth Busted: Split testing is only one of the many tactics used when optimizing a site.
Myth #2 – You Should Only Run Iterative Tests
Iterative testing is a useful split testing methodology, but it is not the only way to run tests.
An iterative testing program is when each new test is born out of the previous learnings. In general, iterative tests make small page tweaks, implement the winner, and test another small change.
Chris Goward, CEO of WiderFunnel refers to iterative testing as Evolutionary Site Redesign (ESR). Though this is a valid practice, it should not be your only testing approach.
Image Credit: WiderFunnel
The above image simplifies the concept of iterative testing, but makes some unrealistic assumptions.
The first is that each test will provide you with some level of website success –– many tests actually flat line or don’t provide a lift. If tests don’t provide any lift between each variation, your website isn’t becoming more successful!
One of the major problems with iterative testing is that it relies on smaller scale changes. More often than not small changes in elements entail small changes in results.
Sure, there are plenty of case studies where a button color increased clicks by 400%, but those types of results are generally:
- Impossible to replicate
- An indication that something is broken in your testing process or technology
Optimizers need to adopt a mixed approach with their testing technique between innovation and iterative tests. I’ve been a huge advocate for the innovation technique (the radical redesign of a page), when it’s appropriate.
Iterative advocates -– before you get your pitchforks, I have a very serious question:
If the page below were a new client’s page, would you really suggest ‘continuous design improvement’?
Screenshot: MrBottles.com
Myth Busted: Sometimes a page is just broken and you need to blow it up and start again. When the updated variation wins, you don’t just stop optimizing. You run further iterative tests to squeeze out more cash.
Myth #3 – You Should Test Everything
This is one of the worst myths of them all.
Not everything should be tested!
If you take one point away from this article, please let this be the point. For every split test you run, there is an infinite amount of other tests you could have run.
You want to make sure you are testing on pages that will bring in revenue and testing elements that matter! There are two types of pages you should never test:
- The broken page that just needs to be fixed.
- A page that has no impact or an inconsequential impact on revenue.
As marketers it is easy to get lost in all of the metrics we find oh-so-interesting. As much as marketers love metrics, it’s important to remember that bosses/clients love money.
Myth Busted: If your test can’t be related to the bottom line somehow, you shouldn’t be testing.
Myth #4 – Everyone Can/Should Split Test
I’m going to go out on a limb here and say it…
If your site is traffic ‘challenged’, you should not be split testing.
If you can’t get at least 100 converting actions on each variation, don’t bother running the test. If you do run the test, be very careful with the data because it is probably wrong.
Since split testing and CRO aren’t the same thing, you can still optimize your low traffic site and use other verification methods including:
- Qualitative insights – Insights gathered from non-analytical sources such as heatmaps, user surveys, or personas.
- Persona construction – A semi-fictional character depicting your ideal customer, in general you create several to account for different customer types.
- Real time personalization – A type of user segmentation that dynamically inserts or provides specific content for that user segment.
- Sequential Testing – A type of test comparing the results from two distinct dates. This method is often frowned upon, but is a good way for traffic challenged sites to get an idea about what’s working. Use sparingly if at all.
- Split tests using micro conversions as indicators – Micro conversions are higher level conversions ,e.g., clicks, on higher funnel pages, e.g., homepage. They are used as indicators when you run a test that doesn’t have enough deep funnel conversion data. For example: if you are an ecommerce site with low traffic, you cannot run a test on your homepage and measure these changes at the sales level. When testing, you can look at the these higher level conversions as an indication of success.
Myth Busted: Tests with an insufficient sample size wastes time and money on an optimization medium that will likely give you bad data.
Myth #5 – Test Results Will Remain Constant
Test results are a snapshot of a test’s performance for a period of time. There are various factors that can cause your conversion data in the future months to change with the biggest factor being time.
Seasonality will always impact conversion rates, and a test that was run in November will have very different results than the very same test in March.
Why?
Because your visitor’s intentions are vastly different during these two periods of time!
Whenever you make your winning variation live, you need to periodically monitor the conversion rates. I recommend this for two reasons:
- You will identify imaginary lifts
- You will be able to identify more optimization opportunities
An imaginary lift is when a test shows you these amazing results that don’t pan out.
Normally this is due to a false positive caused by low sample sizes, test scale issues, or poorly quantified metrics. If you catch imaginary lifts early, you might just save your job.
There is nothing more frustrating than being told to expect a 15% increase in sales and then never seeing the payout.
When you monitor these pages, you become a step ahead of the game. Time isn’t the only thing that degenerates conversion rates:
- traffic source changes,
- offer appeal,
- design paradigm shifts
…are among some of the other factors that will depress your champion variation.
When you monitor these pages, you can get an idea as to why the conversions are dipping and then start a new test campaign to remedy the situation.
Myth Busted: Due to this volatility of your customer, you shouldn’t take your test data and make 12-month projections with that data.
Myth #6 – Split Testing Will Fix My Awful Conversion Rate
Here’s the deal…
Split testing is only a validation tool and nothing else.
Your awful conversion rate is likely due to other issues, and split testing will help you prove what’s broken.
You’ll never be able to merely split test your page into a conversion machine. You need to research and analyze what’s depressing conversions and develop a solution. The solution comes from the research, not the test.
It’s also important to note that you can’t simply optimize a page to be a real winner either. At the very foundation what will impact your conversions is your offer and its appeal to your market.
Myth Busted: No amount of CRO will fix a site with a broken offer.
Myth #7 – The Test Is Done When My Tech Says So
Since it’s so easy to use tech tools, this is in the running for the most damning myth of them all.
Testing tech numbers can lie –– yes, you heard me right.
Your testing tech is great at crunching numbers, but these are simply numbers in a vacuum. It’s our job as marketers to put these numbers in context!
What’s wrong here?
The sample size is just too darn small!
Look at the top variation, the one with 3 conversions. The tech is telling me that this variation is the winner at 95.9% confidence rate (95% is the industry standard). If I only used the tech as my standard for calling a test, I’d be in trouble.
What’s truly disturbing is that 25% of marketer’s who run split tests call that test as soon as the tech indicates.
This myth has even permeated into some of the top marketing groups I’ve had the privilege of being a part of! Just a week ago I saw this answer to the question of, “How many visitors should I have before I call a test?”
Remember, these are some of the brightest people in the industry!
However, the testing technologies have positioned themselves in such a way that we take their approval at face value. Please, be smarter than your testing tech and apply the numbers.
Here are a few ways to make sure you aren’t being duped by unapplied statistics:
- Aim for ~100 conversion activities per variation
- Always find out the set sample size necessary, this will help you schedule your test (you can use any of these calculators: Optimizely, and VWO)
- ALWAYS complete the week (or better yet a buying cycle). If your test is scheduled to be ‘significant’ in 10 days, make sure to run the test for 14 days to account for daily variations
- Don’t just call a test because your tech told you to! There is always a lot of volatility at the beginning stages of a test, and your tech might call a winner because the sheer percentage difference is so large it seems significant.
- Don’t add new variations or remove variations during the testing period. You’ll need to restart the test to do this.
Myth Busted: Testing technologies only report numbers –– we need to apply them.
There are a lot of split testing myths out there and despite these misconceptions, split testing is still one of the most powerful tools in an optimizer’s arsenal.
Ultimately marketers need to understand that the role of split testing is to validate their optimization efforts on campaigns with sufficient sample sizes– that’s it!
Have a question?
Ask the DM team and 9,036 other members in the DM Engage Facebook Group!
Not a DM Lab Member? Learn more here.
