Website testing is conceptually not very complicated. You’re simply showing different versions of a website and tracking which one works best. When thinking about the website testing opportunities, it can help to look at your pages as stages in a decision funnel from the perspective of your prospect.
There are several different types of tests you can run. And if you run a lot of tests, you’ll likely end up running each type at some point. Which test you use will be determined by the specific purpose or outcome your are seeking.
The most common type of tests is an A/B test, or more accurately an A/B/n test, when there are more than two variations. This is were you’ve simply created variations of a page to test against a control page. A/B tests can be split into two types. One is variable cluster tests and the other is isolation tests.
With an isolation test, in contrast, your variations focus on one specific change or hypothesis to test.With these you’re more likely to get learning, and maybe a marketing insight, but the downside is your tests may take longer to run if that one specific change doesn’t make a big impact on the conversion rate. It will involve continous adjustment and improvement. If you have low volumes of traffic isolation tests may take an undue length of time to make substantial improvements.
Variable cluster tests are where you change more than one thing within a single variation. You may get a bigger impact and faster results from these tests, because the changes are more dramatic, but you won’t learn as much from these tests. With this type of test, even if you get a winner, you won’t know exactly which of the many changes produced the biggest impact. That said, using measurement tools like heat maps may give you the insight you need to understand what visitors are responding to.
These are more like isolation tests except that you run several variations on different areas of the page at once. Your multivariate testing tool will compare all the combinations of all the variables, in all the sections against each other to find the best combination. These types of tests usually have more variations being tested than ABN tests, and a lot of times take much longer to run. They do serve a purpose, especially when you want to see interaction effects on very high traffic pages.
But keep in mind you, you probably don’t want to think about multivariate testing if you have fewer than, say, half a million monthly visitors, because they take so much traffic to produce a result.
These are similar to AB tests but instead of just testing one page, you send visitors down a different set of pages. These are often used in defined paths like checkouts or multi-step sign up forms. Or you may want to set up dependent tests where certain messages change across your website depending on the variation the visitor has entered into.
These can be very interesting tests to run, but are more technically involved to set up and need to be planned really carefully.
These vary a consistent section of content across your website. They’re also technically like ABN tests, but I refer to them separately because they’re quite different to plan and run than a typical single ABN test.
Choosing the right test
The type of test you choose should be driven by your hypotheses and also the volume of traffic on your site. Decide which type will best prove out whether your hypothesis is correct? If you want to test your landing page headline, for example, a simple ABN test is probably best. But if you want to try a different order of the links in your navigation, or maybe a new product detail page layout entirely, you’d better go for a site-wide test to maintain consistency for your visitors.