by Remember when you went to the grocery store, and the employees had placed everything in a different layout since your last visit? You’ve just been through an A/B test, congrats! Although this is a variation test, it isn’t the most optimal nor rigorous setup for an A/B test. To make it more reliable, they would need to build an entirely new store and silently redirect half of the customers there. The other half would still be sent to the original store, and only after that, the results would be evaluated. This is, of course, not a reasonable option, which leaves them with this more mild approach.
As you can see, there are terms, methodologies, and conditions you need to respect to make a proper test. By the end of this article, you will have a better understanding and be more knowledgeable about this important (but not so talked about) topic in Web Development. So, let’s cut to the chase and get started.
What is an A/B test?
To start off, we have to make some clarifications. The word “test” in A/B tests, has nothing to do with the more common type of test we usually talk about, those are automated tests. They do share some similarities, but those stop right there. The purpose of AB testing is to extract value from a comparison, not to check if something is working as expected.
With this in mind, let’s move on to what is an A/B test. Also known as split tests, A/B tests make a comparison between one or more versions of a variable (web page, page element, apps, etc.) to see which one performs better in relation to another. The definition of “better” depends on the hypothesis raised in the test, for example: If the button switches the background color from red to blue, users should interact more with the new button. With that, we can test if the statement is true or not. The “A” in “A/B” usually refers to the original version, or “control”, whereas “B” refers to the “variant”, or a new version of the original.
Why should we use A/B tests?
To be more practical, I’m going to give you a real example. Going back to the first scenario, a long time ago, someone working at a grocery store came up with an idea: what if we change where we place the products to see if this affects sales? After permission was granted, they proceeded with what would one day be the standard procedure for shops, and later on even go to the digital world, becoming the topic of this very article.
They had a hunch: the majority of baby diaper sales came from male office workers at the end of the day, probably going back home. The explanation for this behavior was: their partner called earlier and told them they were almost out of it and needed more.
With this in mind, the hypothesis showed up: If we place objects of interest for adult male workers beside the diapers, that should increase the sales because those objects would be closer and easier to get.
They decided to put beers to induce the tired office men into buying just one to relax after an exhaustive day. And it worked. Beer sales had a noticeable increase after that, usually bought with a box of baby diapers, as proposed.
That was a real example of how A/B tests work, it’s all about experiments to achieve a certain goal (typically to increase sales). As you can see, A/B tests have way more value to the business than software development itself, the opposite of automated tests. However, it doesn’t mean that they are not important. They do give a better direction to continuous improvement of the product, as well as more knowledge about clients and the different target audiences.
From the point of view of a software engineer, this kind of test seems a little bit superfluous, but let’s imagine a scenario: each developer on your team (including you) has now a new task to do: a complete redesign of many pages on the website (aiming to increase sales). Everything is good, until the next month when the sales team decides to go back in their decision 🥲
Spending all that time could be costly, and making these changes is likely to delay some important releases related to the core of the business. So, let’s imagine again: what if little by little we tested some small (even big) changes to understand if it’s worth the effort? It will certainly be better than losing hours for nothing. This scenario brings us to one of the most significant reasons why A/B tests are significant: fail early, fast, and without so much damage.
How to do A/B tests?
There are many ways to do an A/B test, and in this section, we will cover some of them. We won’t get into the implementation details, instead, we’re going to take a more general view and tell you how to start testing.
First, you’ll need a hypothesis to test. It can be anything, but there is a template phrase that can help you assemble a better hypothesis: If the proposed change here would the expected result here because why the test would work. This is not a rule, but can help you come up with better-formed proposals.
Next, you’ll need to have options to perform a test of the given hypothesis. To refresh your memory,
” A/B tests do a comparison between one or more versions of a variable (web page, page element, applications, etc.) to see which one performs better against the other”
so you must have these two versions (or more) of a variable available to show to the users.
Splitting URL test
Let’s see a very practical way to start: the split URL testing . This technique consists in splitting the target audience by releasing two versions of the page under different URLs and observing which one was better. We can do this by changing the final URL for the user. For example, if the “control” version is available on https://example.com/exA then a new URL that hosts the new variant is created on https://example.com/exB so part of the target audience sees “A” while others see “B”.
Advantages of splitting URL test
- Very useful when you want to do a large redesign of a page
- It can be used to test other non-visual changes (it’s odd, I know), but let’s imagine that your back-end team made some changes on their side, and they don’t know if it will impact the front-end. This allows you to test without too much damage to the users.
Multivariate test
Another known way to test is the multivariate tests. That one is a bit more complex than our last example, as we’re going to have more than two variations being tested simultaneously to understand which combination performs the best out of all the possible permutations.
Take a look at the image below:
When accessing the website, randomly (or not) you’ll see one of these versions on your screen, while other users will see a different version.
Advantages of Multivariate testing
- This type of test will save you a lot of time. Usually, these variations are testing the same hypothesis, but with different approaches, so you can test everything at the same time to analyze which approach will be used
- Another great advantage is the possibility of putting only the best parts of each variation and trying to reach a common denominator, resulting in a more efficient page in its purpose.
As mentioned before, it’s not possible to cover every kind of A/B test. There are some tools that can help you with testing like Firebase A/B Testing, Google Optimize, and the list goes on, as we have a lot of them available.
Pros & Cons
Pros
- Gather intel about user’s preferences
- Test updates methodologically, increasing the results’ reliability
- Optimize subjects by showing potential improvements
- Detect unknown bottlenecks
- Fail-safe option to possibly disrupt updates
Cons
- More variants equals more dev time spent
- Part of the work will be discarded
- Requires adding more complexity to the app presentation rules
Conclusion
To summarize, A/B testing can be really helpful if you take the extra time to do it, as anything in development has tradeoffs. Overall, it depends on how much value it offers to your project and if the team is willing to commit to it.
REFS:
https://www.optimizely.com/optimization-glossary/ab-testing/
https://adoric.com/blog/a-b-testing-how-it-works/
https://devcycle.com/blog/what-are-the-benefits-of-a-b-testing#:~:text=There%20are%20several%20benefits%20of,your%20site%20or%20mobile%20app.
We want to work with you. Check out our "What We Do" section!
