Rightio want to help you look after your home; it’s a 24/7 service that enables its customers to find and book reputable local tradesmen. We have just finished a piece of work testing their landing pages, with the aim of helping them to better understand what drives customers on their website to actually make a booking.
The project was an interesting one, as it had lots of different moving parts, including elements of offline conversion, as well as different regions and demographics. With this in mind, I’d like to share how we tackled analysis of the project, including some of the tools and services we used, as well as most importantly, a bit about our results.
Different Regions, Different Messages
We got off to a great start – Rightio are very marketing-savvy, with a good grasp of pay per click (PPC) optimisation. The team had already developed hundreds of different landing pages that were suited to the many different regions they are active in, paired with hyper-localised PPC adverts and keyword bidding.
So, during the first phase of the project (‘Think’), we identified that we would need to run a multivariate test in several of these regions, as someone in Leeds might have different needs compared to someone in London.
Next up, we needed to consider the types of messaging we would use within our test. We decided on a multivariate test, as this would allow us to understand how the combination of messaging within components on the site influenced conversion. From Rightio’s analysis, we identified three key themes coming through as concerns and potential conversion push points for users:
Testing a Landing Page for Brighton users, with ‘Trust’ as the key conversion point.
We also included a ‘Wildcard’, which combined these different messaging points to help us understand if one core messaging theme was important, or if it was a blend of multiple.
Running the test
So, we had 4 regions to run a test in and 4 variants, as well as of course the control page. This meant we had 20 landing pages to work with. Rightio’s conversion takes place offline (a user picks up the phone and calls them about a plumbing leak), so we needed to understand how this would be reported. The solution was that the in-house team implemented new phone numbers for each variant, as well as a ‘clean’ control number which wouldn’t appear anywhere else.
We also used Optimizely to manage our traffic allocation. When building the pages, it became clear that it would be easier to use Optimizely’s redirect traffic functionality, which allows you to redirect a set percentage of users per landing page variant.
Optimizely also allowed us to easily integrate Google Analytics, so we had custom dimension tracking (functionality which allows you to powerfully segment users) for each variant in place.
Optimizely also integrates with Crazy Egg, which uses heat maps so you can see where exactly your visitors are engaging with your website.
Outcomes for Rightio
Getting to a point where our results had statistical significance was key, but as our conversion took place offline, we couldn’t use Optimizely’s inbuilt tool for measuring this, so instead we used a ‘two-tailed test’. This can be a controversial topic within statistics and it’s important to understand there are nuances between one and two-tailed tests (more on that here).
This methodical approach to testing meant we were able to finalise our results with confidence. We tested our findings for statistical significance and found that across the board, at least one variant outperformed the control, per region, giving Rightio a clear way forward.