[et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″]
Prioritizing ideas based upon potential impact and test run length
[/et_pb_text][et_pb_image src=”https://33sticks.com/wp-content/uploads/2020/09/image-2-1.jpg” title_text=”image (2)” _builder_version=”4.4.7″][/et_pb_image][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″ custom_margin=”-200px||||false|false” custom_padding=”69px|||||”][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″ min_height=”2327px”]
“Sound strategy starts with having the right goal.” Michael Porter
Whether you are brand new to A/B testing, or a seasoned professional, there are always ways to step up your game! Here is an effective tactic to help you kick-start your optimization program, or kick it into the next gear.
So, you have a list of 50 test ideas gathered from across your company, and have compiled those ideas into your optimization backlog. What next? How do you pick which ideas to test first? You could have key team members vote on those ideas. You could sort the ideas by level of effort and traffic. You could listen to your HIPPO’s opinion. Or, you could step up your game and put some more strategic thought and planning into this. The reason you are testing in the first place is to move the needle on your KPI. So, you want to test ideas that move the needle the most. But which ideas are those?
Let’s say you have a test idea to simply change the title of your call to action button on your Product Detail Page from “ADD TO CART” to “BUY NOW.” A simple, yet potentially high impact, test idea. If you voted for the test idea, one team member could shoot it down and say “No way, this test will never move the needle on site Revenue.” If you used LOE and traffic, you would come up with a different answer because it would be a low level of effort and high-volume page, making it a no-brainer. Two different answers to the same test idea.
Let’s help put some objectiveness and data behind your choice instead, and remove the guess work from your process. The first thing to decide is:
Who is my intended audience for this test?
Sounds like a simple question, but requires a few levels of digging. Is it all visitors to your site that are put into the test experience, regardless if they see the test variation or not? Let’s say that is 10,000 daily visitors. That sounds somewhat arbitrary. Is it visitors to your site that reach the actual page your test is on, in this case the Product Detail Page? So they may or may not have seen the test variation. Let’s say that is 25,000 daily visitors. Sounds like we’re getting close the intended audience. How about all visitors that reach the Product Detail Page? This is getting very close now. Let’s say that is 15,000 daily visitors. One step further and we could say all visitors that reach the Product Detail Page AND click on the BUY NOW button (See Figure 1). Perfecto! Let’s say this is 8,000 daily visitors.
Now we need to decide:
What do you do with your new audience?
Go into your Analytics tool, and create a segment for these visitors. Then, let’s see how well this particular audience converts on the site. It turns out, visitors who visit a Product Detail Page AND click BUY NOW show a 4.5% conversion rate in your analytics tool. This is compared to an overall site conversion rate of 3.1%. You now know (1) potential size in daily visits and the (2) conversion rate for all of your test ideas. This is a great way to help your prioritize which tests to run on the site (along with Level of Effort). You now have your specific audience that you want to show the test experience to, and you also know how that audience converts on your site.
Now what?
Let’s estimate the lift we expect the test variation to produce?
From your past testing experience and from talking with the product and business owners in your company, you predict that your test experience will drive a 5% increase in conversions, as compared to visitors that are not in the test experience.
Finally,
Estimate how long your test should run
We can use the (1) 8,000 daily visitors, (2) 4.5% conversion rate for this particular audience and (3) the 5% estimated lift you expect to see to determine approximately how long you should run your test on this site. There are many tools out there to help you estimate test run length. Adobe has a good one HERE. After plugging in the numbers, your test will take approximately 5 weeks to reach a 95% confidence level (Figure 2). You can now add this to your prioritization and test planning as well, so that you can effectively manage collisions on your site while testing…which will be the topic of my next blog.
Now take this approach to all of the ideas on your backlog, add in Level of Effort (Dev and Creative), rank order the list, and start testing!!
[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_code _builder_version=”4.4.7″][author] Jason Boal [author_image timthumb=’on’]https://33sticks.com/wp-content/uploads/2020/02/Profile-Pic-33-Sticks-2-150×150.jpg[/author_image] [author_info]Jason has over 10 years of experience working on both the client and agency sides, and across Retail, Financial Services, and Non-Profit industries. He always looks forward to helping clients build upon and improve the customer digital experience. He follows a data-driven strategy, focused on constantly learning more about the customer digital experience from both quantitative and qualitative information. Jason’s philosophy is, “taking data and developing a strategy centered around people, process, and technology will lead to tremendous results.” [/author_info] [/author][/et_pb_code][/et_pb_column][/et_pb_row][/et_pb_section]