[et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″ custom_padding=”||2px|||”][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″]

Running Mutually Exclusive Optimization Activities

[/et_pb_text][et_pb_image src=”https://33sticks.com/wp-content/uploads/2020/10/denys-nevozhai-7nrsVjvALnA-unsplash-1024×768.jpg” title_text=”denys-nevozhai-7nrsVjvALnA-unsplash” _builder_version=”4.4.7″][/et_pb_image][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″ custom_margin=”-202px|auto||auto|false|false” custom_padding=”69px|||||”][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″ min_height=”1716px” custom_padding=”||0px|||” hover_enabled=”0″]

In my last blog post, I discussed how to set yourself up for success and properly plan and prioritize your A/B testing backlog. Now, we’ll take a look at an element of test planning which is critical to running a high velocity A/B testing program – managing potential test collisions. You and your team put an incredible amount of time and energy into getting A/B tests live on your site. Let’s talk about how to get the cleanest read possible on your test results data.

The risk of not running mutually exclusive tests is in pushing test winners live to the site on the basis of misleading data. Let’s say you run a test and change the call-to-action on your product page, and the test variant is a winner, so you push it live to the site. Turns out, at the same time you also ran a test on your category page, presenting huge sale items to your loyal visitors. How did the sales item category test influence the results of your CTA test? Did the sales items cause the CTA test to be a winner, or was it the CTA change itself?

First, it’s important to understand that many testing programs simply push all of their A/B tests to 100% of site traffic. In other words, any visitor that comes to the site is qualified to enter the test, regardless of which page they land on or which site they come from. Sometimes, this doesn’t pose an issue, if you are only running one test at a time on your site.  However, if you’re trying to up your game and drive more testing wins, that approach just won’t work.  Here’s the problem with the approach when running multiple tests.

Let’s say you have 2 test ideas that you need to run at the same time, based upon business needs and marketing schedules. The first test is placed on your Product Detail Page, and changes the location of our primary call-to-action button. The second test is placed on your Cart page, and reformats your Promo Code input box.

So you run each test live on the site, and now you are pulling results to review with your business partners in an upcoming A/B testing governance meeting. You are super excited to see that the Cart test has produced a winning test variation! The reformatted promo box has driven a 15% increase in revenue per visitor, as compared to the control treatment. The PDP test variation, the new CTA placement, also had a lift in revenue per visitor for the test treatment, but the lift only reached 75% confidence.

During your readout of the results, a key stakeholder asks a seemingly simple question – “What impact, if any, did the PDP test have on the results of the Cart test?” You pause for a moment, and realize that 100% of your site traffic experienced both the PDP and Cart tests. You reply that you will need to do a deeper dive in your analytics tool to determine that impact, it may take a few days to figure it out. Your stakeholder is disappointed, as she needed the answer that day for an upcoming Board meeting.

Here’s a test planning and architecture method you can employ fairly easily to ensure that never happens again. This method is specific to Adobe Target, but other testing tools have can be used with a similar approach. You will create a script in Adobe Target which will allow you to run mutually exclusive tests on your site. So, if a visitor qualifies for one test, you can exclude that visitor from qualifying from any other tests. In the example above, you will ensure that any visitor that qualified for the PDP test, would not be included in your Cart Promo Box test.

Here’s an example of what the Profile script would look like in Target. In this example, a user would be assigned a random number between 0 and 99, and would then either fall into GroupA or GroupB.  When you create the Activity in Target, you would then select GroupA audience for the PDP test, and GroupB audience for the Cart test.

.                   

Running mutually exclusive tests on your site will allow you to get the cleanest read on your test results. Good luck implementing this within your testing process!

[/et_pb_text][et_pb_code _builder_version=”4.4.7″ custom_margin=”-123px|||||”][author] Jason Boal [author_image timthumb=’on’]https://33sticks.com/wp-content/uploads/2020/02/Profile-Pic-33-Sticks-2-150×150.jpg[/author_image] [author_info]Jason has over 10 years of experience working on both the client and agency sides, and across Retail, Financial Services, and Non-Profit industries. He always looks forward to helping clients build upon and improve the customer digital experience. He follows a data-driven strategy, focused on constantly learning more about the customer digital experience from both quantitative and qualitative information. Jason’s philosophy is, “taking data and developing a strategy centered around people, process, and technology will lead to tremendous results.” [/author_info] [/author][/et_pb_code][/et_pb_column][/et_pb_row][/et_pb_section]

[et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″]

Prioritizing ideas based upon potential impact and test run length

[/et_pb_text][et_pb_image src=”https://33sticks.com/wp-content/uploads/2020/09/image-2-1.jpg” title_text=”image (2)” _builder_version=”4.4.7″][/et_pb_image][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″ custom_margin=”-200px||||false|false” custom_padding=”69px|||||”][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_text _builder_version=”4.4.7″ min_height=”2327px”]

“Sound strategy starts with having the right goal.” Michael Porter

Whether you are brand new to A/B testing, or a seasoned professional, there are always ways to step up your game! Here is an effective tactic to help you kick-start your optimization program, or kick it into the next gear.

So, you have a list of 50 test ideas gathered from across your company, and have compiled those ideas into your optimization backlog. What next? How do you pick which ideas to test first? You could have key team members vote on those ideas. You could sort the ideas by level of effort and traffic. You could listen to your HIPPO’s opinion. Or, you could step up your game and put some more strategic thought and planning into this. The reason you are testing in the first place is to move the needle on your KPI. So, you want to test ideas that move the needle the most. But which ideas are those?

Let’s say you have a test idea to simply change the title of your call to action button on your Product Detail Page from “ADD TO CART” to “BUY NOW.” A simple, yet potentially high impact, test idea. If you voted for the test idea, one team member could shoot it down and say “No way, this test will never move the needle on site Revenue.” If you used LOE and traffic, you would come up with a different answer because it would be a low level of effort and high-volume page, making it a no-brainer. Two different answers to the same test idea.

Let’s help put some objectiveness and data behind your choice instead, and remove the guess work from your process.  The first thing to decide is:

Who is my intended audience for this test?

Sounds like a simple question, but requires a few levels of digging. Is it all visitors to your site that are put into the test experience, regardless if they see the test variation or not? Let’s say that is 10,000 daily visitors. That sounds somewhat arbitrary. Is it visitors to your site that reach the actual page your test is on, in this case the Product Detail Page? So they may or may not have seen the test variation. Let’s say that is 25,000 daily visitors. Sounds like we’re getting close the intended audience. How about all visitors that reach the Product Detail Page? This is getting very close now. Let’s say that is 15,000 daily visitors. One step further and we could say all visitors that reach the Product Detail Page AND click on the BUY NOW button (See Figure 1). Perfecto! Let’s say this is 8,000 daily visitors.

Now we need to decide:

What do you do with your new audience?

Go into your Analytics tool, and create a segment for these visitors. Then, let’s see how well this particular audience converts on the site. It turns out, visitors who visit a Product Detail Page AND click BUY NOW show a 4.5% conversion rate in your analytics tool. This is compared to an overall site conversion rate of 3.1%.  You now know (1) potential size in daily visits and the (2) conversion rate for all of your test ideas. This is a great way to help your prioritize which tests to run on the site (along with Level of Effort). You now have your specific audience that you want to show the test experience to, and you also know how that audience converts on your site.

Now what?

Let’s estimate the lift we expect the test variation to produce?

From your past testing experience and from talking with the product and business owners in your company, you predict that your test experience will drive a 5% increase in conversions, as compared to visitors that are not in the test experience.

Finally,

Estimate how long your test should run

We can use the (1) 8,000 daily visitors, (2) 4.5% conversion rate for this particular audience and (3) the 5% estimated lift you expect to see to determine approximately how long you should run your test on this site. There are many tools out there to help you estimate test run length. Adobe has a good one HERE. After plugging in the numbers, your test will take approximately 5 weeks to reach a 95% confidence level (Figure 2). You can now add this to your prioritization and test planning as well, so that you can effectively manage collisions on your site while testing…which will be the topic of my next blog.

Now take this approach to all of the ideas on your backlog, add in Level of Effort (Dev and Creative), rank order the list, and start testing!!

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.7″][et_pb_row _builder_version=”4.4.7″][et_pb_column type=”4_4″ _builder_version=”4.4.7″][et_pb_code _builder_version=”4.4.7″][author] Jason Boal [author_image timthumb=’on’]https://33sticks.com/wp-content/uploads/2020/02/Profile-Pic-33-Sticks-2-150×150.jpg[/author_image] [author_info]Jason has over 10 years of experience working on both the client and agency sides, and across Retail, Financial Services, and Non-Profit industries. He always looks forward to helping clients build upon and improve the customer digital experience. He follows a data-driven strategy, focused on constantly learning more about the customer digital experience from both quantitative and qualitative information. Jason’s philosophy is, “taking data and developing a strategy centered around people, process, and technology will lead to tremendous results.” [/author_info] [/author][/et_pb_code][/et_pb_column][/et_pb_row][/et_pb_section]

“By failing to prepare, you are preparing to fail.” ― Benjamin Franklin

A few years ago, I was approached by a hotel chain to help figure out why their booking rate was lower than the industry average. It was a unique opportunity to analyze a booking funnel and provide insights that would directly inform them with data-driven insights that would be inputs into UX and creative changes planned for later in their fiscal year.

1. Stakeholder Interview Preparation
I started the analysis by taking out a pen and my notebook and writing down all my questions about the business. I put myself in the position of the CEO of the entire hotel chain and thought what questions they might have. I thought about what the marketing team was doing, why hadn’t this been prioritized earlier, what demand generation team would have questions about, and every other permeation of team or output that might be relevant to my analysis.

Instead of jumping straight into the analytics tool and pulling reports on data, I put pen to paper and did my due diligence in prepping. I forced myself to not even think about data and just think in terms of business questions. I wanted to try to anticipate all of the needs of the team and nuances BEFORE I interviewed the team.

The next thing I did was set up stakeholder interviews with pertinent members of the team. CMO, Marketing Directors, Channel Managers and more.

2. Stakeholder Interviews
Each interview consists of a lot of back and forth to understand the individual’s role or department needs to learn as a result of your analysis. I have a number of questions I typically use throughout the interviews that help me get this insight. It should go without saying that not everyone gets the same questions and a skilled analyst will know how to effectively pivot throughout this part of the process and follow-up or dig deeper on any response that yields interesting takeaways.

• What is important to you?
• How does your job function interact with the booking funnel?
• What specific questions do you have about the booking funnel?
• Why they haven’t been able to do this analysis in the past or why it was important to do it now?
• What customer segments are important to you?
• Are there any known challenges with a particular advertising channel?
• Are their known issues with any of the data?
• What biases exist within the company, department, or team about the booking funnel?

I finished all of the interviews with the same question “If you could describe the perfect output from this analysis, what would that look like?” In essence, they’re now telling me exactly what they expect from the analysis – not the results themselves as nobody knows what those will be at this point – but what types of results they need to do their job.

Throughout that process, I changed my language from ‘you’ or ‘yours’ to ‘me’ and ‘we.’ I became a part of their team for this analysis by fully immersing myself into their culture, their language, and evaluated the teams’ needs.

You might be thinking – “Wait, hold on, Bryant…didn’t they tell you the objective before you started…‘Why is our booking funnel underperforming?’ Why not just get the conversion rate and start running reports to find the answer?” The simple answer is “Yes, they did.” They did tell me they wanted to know why their site wasn’t performing to industry standards.

They also wanted the best answer possible, they hired ME to do this for them. They didn’t tell me to get them a conversion rate and do basic analysis. They were counting on me and my team to provide them with insights they wouldn’t be able to get any other way. In my own prep for the interviews, I had a good sense of what they needed and in fact, a lot of their responses were in line with what I had anticipated. But in doing those interviews I garnered insight into their culture, their priorities, and internal language on how the team spoke. I also built rapport with each stakeholder and member of the team I would be working with.

All of that were filed away in my notes to use later in my readout to them when the analysis was complete.

What if you’re not a consultant and you’re already ‘in tune’ with the culture, organization and business challenges? Should you just skip this step? Absolutely NOT. By interviewing members of the team you work with regularly and others you might not – you’ll hear nuances in the requirements that focus your thinking and efforts into a single analysis. You’ll potentially hear from your boss a requirement that wasn’t detailed in the original request. You might hear from the VP a particular way he articulates ‘conversion’ that you wouldn’t have considered – or a way they evaluate customer ‘engagement’ that’s different than how you usually report.

The point of all of the above is – DON’T ASSUME ANYTHING – except that you don’t know everything. Put in the time and effort to fully evaluate this analysis BEFORE doing anything else.

3. Document Everything
During your interviews you should be taking copious notes throughout. Some of my greatest questions and insights come while someone else is talking and it triggers a thought. There was a time I was interviewing a retailer about a specific analysis they wanted on User Journeys from Email to their landing page for a specific promotion sign up. The VP told me about how it was a new promotion they’re trying to extend the black Friday shopping day. This particular promotion was new in that they were trying a new email vendor in addition to a new designer on the team he had personally recommended for hire.

In that moment – I picked up two details about this analysis I needed to be cognizant of. First – this was a new email vendor and there’s potentially some data issues I’ll run into and need to be extremely careful when analyzing that data. Second – there’s personal bias in seeing this succeed based on the designer that came up with the concept of the landing page. All of this was something I made specific notes about in my notebook to review later.

Organize all of your interviews by day, time, stakeholder, and topic how it relates to your deep dive analysis. Block 15-30 minutes after the interview to collect your thoughts and organize your take-aways from the interview. By not going directly into the next meeting or task, you’re giving your brain a chance to fully digest the information you received.

What works really well for me is to take notes by hand during the interview and type them up afterwards – I find that I’m able to focus more on the overall concept of what is being discussed in person and then put it in words that make sense to me afterwards. If I’m typing during a meeting or scheduled interview, I find I’m trying to capture EVERY WORD instead of listening to what is being talked about. This might not be the case for you – or you might try switching it up a little bit as your listening and interpretation skills are paramount when you’re in this first step of Evaluate.

Finally, take a glance at all of the responses you received regarding the question “If you could describe the perfect output from this analysis, what would that look like?” Read each of them and write out for yourself the ‘perfect analysis’ and what it would look like and entail.

Once you’ve done this – you’re ready for the next stage – IDENTIFY.

Pro Tips and Takeaways:

1. Document all business questions from each stakeholder interview
2. Prioritize business questions based on importance and value to the analysis
3. Describe what a ‘perfect analysis’ would look like using the details from business questions and interviews

A few days ago, I found myself at an intersection, both literally and figuratively. I saw a young man in a motorized wheelchair halfway out into traffic, his wheelchair had stopped working. I sat there thinking, “I really should do something. He needs help. I really should do something. It’s time for me to do something.” For some reason,I hesitated, for some reason I was worried what people would think about me so i hesitated. Thankfully another motorist hopped out of his car and helped get the young man safely across the road.

The last several weeks following the murder of George Floyd, I’ve been thinking deeply about the disturbing amount of racism that is not only rampant throughout the world but in my own communities. I had a lot of good thoughts, wished for a lot of good things to happen, yet as with the young man stranded in traffic, I hesitated, I didn’t look for a solution, I figured someone else would hop out of their car to carry my part of the load.

Recalling the moment at that intersection still pains me. Not taking more decisive action to fight against racism pains me. Hesitating to do good pains me and I’m ready for a change.

So I thought deeper. Why do I hesitate to do what I know is right? Why am I so fearful of taking action?

Is it because deep down I crave social acceptance and I fear losing my “fans?” Do I fear driving a deeper wedge between myself and family members that don’t see eye-to-eye with me on racial issues? Am I worried that I would offend people in my digital analytics circles and they wouldn’t want to work with me?

The answer was that it is ‘ALL OF THE ABOVE.’

But no more.

As I sat at this intersection of my life, as if meant to be, a message from Eric Peterson, Founder and CEO of Analytics Demystified, hit my inbox. He had been thinking deeply as well and he was ready for action. He said he was going to start raising funds for a worthy cause, Black Girls CODE, and wanted to know if I would join him.

Learn more about Black Girls CODE here: https://www.blackgirlscode.com/

YES! No hesitation this time.

I’m ready to stand up, I’m ready to replace hesitation with action. I’m ready to do what I can with my time, talent, and resources to help those that the system was designed to keep down. I’m ready to be an anti-racist!

Here is what we are doing:
My first step will be helping to provide a greater opportunity for some very bright black girls who want to become coders but face a more difficult hill to climb than others due to hundreds of years of racist public policy that was designed to hold them back. So, on behalf of myself personally and 33 Sticks, I am donating to Black Girls CODE, a 501(c)(3) non-profit that is working to increase the number of women of color ages 7 to 17 who are innovators in STEM, leaders in their communities, and innovators in technology. 

And so here we are, ready to take action and Eric and I would like your help.

Here is how you can help:
We are asking each of you reading this to join us in donating to Black Girls CODE and to rally for others to do the same.

To encourage your donations, Eric and I will match dollar-for-dollar up to $20,000 USD in donations over the next 14 days.

Our goal is to work with you to raise $40,000 USD for Black Girls CODE to help bring more diverse voices into technology and to create opportunities today that have the potential to positively impact the world in the future.

To help us and to take advantage of our $20,000 matching efforts is simple:

1. Decide how much you can contribute, knowing that Eric and I are matching you dollar for dollar
   
   
2. Go to www.blackgirlscode.com and make your donation
   
   
3. Once you make your donation, either forward us your donation confirmation email (which is also your tax donation receipt), or simply take a screenshot photo of the receipt or the ‘thank you’ page and forward to blm@33sticks.com. But make sure the amount is showing so that we can match it!
   
   
4. Track our collective progress online at https://tinyurl.com/demystified-33sticks

No donation is too small! If you can give $5 it’s like giving $10! If you can give $50 it’s like giving $100!! Eric and I are confident that if we are able to rally our communities to raise $40,000 for Black Girls CODE that together we can have a positive and meaningful impact on their efforts to make our little corner of the world a more diverse, a more inclusive, and an overall better place.

It’s time for us to stand up, it’s time for us to fight for racial equality, it’s time for us to move from good thoughts to good actions. It’s not fair to wait for someone else to jump out of their car to help someone who needs our love and support. Now is our time!

Please join me. Thank you.

-jason

[author] Jason Thompson [author_image timthumb=’on’]http://i2.wp.com/33sticks.com/wp-content/uploads/2015/02/jason_250x250.jpg?zoom=2&w=1080[/author_image] [author_info]Jason is the co-founder and CEO of 33 Sticks. In addition to being an amateur chef and bass player, he can also eat large amounts of sushi. As an analytics and optimization expert, he brings over 15 years of data experience, going back to being part of the original team at Omniture. [/author_info] [/author]

A lot of information exists in our industry about telling stories, building presentations or deploying analytics. There is less information (that I’ve found) that teaches someone to think like an analyst or how to derive insights from data. I believe it’s less available because it’s not exactly simple to teach. In many cases, people either have an analytical mind and can see things in data naturally – or they don’t. Most people gravitate towards things they’re a little naturally gifted in. That said, I still find the challenge of teaching how to derive insights from data a worthwhile endeavor and that’s the focus of this blog series – Analyze Like a Pro.

I started my digital marketing career in 2003. Yahoo was king. AOL was relevant. Netscape still existed. Google was not yet a publicly traded company and there was still a competitive market for search engines. Amazon was not yet profitable. eBay was all the rage and ecommerce was just starting to blossom. This is the state of the industry when I started my career.

I started in small business consulting for web design, entrepreneurship and digital marketing. I have owned and sold a couple of small businesses including a website where I sold children’s bedroom décor and bedding. I have had multiple failed businesses where I had online stores for Swiss watches, truck tents, and polo shirts on eBay. I learned how to optimize websites for search, pay for advertising through Overture’s revolutionary ‘pay per click’ model for advertising in search.

About 5 years into my career I made a decision to pivot from small business consulting to web analytics and data analysis. Instead of financially challenged business owners looking to squeeze just a few more sales each month on a small e-commerce website or push a few more orders through their eBay store…I was working with brands like Samsung, Microsoft, Electronic Arts and Hewlett Packard. I saw how dramatically money was spent in advertising simply to ‘get eyes’ on a website and marketing budgets that were previously incomprehensible to me. My world was changing, and it was the beginning of the global shift of media dollars from traditional advertising (TV, Print, Radio) to digital formats.

I was assigned a new account and we were to analyze ‘website performance’ for a major retailer. I recall vividly being on a conference call at 6pm with one of my clients. We planned a comprehensive review of their entire website. We analyzed their home page effectiveness, landing pages, product pages, internal search, advertising reach and conversion, and many other dimensions of their website. We had spent weeks on this analysis, had a presentation with 50 or so slides in it and it was beautiful. We presented our material and walked them through the data and insights we had found. We spent nearly two hours on that call, answered questions and ultimately gave them something they had never received before.

I thoroughly enjoyed the work and enjoyed it even more that the insights I was able to glean from their data provided them with answers to questions that had gone unanswered for years in their business. They could now make effective business decisions based on facts – instead of intuition – and it was a result of effective analysis and insights generation.

It’s been quite a journey since I started in this field and I’ve learned a lot. I’ve succeeded a few times here and there, I fail a lot, and I try to keep learning. I’ve had the pleasure and experience to work with Tech, Retail, Hospitality, Healthcare and Financial industries. Throughout this journey, there have been opportunities I’ve had to lead others, formally and informally as they’ve started or transitioned their career into this field. In training others, I’ve found a structure to train new analysts or retrain seasoned analysts into a framework that consistently yields insights from data analysis. I want to share the key components of this framework throughout this next series of blog posts. I can assure you with 100% certainty that if you apply the steps in this framework, you WILL find insights in your data.

1. Evaluate
2. Identify
3. Validate
4. Mine
5. Synthesize
6. Action

>>NEXT POST: Step 1 Evaluate

I hope you learn something new or it reinforces some work you’re doing. Either way – I’d like to know how it’s impacting you, if you have questions or need some guidance – so drop me a line on LinkedIn or email me at Bryant.hoopes@33sticks.com.