Skip to main content

How to run 25,000 A/B tests in 2021

Hi MD,

There's a company that has grown twice as fast as S&P 500. Beating the market consistently for more than 10 years is evidence that what they're doing isn't a fluke.

NL_12_2020_Image_1

This company - whose stock price is shown as the blue curve in the graph above - must be having a secret sauce for growth.

What is it?

Well, the company in question is Booking.com, and they are known for their culture of experimentation. They collectively run more than 25,000 A/B tests annually - which means they're launching ~70 tests every single day. Sounds crazy, but according to their leadership, this A/B testing craziness is one of the critical ingredients in their growth.

To see how this culture of experimentation has played a role in their growth, look at their hotel listings page.

NL_12_2020_Image_2

Everything highlighted on the listing page is there either because an A/B test found it made an impact on their business or is currently getting A/B tested. More importantly, everything that you don't find on the page is not there because an A/B test proved that it actually hurt their business or made no difference.

To learn more about how Booking.com scaled its testing program to 25,000 tests a year, you can check these two resources: So, how can you too run 25,000 tests next year?

Well, I understand that not every website has traffic levels of Booking.com. But if not 25,000 A/B tests a year, a site with one-hundredth of Booking.com's traffic should at least be running one-hundredth the tests - say 250 tests a year, which is almost 20 tests a month.

Do you run 20 A/B tests a month? What about 10 per month?

Unless you're an agency or an outlier company, my guess is that the answer would be no.

What I see across many businesses is running occasional A/B tests and if they have a program, they'd be perhaps launching 2-3 A/B tests a month. That's about it. Finding exceptional cultures where experimentation is taken seriously is a rarity. But why?

I think the answer boils down to whether there's a commitment from the leadership. Experimentation is hard. Coming up with ideas, setting up tests, investing in infrastructure, getting variants designed and coded, performing quality checks, and interpreting results take real effort. Combine this with the emotional cost of most experiments not working out, no wonder so many organizations just give up on it too soon.

But giving up on experimentation is giving up on a high-leverage activity that can trigger business growth. This is because an increased conversion rate is only the most direct benefit of experimentation. A full commitment to experimentation transforms the entire culture:
  • People in the org start trusting data more.
  • A higher percentage of decisions start getting questioned on whether they're someone's pet projects or actual business-needle moving.
  • You end up having fewer failed launches.
  • And, the best part, the organization gets exposed to deliberate serendipity. Even if an A/B test didn't win in the traditional sense, most A/B tests end up increasing the amount of insight the org has about customers and markets.
Leaders at Booking.com understand how easy it is to give up on experimentation and that is why they've done things like:
  • Enabling everyone in the org to launch an A/B test without gatekeepers.
  • Having processes in place that ensure that no changes are made to any page until they're A/B tested.
  • Make experimentation evangelizing team report into company's leadership.
A culture that prioritizes understanding customers better than competitors is the real benefit that experiments create. The actual conversion rate increase is simply a nice cherry on the top.

So, let me ask again: what would it take for your organization to launch more A/B tests in 2021 than it did in 2020? What's preventing your organization from adopting experimentation as a central part of your growth strategy?

Do reply to this letter as I read and respond to all emails :)

Regards,
Paras Chopra
Founder and Chairman, VWO

@paraschopra on Twitter

PS: These letters are getting cross-posted on our website too. Read my past letters here.



VWO (A Wingify product) 1104, 11th Floor, KLJ Tower North B-5, Netaji Subhash Place, Pitampura, Delhi – 110034, India
This email was sent to bablu.sahil786.rockmoney@blogger.com. Update your email preferences to choose the types of emails you receive.

Comments

Popular posts from this blog

ria money transfer- Online Money Transfer ria money

Riya Money Transfer is one of the largest money transfer companies in the world. With more than 317,000 locations in 146 countries around the world and an easy-to-use online dashboard, Rhea is convenient for both sender and recipient. Founded in New York in 1987, Rhea began with the simple goal of providing simple and reliable shipping services to people who ship home with an emphasis on friendly service. Over the years, Rhea has grown as a global company, expanding (and continuing to expand) to its worldwide location. In addition to remittance services, Riya Money Transfer offers bill pay, mobile recharge, prepaid debit cards, check cashing, money orders, and more. With a strong presence on almost every continent in the world, Riya is a brand that millions of people trust. Finally, as a company that places great value on the safety of its customers, you know that your money is always in good hands. The good thing about RIA Money Transfer $ 0 transfer fee when you use a bank account to

Youtube to Video Mp3 Mp4 Converter Download

Youtube Link to Video Mp3 Converter Download.  1. Youtube to mp3 Dowloader Converter.  You can download app here,  csc Best downloader app.  Youtube to mp3 converter This fast and easy to use converter allows you,  Download YouTube video mp3 for free wherever you are,  G0. Just one YouL ube link coPy, rewind and get ready to Listen to your mp3 file. So fast and easy! Instruction,  The main thing you need to do before getting ready,  The to-use mp3 has to be copied to a link first. Then go to our site 2conv.com and paste this link Special area in tne. Convert press to mp3. Your MP3 is ready to listen and enjoy! Don't forget to share this amazing ,opportunity Your friend. They will be very grateful to you! Enjoy Music with 2conv - Your Online YOu l ube to mp3 converter,  Are you tired of trying a fast and sociable to mp3, YouTube converter music that will allow you,  Download videos and favorite music tracks? And yours,  Tries to find an easy option to get your files,  Tre

SBI Clerk Exam Results

  SBI Clerk Results 2021 Out, Clerk prelims Marks out.  Table of Contents SBI Clerk Score Card 2021 Out SBI Clerk Prelims Score Card 2021 SBI Clerk Score Card & Marks 2021 SBI Clerk Score Card 2021 Out SBI Clerk Prelims Score Card 2021: State Bank of India has released the SBI Clerk Prelims Score Card 2021 along with the release of SBI Clerk Prelims Result on 21st September 2021. The candidates who have appeared in the prelims exam on 10th, 11th, 12th, 13th July and 29th August 2021 can check their SBI Clerk Score Card from the official website of SBI @sbi.co.in or directly from the below link mentioned in the article. SBI Clerk Prelims Score Card 2021 SBI Clerk Prelims Score Card 2021 Organisation     State Bank of India Exam Name     SBI Clerk 2021 Post Name     Junior Associates (Clerical Cadre) Vacancies     5454 SBI Clerk Exam Date     10th, 11th, 12th, 13th July and 29th August 2021 SBI Clerk Prelims Score Card     21st September 2021 SBI Clerk Prelims Result     21