Skip to main content

No, that won't work

Hi MD,

Last week I interviewed Ian Harris, a professor of surgery, for my podcast. His research has found that over half of all surgeries are ineffective.

Yes, you heard that right: the majority of surgeries don't work (as compared to a placebo). The list of what doesn't work includes commonly done surgeries such as those done for knee pain, back pain, hip replacement, and so on. (If you don't believe me, I highly recommend watching this talk by the professor.)

Surgery: the Ultimate   Placebo


This claim indeed sounds shocking, but recall that the most common surgical procedure in human history was bloodletting. In this practice that lasted for thousands of years and was common until the 1850s, the doctor would make a cut in the body and simply let out the blood of the patient.

Everyone thought this should work (partly because everyone did it) and many different variations of bloodletting were iterated and compared with each other. But, amazingly, a question that wasn't asked until the 19th century was whether performing bloodletting works better than not doing it at all? Once this question was asked and a randomized control study was done, it was quickly understood that bloodletting doesn't work (but may actually actively harm the patient).

Let me reiterate… bloodletting was performed for over 2000 years before it was discovered that it doesn't work.

This makes you wonder - what else that we do commonly doesn't actually work?

Actually, the sad part is that we don't know what doesn't work because, apart from medicine and science, we don't perform enough randomized control trials (which are nothing but A/B tests in the marketing parlance). Even in the field of surgery, a systematic review of ~10,000 surgical procedures discovered that an A/B test was conducted for only 1% of such procedures. The remaining 99% of procedures were simply assumed to work.

In our day to day work as marketers, product managers, or leaders, we take so many decisions that feel right. Surely, that "best practice" change should work. Or this new campaign makes sense, so it should be run. But does it really work? What evidence do you have that if you hadn't done anything, that metric wouldn't have gone up?

Like the human body, businesses and markets are complex systems. There are thousands of factors that influence whether a particular action will have the desired impact. If the suggested action is our idea, we're biased to believe that it works.

However, learning from the ineffectiveness of bloodletting and many such surgeries that are still performed, the right attitude should be that best practices and genius ideas may simply not work (or, worse, may actively harm).

Unless a properly set up randomized control trial (or an A/B test) gives clear evidence of harm or benefit of a particular action, it would pay to believe that it probably doesn't have any impact. The default orientation towards a new idea should be: no, it doesn't work (unless you can prove that it works).

In other words, yes you should really be planning to run 25,000 A/B tests in 2021.

Do you have a story of an A/B test that debunked a widely held belief? I'd love to know about it. I read and reply to all emails :)

Regards,
Paras Chopra
Founder and Chairman, VWO

@paraschopra on Twitter

PS: These letters are getting cross-posted on our website too. Read my past letters here.



VWO (A Wingify product) 1104, 11th Floor, KLJ Tower North B-5, Netaji Subhash Place, Pitampura, Delhi – 110034, India
This email was sent to bablu.sahil786.rockmoney@blogger.com. Update your email preferences to choose the types of emails you receive.

Comments

Popular posts from this blog

Umbrella type Fish Trap

Jiophone Next Launched - Jio and Google lunch new Jio phone

  The world's cheapest Android smartphone is going to be the Jio Phone Next Lunch.It will be lunched before Diwali, but before that the company has shared its video. The video explains how to make the phone. This phone is being made in India only.Jio phone is being made in India is made for India. Made by India.Let me tell you what is special in this phone. This phone will get Pargati OS which will be Android powered.This is the best operating system, it has been specially made for India. Both Jio and Google have jointly prepared the phone. The phone's OS is named Pragati because the phone wants to bring progress in the country, the country will be the first Android phone to get it at a low price. Users will get a better experience even after the low price. The processor of Jio Phone Next has been prepared by qualmqoure. This smartphone will give a new opportunity to every citizen of the country to connect with digital.This phone is going to change the lives of lakhs of recrui

Vivo V23 5G and Vivo V23 Pro 5G with 50MP Dual Selfie Camera launched in India

Print  This phone is available for purchase on the company's official website, Flipkart and offline retailer stores. The pre-booking of Vivo V23 5G and Vivo V23 Pro 5G phones will start from January 5. While the base variant will be available for purchase from January 19 and the Pro variant.  Vivo V23 5G and Vivo V23 Pro 5G smartphones have been launched in India. Both these smartphones come with Fluorite AG glass bank panel, which has been claimed that this back panel changes color as soon as UV rays fall in the sunlight. The Vivo V23 55G and Vivo V23 Pro 55G smartphones are powered by MediaTek Dinance 920 and MediaTek 1200 processors, coupled with 12GB of RAM. These two Vivo phones come with 5G connectivity. In these, full-HD Plus AMOLED display and dual selfie camera have been given. The primary camera is of 50 megapixels. Vivo V23 5G, Vivo V23 Pro 5G price in India, availability The price of 8 GB RAM 128 GB storage variant of Vivo V23 5G phone is Rs 29,990 in India. Apart fro