Twyman’s law

Hi MD,

Twyman's law states that any data or figure that looks interesting or different is usually wrong.

Sounds unbelievable, isn't it?

But, it's true. I saw this in action recently and wanted to share that story with you.

In June, we ran a test on our homepage and while I was looking at conversion rate by segments, I noticed that users from Windows had a 400% higher signup rate for VWO free trial as compared to users using Mac OS X.

Now, that's baffling and our team spent a good deal of time trying to understand why was that happening. Someone in marketing hypothesized that perhaps Mac OS X users have a better design aesthetic and our homepage wasn't appealing to them. Was it true?

When we dug into data, we realized that our recently installed automated QA service creates signups on the homepage every hour or so (to ensure the form doesn't stop working) and guess what, that automated service used Windows.

After removing such QA signups from data, Mac OS X and Windows conversion rate became comparable.

This is a perfect example of Twyman's law. Remember, if the data is too good to be true, it's probably wrong.

Many extreme results are more likely to be the result of an error in instrumentation (e.g., logging), loss of data (or duplication of data), or a computational error.

Hope this mental model was new to you (it certainly was to me!).

If you also have a story related to exaggerated data to share, hit reply, and send me an email. I read and reply to all emails :)

Cheers
Paras Chopra

Founder & Chairman, VWO

PS: I'm planning to send one such idea/story every month to our list. (Aren't we all sick of faceless traditional marketing newsletters? VWO's newsletter is going to be personal going forward.)

PPS: If you want to learn more about Twyman's law, Ronny from Bing's experimentation team spoke about it in his talk.



VWO (A Wingify product) 14th Floor, KLJ Tower, Netaji Subhash Place Delhi 110034 India
This email was sent to bablu.sahil786.rockmoney@blogger.com. Update your email preferences to choose the types of emails you receive.

Comments