Tag Archives: statisticians

Why I think p-value is less important (for business decisions)

This article only stands for my own opinion.

When the Internet companies start to embrace experimentation (e.g. A/B Testing), they are told that this is the scientific way to make a judgement and help business decisions. Then statisticians teach software engineers the basic t-test and p-values etc. I am not sure how many developers really understand those complicated statistical theories, but they use it as way -- as a practical method to verify if their product works.

What do they look at? Well, p-value. They have no time to think about what "Average Treatment Effect" is or what the hell "Standard error and Standard deviations" are. As long as they know p-value helps, that's enough. So do many applied researchers who follow this criteria in their specialized fields.

In the past years there are always professionals claiming that p-values is not interpreted in the way that people should have done; they should look at bla bla bla... but the problem is that there is no better substitute to p-value. Yes you should not rely on a single number, but will two numbers be much better? or three? or four? There must be someone making a binary Yes/No decision.

So why do I think that p-value is not as important as it is today for business decisions as what I wrote in the title? My concern is that how accurate you have to be. Data and statistical analysis will never give you a 100% answer that what should be done; any statistical analysis has some kind of flaws (metrics of interest; data cleaning; modelling, etc.) -- nothing is perfect. Therefore, yes p-value is informative, but more like a directional thing. People make decisions anyway without data, and there is no guarantee that this scientific approach provides you the optimal path of business development. Many great experts have enough experience to put less weight on a single experiment outcome. This is a long run game, not one time.

Plus the opportunity cost -- experiment is not free lunch. You lose the chance to improve user's experience in the control group; you invest in tracking all the data and conduct the analysis; it takes time to design and run a proper test, etc. If it is a fast-developing environment, experiments could barely tell anything about the future. Their predictive power is very limited. If you are in a heavy competition, go fast is more important than waiting. Therefore instead of making adaptive decisions based on each single experiment, just go and try everything. The first decision you should make is how much you want to invest in testing and waiting; if the cost of failure product is not high, then just go for it.

I love these cartoons from the book "How Google Works". You have to be innovative in the Internet industry because you are creating a new world that no body could imagine before.

Screen Shot 2015-02-06 at 12.48.06 AM Screen Shot 2015-02-06 at 12.48.13 AM Screen Shot 2015-02-06 at 12.48.45 AM Screen Shot 2015-02-06 at 12.49.05 AM

to be an eBay seller...and explore the social network effect

It is interesting to work for eBay; however, most of the time I am looking at the massive data without really thinking about how they are generated... for instance, I have no idea how difficult it should be to be a successful seller on eBay. It somehow sounds a little wired that we talk about mechanism design (say, whether eBay's ecological system is better than Taobao's) without personal experience.

To step in, I should try - and there was a precious opportunity for me to do so! After the China-R conference in Beijing, two professors from Australia (Graham Williams and John Maindonald) asked if they could buy additional packs of the playing cards somewhere so they could share with their colleagues and friends. Then why not on eBay?

Then I typed "ebay.com" in my browser... log in...sell...The photos were prepared several months before... upload... done!

However, when I was trying to copy the link...it was so long!

http://www.ebay.com/itm/R-Language-Playing-Cards-Postcard-Suit-Best-Gift-Statisticians-/321134725623

Then I noticed the lovely "share to twitter" button on the left ^O^ Let me just use this one!

A simple click leads to a new tweet... of course I need to at some useR... Yihui should be the best choice 😛

Not sure what has happened during the night... when I wake up next morning, they are almost sold out (I listed 10!). Well... then the painful shipping process comes. Thanks to the fantastic e-packet tool(http://shippingtool.ebay.cn), it was pretty convenient to complete shipping online. OK, the postman will take care of them. Goodbye, my cards...

The special benefit is that now I realize how powerful the social network could be. Without spreading information through twitter, it would be much slower if we manually copy and paste the URL in emails. Social network sites control it just in the right way - "word of month" helps customer selects goods based on their reliable information source. Some comments:

Karthik Ram@_inundata 30 May

Totally bizarre #rstats playing cards for sale on @ebay. http://bit.ly/18BmwTg 

Geoff J@geoffjentry 30 May

@_inundata I once bought a "used wife" on @ebay yet somehow this is odder.

Ok...is it that wired?

BTW, the integration of eBay site and twitter is pretty good. Once the link is attached the listing image also appears. See below.2013-06-04 14_11_43-Karthik Ram (_inundata) on Twitter