Saturday, January 5, 2013

Data versus Intuition

Interesting article in the New York Times recently, about how businesses make decisions.

"Sure, Big Data Is Great. But So Is Intuition." by Steve Lohr

I see this all the time, the excessive reliance on "data."  Gather all the data and then the questions answer themselves.  But people don't realize that all this "data" is really just a set of estimates and models.  It's not exact.  Furthermore, a lot of the time even with a lot of data, the decisions are close calls.  Point is, we can never get away from making human judgment calls.  There are those who think we don't have to do that, those who don't really understand the assumptions made in gathering the data.  I disagree.

The final paragraphs in the article provide a good summary:

"Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?"

"At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.  'I don’t worship the machine,' she said."