Why Sifting Big Data Requires Humans

Top 11 Highly Requested IT Skills

My colleague Loraine Lawson recently wrote about the importance of being an “explorer” with Big Data  rather than a “scientist.” That was the same day I wrote about the importance of curiosity.

I’m reminded of an interview I did a while back with Jack Phillips, CEO of the research firm International Institute for Analytics, in which I asked him whether more sophisticated technology would make human analysts less important. He countered that humans are more important than ever — it’s all in knowing which questions to ask, he said.

In Loraine’s post, Jill Dyché, the vice president of Thought Leadership at SAS, made the point that sometimes you don’t know the right questions, but can make big discoveries through what she calls “low-hypothesis exploration.”

Columbia history professor Matthew Jones, who is studying the history of data mining, put it this way in a blog post:

“Data science depends utterly on algorithms but does not reduce to those algorithms. The use of those algorithms rests fundamentally on what sociologists of science call ‘tacit knowledge’—practical knowledge not easily reducible to articulated rules—or perhaps impossible to reduce to rules. Using algorithms well is fundamentally a very human endeavor— something not particularly algorithmic.”

A Forbes article says that requires two things: expertise and experience in a specific domain. It says that domain would be deep understanding of the tools they use, though it would seem that could also mean deep understanding of the business and its markets.

At the Silicon Valley Comes to Oxford annual event, Stephen Sorkin, vice president of engineering for the Big Data analytics company Splunk, took on the idea that nuggets of pure gold would be gleaned from oceans of data, reports The Wall Street Journal. It quotes Sorking, saying:

“It is not going to happen magically. The software only finds correlations, not causations. In order to find causal relationships you have to do work. If you take any sufficiently large data sets, you are going to find correlations. You need a human in the loop to work out which are important.”