Ironically, one of the most common barriers to companies adopting analytics isn’t the business case, but executives’ faith in their own memory.
Roei Ganzarski, CEO of the predictive analytics firm BoldIQ, told Information Week that he daily encounters leaders who reject the idea of using analytics over their own business intuition.
Ah, man versus the machine — it’s an old, old debate for literature, philosophy, and more recently, brain researchers. To be honest, the more I read, the more I think the gig is up for business intuition.
As least, that’s what the data says, according to Andrew McFee, co-director of the Initiative on the Digital Economy in the MIT Sloan School of Management. According to McFee, “…there have been a raftload of studies comparing the predictions of human experts vs. those of algorithms, and …in the great majority of them the algorithms have been at least as good as or significantly better than the humans,” McFee wrote in a Harvard Business Review blog post. “In a meta-analysis conducted by William Grove and colleagues of 136 research studies, for example, expert judgments were clearly better than their purely data-driven equivalents in only eight cases.”
In fact, with what we now know about the brain, the harder it gets to justify intuition over algorithms. From memory formation to mental fatigue brought on by too many choices, research tends to favor the idea that we are much more fallible than we would ever want to believe.
Marcelo Gleiser, a theoretical physicist, natural philosopher and professor at Dartmouth College, expressed this problem eloquently in a recent NPR post.
“We can thank the brain for tricking us into building a sense of the ‘real,’” Gleiser writes. “What we call reality is the result of our brain's very complex integration of external stimuli: sights, sounds, tastes, touch and smells. We perceive nothing in the actual present.”
To be fair, the same is somewhat true about analytics, despite all the hype about real-time analytics. The difference is that analytics is based on a collective sense of reality — not one person’s reality. Or as Barrett Thompson, GM of pricing excellence solutions for Zilliant, told Information Week, predictive analytics is “…the distilled wisdom and experience of five hundred salespeople who encountered tens of thousands, or hundreds of thousands, of unique selling circumstances.”
The pro-intuition crowd isn’t without a champion, however. Analytics expert Tom Davenport has written about the important role of intuition in Big Data projects.
I checked out Davenport’s full HBR post. It’s worth noting that his examples aren’t so much about decisions business leaders made, but rather about new ideas or theories they had that were confirmed or, in many cases, actually made possible by Big Data.
To me, Davenport’s examples are creative acts — which actually fall outside the domain of many executive decisions, if you think about it — even the strategic one.
As Information Week notes, what we’re talking about is significant, but daily, decisions that are largely based on guessing—not facts—even though you have access to the data.
- Which route is the most cost-effective for our delivery drivers?
- Where are we making overpayments or fraudulent charges?
Businesses often have this data, but struggle to analyze it. This is where executives play John Henry, stubbornly fighting the machine despite what’s best for business. But in such cases, Big Data analytics outplays intuition.