Lately I’ve seen several blog posts that I’ve mentally labeled as “Lectures on why Big Data can’t fix stupid.”
These posts are essentially cautionary tales about not expecting Big Data to solve your deep, organizational problems.
They vary in degrees; some are more or less in favor of investing in Big Data as long as you don’t expect too much, while others spend more time “raging” against Big Data investments as the cost of smaller, concrete steps troubled organizations can take to create useful change.
IDG posted one of the former type a few weeks ago, titled “The Big Data Fairy Tale.” It’s the tale of a data analytics project initiated by the NYPD under Mayor Giuliani, with help from CompStat. The idea was to map crime by geography while also charting officer performance by “quantifying criminal apprehensions.”
My cop-speak is a bit rusty, but I’m pretty sure that means they’re making crime reports and arrests.
“The key to success was not the data or analysis, but that the organizational management that used the data and analysis was effective,” the post notes. “Processes, structures and accountability were set up to drive the transformation.”
Oh, yeah, and you have to find the right people and get the wrong people out of the way. And here we come to the moral of the Fairy Tale:
In the same way that Giuliani fired one of the precinct commanders when he showed up drunk at the first CompStat meeting, big data systems require a complementary management philosophy to ensure whatever transformational insights are derived get implemented and controlled.
Just last week, the Harvard Business Review published a classic “rage against the Big Data in favor of small data” piece by Robert Plant (no, not that one), a Miami School of Business Administration professor. It’s well-summed up by the second paragraph:
So why do companies spend millions on big data and big-data-based market research while continuing to ignore the simple things that make customers happy? Why do they buy huge proprietary databases yet fail to use plain old scheduling software to tell you precisely when a technician is going to arrive?
“Big data is today's panacea, the great new hope for unlocking the mysteries of marketing,” Plant warns. “Companies would do better at satisfying and retaining customers if they spent less time worrying about big data and more time making good use of ‘small data’ — already-available information from simple technology solutions — to become more flexible, informative, and helpful.”
Maybe they don’t know how to use the small data. Dane Atkinson, the CEO of SumAll, says that’s a big part of the problem he encounters in his data work with small companies.
“Everyone we've interviewed who is striking out and building their own small business is sharp, aggressive, hungry, the best kind of people,” Atkinson said. “But they usually have very little data, very little math experience.”
That’s where SumAll comes in, of course — helping them break up the customer data and make sense of it, and he makes a pretty good sales pitch out in this Business Insider interview, if you’d like to hear it.
But what about those bigger companies that have the technology, that have the data analysis and still are getting the small data wrong? Can Big Data help them?
I’m going out on a limb here to say, “No.” Some business problems can’t be fixed by technology or even the best IT staff. That’s why so many techie people have t-shirts that say “ID-10T error” and PEBKAC shirts.
He’s not a professor or analyst, but I think comedian Ron White best sums up the real problem at some companies:
You can't fix stupid. There's not a pill you can take; there's not a class you can go to. Stupid is forever.
So, no, Big Data won’t fix what’s wrong at these companies. But then again, I suspect well-meaning posts, however spot on, won’t help, either.