You Can’t Fix Stupid and Other Arguments Against Big Data

Loraine Lawson

Lately I’ve seen several blog posts that I’ve mentally labeled as “Lectures on why Big Data can’t fix stupid.”

These posts are essentially cautionary tales about not expecting Big Data to solve your deep, organizational problems.

They vary in degrees; some are more or less in favor of investing in Big Data as long as you don’t expect too much, while others spend more time “raging” against Big Data investments as the cost of smaller, concrete steps troubled organizations can take to create useful change.

IDG posted one of the former type a few weeks ago, titled “The Big Data Fairy Tale.” It’s the tale of a data analytics project initiated by the NYPD under Mayor Giuliani, with help from CompStat. The idea was to map crime by geography while also charting officer performance by “quantifying criminal apprehensions.”

My cop-speak is a bit rusty, but I’m pretty sure that means they’re making crime reports and arrests.

“The key to success was not the data or analysis, but that the organizational management that used the data and analysis was effective,” the post notes. “Processes, structures and accountability were set up to drive the transformation.”

Oh, yeah, and you have to find the right people and get the wrong people out of the way. And here we come to the moral of the Fairy Tale:

In the same way that Giuliani fired one of the precinct commanders when he showed up drunk at the first CompStat meeting, big data systems require a complementary management philosophy to ensure whatever transformational insights are derived get implemented and controlled.

Point taken.

Just last week, the Harvard Business Review published a classic “rage against the Big Data in favor of small data” piece by Robert Plant (no, not that one), a Miami School of Business Administration professor. It’s well-summed up by the second paragraph:

So why do companies spend millions on big data and big-data-based market research while continuing to ignore the simple things that make customers happy? Why do they buy huge proprietary databases yet fail to use plain old scheduling software to tell you precisely when a technician is going to arrive?

Why, indeed?

“Big data is today's panacea, the great new hope for unlocking the mysteries of marketing,” Plant warns. “Companies would do better at satisfying and retaining customers if they spent less time worrying about big data and more time making good use of ‘small data’ — already-available information from simple technology solutions — to become more flexible, informative, and helpful.”

Maybe they don’t know how to use the small data. Dane Atkinson, the CEO of SumAll, says that’s a big part of the problem he encounters in his data work with small companies.

“Everyone we've interviewed who is striking out and building their own small business is sharp, aggressive, hungry, the best kind of people,” Atkinson said. “But they usually have very little data, very little math experience.”

That’s where SumAll comes in, of course — helping them break up the customer data and make sense of it, and he makes a pretty good sales pitch out in this Business Insider interview, if you’d like to hear it.

But what about those bigger companies that have the technology, that have the data analysis and still are getting the small data wrong? Can Big Data help them?

I’m going out on a limb here to say, “No.” Some business problems can’t be fixed by technology or even the best IT staff. That’s why so many techie people have t-shirts that say “ID-10T error” and PEBKAC shirts.

He’s not a professor or analyst, but I think comedian Ron White best sums up the real problem at some companies:

You can't fix stupid. There's not a pill you can take; there's not a class you can go to. Stupid is forever.

So, no, Big Data won’t fix what’s wrong at these companies. But then again, I suspect well-meaning posts, however spot on, won’t help, either.

Add Comment      Leave a comment on this blog post
Oct 9, 2012 9:09 AM R. Lawson R. Lawson  says:
Perhaps the problem isn't the desire to want to connect dots that can only be connected by accessing data in two different data sources, but the way in which those dots are connected. I'm not sure how that particular project went down. My guess is the mayor said something like "I want to know at a high level what crimes are occuring, where, and how our officers are responding". That requirement makes its way to the CIO. The CIO quickly arrives at the opinion that this isn't something that can be handled in-house, and calls in a contractor. One of my buddies runs an outsourcing firm, filled with low wage foreign guest workers. Let's call them. Said outsourcing firm hooks the city on a lowball quote, knowing that once the project gets going the client will double down rather than cancel the project as a failure. Instead of solving a small business problem, analysts identify "opportunities" to fix a host of other "problems". And so begins a "big data project". Costs soar. The story could have turned out differently . . . (continued) Reply
Oct 9, 2012 9:18 AM R. Lawson R. Lawson  says:
Based on what I have heard (armchair quarterback time), we have a bunch of information that should be public knowledge anyways. It's tough for me to give any meaningful analysis based on news reports. But let's pretend I understand the scope of the problem for a moment. We have at least two different databases with source data. We have officer arrest reports with geodata logged in one database (or more... who knows). Some of this data should probably be made available to the public. We have another database with officer trip logs, perhaps logged by dispatch. I assume that the NYPD knows where each vehicle is at a given time, and there is data tracking the geo coordinates from origin to a crime or accident scene. Omit some private data, and we probably have more data that should be made available to the public. Where am I going with this? You don't need to hire an outsourcing firm and pay them a quarter billion dollars when the public will gladly shape this data for you. Expose data through restful services. You provide transparency of a public agency, and the public does some of the heavy lifting for you. Checkout They are doing this already. Reply

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.