The High Costs of Big Data: Will Monetizing Lead to Regulations?

It’s never been cheaper to obtain the technologies needed to process Big Data, but alas, that doesn’t change the fact that it’s still very expensive.

“Even with Amazon Redshift’s aggressive pricing, NASA would have to pay more than a $1 million for 45 days in data storage costs alone,” Bruno Aziza wrote in a December Venture Beat guest blog. “This number is consistent with New Vantage Partners survey, which evaluates the average big data project to cost between $1 million and $10 million.”

Aziza is the vice president of marketing at SiSense, a Big Data analytics company. And he’s not the first or the only one to note that Big Data costs, even with the open source options, can be prohibitive.

Even the Library of Congress can’t afford the distributed and parallel computing programs it’d need to process its collection of 170 billion plus tweets. (Although experts say there may be ways to combine open source Big Data tools with other data management techniques.)

There’s no denying Aziza’s point: Most companies don’t have that kind of IT budget. His conclusion is that Big Data discussions should stop focusing on these extreme use cases and spend more time looking at how Big Data tools can help with the data problems most companies face.

“Sometimes, I wonder what would happen if we changed the definition of big data,” Aziza writes. “What if, instead of focusing of the proverbial 3 V’s (velocity, volume and variety), we tried something like this: ‘Big data is a subjective state that describes the situation a company finds itself in when its infrastructure can’t keep pace with its data needs.’”

Like I said, it’s not the first time the economics of Big Data — even with its open source solutions — has been called out. But overall, this issue is downplayed or ignored in the industry.

I suspect the assumption has been that the costs would go down as more vendors entered the space.

And certainly companies are rushing into the space of Big Data. The blog Beautiful Data published a list of “10 Hot Big Data Startups to Watch in 2013” and Big Data firms ranked on Inside Analysis’ Robin Bloor list of “10 Companies and Technologies to Watch in 2013.”

But judging from Aziza’s estimates, it’s going to take competition for cloud computing platforms such as Amazon to really drive costs down, assuming that’s even possible.

What seems more likely is that companies will find other ways to make Big Data projects pay off.

That’s Gartner’s contention. In a recent press release, the IT research firm predicted that 30 percent of businesses will “monetize” their information assets directly by 2016.

That headline surprised me, because I thought 30 percent seemed low. But then I read the full press release, and got the larger point. It’s not just that 30 percent will monetize their data.

No — it’s specifically that (emphasis mine) Gartner’s predicting “The financial demands of storing and managing Big Data will lead 30 percent of businesses to directly or indirectly monetize their information assets by trading, bartering or outright selling them by 2016.”

“The need to justify the expense of accumulating and managing huge volumes of data has led many organizations to consider monetizing or productizing their information assets,” Doug Laney, research vice president at Gartner, states in the release.

Since most companies aren’t in the business of information, they’ll resell it to third parties, giving rise to a growing group of “information resellers,” Gartner predicts. Ray “R” Wang said something similar in December, but he called them “information brokerages.”

I have two reactions to that, and I suspect most people will share them:

1. How do I get in on those startups?

2. Wait, what? They’re going to sell my information even more than they do now???

And that second reaction is exactly the one Gartner says could trigger a public backlash, which in turn could trigger tighter regulations.

But business hasn’t historically excelled at knowing when to stop, especially when it comes to profits and respecting privacy. And as I’ve already pointed out, most of that Big Data probably doesn’t really belong to organizations in the first place. So here’s my prediction: Regulations are exactly where this will wind up.