Three Common Myths About Big Data Success

The three fairly pervasive ideas about what it takes to succeed with Big Data:

1. You absolutely need Hadoop, or at least some sort of NoSQL, in-memory, new-fangled approach to data.
2. Legacy systems like data warehouses are irrelevant.
3. Don’t even think about Big Data until you’ve hired a consultant or data scientists.

All are misconceptions, according to Tom Davenport, the President’s Distinguished Professor of IT and Management at Babson College, a research fellow at the MIT Center for Digital Business, co-founder of the International Institute for Analytics, and senior advisor to Deloitte Analytics and, to simplify, a well-respected analytics-brainiac.

The Hadoop Challenge for Business Intelligence and Analytics Users

Davenport draws on interviews with 15 organizations to go mythbusting in a recent Harvard Business Review Blog post. The organizations come from a variety of industries, but it’s worth noting that he did obtain most of his interview contacts from Teradata Aster, which also sponsored his study. That said, Davenport asserts that the company “did not influence the outcome of the study or the content of this document.”

It’s hard to see how that wouldn’t influence the study, since Teradata Aster is one of the “data discovery platforms” many of those interviewed rely on for their Big Data projects.

Just because there’s an innate bias doesn’t make this useless, though. For one thing, lots of companies are in the same predicament when it comes to Big Data: They have invested in legacy data warehouses, they have staff expertise in SQL programming, and they want to effectively leverage both with Big Data.

The surprisingly good news is, Davenport found that these tools do help companies program Big Data applications with existing languages — most importantly, SQL.

“I also learned that companies with existing data warehouse environments tend to create value faster with big data projects than those without them,” he writes. “Your existing analytical tools—SAS, SPSS, R—will also still be useful with big data.”

Likewise, large companies say existing IT staff are instrumental in Big Data implementations, particularly when they’re teamed up with people with additional expertise.

“The large companies I interviewed about big data projects said they were not hiring Ph.D. level data scientists on a large scale. Instead they were forming teams of people with quantitative, computational, or business expertise backgrounds,” he said. “They felt the need to educate some of the project team members on big data technologies such as Hadoop and scripting languages. But they were not in desperate straits from a big data talent perspective.”

Davenport also says sound project management practices, a clear business objective and change management expertise are significant traits of successful Big Data implementations.

It’s worth noting that Phil Simon, a business consultant and author, disagrees slightly with Davenport in the comments. His contention is that Big Data works best when it’s embedded into an organization’s DNA. I shared some of his views in “Three Signs You’re Not Ready for Big Data.” You can also read his previous HBRB post.

If you’d like to find out how organizations are leveraging legacy skills and investments with Big Data, you might also check out Davenport’s report. The executive summary is beefy and available for free download. It includes a list of the typical applications for Big Data and a checklist of capabilities to consider when investing in discovery platforms.