Big Data Lessons from the Real World

Big Data: Not Just for Big Business Anymore

It’s one thing to read the hype about a technology project, but quite another to actually do it yourself. We’ve all learned that technology seldom works as well as we were told.

That’s why I really wonder what it’s like to do a Big Data project from scratch. No one has ever said Big Data would be easy, but so many of the success stories come from Internet companies whose business is Big Data, I have to wonder what the reality would be for a company that’s not, say, Google.

Would it be ridiculously hard? Prohibitively expensive? Pretty nigh impossible?

What’s it really like to do Big Data?

A recent Network World article gives us an inside look at the challenges learned from three real-world Big Data roll-outs at:

• Klout, the social media company
• ACE Group, an insurer
• Vanguard Health Systems (now part of Tenet Healthcare)

Employees from the companies shared their experiences during a panel discussion at this month’s The Big Data Conference in Chicago.

Now Klout, obviously, is ahead of the game in that its founders all have extensive background in data and large data-related projects.

Even so, CTO Virendra Vase said the company struggled to find talent and discovered that the management and configuration tools were still immature, requiring the team to do a lot of tool development.

Brian Barnes, vice president of consumer applications at Tenet Healthcare, cautioned conference goers not to underestimate the amount of vendor management or systems integration that Big Data analytics requires.

The company wanted a best-of-breed approach, which ultimately meant using eight different companies.

Using that many vendors also meant integration became a major issue.

“Systems integration has been a huge challenge for us,” Barnes told the audience.

It’s not all doom-and-gloom, though. The panel also shared their suggestions for how to handle Big Data, including:

• “Over-communicate” your intention, vision and success stories.
• Establish a steering committee.
• Let business lead.
• Develop talent in-house.
• “Embed” your Big Data talent within the business units.