In a conversation with David Foote, co-founder and CEO of market research firm Foote Partners, he told me that the most crucial issue companies face with Big Data isn’t really a skills issue at all.
There has been concern that lack of talent for Big Data could limit companies’ ability to deal with their massive data stores. Foote contends there are bigger barriers.
“When you look at companies that have failed with Big Data, the reason why they failed was the same reason they failed at EA (enterprise architecture), why they failed at project management, at early BI, data warehousing …” he said.
Ultimately, to make it succeed, you have to realize it’s not a tech problem. You can’t just throw tech at it. Companies can spend a lot of money on it, but even if they have the right tools and can afford Hadoop people, the reason they fail is they have cultural, organizational structure that is not used to transparency, not used to sharing.”
Companies that effectively use their data — think Facebook, Yahoo and the like — are relatively young and have an open culture of sharing data horizontally across the organization.
My colleague Loraine Lawson pointed to a SAS white paper with this stat that seems to back him up:
… isolation of business analytics to a single department or function — commonly referred to as “siloing” — is more than three times as common (31 percent vs. 9 percent) among these less-effective users of analytics than it is in companies that are “very effective” in their analytics usage.
Foote has been studying vendors of electronic health record systems lately and said that 40 percent of the value creation from using EHRs comes simply from creating transparency.
Companies successful with Big Data also are fact-based cultures that tie performance measurement into every aspect of the business. They often use centers of excellence to collaborate on using the data laterally, rather than allowing silos or using the data politically, he said.
Some companies are trying to reinvent the wheel on basic analytics, he said, while others get hung up on data quality or governance issues.
“They can clean data all they want, but it doesn’t really matter. They’re not going to be able to use it effectively if they have people in the organization opposed to transparency and cross-enterprise activity,” he said.
“And sometimes, we tell companies that the only way they’re going to solve this is through attrition. Some managers have to leave before this will work.”
