Companies are learning the hard way that there’s a downside to data democratization: more data silos.
“On the heels of the consumerization of enterprise software and the growing ubiquity of easy-to-use analytics tools, silos appear to be coming back in all their former collaboration-stifling glory as individual teams and departments pick and choose different tools for different purposes and data sets without enterprise-level oversight,” writes Katherine Noyes in a recent Computerworld article exploring this growing problem.
It’s hard to hear in this age of Big Data and data lakes, but in hindsight, it really isn’t surprising. SaaS made it possible for the lines of business to choose their own applications with nothing more than a credit card. Then Apple tipped the balance on personal devices. Finally, Amazon and others democratized storage and Big Data processing power. It only makes sense that analytics — and more data — would leave the centralizing influence of IT and segregate into silos.
“You can easily replicate the same issue you’ve already got with spreadsheets: Multiple versions of the truth, except it looks prettier,” Forrester analyst Martha Bennett told Noyes.
These silos create a multitude of data sins, including duplication, errors, conflicts and compliance problems. Perhaps most significantly, departmental silos make it impossible to determine who’s right or to generate enterprise-wide insights.
“Most importantly, the organization cannot become truly data driven in its decision-making, which is likely sooner or later to lead to competitive advantage,” Bennett added.
This isn’t the first time IT has faced this problem.
Data Lessons from the Past
The current thinking is that data lakes will solve the silo problem, but not so long ago, ERP and data warehouses also promised to resolve data silos.
“ERP suites were designed to be a replacement for separate departmental applications,” IDC analyst Henry Morris, senior vice president, told Noyes. “Several things thwarted this vision.”
First, organizations wound up with multiple ERP suites. Second, cloud and SaaS happened, giving LOBs even more options for opting out of the established ERP systems. Third, cheaper online tools made it easier for LOBs to purchase new apps without corporate approval — or even IT’s knowledge. And of course — there’s Excel. Business users still use Excel for DIY analytics, but now they can acquire even more data to squirrel away from others.
Rick Sokolosky, the life sciences practice leader at NewVantage Partners, reminds us that organizations also thought data warehouse could solve the silo problem. Instead, they’ve exacerbated silos and created “semi-pro data warlords,” he writes in an Information Management column.
Data warehouses extract the “needed” data and lock it down. But in a world where business decisions are made in real time and new data sources emerge constantly, no one has time for a complex system designed with one purpose, he writes. The average data warehouse change takes nine months and over $1 million. That’s not agile.
“There is only one problem. It doesn’t come close to delivering access with the speed and agility that businesses really need from their data,” Sokolosky states. “Where did we go wrong? We forgot about the data.”
What These Mistakes Can Teach CIOs Now
Enterprises can learn three lessons from IT’s past. The first is admittedly counter-intuitive: Experts say IT should loosen up. When you build analytics systems and data lakes, think less about controlling analysis and more about creating quality source data that the lines of business can trust.
To do that, realize that the principles of data warehouses are no longer useful with data lakes, Sokolosky writes. So go back to the source data and focus on service-enabling it, rather than locking it down through traditional integration, he advises.
“Business requirements should drive what sources are taken in their entirety into the environment,” he writes. “For even better results, that source data should be lightly integrated and standardized into easily understood subject areas for greater usability.”
Forrester VP Boris Evelson made a similar recommendation in Noyes’ article, but he adds a second suggestion: Focus on building a data hub based on a low-cost platform such as Hadoop (a.k.a., the data lake), then add BI applications as spokes off the hub. Most of the enterprise data should be in the hub, even if that means giving up some best practices now.
“Here you give up some of the controls,” he tells Noyes. “The data may not be very clean or integrated, but it’s all in one place.”
Third, remember the hard-won lessons of IT/business alignment discussions. A panel at the Canadian Telecom Summit stressed that data and technology must bow to customer and business needs, reports IT World Canada.
“I’m a huge fan of data analytics, and big data, but you can’t do it in a way that ignores some basic business tenets like, ‘start with your customer, what your customer wants, what can you deliver,’” said Ann Cavoukian, executive director of Ryerson University’s Privacy & Big Data Institute. “If you don’t have that dialogue, you’re going to lose out.”
Loraine Lawson is a veteran technology reporter and blogger. She currently writes the Integration blog for IT Business Edge, which covers all aspects of integration technology, including data governance and best practices. She has also covered IT/Business Alignment and IT Security for IT Business Edge. Before becoming a freelance writer, Lawson worked at TechRepublic as a site editor and writer, covering mobile, IT management, IT security and other technology trends. Previously, she was a webmaster at the Kentucky Transportation Cabinet and a newspaper journalist. Follow Lawson at Google+ and on Twitter.