Leaders of business intelligence (BI) projects should push for a revamped data architecture that supports more integrated data, even if it means looking at a Big Data option, according to a recent InfoWorld column.
In “Why BI projects fail — and how to succeed instead,” software consultant Andrew C. Oliver says it’s essential to be able to integrate large amounts of data. BI tools tend to be resource-hungry, he adds.
So, rather than viewing technologies such as Hadoop, data lakes, enterprise data hubs and data warehouses as “trendy,” you should view them as essential to BI success, argues Oliver.
“A successful BI project does not forget about either business integration (more later) or data integration,” he writes. “Your requirements should dictate what, how much, and how often (that is, how ‘real time’ you need it to be) data must be fed into your data warehousing technology.”
Oliver, who is the president and founder of a Big Data consulting company, Open Software Integrators, acknowledges that this can cause furrowed brows in IT. So BI leaders offer something in exchange.
“As part of the pitch, you promise IT to stop doing point-to-point and start with hub-and-spoke integration to the data lake,” he says. “In other words, you need one big feed to serve all instead of hundreds of operational, system-killing little feeds that can’t be controlled easily.”
An integrated data repository isn’t enough, however. As this IT Business Edge slide notes, it’s actually a “BI worst practice” to set up a data warehouse and assume that alone will drive business users to the information.
The slideshow draws on material from Information Builders and argues that at times, you might even want to forgo a big repository for all your data.
“The truth is that not all BI applications require a data warehouse,” Information Builders noted in the slides. “Many BI applications are better served with integration and portal technology that allows data to reside where it currently exists and pulls it on an as-needed basis. Unfortunately, many organizations fail to assess whether or not a data warehouse is the right solution to their challenge before starting down the warehouse path.”
That can especially be the case when you’re dealing with something as involved as Hadoop, where it can be hard to track down the right data for the job — even for data scientists. A recent study found that 76 out of 100 data scientists said Hadoop is too slow, takes too much effort to program or has other limitations, CIO.com reports.
Part of the problem with being able to integrate everything is that you aren’t focusing your efforts on what matters. You can avoid that trap by following the advice of John Lucker, a principal and Advanced Analytics and Modeling National practice leader for Deloitte Consulting. Lucker warned about this problem two years ago during an interview:
… I have seen is a lot of organizations really lose sight of the whole 80/20 rule around data hygiene and what data needs to be available to them. So I think what ends up happening is organizations never quite get done with what they want to achieve with BI and making data available to an end user. So, I think, focus on the 80/20 rule around data availability integration is important.
The best way to do that is to sit down and discuss the goals of your BI and analytics program. That will help you pinpoint which data you must have, which data you’d like to have, and even which external data the organization may need to obtain. I would even argue it’s an essential step if you’re concerned about costs, since experts say integration work can drive up the price tag on BI projects.
Another option to consider is self-serve integration, which is a huge trend among integration vendors and end-user organizations this year. An Aberdeen Group study released in January showed that 64 percent reported that their companies were pursuing a “self-service BI strategy” to address the endless demand for more data.
Loraine Lawson is a veteran technology reporter and blogger. She currently writes the Integration blog for IT Business Edge, which covers all aspects of integration technology, including data governance and best practices. She has also covered IT/Business Alignment and IT Security for IT Business Edge. Before becoming a freelance writer, Lawson worked at TechRepublic as a site editor and writer, covering mobile, IT management, IT security and other technology trends. Previously, she was a webmaster at the Kentucky Transportation Cabinet and a newspaper journalist. Follow Lawson at Google+ and on Twitter.