In this age of Big Data and distributed enterprise architectures, it isn’t hard to convince the average IT executive of the need for data warehousing. The difficult part comes in the design and implementation phase.
When the success or failure of a data management and analysis venture depends strictly on its ability to interact with constantly shifting and increasingly complex infrastructure, there are usually only a few ways to get it right, and many ways to get it wrong.
This may explain why some analysts are starting to question the efficacy of warehousing. As experience with the technology grows, it is only natural that shortcomings will emerge. But as consultant Joe Caserta explained to Information Management recently, it’s not that enterprise data warehousing (EDW) is losing its value, but that it tends to perform better in some environments than others. In general, though, he recommends a crawl-walk-run approach to warehousing, as it can be quite overwhelming for an enterprise to jump from simple database management straight to a full EDW platform, particularly if the move concurs with rapid expansion to virtual and cloud environments.
To be sure, there is no shortage of EDW platforms available for those who are ready to take the bull by the horns – and they are getting more powerful all the time. Teradata, for example, recently upgraded its already cutting-edge EDW 6700 platform with new eight-core 2.6 GHz Xeons and InfiniBand storage networking, said to produce a 10x increase in throughput over existing Ethernet-based versions. The system can also scale up to thousands of concurrent requests and is available with storage tiering software that prioritizes hot and cold data among various solid-state and mechanical disk drives.
Still, state-of-the-art technology is only part of the warehousing puzzle. The rest falls to a long list of intangibles like proper implementation, training and the overall support of the user community, according to memeburn.com’s Michael Clark. To that end, enterprise executives should look long and hard at any platform’s operational efficiency, user-friendliness and support for recognized standards. After all, a warehousing system will only be as effective to the degree it can integrate into existing data infrastructure and workflow environments.
In the end, then, does warehousing put us on the path to creating the “enterprise brain,” asks tech consultant Jim Harris. If you think about it, the warehouse serves not only as the repository of institutional knowledge, but provides the interconnectivity between the various data silos that exist in most organizations. Executives should beware, then, of introducing too much structure to the warehousing process, given that the human brain often achieves the greatest insight during its chaotic, unstructured moments.
Putting visions of SkyNet aside for the moment, data warehousing will most certainly prove to be a valuable tool even for small, more focused organizations. As more and more business activity takes place outside the traditional enterprise – in the cloud, on mobile platforms – the need to harness and analyze disparate data sets will only gain in importance.
In fact, this is likely to remain one of the top priorities as the enterprise makes the transition to more dynamic data infrastructure. After all, deploying new technologies is easy. Integrating them is not.