Data Integration Remains a Major IT Headache
Study shows that data integration is still costly and requires a lot of manual coding.
We tend to to talk about silos as the original sin of data management. In the beginning was the mainframe, and then came the fall: Satan, lurking in the silos, introduced departmental computers, which begat millions of rogue spreadsheets, databases, even data marts and lo - pandelerium.
But do silos get a bad rap? Are silos innately evil or is there a good and necessary side to silos?
I look at the world through an integration lens, so it's hard for me to comprehend, but recently I've read not one, but three (!) pieces defending silos.
Silos, the article puts forth, allow departments to "achieve their unique goals."
"Silos are a necessary part of an organization, but they need to be challenged into sharing information," quips Sean Canning, senior vice president of customer management at Firstsource Solutions, in the article.
I guess that's one opinion. However, I'm partial to the description offered by Harvard Business School Prof. Ranjay Gulati, who called silos "a necessary evil." The article gives six steps for integrating information across departments. The key takeaway here is to recognize that one department does own the data, so act as a diplomat, not a dictator.
The second defense of silos surprised me a bit, since it came from Joe McKendrick of ZDNet. While he acknowledges the problems of silos - duplication, extra work, the lack of access to critical information across the organization - McKendrick says we still need to respect that there's a time and place for silos. "Sometimes, it's better to live with silos than fight with them," he writes.
He harkens to a more complete defense of silos published recently on ReadWriteWeb, which interviewed Anjul Bhambhri. Bhambhri is IBM's VP for Big Data, so his discussion of silos is related to that particular type of data.
What tickled me about this particular piece is that Scott Fulton manages to blame silos on IBM while talking to the IBM vice president:
the reason for database siloing dates way, way back to the 1970s and '80s when computing products were purchased on a department-by-department basis. In the mainframe era - which IBM helped the world inaugurate (so it's your fault) - computing products were purchased, deployed, configured, programmed by the people in finance, in budgeting, in human resources, in insurance management, in payroll. The archival data that has amassed from this era is based on these ancient foundations.
But Fulton also points out that, really, IT silos serve a very real purpose. Often, departments are keeping the data hidden for very real reasons, such as regulations or policies. In other words, it's not just about being petty - they're controlling the data for a real business reason.
Bhambhri responds with an answer that nicely sums up why IT might want to back off its crusade against silos:
Data is going to be where it is, in an enterprise. There may be department-level decisions that were made, department-level applications that are running on top of it. Nobody's going to like some guy coming in saying Let me bring this all together.'
So, how do you leave the data alone and still make it accessible? Bhambhri's suggestion is to use data federation and information integration, which allows you to leave the data where it is, but federate the queries.
"Heterogeneity is a reality, and we have to accept it and provide the technology that actually takes advantage of that heterogeneity, and respect the decisions that the customers have made," Bhambhri says.
Data federation or virtualization isn't always going to be the right answer to dealing with silos, but it's certainly one to keep in your data integration toolbox. As these articles point out, silos can be necessary when it comes to owning and governing data. Data virtualization lets you open up the data to the organization, without ripping or replacing complicated systems.
I have to admit that, in some cases, it makes sense to make peace with silos.