Big Data, Hadoop, cloud, MDM - these were the data management topics that dominated headlines in 2011. But as I looked back over a year's posts about integration and all its related disciplines, what I realized is 2011 launched a more far-reaching trend: the redefinition of integration itself.
The first shoots of change emerged, appropriately enough, in March, with a Gartner prediction that data integration vendors might soon be able to compete for projects traditionally opened to only database-management system (DBMS) vendors.
The reason for this prediction is that database management systems are being pressured to add context and insight to the data they're storing - and metadata management would be key to doing that. Since many data integration vendors already offer metadata management, it made sense that they would compete with DBMS for control of that information management layer, according to Gartner.
"As projects become increasingly complex they require deployment of many technologies from these DI toolboxes, a significant proportion of the overall time on the project is spent resolving technology integration issues that could be better applied to DI requirements that support the business," he wrote in a BeyeNetwork article.
His suggestion: a move to a data management platform, which would add data quality and master data management (and perhaps governance) to data integration tools.
At about the same time, IT consulting veteran Rajan Chandras said in an InformationWeek column that BPM and data governance should be combined, rather than handled as standalone projects.
Within a week of that post, the TDWI released "Next Generation Data Integration," which offered a broad re-evaluation of data integration, in particular pointing out that data integration is absorbing other data management disciplines, including data quality, data federation, replication and even MDM.
It also included this surprising news: More companies are shifting away from hand-coding as their standard approach to handling integration and adopting DI tools and solutions. As I wrote then, that's a significant change from the status quo, since vendors have been telling me for years their number one competition isn't each other, but hand-coded integration, handled either in-house or outsourced.
In May, Dana Gardner of Interarbor Solutions lead a panel discussion about whether IT needed to rethink data integration, and the answer was an unqualified "yes." The panel pointed out IT no longer has 3 to 5 years to bring in and integrate new applications. Gardner pointed out we needed to find a way to circumvent the integration bottleneck caused by hand-coding and elevate integration "out to a wider group of individuals."
But the trend isn't just about tools. In June, Julie Hunt called attention to the need for convergence in all information disciplines.
"The solutions surrounding the management of information, data and intelligence are emerging from artificial silos to acknowledge the greater overlap and interrelationships of these technologies and practices," Hunt wrote. "Deliberate initiatives to tighten up the convergence of these solution spaces will not only improve the better overall functioning of all of these solutions but will help both IT and Business users see how it all works together."
By July, I found myself writing about what the Bloor Group called Information-Oriented Architecture - a sort of grand unified theory of managing information.The shift toward integration's redefinition and, really, the integration of all-things data only accelerated as the year wore on. Gartner's fall release of the Data Integration Magic Quadrant contained fewer companies, but with more capabilities.
Most recently, Ken Vollmer of Forrester called for "holistic integration," which is to say the integration of all three branches of integration: data, application and business process - into one overarching enterprise approach to integration.
So the idea of what we mean by "integration" has evolved substantially this year, from a focus on separate tools and particular types of integration to a more ... well, integrated and strategic approach. I leave 2011 with a new concept of what integration can encompass, what it can mean and, more importantly, what it can achieve. Hopefully, 2012 will continue the discussion and help start us on the path toward making these ideas a reality.