Fair Warning on Combining Data Integration and Agile Methods

Loraine Lawson

You see the word "agile" a lot in tech sectors these days.


Sometimes, it means just what you think it'd mean-agile, as in nimble, moving quickly and lightly. More recently, it's being used a sort of synonym for IT/business alignment.


But, more often than not, agile is referring to agile development.


My understanding of it is largely based on what I've gathered in passing, and what I've read on Wikipedia. Books have been written on the subject, which as I understood it first took hold in 2001 with the writing of the "Agile Manifesto." But, for those who don't care to go deeper, my basic business-friendly definition is that it involves a cross-functional, team approach and iterative and incremental development-as opposed to pronouncing something finished and complete. It strikes me as a very Total Quality Management/Six Sigma/process-improvement-of-the-month approach.


I have to say, it sounds like a good deal for the business in a lot of ways. First, it reaches out to the business in an effort to be cross-functional and it can create a faster turn-around on projects-with the understanding that you're not necessarily getting a bugs-free, finished application on the first go-round. (When did you ever?)


Not surprisingly, there are those who are starting to question whether the agile methodology can be applied to data integration. I've noticed much of this discussion is tied to business intelligence, where agile development seems to be taking hold.


I'm not going to lie to you-this discussion can become very geeky, very fast, which is why I haven't written about it until now.


Recently, Philip Russom, senior manager of research and services at TDWI, tackled the topic and I think he does a nice job of highlighting the pros and cons of applying agile development practices to data integration and data warehousing. That said, it's still a piece written primarily for developers and their managers, so I'll summarize a few key points.


Why is agile right for BI? Russom points out that agile development is designed to speed up development for operational and transactional applications that automate a business process. As it turns out, the reports and dashboards generated by BI applications are very similar, so it's easy to transfer agile methods to BI.


The problem is, data warehouses and data integration are different because "they focus on data and the repurposing of data, in the context of long-term infrastructure that will be shared by many teams," writes Russom.


In other words, while applications and BI reports are written for specific processes, data is not and, in fact, will be shared for many teams. Therefore, data integration and data warehouses require a broader vision, so to speak. They're not standalone projects, but support a wide range of applications, reports, dashboards and other enterprise-wide needs.


There are other issues as well, including data models, standards, governance and other requirements. This makes it challenging to apply agile methods to data integration and data warehouse.


That said, Russom says it is possible to apply agile methods to data integration and data warehouses-provided you understand that neither will be as "agile as BI."


He wraps up the piece with a list of accommodations that will need to be made for data warehouses and data integration.


In this day and age, agility-meaning, nimble, quick and light-can be a strategic advantage. Agile development can help deliver that, particularly with BI projects, but-and this is the critical take-away here-it needs to be adapted at the data integration/data warehouse level. If your development team is pushing agile methods, make sure they're willing to work out a more accommodating approach with the data integration and data warehouse team.


You might also want to read "Why IT and Business Should Care about Agile Integration," a post about how data integration can be made more agile - in the traditional meaning of "agile."


If you'd like to read more about agile methods and data warehousing, check out this Q&A with Ralph Hughes, chief systems architect at Ceregenics and a speaker at the TDWI World Conference seminars.

Add Comment      Leave a comment on this blog post
Jun 23, 2010 1:24 AM Peter Peter  says:

Finally someone who understands the difference between BI and Data Integration!  I have been in BI/DW for years including 10 years with IBM and I have yet to see Agile successfully applied to the Data Extraction, Transformation and Load components when building the enterprise data warehouse, including the database design.  I have debated this with many of my IT peers who use Agile in traditional application development but up until now it seemed like I was the only voice.   Clearly agile is well suited for data mart, data analytics and report development.  Unfortunately most of the articles, save for the most recent TDWI one referenced in Loraine's article,  fail to differentiate Data Integration.

Thank you, thank you, thank you !


Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.