Resolving Data Quality Issues in Flight

Michael Vizard

The trouble with data quality is that the whole process of managing it is disassociated from how the data is actually used, so what winds up happening is that it never gets done.

Instead, data quality becomes another exercise in IT good housekeeping that nobody ever quite gets around to doing. Against that backdrop, it's worth noting how SnapLogic, a provider of application integration software, has teamed with Trillium Software, a provider of data quality software that is provided via a hosting service, to integrate data quality management directly into the application integration process.

According to Clark Newby, SnapLogic senior vice president of marketing, IT organizations can set up a SnapLogic server to automatically route data being transferred through the Trillium service, which essentially makes the whole data quality management process transparent to the average end user.



The Trillium integration is part of the summer 2011 release of the SnapLogic server, which also includes an instance of the open-source CouchDB database for managing documents and improvement to the way SnapLogic can run in-memory to boost overall performance.

As Newby points out, new technologies such as cloud computing and Big Data tend to turn bad problems like data quality into full-fledged epidemics. If IT organizations want to cure data quality issues, or at least make the problem manageable, they are going to have to find new ways to automate the process in a way that, as far as the rest of the business is concerned, makes the problem essentially go away. After all, rightly or wrongly, the rest of the business thinks data quality is an IT issue so the IT department might as well take it upon itself to solve the problem in the least-obtrusive way possible.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Add Comment      Leave a comment on this blog post
Jul 14, 2011 8:57 AM Jim Pennington Jim Pennington  says:

Agreed that IT should take Data Quality responsibility on.  However, fundamental changes in how IT is measured have to take place.  Every project in the Project Queue is addressed in ways that add to the data quality problem, effectively creating more costs for future projects and more complexity for IT. 

With this responsibility, IT has to change how it is measured by the business as being "successful".  The success of every project must also be measured on the enterprise level improvement or alignment with enterprise data quality consistency standards.  And before that is done, there must be an Enterprise Architecture that sets these standards based on true architecture based on work at the metaphysical level, not the physical/logical stuff that people try to pass off as architecture.  Secondly, that architecture has to lead to an effective data governance program.  We can't "tool" our way out of the current data quality mess by creating more of a mess at the same time we are trying to clean up the mess.  This is like a dog chasing its tail.

The goal is to stop building enterprise data liabilities and start building enterprise data assets.  Today, data is not an asset.  The architecture at the metaphysical level is not there, and does not drive the quality that is achievable at design time and transaction execution time.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.