Bad Data Quality, More Problems for the UK

The Hadoop Challenge for Business Intelligence and Analytics Users

I’ve seen some hefty price tags associated with poor data quality, but I have to say, last year’s figure from the Ministry of Defence may take the prize. The UK agency was told “it was at risk of squandering a £1 billion in investments in IT because of dire data quality” last year, according to Martin Doyle, the Data Quality Improvement Evangelist for DQ Global.

This year, another UK agency, the National Health Service (NHS), is under scrutiny for sharing data without consent. Names and addresses may have been taken from the database and sold for studies, which meant it was uploaded to third-party cloud storage services, according to Doyle.

As if that weren’t bad enough, the NHS is also working on a project called Care.data, which is a centralized hub for patient care records. The NHS has “problems recalling exactly who has all of this patient information already, suggesting it has bigger problems to solve,” he writes. This issue has triggered a backlog in patient care.

“Data is so vital in modern life that poor data quality can be catastrophic for millions of people, and the sooner we realize, the sooner we can face the challenge head on,” Doyle writes. “We see this time and again; large organizations plough money into projects that should make information management easier, but the data flowing through the system is not of a high enough standard to make investment worthwhile.”

Let’s hope the UK’s data troubles aren’t a sign of things to come in the U.S., as we move toward electronic medical records.

SAS Institute’s Big Move on Big Data

Ventana analyst Mark A. Smith recently offered an assessment of SAS Institute’s Big Data portfolio. In some ways, this has been a natural expansion for SAS, which always dealt with large datasets. Thanks to investments in the last few years, Smith notes that the company’s in-memory infrastructure can, among other things:

• Operate within Hadoop
• Run MapReduce jobs
• Access various Hadoop distributions
• Use Hadoop’s Pig and Hive interfaces
• Extend that to data and visual discovery and exploration tools

In an Information Management post, Smith discusses how SAS addresses data integration, noting that it is a key issue for 47 percent of organizations using Big Data analytics. He also points out that Big Data quality and consistency is a “significant challenge” for 56 percent of organizations, and explains how SAS’s solutions address these pain points.

You may not be in the market for something as robust as SAS, but Smith’s piece also explains how all the vendors play in the broader Big Data market.