Five Integration Challenges for Big Data

Big Data Makes Organizations Smarter, but Open Data Makes Them Richer

How important is Big Data? The answer seems to depend on whom you ask.

Talend, an open source data management company, recently surveyed 231 professionals and found that “Big Data has moved past an early adopter stage, with 41 percent of organizations having a Big Data strategy in place.”

It’s worth noting that Talend targeted people who are involved in the delivery of data solutions for their company. Obviously, that’s a niche audience that’s going to understand Big Data and its potential.

Now consider the results of another recent survey, this time by The Register, which used Freeform Dynamics to survey 502 of its own readers, all IT professionals. This is still a group that should at least be familiar with Big Data, since The Register regularly writes about it.

So it’s telling that less than half of the respondents indicated a clear understanding of what the term “Big Data” even means.

Obviously, if you’re a company like Talend, you probably care more about that data professional slice — after all, they’re your advocates within organizations, the people who will ultimately influence the decision-makers about which products and projects are worthwhile.

And it’s certainly not insignificant that for this group, 68 percent say the number one business driver for Big Data is to increase accuracy and depth of predictive analysis, followed by revenue optimization (51 percent) and new revenue generation (48 percent).

But it’s also worth noting that when you’re dealing with a broader IT audience, structured data ranks higher in terms of importance, growth and usefulness. It’s certainly also worth noting that even though organizations should be well-versed in using structured data by now, they’re still struggling to do so. As The Register, a UK site, points out:

There is also a widespread acknowledgement by many that they do a poor job of making full use of both types of data, with the effective exploitation of unstructured sources at remarkably low levels. Taken together, these results show that there is obvious potential for both ‘traditional’ business intelligence/data analytics tools and Big Data solutions to be utilised for the benefit of very many organisations.

The article also points out that integration will be a major concern as organizations try to integrate Big Data with the structured data stored in relational databases. In particular, the article notes five integration-related challenges:

1. Integrating both environments without adversely impacting either.
2. Feeding any Big Data results into existing information management and BI or analytics environments in a way that’s useful.
3. Avoiding silos of capabilities, management tools and processes.
4. Keeping integration costs down while still being able to access and use the data quickly.
5. Integrating the two systems so organizations can manage data security, governance and regulation.

You really can’t underestimate the importance of that last one, especially when you consider Talend’s survey, which found the most common applications being integrated all involve tricky compliance and privacy issues:

• Financial transactions (48.2 percent)
• Social media and Internet text (48.2 percent)
• Web logs (35.8 percent)
• Call detail records (28.4 percent)