Thinking Through the Challenge of Social Data

Loraine Lawson
Slide Show

Social Media: Measure, Monitor and Mean It

Highlights and suggestions from Burson-Marsteller's Global Social Media Checkup.

Enterprises are so used to dealing with data, you'd think adding text from social media would be no big deal.


And yet, few organizations have managed it. That's because there are actually real technology challenges when it comes to integrating social data, writes Jackie Kmetz (@speedyjackie), a "Social Intelligence Crusader" for Visible Technologies. Kmetz is writing a series on the "Top 5 Challenges of Social Data Integration." She's listed four of the five, and, not surprisingly, each the posts points out that Visible Technologies offers tools that handle all of this for you.


Despite the expected sales pitch, the posts do point out the very real problems organizations will face as they try to bring social media data into their existing software and systems.


You may have thought of some of the problems she points out, but not thought through what a challenge it could really be. At least, that was my experience reading it. For instance, you know that the amount of social media data is huge - from blogs to Twitter posts, it's an unending stream of chatter.


What you may not have thought about is how this could overwhelm your existing enterprise systems, which tops Kmetz's list of integration challenges for social media. It's one reason why social media is such a popular use case for Big Data. But it's about more than just processing the stream of data.


"The second issue is that most enterprise applications are built to process data in less frequent batches, such as end of day reports," Kmetz writes. "This creates an obvious mismatch in capabilities when organizations need to listen, learn and engage in real time with consumers. The potential hit to network performance for the entire organization could be debilitating or extremely cost prohibitive."


That's where social media monitoring systems come into play. They handle the deluge of data, picking through what's relevant and dumping the rest. Another integration challenge: Integrating social data into your existing analytics and BI systems. The reason this is such a challenge, she explains, is because these systems typically draw on structured databases - but social media is unstructured, and that makes it harder to integrate, since databases and legacy CRM systems are designed for structured data.


Her third social media integration challenge is the use of APIs (application programming interfaces). She notes this isn't a skill you'll find in the average IT department, but I suspect it's one they could quickly learn, since software systems do use APIs. No, the catch here is the sheer volume of APIs you have to manage, and the fact that many of the social media APIs are open and may change without notification, breaking your connections.


In most recent installment of the series, Kmetz writes about the unique data quality issues involved with using social data.


"The social data that is integrated into existing systems must be high quality, validated and consistent or enterprises risk polluting business applications downstream. As we discussed earlier, social data comes from a diverse ecosystem of sites in an unstructured format," she writes. "This data requires parsing, cleansing, standardizing, validating, de-duplicating and enhancement with additional metadata prior to incorporating it into your existing systems."


Again, an excellent challenge IT will need to think through before connecting with social data. As she points out, Visible (and I'm sure others) offer a tool for this, but it's still important for IT to do its due diligence and not over rely on packaged solutions.


Next week, she should conclude the series with the fifth integration challenge. So far, it's been worth the read, so I'll be interested in seeing the last in this fascinating list.

Add Comment      Leave a comment on this blog post
Feb 29, 2012 2:13 AM Corey Corey  says:

thanks for adding the top 5 challenges of social data integration. I am dealing with this topic now and there are not many articles on this.  You are right most are not dealing with this at all.

Feb 29, 2012 6:13 AM Jackie Kmetz Jackie Kmetz  says:


Thanks for the positive reception to my blog posts and sharing them with your audience.  As Corey said, there just isn't much discussion about it.  I have speculated that it is partly due on the business side to the fact that they aren't ready to reach into their pockets for the investment because of resources required and the still perceived "newness" of social and its ROI. 

Here is the final part of the series that posted today.  Enjoy!


Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.