IoT Data: Insights Are Hard to Achieve in an Ocean of Possibilities

Loraine Lawson
Slide Show

The Challenges of Gaining Useful Insight into Data

The Internet of Things (IoT) will create oceans of new data — but finding meaning amid the swell will be challenging.

That’s what some companies are finding when they deal with unstructured data in general. Despite the success stories, a recent Forrester survey shows our ability to manage and use unstructured data significantly lags behind traditional, structured data. Forrester found that on average, enterprises use about 35 percent of their structured data for insights and decisions. By comparison, only 25 percent of unstructured data is leveraged for those tasks.

Perhaps that doesn’t seem so bad, given how long enterprises have invested in structured data. But focus on only the IoT data, and the picture becomes even murkier. A recent Dimensional Research study, sponsored by IoT analytics vendor ParStream, asked 203 IT professionals, including systems integrators and IoT stakeholders, about their IoT projects. They found that only 8 percent agreed that they’re fully capturing and analyzing IoT data in a timely manner — even though an incredible 86 percent said data is important to their IoT projects.

Despite the low number of IoT projects capturing data, 70 percent of respondents said improved data would lead them to better, more meaningful decisions. I have to wonder how that’s going to happen if only 8 percent are actually capturing and analyzing said data in a timely manner.

It seems the reality of collecting and using IoT data is much, much trickier than success stories would lead us to believe.’s guest column, “Are you ready for the Data of Things,” explains why the Data of Things (DoT) is much harder than the Internet of Things.

“DoT will be disparate, messy, in constant flux and inherently incomplete in nature,” writes Kumar Srivastava, senior director of product management at cloud Big Data analytics company, ClearStoryData. “DoT will be sparse, compacted and compressed. It will require being contextualized, decorated and described with data and metadata from other data sources.”

On top of all that, the quality of the data will be “unpredictable,” due in part to variances caused by the IoT’s routing mechanisms, he adds.

Organizations will have to offset these problems by blending or harmonizing IoT data with other data sets. These additional data sets will provide the context — time, environment, location, or other related details — about the IoT data, Srivastava explains.  

Honestly, it sounds like a nightmare. IoT data might not be worth the effort if it weren’t for the capabilities it promises.


“DoT will enable micro segmentation, micro messaging and micro behavior management,” Srivastava writes. “That is, enterprises will be able to control the behavior of micro segments across their network of connected devices, apps and systems.”

Of course, these micro segments will also strain your existing data pipes and processing systems. But, at least you know what you’ll get in exchange.

It seems clear that organizations understand the value of IoT data, but they’re unclear how to actually get from point A (collect the data) to point C (derive value from the data). Until that’s resolved, IoT projects may prove more frustrating than useful.

“The intersection of IoT and Big Data is creating tremendous business opportunity, but the reality of today’s projects is that this potential has yet to be fully tapped,” Diane Hagglund, Dimensional Research principal, told ParStream.

Loraine Lawson is a veteran technology reporter and blogger. She currently writes the Integration blog for IT Business Edge, which covers all aspects of integration technology, including data governance and best practices. She has also covered IT/Business Alignment and IT Security for IT Business Edge. Before becoming a freelance writer, Lawson worked at TechRepublic as a site editor and writer, covering mobile, IT management, IT security and other technology trends. Previously, she was a webmaster at the Kentucky Transportation Cabinet and a newspaper journalist. Follow Lawson at Google+ and on Twitter.

Add Comment      Leave a comment on this blog post
May 7, 2015 1:44 PM dcunni dcunni  says:
Hi Loraine, I think you're going to see more and more about the intersection of IoT and Big Data. Not sure DoT is the right way to say it though. I prefer, "You can't get Things in a data warehouse!" SnapLogic introduced a set of Snaps this week for MQTT and plans to support additional IoT protocols in the coming months. It's IoT meets iPaaS. I blogged about the announcement here: Reply
Apr 9, 2016 11:42 AM jpnorair jpnorair  says:
The only IoT protocol/stack that is designed from the get-go for DoT is DASH7. Haystack Technologies is the company that invented it, and worked it through alliance and standardization activities for several years now. DASH7 implements a JSON-ish distributed database right on the sensor endpoint, along with real-time notification and query features. It also runs in the same power-profile as BLE, so years on a battery. It took nearly a decade of engineering effort, from contributors around the world, to make it work. MQTT is little more than an application protocol. If you have a lousy stack underneath it, the performance of MQTT becomes lousy. The only IoT stack that can make MQTT work in such challenging environments is DASH7. Reply

Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.