Big Data, Cloud Pushing Adoption of Data Archiving

2014 Big Data Outlook: Opportunities and Challenges

In an unexpected twist, Big Data is driving adoption of data archiving, according to Gartner.

When people first started talking about Big Data technology, some said it would eliminate the need to worry so much about archiving or, at least, Hadoop clusters would take on that role. Ironically, it’s the increased adoption of Hadoop that is now forcing organizations to look at data archiving, CMS Wire reports.  Growth of structured data is particularly a concern as organizations try to separate out useful from non-essential data, the article notes.

Don’t worry, it’s expected to come full circle. The article notes that Gartner’s latest report, the Magic Quadrant for Structured Data Archiving and Application Retirement, predicts that the archiving needs will be so robust by 2017, 75 percent of structured data archiving applications will have to incorporate Big Data analytics.

The Big Data giveth and taketh, it seems.

I haven’t seen the actual report, but have only read CMS’ write up. Honestly, I’m curious about how this problem would be caused by Hadoop, since structured data was on the rise, with or without Big Data solutions. As one expert told me, organizations already had a Big Data problem before any of the Big Data tools emerged.

Gartner also found that data center consolidations, a shift to cloud-based applications, and organizational mergers and acquisitions are all factors in retiring redundant or legacy applications. This, in turn, gives organizations a chance to re-assess the cost of storing all this data.

At least, that’s the theory, but the article points out that in many enterprises, archiving decisions are more about chance than an actual policy-driven, coherent plan.

I suspect this goes back to the practice of archiving traditional, unstructured data based on the passage of time. I would think unstructured data requires a more sophisticated criteria. It will be interesting to watch over the next few years to see when and how Big Data affects regulatory compliance and storage.

Actually, this highlights an even broader problem: In general, most organizations simply do not have adequate policies about how to handle data and information, according to data management veteran Malcolm Chisholm.

 “The average enterprise does not have nearly enough policies for data management, and this famine of policies has an enormous detrimental impact on the way the average enterprise manages its data,” Chisholm writes in a recent Information Management column.

For CIOs, the good news is: It’s not IT’s job to fix this, he explains. Using operations as an example, he outlines why business units need to establish better data governance and information management policies.