There's a lot of legitimate concern these days about the ability of IT to process, store and manage huge volumes of Big Data. But if Jeff Jonas, chief scientist for the IBM Entity Analytic group, is right, managing massive amounts of information may be roughly akin to breaking the sound barrier: There's definitely a lot of turbulence along the way, but once you attain enough speed to break through, the ride actually gets a whole lot smoother.
The theory behind this thinking is that as more data gets collected, the more context there should be about that data. Jonas says that means that processing each new piece of data should not only get faster, because the system will already know the relationship between that data and information it has already processed, the amount of data being stored will start to fall because the systems will be able to make sense of the data in ways that recognize duplicate data that does not need to be stored over again. That capability, adds Jonas, should also revolutionize the way we think about business intelligence by making it easier to correlate information across massive amounts of Big Data.
This process, says Jonas, is similar to the way the human mind works on puzzles. As more pieces of the puzzle come into place, it takes less time to figure out where the next piece of the puzzle should go.
Jonas has more than a passing familiarity with working with large amounts of data. He founded Systems Research & Development (SRD), which IBM acquired in 2005. Prior to that acquisition, his team developed the technologies that the gaming industry now relies on: thwart card counting teams such as one that was formed by students from the Massachusetts Institute of Technology (MIT), which became the basis for the movie "21." Since then, Jonas and his team have worked with law enforcement agencies to identify rings of child pornographers, as well as links between different terrorist groups.
None of this means that there isn't a long way to go between where IT is now and the data-processing nirvana described by Jonas. Some estimates would suggest that we're going to need machines that are 10,000 times faster than what we have now. The good news is that if Jonas is right, our processing requirements may not be heading up and to the right on a linear basis forever. In fact, it's conceivable that as we start to collect massive amounts of data, managing it all could theoretically become a whole lot easier.