Informatica Rises to the Big Data Management Challenge

Managing data even before it got “Big” was never all that easy. Now that IT organizations are not only starting to realize how much data there really is, thanks to technologies such as Hadoop that make it affordable to store it all, data management is rapidly becoming a crisis. The challenge is going to be finding a way to make any given subset of all that Big Data available at the just the right time that an application wants to invoke it.

At the Strata + Hadoop World conference this week, Informatica announced that it intends to rise to meet that challenge with the launch of Informatica PowerCenter Big Data Edition, a framework for managing data large and small across Hadoop and more traditional enterprise data repositories.

According to John Haddad, director of product marketing for Informatica, Big Data is really at the nexus of three converging megatrends: cloud computing, mobile computing and social networking. Trying to manage that volume of data using existing data management approaches simply won’t scale, says Haddad. What’s required is a new approach to data management based on a visual set of tools that extract, load and transform (ETL) data at very high rates of speed.

The goal, says Haddad, is to provide a layer of data management services that essentially masks the complexity of underlying data repositories such as Hadoop.

In a world where IT organizations are moving from actively managing 10 to 20 percent of their available data to managing all of it, the challenges are nothing short of immense. Unfortunately, most IT organizations are not going to get more people to throw at the Big Data management challenge, which ultimately creates a need for a data management framework that allows the existing IT staff to effectively manage massive amounts of information that will continue to grow exponentially as far as the eye can see.