The problem we have with data is not our ability to store and manage it, but rather our inability to correlate it. Without correlation, the data has no meaning. And if the data has no meaning, business people start to ask some pretty tough questions about the value of IT.
One reason that we can't correlate data is because there are so many conflicting types of information. We have reams of structured data that resides in a database, and with each passing day we add massive amounts of unstructured data in the form of e-mail and a host of emerging Web 2.0 applications.
And yet, all most business people really want to be able to do is look at all the structured data about a customer alongside all the unstructured to determine what exactly is going on with that customer.
One company providing a framework to do that is Attivio, which provides a framework for delivering end users something relatively simple like being able to dynamically mash data together or build complex composite applications.
The Attivio Active Intelligence Engine (AIE) is designed by a team of developers led by company CTO Sid Probstein that built a framework for managing information that, rather than forcing everything into the same schema architecture, pulls data in its natural format into a centralized system. Database tables are presented in full alongside e-mail messages because there's no need to normalize data or flatten everything to work with a proprietary search engine.
AIE has a search engine that users can invoke to find data in the system, but the real value the system provides is that it is agnostic when it comes to schema, which ultimately makes it easier for end users to analyze multiple sets of data in the context the data was first gathered, said Probstein.
AIE includes application programming interfaces that allow developers to build applications on top of the core engine or integrate the data within AIE with other applications. A forthcoming AIE 2.0 offering, which is in limited release now, adds faster indexing, SQL and JDBC support.
Attivo CEO Ali Riaz says the real trick to building AIE was bringing together developers with background in structured and unstructured data and letting them develop a solution to managing data that leveraged each other's disciplines.
At the end of the day, says Probstein, the fundamental issue that Attivio is trying to solve is that most knowledge workers spend more time gathering data than analyzing it. Unfortunately, it seems like the vast majority of our enterprise applications are about gathering more data than ever, rather than trying to make sense of it.