Reinventing the Business Intelligence Process

Michael Vizard

Data without any context isn't all that useful. Unfortunately, most of the data that we do collect in the enterprise has little to no context.

To compensate for this, IT organizations have been investing billions, maybe even trillions, of dollars in data warehouses and business intelligence applications. We are collecting more data than ever, which in turn is in straining not only our ability to make sense of that information, but also our storage budgets. What if we took an entirely different approach to collecting data in the first place that was inherently more efficient in terms of assigning context to information? That's the strategic goal of a G2 project that Jeff Jonas, IBM distinguished engineer and chief scientist for Entity Analytics, is working on.

Jonas argues that one of the reasons that IT is fundamentally flawed is that we keep collecting data and then trying to ascribe context to it after the fact. Working with other elements of IBM, Jonas is developing a system where context is ascribed to each piece of data as it is collected. That new data then informs the rest of the applicable data sets about its context, which in turn would continuously update the context surrounding the data set, but also inform other related sets of data about changes in context. IBM acquired much of the capability to do this when it bought SRD, which was founded by Jonas, in 2007. Under this new project, Jonas is researching how to apply that technology, currently offered in the form of IBM's Identity Insight Server, to a much broader range of entities, including names, places and events.

On one level, the idea is pretty simple. In order to acquire knowledge, systems need to correlate information. To achieve context today, we take samples of data sets and make a "best guess" about the context of the information. That best guess, however, is very dependent on both the quality of the information and the assumption that there are no missing variables that could render the data model irrelevant. Given how often end users complain about how IT reports don't reflect the true state of the business, clearly the amount of poor quality data coupled with missing critical components of a data model is having a negative impact on the value that end users place on BI applications. Many of them would much rather rely on spreadsheets, where they can better assign all the variables, than a report from IT. It's only when the problem scales beyond human ability to process on a spreadsheet that we see business users acquiescing.

Jonas says we could eliminate much of this distrust of the data by assigning context to each piece of data as it's collected. Each new piece of data would then be, first, identified in context with existing data in the system and then used to update the rest of the system. In turn, that would not only reduce processing requirements because each new report would not have to reprocess the entire data set, it would let us work with much larger sets of streaming data using less processing power to actually provide more real context than ever in less time.

More importantly, it would become possible for systems to have a memory of processes, thus overcoming what Jonas describes as the "corporate amnesia" that plagues all our IT applications. In effect, systems would be able to correlate data sets representing separate processes to identify patterns of interest in real time without having to wait to respond to a specific query.

We're still in the very early days of reinventing how we process data. The issues at hand here are as much cultural as they are technical. But as the former CEO of SRD, Jonas says IBM has the best seat in the house to realize this vision now that the company, under the auspices of its "Smarter Planet" initiative, has added SPSS and Cognos to its already formidable investments in business intelligence. And to its credit, while other vendors are touting the importance of business intelligence, IBM seems to be the only vendor spending billions of dollars to fundamentally try to improve it.

Add Comment      Leave a comment on this blog post
Nov 30, 2009 8:52 AM @chrisspivey @chrisspivey  says:

I think there's a tremendous amount of value and insight in a very few words here!

If "user requirements" for a system include context and processes are truly documented, i think IT has a chance.  Hammer and Champy define process as 'a collection of activities that takes one or more kinds of input and creates an output that is of value to the customer.'  If the customer can define and document context, the system can be programmed  to deliver the output required.  the trick, I believe, is in helping the end users define "context".

I'd be interested in further discussion about frameworks and experience in defining "context".  If they can "assign the variables" in a spreadsheet, then how do we translate that into application code?

Thanks for a thought provoking post Mike!

Dec 1, 2009 6:23 AM Jeff Jonas Jeff Jonas  says: in response to @chrisspivey


You will find my thinking about exactly how context is determined here in this link:

     Context: A Must-Have and Thoughts on Getting Some

And you might find this one useful too:

     It's All About the Librarian! New Paradigms in Enterprise Discovery and Awareness

And by the way, context is not primarily domain specific - it is at least at its most basic level very universal.

Dec 1, 2009 6:35 AM Jeff Jonas Jeff Jonas  says: in response to Arnold Kransdorff


Glad you commented and we have connected.  I have been using the term enterprise amnesia for a while now but I see you have been using the term corporate amnesia much longer.  Kindred spirits ... although I may be the late bloomer.

Re: learning is incremental, oh I do agree.  Here are a few thoughts I have had on exactly this point:

Smart Systems Flip-Flop

Accumulating Context: Now or Never

Big Breakthrough in Performance: Tuning Tips for Incremental Learning Systems

Would be curious how to capture and characterize human observations in an easy manner with low overhead to us humans!

Dec 1, 2009 6:45 AM Arnold Kransdorff Arnold Kransdorff  says: in response to @chrisspivey

Jeff Jonas is spot on when he points out that data/information, even knowledge, needs context to be useful. It's a field I've been working on for 25 years. Without an accurate and contextual organizational memory (OM), effective experiential learning is impossible. In truth, experiential non-learning is the norm in today's workplace, no thanks to individuals' short, selective and defensive memories and the attending modern phenomenon of the flexible labor market, which has ensured that the awareness of 'context' walks out of the front door on average every four to five years.

The work I€™ve done marries up two separate disciplines €“ the better management of organizational memory (OM), which addresses the contextual component of what€™s missing in the collection of data/information/knowledge, and experiential learning, using the acknowledged reflective methodology of Professor David Kolb that I€™ve adapted to modern workplace practice. It provides a formalized way to apply prior practice to capitalize on how most learning takes place €“ incrementally.

The processes are outlined in my three books on the subject, below.


Arnold Kransdorff.



+ "Knowledge Management: Begging for a Bigger Role" (Business Expert Press 2009)

+ "Corporate DNA" (Gower, 2006)

+ Corporate Amnesia (Butterworth Heinemann 1998)


Post a comment





(Maximum characters: 1200). You have 1200 characters left.



Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.