You Can't Know the Unknowns in ROI Analyses-but You Should Try

Ann All

In today's post about companies successfully using social CRM, one of the examples I cited was Entarasys Networks, a company where the sales team closed a record number of deals in the first quarter after it implemented the Salesforce.com Chatter collaboration tool. An item on Software Advice suggested the sales increase was due to the team working together more closely.


I'm sure that had an impact, but what else was going on that quarter? Did the company offer some especially enticing promotions? Did a competitor go out of business? Did a market force, perhaps a regulatory change, mean more customers needed its data networking services?


It's hard to say. And that's a fundamental problem with ROI analyses.


Steve McDonnell makes this point on Tibco Spotfire's Trends and Outliers blog. He's writing specifically about business intelligence, which has a notoriously hard-to-quantify ROI. But the advice he offers applies just as well to other technology projects.


If you can't create a controlled experiment to demonstrate a cause and effect between BI and a measurable metric like sales or profitability, McDonnell says, consider other factors that might influence success and include them in your analysis to let people know you considered them. This will "probably help increase the validity of your business case when you present it to others," he writes.


When I interviewed IT consultant Freddy Fam, he mentioned the importance of determining the factors-both internal and external-that could influence ROI in order to isolate them from the ROI determination. Yet this often ignored, he said:

Typically, when a ROI is carried out, the data collection is normally based on quantitative data and not so much of qualitative data. From qualitative data, one can sometimes realize that the quantitative data that was collected does not match with what was collected during employee interviews and observation. This means the data contradicts each other. Like I said, it's the practice of simplicity. Isolating project interventions takes more effort and time to execute, especially when it comes to large projects, but this cannot be ignored. Ignoring it would be the same as not carrying out ROI measures because the result is "polluted" with impurities.

Obviously determining ROI can be tricky. But, as IT Business Edge's Loraine Lawson wrote, don't be tempted to ignore it.


Remember Secretary of Defense Donald Rumsfeld's famous (or infamous) quote? He said:

There are known knowns. There are things we know we know. We also know there are known unknowns. That is to say we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know.

It's the unknown unknowns that usually present the biggest problems-in ROI analyses or anything else. I don't think there is really any way to account for those. But smart IT organizations should do their best to try and think through the known unknowns.

Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.


Add Comment      Leave a comment on this blog post
Feb 26, 2011 9:17 AM Christopher Christopher  says:

This is exactly one of those things that upper-end IT people with no real world experience have a trouble understanding. It's hard to measure virtual increases in productivity. They just don't get it. There is no bar graph or count of things through the usage of say social media etc.

It's like backups & disaster recovery, there is no 100% foolproof backup. If these folks had more real world experience prior to jumping into this stuff, maybe they would understand the difficulty in gather ROI on some implementations.


Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.

Subscribe Daily Edge Newsletters

Sign up now and get the best business technology insights direct to your inbox.