Getting Smart About Data Mining

Share it on Twitter  
Share it on Facebook  
Share it on Linked in  

New developments in business intelligence and data mining are making it possible for even non-specialists to garner valuable insights by drilling down into information in ways that wouldn't have been possible in the recent past.


Yet many companies still find it more costly and complicated than anticipated to build data warehouses. And they often fall short of expectations. Companies expect too much of their analytics tools, says Peter Fader, a professor of marketing at the University of Pennsylvania's Wharton School, in an interesting Baseline Q&A. While data mining is good at spotlighting the differences between groups of people, he says, it can't predict the specific behavior of individuals.

There's a tremendous amount of intractable randomness to people's behavior that can't be captured simply by collecting 600 different explanatory variables about the customer, which is what data mining is all about.

Much like scientists who think they can discover the cause of every disease if they just get enough of the right kinds of data, companies suffer from the delusion that they can use data mining to answer any question about human behavior, Fader says.


Many wrongly focus on one-to-one marketing, because they think the data reveals enough about customers to create promotional offers targeted to their personal needs. Yet this type of marketing only works if the company already has a close relationship with the customer.


Otherwise, Fader says, "the cost of trying to figure out what specific customers will do far outweighs the benefits you could get from that level of detail."


He advocates the use of probability models, which can be crafted using simple Excel sheets, to refine data before introducing data mining tools to help determine customer propensity for certain types of behavior. While spreadsheets have gotten a largely negative rap in recent months, Fader says they provide a logical starting point for more sophisticated analysis.