Crowdsourcing Can Help Companies Find Scarce Data Analysis Skills

Share it on Twitter  
Share it on Facebook  
Share it on Linked in  
Slide Show

Nine Predictions for the Analytics Industry in 2011

The competitive gap between analytical innovators and those who do not invest in analytics will widen over the coming 12 months.

When I wrote about Netflix's 2009 competition to find an algorithm to improve the accuracy of its recommendation system, I noted there were plenty of winners other than the obvious one, the international team of statisticians and engineers that snagged the million-dollar prize.


Although his team did not win, the CEO of data analytics company Opera Solutions said the company got "a $10 million payoff internally from what we've learned" by using improved modeling and analysis techniques it created for the contest with its paying clients. And Netflix got a pretty sweet return on its investment. Companies like Netflix could expect to pay $1 million to hire five researchers for a year. As Darren Vengroff, a former lead researcher for Amazon's recommendation engine, said in a Forbes article, Netflix "spent the same amount and got thousands, probably millions of engineer-years."


So it's not too surprising to see other companies trying to attain similar R&D returns by offering big bucks to folks who can help figure out the answers to complex data analysis questions.


As The Wall Street Journal reports, California physicians group Heritage Provider Network Inc. is offering $3 million to the participant who develops the best model to predict how many days a patient is likely to spend in the hospital in a year's time. Contestants will receive "anonymized" insurance-claims data to create their models.


Heritage executive Jonathan Gluck says the goal is to reduce the number of hospital visits, by identifying patients who could benefit from services such as home nurse visits.


Like Netflix, Heritage decided to try the contest format to maximize its R&D spending. It also hopes to attract creative thinkers it might not otherwise have known about, says Gluck.


Heritage is working with Kaggle Pty Ltd., an Australian startup that charges companies about $10,000 (plus consulting fees) to design contests and prepare the raw data. It's conducted some 15 contests in the past 11 months, including one for Ford Motor Co. Its client roster also includes the National Aeronautics and Space Administration and Wikipedia.


Such contests might become even more popular, given the apparent shortage of people with data analysis skills. The percentage of job starters on LinkedIn with titles related to analytics and data science has grown more than 40 percent in the past year, according to The Wall Street Journal. Kaggle CEO Anthony Goldbloom says he found a recent hire by reviewing a contest leader board.


IT Business Edge colleague Susan Hall discussed the dearth of data analytics pros in her recent interview with Jack Phillips, CEO of the research firm International Institute for Analytics. Not surprisingly, he told her health care is one of the industries most keen to use analytics as a competitive differentiator, along with banking, insurance, telecommunications, retail, energy and utilities, transportation, and media and entertainment. It's difficult to find qualified folks, as the requirements for analytics positions are still evolving.


Phillips said professional organizations and universities are just beginning to address the shortage. Like many emerging IT roles, data analytics pros will need a healthy dose of general business smarts. He said:

... At the very core, you're talking about statistics, econometrics and mathematics. Then couple that with a business background that gives you the top-level "what questions are you trying to answer? What business problems are you trying to solve?" So the curriculum at that early level is being absolutely savvy in regression analysis, econometrics, mathematics, but also have a very firm sense of where analytics can really have an impact-either in increasing revenue or decreasing costs. Those are the two big levers that analytics can pull on.

The IIA predicts a Chief Analytics Officer role will emerge to help guide companies' data strategies in 2011, Phillips said.


The kinds of contests conducted by Netflix and the Heritage Provider Network shouldn't be seen as panaceas, warned several academics in an MIT Sloan Management article earlier this year. While they do generate a larger pool of ideas than can be produced internally, organizations must be willing to invest the time and money to build this kind of open innovation approach and also to resolve any intellectual property ownership issues that might result from using ideas contributed by outsiders.


Legal issues can be potentially hairy, as seen by Netflix's decision to scratch a second innovation contest after the first contest led to a lawsuit over giving outsiders access to Netflix customer data. Companies must learn when to use internal resources and when to use external ones, wrote the academics in the article.