Crowdsourcing Big Data Analysis

Susan Hall
Slide Show

Big Data Analytics

The first steps toward achieving a lasting competitive edge with Big Data analytics.

FINS has a piece on the rise of temp work as companies bring in staff for specific projects and use contracts to try out workers to see if they're a good fit for the organization. That's certainly not new in IT. In InformationWeek's 2011 IT Outsourcing Survey, which I wrote about in April, 72 percent of respondents said they use "gig" contractors, temporary workers who perform key tasks on a long-term basis. But companies often can't afford, or don't have permanent need, to hire some of these project folks outright.


A startup called Kaggle has come up with an interesting solution to the difficulty in finding analysts who can glean insights from companies' volumes of data. Kaggle, which just raised $11 million in venture capital funding, crowdsources Big Data problems by setting up contests among Ph.D.-level data scientists, according to GigaOm. The company says it has a community of 17,000 people around the world working on big problems.


The New York Times quotes Kaggle founder Anthony Goldbloom, saying:

We're really making Big Data into a sport.

Among the contests:

  • A health plan is offering $3 million to the person who can, from years of medical claim data, create a formula to predict who will be hospitalized in the next year so that doctors can intervene.
  • A bank is offering $5,000 for a better way to predict from credit score data whether a person who gets a loan will be in financial distress within two years.


The Times reports Kaggle participants have outperformed academic research on how long people with HIV would live, found new ways to map dark matter and revealed that people who order vegetarian meals from airlines are more likely to show up for the flight. For the data scientists, it's a way to bring attention to their skills - and score some cash.


Kaggle charges the companies a fee for the competitions and is looking at taking a cut of the prize money in the future. For companies that don't want to put their data sets online, it offers private competitions in which every participant gets paid. Only scientists who have passed a background check, are held to non-disclosure agreements and have "performed extremely well" in other Kaggle competitions will be allowed to work on the private contests.

Add Comment      Leave a comment on this blog post

Post a comment





(Maximum characters: 1200). You have 1200 characters left.




Subscribe to our Newsletters

Sign up now and get the best business technology insights direct to your inbox.