My colleague Loraine Lawson has been all over the rise of Hadoop for managing massive amounts of data.
Speaking at Hadoop World this week, Larry Feinsmith, managing director, office of the CIO at JPMorgan Chase, called Hadoop hugely promising, maturing quickly, and predicted it might overlap the functionality of relational databases over the next three years. InformationWeek finds credibility for Hadoop from the massive financial services firm, which has 245,000 employees, 24 million checking accounts, 5,500 branches and 145 million credit cards in use. It also has 25,000 IT employees. Yet Feinsmith refers to Hadoop skills in terms similar to that of my colleague Arthur Cole: "a rare and expensive commodity."
There are lots of SQL skills, SAS skills, and SPSS skills, but there are not a lot of [Hadoop] MapReduce skills.
Hadoop's headed for wide adoption - the article points out that mega-vendors IBM, Microsoft and Oracle have jumped into it this year. It calls Hadoop "a sound career path, much like SQL was 30 years ago."
The article quotes Jeremy Lizt, VP of engineering at Rapleaf, an early adopter of Hadoop, talking about the benefits for companies that train staff on Hadoop. And for those learning it, he says:
I think intelligent technologists will pick up Hadoop very quickly.
But the job security won't be limited to those who design and run the infrastructure. Feinsmith says scalability isn't the problem, but data security, entitlement and privacy provisions for a centralized resource provide big challenges. I see "big challenges" to mean job opportunities. And then there's the issue of gleaning valuable insight from all this data, which is creating a hot job market for analytics pros as well.