Hadoop Skills: Not the Same Old Java

I've been asking my colleague Loraine Lawson a lot of questions lately about what Hadoop is and isn't, trying to better understand it and the skills required. Probably a lot of folks are in the same boat.

She kindly passed along this SearchSOA piece, which says that though Hadoop requires Java skills, Hadoop probably will be a different type of Java development project. It says Big Data analytics shares similarities with traditional SOA and batch-oriented development.

Helena Schwenk, analyst at MWD Advisors, told the site:

Hadoop is "not about real time operational [business intelligence], but more about the discovery, exploration and analysis of large amounts of multistructured data."

She said it calls for more than a single platform or engine, but rather a variety of technologies, components and architectures. Among the skills and experience she mentions:

• Experience in large-scale distributed systems
• Knowledge of languages such as Java, C++, Pig Latin and HiveQL
• Predictive modeling, natural language processing and text analysis
• Data management
• Integration of structured and unstructured data
• A range of data-latency demands
• Architectural support for scalability and high-speed processing

It's unlikely you'll find all those things in a single person and it's quite likely you'll need to broaden skills on your staff with more training, she says.