Data Integration Remains a Major IT Headache
Study shows that data integration is still costly and requires a lot of manual coding.
You're probably familiar with Pandora, the leading Internet radio service where you can use an artist or even just a song to create your own radio station on the fly. I fell in love with Pandora about six months ago, when I discovered my Alison Krauss station would - without fail - lull my baby boy to sleep.https://o1.qnsr.com/log/p.gif?;n=203;c=204663295;s=11915;x=7936;f=201904081034270;u=j;z=TIMESTAMP;a=20410779;e=i
What you may not know is that Pandora isn't just an Internet-based service - the company is completely dedicated to the cloud. It uses all hosted apps, including Salesforce.com, Zuora, Operative, Box.net and so forth. It's an approach that is low cost, flexible and easy - but as you can imagine, a completely cloud-based approach brought with it unique integration challenges.
One of the more troublesome problems it created was reconciling data between Salesforce.com's sales, leads and account data and the data in Operative, a cloud-based, end-to-end, advertising and business management platform. Pandora actually had to do this manually, which, of course, was a huge source of frustration and a time waste.
Pandora turned to SnapLogic, which offers both a cloud-based and an on-premise data integration solution.
Pandora is one of SnapLogic's more interesting use cases, according to John Schuster, vice president of engineering at SnapLogic. Schuster recently shared his views on how cloud integration is challenging existing integration approaches, such as ETL and EAI.
"They made a philosophical decision to not use any legacy, on-premise business applications. They've found the best of breed cloud applications that they want to run their business on and they've constructed SnapLogic pipelines to pump the data from and to all the applications that need it. This does happen in near-real time," Schuster said.
Most integration solutions can be put into some sort of camp. Oh, sure, you may really have to dig deep to find, but buried down there somewhere is some sort of enterprise service bus, ETL, enterprise application integration approach, or some sort of recognizable integration approach.
SnapLogic is one of those unusual companies that's challenging to categorize. It has ETL capabilities, like standard transformations, joining sources, filtering. But they don't call themselves an ETL. Schuster said it's also like an ESB, in some ways, but SnapLogic isn't really used in the high-speed transactions you might see with ESBs. In fact, it's often used in conjunction with an ESB.
And it's not exactly data virtualization, although Schuster says it is "well-suited to feed the data into that layer on the back in so that a data virtualization product could in fact take the data and allow the customer to provide governance, security controls, logging, all the things to the data."
Gartner, in its Magic Quadrant for Data Integration Tools, simply says this: "DataFlow supports real-time and federated integration of data with a focus on diverse data sources, including SaaS- and cloud-based sources, and via Web-oriented architectural approaches."
However you describe it, one of its strengths, according to Schuster, is that it's built with REST-based concepts, and uses REST internally to communicate between components. This makes it easy to distribute in clouds, private and public, and across firewalls, he added.
That ease of connectivity is one thing that prompted Pandora to choose SnapLogic.
"Their whole company is running on SnapLogic technology. That's really exciting for us and not only is it a fun, complex example, but I personally find it impressive that we're being deployed in mission critical applications and business process in a company that's so forward thinking and has only cloud applications," Schuster said.