The Forgotten Critical Step on the Way to Real-Time Data Integration


If you're considering a real-time data solution, you'd be well advised to first add high-availability data recovery.


I learned that during a recent interview with Sami Akbay, vice president of marketing and product management for GoldenGate Software, which provides both types of solution. How I learned it is a bit of a funny story, actually.


Akbay's PR people contacted me after reading about Ken-Hardin's frustrating attempt to hunt down an Espresso four-piece high-top dining set, sold by a leading national discount retailer. They thought GoldenGate could provide some insight into why retailers still don't have access to real-time information, despite UPC scanners and RFID tags and all manner of technology that would seem to make hunting down an Expresso four-piece high-top dining set easy.


They were right. Akbay did a great job of explaining how most retailers do business, opting for overnight ETL runs instead of using real-time data integration. He offered a reason for why more retailers aren't using real-time data:


"It's adoption. It takes time for people to move to real-time technology. ETL has been in use since 1990, whereas this type of technology like ours has only been available in the last three or four years. We do work directly with ETL vendors because people do have millions of dollars in investments into their ETL systems, so in some cases, we will just feed the ETL system so they can do a gradual change, as opposed to a big-bang change."


He also offered an example of what happens when a retailer does things right-citing a GoldenGate client where he likes to shop. At the end of the interview, Akbay asked which retailer Hardin was referring to in his blog post, saying he thought he'd contact it to offer a real-time solution. I promised to find out and e-mail him.


We were both surprised, then, when his client and Hardin's big box retailer turned out to be the same company.


Obviously, I couldn't run the interview knowing that, so we arranged a follow-up call. Akbay was understandably a bit sheepish when he admitted he'd been mistaken. He assumed the retailer had opted for the real-time solution, when, in fact, it had only installed GoldenGate's High Availability solution. He thought it planned to move to real-time, however.


The High Availability solution is a data-recovery product. It ensures you've got continuous database availability, so you can get up and running after a system crash or other problems. Akbay described data recovery as a critical step before adding real-time data integration:


"When you start using real time information and you're feeding data throughout the day and the system crashes, then going back to last night's information or this morning's information means that you've lost everything that you did since the last time you took a backup. So because of that, when people implement these real-time data-integration technologies, when they start updating the systems all the time, they first have to address the continuous availability or high availability, otherwise if they have a crash, they'll lose information."


Companies often skip this step and go straight to real-time, which can lead to serious problems if something goes wrong. Akbay said the retailer in question is actually taking the more prudent approach by adding high-availability first:


"This is how we should do it and it seems to me that they're taking the approach of build the infrastructure and then add the features and functionality. Whereas, if you jump the gun and do it the other way, you have the risk of losing information, and losing data and inventory information may not be as detrimental, but if you lose customer data, then it could be really detrimental for your business."


The moral of the story, then, is get your ducks in a row with data recovery before you move to real-time data integration.


With the big-box retailer situation cleared up, we ran the original Q&A, "Real-Time Retail Data: What's Taking So Long," last week. For further reading, you might also want to check out this Enterprise Systems interview with Akbay, which explores how an operational BI framework ties into real-time integration and best cost practices for implementing real-time data integration.