You might have read the IDG story in The New York Times earlier this week that a Singapore bank suffered a massive IT failure. Since I live in Singapore, first-hand accounts from friends unable to withdraw much-needed funds from ATM machines were filtering to me even before the first news reports were written.
And a massive failure it was; it appeared that all IT aspects of DBS Bank's operation were affected, ranging from services at bank branches, Internet and mobile banking facilities, the use of ATM machines and an inability to pay by credit card or a popular electronic debit card system called NETS.
In fact, it is hard to believe that such an all-encompassing failure could even happen. According to a report from local newspaper, The Business Times, DBS Bank has a customer base of 3.2 million and operates 81 branches and 970 ATM machines. In addition, the bank has 1.35 million Internet banking customers, which gives it 35 percent of the Singapore market.
With ongoing investigations, actual technical information of what really happened is not available at the moment. What is known though, is that the bank has a 10-year outsourcing agreement with IBM valued at SG$1.2 billion (about US$870 million) when it was signed in 2002.
While it is too early to start pinning the blame on any party at this point, the sheer magnitude of the failure suggests there actually were multiple failures that cascaded out of control. Also, reports so far indicate that "multiple levels of redundancies" were in place, though they apparently did not kick in. In addition, a redundant facility exists, though, again, it was not clear why it wasn't activated.
The repercussions are likely to be severe as uptime for online banking services for banks in Singapore translates to about 99.99 percent systems availability or better. This works out to be less than 53 minutes of downtime per year, while the total outage in this instance was 7 hours before service was fully restored. The fact that the problem started at 3 a.m. mitigates the business impact somewhat, though it will likely cut little slack with the country's central bank, the Monetary Authority of Singapore.
If the lesson from Amazon's recent outage is about the continued relevance of making business continuity plans, the lesson here is about the need to rigorously test disaster recovery plans and to conduct periodic checks that failover systems continue to work as designed.
And while it clearly does not pertain to banks here, I recently wrote about whether SMBs really need 99.999 percent uptime; you can also check out some practical tips to ensure business continuity here.