I was talking to a security specialist from one of the largest aerospace companies in the world and he told an interesting story. Fortunately, it wasn't about his own company so I can share it. It quickly showcased the amazing opportunity and incredible danger of the public cloud and the fact that the people often trading one off against the other not only generally don't have the authority to do so, they don't seem to realize they are taking extreme risks.
Like all things, there clearly is a balance here. I'm not advocating for or against using this new tool; what I'm suggesting is that policies need to be updated and employees trained in them so that the use of the tool doesn't get a lot of people fired and/or put the company in severe and embarrassing financial distress.
Evidently, some researchers at a major pharmaceutical company had a problem. They had a project that required a rather large amount of computational resources to complete, but the resources weren't available to them. First, they went to IT. Based on how they scoped the project, the budget was set at $50,000 with a timeline to completion of several months. They might have been able to come up with the money but the time was a problem, so they went to the cloud and rented these same resources. They quickly discovered that their initial estimate of what would be needed was off enormously, increased the resources they were renting significantly, and still completed the project in under two weeks (when spread across the proper amount of resources and run during low load times, the actual job took hours). They put the cost of the project on a credit card and it came to about $80. (Granted, I think this is a bit optimistic but I wasn't telling the story.)
That's a $50,000 project with months to complete (and factoring in the fact that they underestimated significantly the project resources and that they didn't realize this until they first ran the job, we are probably talking well over $250,000, and something over six months to account for the reset) vs. $80 and two weeks. Pretty amazing, huh? But a story of huge savings is hardly uncommon.
Remember, I said this was a pharmaceutical company. This industry, because of the nature of the work, has security requirements that the U.S. National Security Agency (NSA) likely envies. While the researchers got the results, they have no idea where, geographically, the work was done or who else might have access to it. It could be up on the Web in a catalog of interesting information in a variety locations at the moment and they would have no way of knowing.
While it is hard to assess the value of the project to the pharmaceutical company, typical projects like this generally are valued in the millions of dollars. So they may have saved over $250,000 but created a potential liability well into seven figures. And therein lies the problem; no one working on the project apparently even considered this. While this is evidently being kept pretty quiet at the moment, you can expect how relatively pissed off a whole variety of folks with big titles would be if they got this story. In many ways, these guys are heroes because they completed a valuable project on time and massively under budget. But instead of being rewarded, they are likely at high risk of being fired.
Perspective: It's a Process Problem
Now to be clear, if they hadn't gone down this path, this multi-million dollar project wouldn't have been worth a thing because it wouldn't have gotten done. Even the multi-million dollar part may be a stretch, but if you've ever been on the wrong side of one of these things, the zeroes on the liability seem to breed and grow exponentially and have little to do with any reality -- because it turns into a witch hunt, with you as the witch.
The company now has an asset it wouldn't have had if this project hadn't been completed, but there may have been a way to do this that might have cost a bit more and protected the data. There might also have been an executive with the authority to sign off on the risk to use the method that was actually used. However, neither option was explored so we may have two unemployed heroes in a few weeks.
The fix is not to stop using the cloud but to put in place policies and resources that balance the financial costs and savings with the reasonable risks a certain path might take. Companies that provide cloud services and comply with corporate security guidelines, private cloud resources that can be made available inexpensively to internal customers, and very clearly understood escalation and approval guidelines can go a long way to insuring your internal heroes survive. A real hero doesn't rescue the princess under budget while pissing off the dragon so it later eats them both. He lives happily ever after, and that is the goal here.
Wrapping Up: Living Heroes
The cloud provides some amazing opportunities but it represents a largely unknown risk to many. The job is to understand the technology and the risk, manage the risk intelligently so saving some money doesn't put at risk a lot more and, in all honesty, indulging in CYA. While it's great to be a dead hero, it's even better to be a living one. Here is hoping that when your chance comes, you'll be one of the latter.