In this Help-Net Security piece, Nick Baskett, the managing director of security firm Matta, says that since 2004 the company has been running Sentinel, a program that tests the technical competence of consultants. The writer dresses everything in very polite terms, but the penetration testing capabilities of the consultants run through the program vary greatly.
Organizations use the Sentinel program to find consultants or rate folks who are working internally. Penetration testing -- adequately defined in this CyTRAP Labs post -- focuses on attempts to identify security problems by prying through tiny cracks in an organization's defense. The results of the Sentinel testing were a bit disconcerting to Baskett:
[P]erhaps the most startling fact of http://www.itbusinessedge.com/blogs/top//?p=165&preview=trueall is that every consultant who has gone through the test has always found vulnerabilities with their tools, which then failed to make it on to their final report.
Like two service stations that come up with different diagnoses of a pinging sound from under the hood, folks paying for penetration testing, either from an outside company or through the IT department, are getting a differing level of service.
Baskett provides examples of the testing. He also points to two possible rationales for the problems: One simply is that testing tool output can be difficult to interpolate. The other, which he said is "really concerning," is that some users didn't appear to read the briefing notes they were given in order to participate.
Penetration testing is a broad category. This post on one level is a review of a book about this discipline. Though it's a bit unorganized, it provides something of great value: a listing of many of the things that penetration testing would aim to find.
Gartner analyst Joe Pescatore, quoted in this ServerWatch piece, suggests that e-commerce companies have outsiders perform penetration testing (which also goes by the much more evocative name of "ethical hacking") annually. Big companies should add specialized tests. The story positions penetration testing -- the next step after vulnerability scanning -- as having grown during the past year. The piece says both commercial tools and freeware such as the Metasploit framework 3 can be used.
This column at Microsoft Certified Professional Magazine by Verizon Business employee and well-known security expert Russ Cooper raises some interesting questions. Running penetration tests isn't necessarily difficult, but it isn't always appropriate or even legal. Cooper mentions that a working group sponsored by the Computer Security Institute has found that penetration tests are illegal (presumably without an agreement with the company being probed). Cooper explores this and the laws and ethics governing this area.
Another well-known consultant, Bruce Schneier, posted this commentary on his blog. He says some security experts say penetration testing is vital and some say it is a waste of time. He claims to come down in the middle and suggests that penetration testing is far more nuanced and covers more ground than many assume.
Schneier's comments suggest he's more on the skeptics' side. He says running a penetration test and then not opting to fix identified problems can lead to legal issues if a breach ensues. (It is fair to say, however, that it may be just as hard to justify not having taken steps out of fear of finding probles.) In any case, Schneier says he is willing to give a free penetration test: "You're vulnerable," he writes. "Now, go do something about it."