The first vital step of any quality measurement program needs to be determining what measurements and metrics you will use to evaluate the performance of the system, once it is up and running. This may sound obvious, but many shops wait until a system is up and running to determine "success metrics." This may result in the need to retrofit some systems to support this report or, even worse, the measurements simply being ditched because they don't fit the implementation.
The National Institute of Standards and Technology addresses the importance of advanced planning in setting security metrics in its special publication, Performance Measurement Guide for Information Security. The in-depth, 80-page report is available free to IT Business Edge members here in the IT Downloads library.
The NIST report lays out the general process for setting measurements on the three key security metrics - business impact, efficiency and implementation - in the figure you see below.
As you can see, the process is cyclical - you're never done setting measurements for a process, particularly one as dynamic as network security, which has to change to address quickly evolving threats. You can see that there is no assumption that you will ever completely implement a security program (steps 4 and 5), and of course any changes to your implementation must change your performance measurements.
However, the entire process keys on review by stakeholders, many of whom are not from inside IT, to determine the goals and supporting measurements of your security program.
The report suggests that your stakeholders panel should include:
Each stakeholder may be assigned the task of developing two or three measurements specific to their line of the business - for example, the CFO may determine the business impact of a Personally Identifiable Information breach, while security engineers would be more concerned with metrics specific to system implementation.
From the goals set by the stakeholders panel, your organization will develop a set of policies and guidelines to enforce the measurements set. These policies will be quite granular, and must be enforced rigorously. (Be sure to check out the Security Assessment Policy Template from our partners at Info~Tech Research Group if you are in the process of setting monitoring policies.)
A key to making sense of all the data that this process will generate is prioritizing your security measurements. Again, making sure that tactics implemented to protect private data are working may be more important than pretty much anything else you have going on. Stakeholders will be involved in this prioritization process, of course. This process goes hand-in-hand with setting targets, or benchmarks, for your defined measurements. Industry standards are most useful in this process, but you might have to rely on a first reading of your implementation as a starting point.
The report goes on to detail the steps for a successful implementation and monitoring project. As with all NIST publications, it is an extensive resource that is well worth the read.