While some people consider hacking a 'black art,' it surely has some scientific component to it due to the required knowledge of computers, operating systems and application software.
The discipline of computer security involves more than just testing how hack-proof computer systems are, it also includes guidelines for creating such systems. However, the penetration testing component of computer security seems to involve at least as much art as science. It often relies on the experience, knowledge and intuition of a particular tester.
To develop comprehensive benchmarks for measuring penetration testing efficiency, one has to have a clear strategy and guidelines for performing the testing. While penetration testing based on in-house standards may be effective, there is no way to compare the service level from two different sources.
Let us provide some background on penetration testing. First, penetration testing should not be confused with vulnerability assessment. The latter might not go beyond automated scanning of Internet-facing or internal systems with vulnerability scanner software and then reporting the results. On the other hand, full-blown penetration testing goes as far as social engineering attacks against company employees. It includes analysis of IT defenses, host protection, communication and physical security, level of security awareness, etc. In fact, penetration testing goes beyond the domain of computer security into the broader field of information security.
Penetration testing should also not be mixed with 'security testing and audit,' which involves having full access to all the systems, and aims mostly at comparing the implemented defenses with written policy documents. For complex or critical systems it is often recommended to do both a security audit during the design phase and penetration testing after deployment. Also, sometimes penetration testing is performed with some advance knowledge about the organization.
Penetration testing is one of the few components of computer security which is truly proactive: it attempts to find the holes before attackers do, rather than to block incoming attacks. However, it is often mentioned that the task of a penetration tester is much harder than that of a malicious hacker. The reason is simple: the attacker has to find just one flaw to get in, while a penetration testing team needs to find them all. But it is not only about software flaws: penetration testing is an effective way to test the success of an organization's security strategy, its design and implementation. Quality penetration testing assesses security on many levels, such as technology, administration and people issues, and can serve as a realistic forecast for how an organization will fare against real-world threats.
It is well known that justifying the return on security investment is not an easy thing to do. In part, it is due to the absence of testing metrics that can clearly indicate to executive management that the deployed security controls and policies are effective against current threats. Penetration testing can compare the current level of security with that at a different moment in time, or for a different organization.
So, having established the need to create guidelines for penetration testing, several attempts to approach this will be outlined below. It should be stressed that creating such a set of guidelines is not an easy task due to the flexible nature of the testing. It is important not to impose extraneous restrictions upon testers so that their creativity is not stifled, yet still keep the test independent from the personal qualities of the tester. 'Best practices' or methodology for penetration testing is probably more appropriate than a solid standard.
The United States General Accounting Office (GAO) Management Planning Guide for Information Systems Security Auditing (https://www.gao.gov/special.pubs/mgmtpln.pdf) mentions penetration testing, but only as a part of a security audit program. It sheds some light on a government approach to security testing and audit, without providing much of a 'what' and 'how.' The National Institute for Standards and Technology (NIST) Guideline on Network Security Testing (https://csrc.nist.gov/publications/drafts/security-testing.pdf) is another government guide with more details. It focuses on the higher-level picture of penetration testing such as phases of the test (planning, discovery, attack, reporting) and goes into some of the details of finding system vulnerabilities. The NIST guide emphasizes the need for penetration testing of critical computer systems.
However, the most ambitious project to bring order to the chaotic world of penetration testing is the Open Source Security Testing Methodology Manual (OSSTMM), freely available at www.osstmm.org.
The project manual contains information on testing within the domains of Internet security, information security, social engineering, wireless security, communication security and physical security. The big advantage of the project is that it covers both the high-level issues of penetration testing such as planning the test, and also the low-level details, down to what to say during a social engineering phone test. In addition, the approximate estimates for reliability of the obtained data are provided: some results might still be valid after several months while others need to be retested sooner. The methodology is structured as a set of sections, modules and specific tasks to be performed.
The modules cover such areas of testing as intrusion detection, password cracking, denial-of-service and many others. The tasks within each module specify attacks to try, information leaks to look for and tools to use. Even social engineering attacks have their own modules and tasks. Another very useful feature of the project is the information templates that can be filled to simplify information collection and report preparation. In case the testing is done with advance information, the guidelines for a security policy review are outlined as well. An important checklist of legal issues to handle before the test is also included.
How successful is OSTMM methodology in bringing just enough order to the fluid phenomenon of penetration testing? OSSTMM project founder Pete Herzog has this to say about the issue: "The OSSTMM offers a methodology which explains WHAT should be tested more importantly then HOW. This means that we had the flexibility to make some things more high level then others in order to take into account, for instance, that it is foolish for us to list every vulnerability to test for. Instead we say, please use two reputable vulnerability scanners - we offer Nessus as a suggestion for one of them. It also means that when we give very low level tasks for testing a firewall, for instance, one of those tasks is to test it for vulnerabilities, while the rest are very specific things to test. Some things in firewall testing don't or haven't changed, whereas new vulnerabilities pop up for a variety of firewalls all the time. This allows us to take the chaos out by defining chaos as merely a variable, susceptible to change frequently."
OSSTMM complies with many security standards such as ISO 17799 requirements for remote system testing and several European standards.
As a conclusion, we stress the importance of legal issues in penetration testing. Detailed written permission is absolutely essential for the penetration test. The permission should include such items as the machines to be tested, the list of techniques to be used, times of testing, specific people and machines involved in attacks, and measures to prevent law enforcement being alerted. In addition, trusting the party that performs the test is of paramount importance, since it will know all of the weak spots of the tested organization that will be discovered during the test.
Anton Chuvakin, Ph.D. is a senior security analyst with netForensics (www.netforensics.com), a security information management company that provides real-time network security monitoring solutions.
