Researchers with Fortify Software reported this week that the wide differences among "black box" testing methods can put organizations at risk without enough oversight.
According to the new paper put out by the code analysis company, many organizations use penetration testers and scanning tools to perform black box tests in order to confirm the security of their software programs, but they often have no way to measure how much of the application is scanned.
During several months of research, Fortify analyzed numerous black box testing methods on the market by placing software monitors at security-critical parts of the application to see how well the tests were able to scan the application.
Black box testing, also known as functional testing, is a method where the internal workings of a product are not known by the researcher from out outset.
Researchers noted that while all tests managed to find vulnerabilities reasonably well, some had deficiencies including failure to cover large sections of applications, failure to find critical vulnerabilities and failure to provide necessary information to fix identified vulnerabilities.
Especially disconcerting to the researchers is that there are no widely available tools to test the thoroughness of a black box test. Without this, it is difficult to judge how secure the application really is, said Mike Armistead, Fortify vice president for marketing and founder.
"You can't improve what you can't measure," he said. "It is a very different thing to say that I found 30 vulnerabilities while testing five percent of my code than to say I found 30 vulnerabilities with my test covering 80 percent of my code."
Click here to email Ericka Chickowski.
