Security firm Cylance Tuesday disputed accusations that it used fake malware that only its Protect product could detect to gain more favorable results over competitors during testing.
Chad Skipper, vice president of industry relations and product testing at Cylance, denied that the company intentionally misled customers and called for changes to security testing that would "consider more real-world scenarios."
A software engineer became suspicious last fall after seven of the 48 malware samples provided by Cylance for testing turned out not to be malware, according to a report by Ars Technica. Only Protect had detected all the samples during testing.
The revelation prompted a sharp rebuke by Carbon Black Cofounder and CTO Mike Viscuso.
“It's unbelievable that businesses today can't trust the people who they rely on to keep them secure," Viscuso said in comments emailed to SC Media. "The actions Cylance has taken puts their customers and our national security at risk."
But Skipper said the company, which recently announced layoffs and saw the departure of CTO Glenn Chisholm, did no wrong. Cylance creates malware samples to use in testing, "we employ the same methods and tools that hackers do, including creating mutations and packing the samples, to better emulate what attackers do for more meaningful testing. We are not running or using any tool that isn't already in an attacker's arsenal," he said in comments emailed to SC Media. "Any time you pack a real file, there is a chance that the original piece of software will break. Some installers use internal checksums that are broken by the packing process, resulting in a valid file that does nothing" though, while it may not run properly "still has the earmarks has the earmarks of malware."
Skipper contended, "this is how it works in the real world and can be seen frequently in real malware, where the resultant mutated sample doesn't operate anymore."
He said the company could "guarantee with 100 percent certainty that the original files we pack are in fact very legitimate malware," but explained that since Cylance doesn't "control the inputs, sometimes this process can result in valid but harmless files being output as attackers change their tactics in how they generate input files."
When the company discovers that "outputs are no longer valid, we make adjustments to our process to remove these to ensure the fairness of results," Skipper said.
When the industry relies on existing and known samples to test their products, "a test does little to nothing to help someone understand how a product may fare in the real world.," Skipper said, calling for security teams to "test for themselves because these tests will be most relevant."
He pointed to NSS Labs and AV-Test as emerging leaders in "evolving testing methodologies to be more real-world."
