Bishop Fox CAST: Review | Security Weekly Labs

This review is part of the April 2021 assessment of the Attack Surface Management (ASM) product category. If you haven’t read the category overview, you might want to check it out – it explains the category’s basics, use cases and the general value proposition. Our testing methodology explains both how we interact with vendors and how we tested these products. In short, ASM products aim to discover and manage an organization’s external digital assets. This approach extends far beyond assets with an IP address, however, including everything from certificates to S3 buckets to DNS misconfigurations. 

Company background

Originally founded in 2005 as Stach & Liu and rebranded in 2013, Bishop Fox is one of most widely recognized security services firms. It employs over 250 and is headquartered in Phoenix, Arizona. Its’ employees have produced numerous books, research, talks and open source tools over the years. While it’s common to see individual creations come out of cybersecurity consulting firms, they occasionally step into the product space as well.

In early 2019, Bishop Fox raised a $25m Series A from ForgePoint Capital to do just that. The initial result of diversifying into the product space is CAST, which stands for Continuous Attack Surface Testing and is the focus of this review.

Understanding CAST

First and foremost, CAST is unlike all the other ASM products we’ve tested. Other ASM vendors started with the data and have been slowly working towards risk analysis and validating findings, using both humans and automated means.

Bishop Fox has come at the problem from the opposite direction. As a fifteen year-old consulting firm, they already had the human side of the equation – the talent. They then saw an opportunity to automate all the manual work of gathering, organizing and analyzing the data - turning it into a product, backed by experts. The final piece was the business model. By leveraging automation, it was possible to deliver a service that constantly monitors companies’ exposed assets year-round without labor costs making the service financially prohibitive.

One issue with traditional penetration tests is that they are point-in-time, typically performed only once or twice a year. The gaps left between these assessments was an opportunity for an organization’s security posture to decline, or at the very least, become unclear to those managing it.

Another issue with traditional penetration tests is that they are time-bound. Typically, the assessment is scoped for a finite number of hours. The penetration tester gives it their ‘best effort’ for that given time. It might be enough to assess 30% of the company’s external assets, or perhaps only 5%. It’s unlikely that a traditional penetration test will come anywhere near touching 100% of an organization’s exposed assets, however.

CAST addresses both these shortcomings. By running continuously, it is reasonable to expect to find all the assets there are to find (within the asset types CAST is designed to discover, of course). As soon as a finding emerges, Bishop Fox’s team of offensive security experts can perform a penetration test on that one small element (they call it a micro-pen test).

Another added advantage is that the typical penetration tester mindset doesn’t enjoy repetition. Part of the workflow for the offensive team is to continually improve the CAST engine to automate and improve testing and decision-making. As a result, the engine is capable of making decisions more and more like a penetration tester does. As it gets more efficient, the CAST service is able to scale more like a traditional software product and less like a human-driven consulting business.

Competitively, in a few years, we may see ASM vendors CyCognito and Randori, who already have offensive experts on staff, meet in the middle and begin to compete with Bishop Fox for larger customers. Currently, however, the customers and priorities CAST is designed for differ from the rest of the ASM market. CAST also pitches itself as an alternative to private bug bounty programs, where it could make a case that it casts a wider net, finding issues that the bug bounty crowds aren’t as skilled in discovering. It’s also easy to imagine CAST as an alternative to hiring additional internal staff to do the same job, less efficiently.

To be clear, Bishop Fox isn’t going to stop selling penetration tests because they’re now selling CAST. There will likely always be a need for point-in-time security assessments. However, CAST makes a lot more sense for enough of Bishop Fox’s existing customers that it was a no-brainer to raise some funding and develop it.

Review summary

While the real selling point for CAST revolves around the skilled staff that both validates findings and trains the 'brain' of the automated system, the customer will be primarily interacting with the web-based portal application. It should be noted that interaction with the experts behind CAST’s mini penetration tests is welcomed whenever customers have questions or wish to discuss findings.

While we give an overview of the CAST platform in the Usage section of this review, it’s likely that most CAST customers will log in to research or verify something only occasionally. Bishop Fox’s ideal customer will likely integrate CAST into an existing vulnerability remediation workflow (e.g. a ticketing system like ServiceNow). This is the primary value of the product – if anything seriously critical shows up, the product will let you know without a security analyst needing to log in every day to tap on dials and gauges.

Target market: Fortune 500 companies with large, complex attack surfaces

Time-to-value: As with other ASM products, the vendor asks for initial seed values (company names, domains) and initial results are available in a few hours.

Maintaining value: As a product with a managed component, maintenance is only necessary if additional companies, subsidiaries or seed values need to be added to CAST for monitoring and analysis.

Total cost: Bishop Fox did not provide pricing, but we’d expect it to be attractive when compared against the costs of building a small internal team to provide the same function as CAST.

Strengths: CAST effectively reduces the validation work in ASM to zero, leaving some lighter analysis, attribution and remediation work for internal staff to handle.

Weaknesses: Market limiting price point.

Conclusion: An view into a more efficient, hybrid future for services and product. When intelligent automation is combined with human services, buyers ultimately win.

Deployment and configuration

The initial setup process is notable, as CAST joins a small club of ASM vendors that not only offer multifactor authentication by default, they require it to be configured on first login. Be the change you want to see, right?

As with other SaaS ASM products, the initial seed values and provisioning are handled by Bishop Fox for the proof-of-concept, which will become the customer’s production instance if they choose to purchase the service.

Usage

The interface is reminiscent of most Attack Surface Management (ASM) products we've reviewed at SW Labs. An intuitive dashboard summarizes findings and assets. Charts break down assets by category and display time-based trends.

Findings can be filtered by severity, status or age. A search interface is also available, though with a service that aims for zero false positives, I wonder if any but the largest customers will have enough findings to justify the use of it (which is generally a good thing, in case that's not clear).

As we drill down into the details of a finding, we begin to see where CAST separates from ASM products that are largely automated. Findings read much like they would in a penetration test report. Finding details, screenshots and evidence of exploit or validation are present. For findings that require some narrative, took multiple steps, or succeeded in pivoting further, all these details are included. As we'd expect, credentials and other sensitive details are redacted, but the screenshots make it clear the staff made the 'phone call' from inside the house.

At this point in the review, it's clear that Bishop Fox isn't selling an ASM product. They're selling a continuous offensive security service - the ASM piece merely enables and feeds that service. Think of CAST as an external penetration test that never ends. As soon as new attack surface emerges, it is analyzed by the CAST system. The system sends anything interesting to the the CAST team and they go to work.

Returning to the interface, assets (called "Targets" in CAST) are organized by protocol, URL, protocol and port. Filters can be applied using any of these elements and it's possible to search by IP or host name.

Like others, CAST records screenshots of any websites it finds. Unlike others, however, CAST helpfully groups similar screenshots. In large environments, there could be tens of thousands of screenshots, many of which will be identical or have only slight variations (e.g. dev, QA and prod versions). Drop into "quick mode" and the arrow keys can be used to quickly flip through groups of screenshots - a sure pleaser for analysts that click and scan through data all day long.

CAST also provides a breakdown of technologies discovered (the classic comparison is BuiltWith - a service that tells you what technologies are used by a given asset, company or app). Here, however, the data isn't nearly as detailed as it is in ASM vendors that focus more on comprehensive data collection than risk analysis and prioritization. It lists technologies, but isn't forthcoming on versions in use and doesn’t link back to the assets (targets) where these technologies were discovered.

One interesting view is a breakdown of targets by hosting provider. At a glance, you can see how many IP-based assets the organization has connected to CloudFlare, hosted by AWS, running on Linode or hiding on DigitalOcean (that's where we all run our Shadow IT stuff, right?).

Lest you think CAST is limited to ports and protocols, the next section of the UI is dedicated to domains, subdomains and networks. Part of the onboarding process with new customers aims to discover all domains used by the organization. We gave Bishop Fox a list of six domains, intentionally leaving some out. The CAST portal currently lists an additional four we never mentioned.

The subdomains view includes a list of subdomains, likely discovered and brute-forced using a variety of techniques. Interestingly, domains determined to be abandoned are labeled as such. While the labels seem accurate, there is no reason listed as to why each domain has been tagged as abandoned.

Support

Security program fit

Bishop Fox, like other products and services focused on discovering vulnerabilities and misconfigurations, fits solidly within the Identify column of the Cyber Defense Matrix.

Conclusion

For the Fortune 500 (or even the Global 2000), we can see this offering appealing to organizations struggling to attract and retain talent for this kind of work, or prefer to prioritize that talent elsewhere. The quality and depth of findings can’t be matched by automated approaches today, and this level of experience is priced accordingly.

We’re intrigued by this development and can’t help but wonder if this is a window into the future of some cybersecurity services. The answer likely lies in the ability for this approach to scale down market, to smaller enterprises.