This review is part of the September 2021 assessment of the vulnerability scanner product category. If you haven’t read the category overview, you might want to check it out. It explains the category’s basics, use cases and the general value proposition. Our testing methodology explains both how we interact with vendors and how we tested these products.
For this review, we tested the following three open-source projects:
Free and open-source software (FOSS) for vulnerability management does not exist in any single project. We tested a few different vulnerability scanners, which is just one activity inside vulnerability management (albeit an important one).
Two on our list are network-based scanners, OpenVAS and Nmap plus Vulners. OpenVAS is, for those not familiar with the history, a fork of the last open-source release of Nessus version 2. Starting with Nessus version 3 in 2005, all subsequent releases of the Nessus vulnerability scanner by Tenable (then “Tenable Network Security”) are closed source.
Nmap is a popular open-source port scanner (and is still available as OSS today). Nmap introduced an open plugin system several years ago, based on the LUA programming language.
Subsequently, many have contributed plugins, including Vulners — an open-source vulnerability enumeration suite of plugins. Vuls.io is the only local scanner on our list, relying on credentialed access to the targets to extract a list of installed software and compare that to lists of documented vulnerabilities.
In an ideal world, you would have both types of scanners deployed on your network and systems. One type of scan identifying and scanning systems connected to the network(s), and another connecting with credentials pulling the list of software and vulnerabilities. In open-source vulnerability scanning these are two separate initiatives, compared to commercial scanners that introduced credentialed and agent-based scanning some time ago.
Installation and configuration of FOSS vulnerability scanners can be a bit of a daunting task. The various projects have different requirements, change frequently and features are added and removed based on the “greater good” or because the developers felt it was a better decision for the project, not necessarily for the users. A strong background in administering Linux or UNIX systems is a huge asset when it comes to FOSS in general, and this is certainly the case for FOSS vulnerability scanners. While you may get away with a basic installation and setup today, the next time you run the scanner or upgrade it, the game could have an entirely different set of rules.
The results varied between projects: OpenVAS provided some of the best results in terms of accuracy and coverage. Nmap plus Vulners provided a very false positive heavy result set that was difficult to parse or integrate with any other solution (commercial or open source). Vuls.io was its own unique experience, and provided you can set it up properly, provided very detailed results with accuracy, however the web interface for sorting and filtering results requires a great deal of study before it can be useful. The most impactful observation is this: You can get some good coverage and accuracy with FOSS vulnerability scanners, but you will have to work hard to turn this activity into any sort of rudimentary vulnerability management system. This means you can collect a large mass of results, but its up to you to put all the pieces together in order to report the findings to the right people and track remediation.
Nmap Plus Vulners
Nmap is the popular open-source port scanner created by Gordon “Fyodor” Lyon in 1997. In 2007, the Nmap team introduced the NSE scripting language for creating plugins, paving the way for people to use Nmap for vulnerability scanning and more in-depth probing of network targets than previous releases. In 2017, the Vulners team released the first beta version of the vulners NSE scripts for Nmap. Vulners NSE scripts take the Nmap output and attempt to match patterns to report any matching vulnerabilities.
Target market: Market can refer to size (e.g., large enterprise, SMB) or vertical (e.g. healthcare, etc.)
Time-to-value: The Vulners NSE scripts are included with Nmap, and depending on how up to date you want to be, running them simply required running Nmap with the correct options. Nmap is available on most Linux, UNIX and *BSD distributions, so this can be run very easily. If you want to run up-to-date versions of Nmap and the vulners scripts it can take one- to two hours to build a Docker container that automates this process. We have published a Docker container build that will download the latest version of Nmap, compile it and install the latest Vulners NSE scripts. If you are familiar with Linux and Docker, this is a task that can be completed in a few hours (depending on how you choose to build the container and your knowledge and experience).
Maintaining value: Using Docker, depending on the speed of your compute device, it takes three- to five minutes to build an up-to-date instance of Nmap with Vulners.
Total cost: Nmap plus Vulners is a very low-cost solution, requiring only a bit of knowledge and experience.
Strengths: Very fast and lightweight scanner as it does not take much effort to build and use.
Weaknesses: Vulners has a high false-positive rate, largely due to the way in it uses basic matching and regex to determine if a vulnerability exists on a given service based on the Nmap fingerprint for a the given service.
Support options: You are on your own on this one (unless you can convince someone to provide an answer via Slack, Discord or Github Issues).
Deployment and configuration
As long as you need — with FOSS tools, this will likely be the longest bit of the writeup and the bit folks are most interested in.
Usage
If you are familiar with running Nmap, using it to enumerate vulnerabilities is as easy as adding some extra command line options. These are very well documented by the community. You can review the results in the Nmap output formations (raw Nmap, CSV and XML). If you want to automatically output HTML, look at Cloudflare’s FlanScan as it outputs an HTML report sorted by vulnerability by IP/hostname. The output is generally not readily available for importing into popular tools, commercial or open source, that allow for more advanced reporting activities such as creating tickets, marking false positives and prioritizing remediation. This is unfortunate as these capabilities would make Vulners output more valuable. However, I believe this is one of the downfalls as the output from Vulners is not easily parable (it comes bundled all in one field with the Nmap results).
Performance
If you are looking for a fast scan, Nmap plus Vulners is the place for you. Nmap, especially more recent versions, in our testing has dramatically improved performance. While we did not specifically test performance, having used Nmap for many years as a user, I found that it performs very well. For larger scans of thousands of hosts over the internet you may consider chaining different tools together (such as massscan, Amass and your favorite website screenshot tool).
Conclusion
For smaller, targeted scanning that required vulnerability enumeration, Nmap plus Vulners really shines. It is highly recommended that you setup and use the latest versions of both as you will get better performance and more vulnerability checks. Nmap plus Vulners does have a high false positive rate, however if you are using it to scope penetration tests or running it regularly against smaller sections of your environment, this combination really shines. If you are expecting a full replacement for your regular scanning of your entire enterprise (large or small), look elsewhere.
Greenbone (OpenVAS)
In 2005, the developers of the vulnerability scanner Nessus decided to discontinue the work under open-source licenses and switch to a proprietary business model. This left Nessus version 2.x as open source and all future versions (3.x and beyond) as proprietary software. Several forks of Nessus were created as a reaction to the discontinuation of the open-source solution. OpenVAS, the Open Vulnerability Assessment System, being the most popular Nessus fork, garnered a lot of attention and support from the security community.
The German-based company Greenbone Networks maintains the open-source project today with the goals of:
- Go beyond plain vulnerability scanning towards a comprehensive vulnerability management solution.
- Create a turn-key appliance product for enterprise customers.
- Continue the open-source concept of creating transparent security technology.
(Reference: https://greenbone.github.io/docs/background.html#history-of-the-openvas-project)
Today the term OpenVAS is one component of the Greenbone Vulnerability Management (GVM) suite available as both open-source and commercial software. There are seven Github repositories now responsible for making up the entire suite of software that allows you to scan your networks for vulnerabilities, collect and manage results and integrate with other solutions.
You can find an architecture diagram that shows how all the various components fit together here (https://greenbone.github.io/docs/background.html). The software components today are available in three different deployment options:
- You can download a pre-configured VMWare or Oracle Virtualbox VM from Greenbone (https://www.greenbone.net/en/testnow/#downloadnow)
- Registration is only required if you want a trial of the Greenbone Security Feed (Paid), otherwise you can use the Greenbone Community Feed (Free). The differences are listed here: https://www.greenbone.net/en/security-feed/
- There are several Docker containers that people from the open-source community have created, some are actively maintained, and some are not.
- You can download, compile and deploy from source code available from Github (this is the most time consuming options but offers the most flexibility and control)
Target market: Greenbone Networks offers vulnerability management for those who wish to try out a scanner all the way up to commercial offerings for enterprises as a cloud-managed solution.
Time-to-value: If you are just starting out with GVM it can be bit confusing as to the best way to get started and deploy a scanner. We found, after a significant amount of testing, that the free version can be deployed using Docker (in just two commands) and ready for use in just a few minutes. Of course, you can download pre-configured VMs from the Greenbone website for testing as well.
Maintaining value: GVM does need to be kept up-to-date, along with all of the plugins (made available as feeds). The Docker container builds do a much better job at allowing you to more easily keep up to date, provided you have these skills on your team to support such a deployment.
Total cost: If you are willing to deploy and maintain the Docker containers for GVM the cost for the software is free and you will have to put minimal time and resources into maintaining the scanner itself.
Strengths: GVM comes with a comprehensive set of vulnerability checks and has some nice features that allow you to re-cast the severity of the vulnerability checks and the ability to really fine-tune the scan policies. The XML report output format is widely accepted by many other tools and solutions for importing.
Weaknesses: The open-source graphical user interface (GSA, Greenbone Security Assistant) is functional and an improvement over previous versions; however, it lacks a modern look and feel. The actions are accessed via small icons in the top left and right of the interface and takes some time to learn which icon allows for which action. The various components all must work together for the scans to run, so troubleshooting could be an issue if problems arise.
Support options: Greenbone maintains a community forum (https://community.greenbone.net/) where volunteers help solve problems and offer advices for the most common issues. If you do not wish to run the open-source version, Greenbone offers both commercial solutions and support options (https://www.greenbone.net/en/services/).
Deployment and configuration
The free and open-source version of GVM can be deployed as a VM, a Docker container, or compiled from source. In our testing it is easy to get GVM running via a Docker container. While there are several different projects to choose from on DockerHub, we found this project https://github.com/immauss/openvas from Immauss Cybersecurity to be the best and most up to date. In fact, after our testing was complete the project released a new version based on the latest version of GVM (21.4.3 at the time of this writing, while testing was performed using version 20.08). The documentation for setup and deployment can be found here: https://github.com/immauss/openvas/tree/master/docs.
(Reference: https://greenbone.github.io/docs/background.html)
Usage
Defining targets, modifying a scan policy and executing a scan with GVM via the Greenbone Security Assistant (GSA) was easy and straightforward (And well-documented here: https://docs.greenbone.net/GSM-Manual/gos-21.04/en/scanning.html).
Once logged into the Greenbone Security Assistant (GSA), you will add some targets. I simply defined my local subnet and gave it a name. From there, I navigated over to the scan policy section. I reviewed the different policies that come pre-defined, immediately I felt in familiar territory as some of the options are very reminiscent of Nessus, and some are new to GSM. The more I dug in the more I realized that I could really be in full control of the scan settings. I initiated a new scan, and I was off to the races. The results were easy to find and review. I really liked how it discovered some default username and password combinations on a few devices in my network, pointed out flaws on a NAS, and some Apache Tomcat vulnerabilities on a management server. Knowing this target network very well, as we use it to test a wide variety of scanners and other security solutions, I believe the results to be accurate.
Performance
The scans completed in a short amount of time; much faster than previous versions I have tested. I even configured a few different scan policies, enabling even more plugins and features, and OpenVAS still completed in a reasonable amount of time.
Conclusion
I plan to use the Docker container version of GSM to perform regular scans of my network. The team at Greenbone has made huge strides in usability and performance. I re-ran some of my testing using the latest version, 21.4.3, and found it to be more responsive and even faster for scans to complete. GSM is, by far, the best open-source vulnerability scanner out there today. The comprehensive set of plugins and security checks, along with the set of features offered in the Greenbone Security manage, that are offered for free is truly amazing.
Vuls.io
Target market: Linux-heavy system implementations run by folks with very experienced Linux administrators and engineers.
Time-to-value: From the time I began reading the documentation to the time I was working with results was around one- to two working days. It takes some time to read the documentation, choose your deployment model, build and configure the software, and setup new SSH key trusts with all your systems.
Maintaining value: It is a manual process to keep all components up-to-date, which means you have to setup cron jobs to continually assess systems and update the vulnerability databases with the latest information.
Total cost: This project is free, and we are thankful, however it requires significant time and effort to implement and maintain. However, you may find this worthwhile as it provides functionality that is not easily reproducible from other projects or free solutions (e.g., you could implement your own using Ansible or take a stab at a Salt Stack implementation).
Strengths: Very accurate (and fast) results from the target systems as the scans are all run using credentials.
Weaknesses: Difficult to setup, deploy and work with the results.
Support options: They have a Slack channel if you get stuck (https://vuls-github.slack.com/).
Deployment and configuration
I originally tried to deploy Vuls.io using Docker containers; however, due to bugs in the builds available at the time, this did not work. I repeated the setup process, using the manual setup steps and retrieving everything from the GitHub repos. After some trial and error, I did manage to get it working.
Usage
Once the configuration file was setup (yes, there is no GUI for this part so get ready to configure some TOML files, very similar to YAML) scanning a system was as easy as running one command. The web interface is a separate installation, but easy to get up and running. However, working with the results will require some study. There are many different options for filtering, sorting and displaying the results. It's amazing what the web UI can do, however it will require some study and tinkering to get the hang of it.
Conclusion: While there is a learning curve with Vuls.io, I do really like the project. They’ve put a tremendous amount of effort into the project, and it shows. The results are comprehensive and accurate, as expected with a credentialed vulnerability scan. The web user interface can be intimidating at first, but you will learn it is extremely powerful, allowing you to run some complex queries against your vulnerability data. I was disappointed to not find the ability to apply patches to the remote systems. This is one feature that is on my immediate list of requirements for the Linux systems I manage on a day-to-day basis. Currently I use a commercial tool and would like to replace it with an open-source tool and was really hoping Vuls.io would include the ability to patch. If you are looking for something to keep your patching system honest, and are willing to put in a little work, Vuls.io could be for you!
Security program fit
Free and open-source software may have a place in your security program, depending on your requirements, environment and the skill that exists in your team. There are certain projects that you may choose to use open-source software, and some where you may want to implement commercial software. The choices are yours and depend on many variables. With respect to vulnerability scanning, you may choose open-source software, in fact we highly recommend the free version of OpenVAS to get you started when building a vulnerability management program. You will need other components, but to help prove the value, open-source software can be a great place to start with vulnerability scanning.
Conclusion
I believe that all the projects in this review have their place, and at the very least deserve to be tested. Nmap with Vulners is a suitable place to start, especially for penetration testers looking to get a fast scan on a small number of targets that they are planning to investigate further. OpenVAS, the free and open-source version, has an impressive architecture and plugin set for a tool that you can easily setup and deploy for free. Vuls.io comes in handy for Linux-heavy shops that are looking to track Linux system vulnerabilities to ensure the patching system is working correctly. These tools are a great proving ground for selling security to management, proving a concept or process, and helping test that your security program is working.
