Patch management was challenging enough before the world was upended by a rapidly spreading pandemic. But with security teams working remotely, and employee-operated devices dispersed across large distances, quickly prioritizing and fixing critical vulnerabilities has become both more difficult and more important.
As the 2017 Equifax breach showed, delays in patching can result in a devastating data breach or malware infection. Then again, if security teams act too hastily and without a plan, they can potentially open up their corporate systems or employee devices to additional exploits due to incomplete patching or careless use of remote administration tools.
“In the immediate future, patch management concerns will extend far beyond the established, known, managed and curated networks into a potentially chaotic mix of uncontrolled system versions and devices of thousands of employees,” says Eric Welling, North American lead for the Accenture Security, Cyber Investigation and Forensics Response (CIFR) group. “The balance between functionality and security is a longstanding consideration, but the extra pressure from COVID-19 will require both agile implementation and a methodical approach to ensuring continuity while remaining secure.”
The new IT landscape that COVID-19 is quickly shaping presents numerous obstacles for patching. Among them:
IT resources are strained, resulting in less time to patch. “With the increase in the number of employees working from home due to the recent coronavirus pandemic, there has been a huge strain on corporate VPN networks and the internal bandwidth required to handle the external traffic,” says a CISO for a Fortune 500 company in the electronics industry, who requested anonymity. “Many network changes, updates and patches are being temporarily put on hold until additional network circuits have been installed to help provide some stability to the increase of external traffic supporting the remote workforce. This temporary interruption of patches will cause additional risk to enterprise endpoints.”
But it’s not just network infrastructure that’s being pushed to the limit. With too much to do, security teams are also running short on manpower and time.
“Lack of time leads to skipping critical steps in the process or best practices,” says Vesh Bhatt, co-founder and CTO of Attila Security. “At a time when IT teams can be overloaded with trying to help with the influx of new remote workers, creating new policies, upgrading existing infrastructure to handle the new load, monitoring the uptick in cyber-attacks, etc., it can become easy for them to turn a blind eye to patch management best practices or policy. Sometimes patch management can be put off completely because ‘everything works just fine,’ ‘the patches don’t offer anything new,’ or ‘we can’t afford any down time.’
Visibility and access into certain systems are limited. This is especially true for devices operating out of employees’ homes.
“…[I]t becomes incredibly hard to have any visibility or direct access into employees’ home networks due to the routers and firewalls in place that an organization does not control,” says Nathan Wenzler, chief security strategist at Tenable. “This means it can be impossible for traditional patch management tools, which typically have administrative access to target systems and unrestricted access to the network segments corporate systems live on, to deploy patches to these remote systems. Even with VPNs in place, if employees are using personally owned systems to access corporate networks, the patch management tools may not have sufficient permissions to successfully deploy and install required fixes.”
Even on-premises servers and systems can present this same problem if, for a strategic reason, they’re not remotely accessible. For instance, “local admins and support personnel are being restricted from working on-prem at manufacturing plants floors,” says the anonymous electronics CISO. “Typically, these systems are segmented from the corporate network and rely on manual patching. These plant floor devices will remain at risk until government shutdown restrictions are removed.”
Software incompatibility issues can also result in a lack of accessibility and insufficient patching. “Incompatible software versions, especially between the OS, VPN, remote monitoring and management tool, patch software, etc., can lead to the loss of remote access to corporate devices,” says Bhatt. “This means you can no longer monitor, manage or support the device without the user having to ship or bring the device back.”
It’s a BYOD party, and the CISO isn’t always invited. Typically, corporate-issued devices run on the same operating system and share configuration settings and universal toolsets for pushing across security updates. But the same can’t be said for employee-owned devices that, under ordinary circumstances, certain companies would not even permit for business use.
chief security strategist, Tenable.
“Security teams may now have to accommodate operating systems they've never had to manage before and deploy far more and far older patches than they may be prepared to deal with,” says Wenzler. “Even if they are able to reach these systems and have the credentials to manage them within their existing corporate patch management tools, there'll be a need to add more patches to the system for deployment, test them if possible, and change the configuration of the central tool to accommodate these new patches, resulting in more work for both the security and operations teams supporting the patch management program.”
Employees using their own unpatched devices to access corporate devices is an especially troubling practice, according to Leigh Metcalf, Ph.D., senior vulnerability research analyst with the CERT Division of the Software Engineering Institute at Carnegie Mellon University. “This can yield an ecosystem of unpatched devices that can spread malware, similar to [how] a lack of personal hygiene can spread COVID-19. Corporations must require automatic patching before allowing these machines to access their infrastructure; otherwise they are endangering their own assets.”
Overuse of remote access tools and protocols poses its own danger. At the 2020 RSA conference, FBI Special Agent Joel DeCapua reportedly revealed that Remote Desktop Protocol – used by network administrations for remote management purposes – constitutes 70 to 80 percent of the initial foothold that ransomware actors use when infecting a company.
Meanwhile, remote administration tools used by IT staffers to troubleshoot individual devices can similarly offer an open door for attackers if they, for example gain hold of an admin’s credentials. Bhatt says that hackers can leverage these tools to “steal your data, install ransomware, or really whatever else they want.”
Ideally, companies should place their RDP servers and remote admin tools behind a VPN, and use host-based security measures and multifactor authentication as additional layers of protection. But the makeshift WFH environment created by the COVID-19 pandemic invites the opportunity for sloppiness.
“In trying to gain access and control over remote-based systems, administrators can introduce a large amount of risk to home networks if they require the existing security controls in place to be relaxed in order to accommodate remote admin tools and services,” says Wenzler. “Not only does this potentially expose employee systems to attackers, but this can create additional liability for the organization should these systems become compromised, as these networks and systems are not owned by the company and may not be directly covered by existing policies. While security teams may be solving the more obvious patching problem, the introduction of new risks may outweigh the benefits of trying to protect the remote workforce via patches alone.”
James Globe, VP of operations with the Center for Internet Security's Multi-State Information Sharing and Analysis Center (MS-ISAC), agrees, noting that “Without proper security measures, such as the principles of least privileges and proper MAC or IP filtering, the use of remote access tools can be… like leaving your house door closed, but unlocked.”
Even before the novel coronavirus upended businesses around the world, unprotected remote connections have represented a major trouble spot. But COVID-19 further “increases [and] shifts the threat landscape, since the number of employees using remote capabilities has increased tremendously,” Globe adds.
As security teams strive to adjust to a new normal for an indefinite period, it is comforting to know that there are ways to lower the risk associated with WFH environments.
“The real key for any organization is to make sure they have a good process in place and that there’s proper testing being done prior to pushing out patches” to ensure that systems will still work after the change is made, says Globe.
It’s not a one-size-fits-all scenario – and what that exact process is will depend on a particular company’s set-up, Globe continues. Nevertheless, some strategies are universal, like communication. To that end, Globe suggests “sending out notifications to users across multiple internal channels (e.g. e-mail, calendar invite, internal message board), letting them know patches are coming…”
Other steps companies can take, according to Globe, are instituting best practices for remote access, including MFA, account lockout, role-based access control, least privilege, password complexity, auditing, logging and more.
Companies may also want to invest in cloud-based, automated remote patch management solutions or mobile device management solutions as a means to securely push fixes from a central server across a complex, scattered network of heterogenous devices. Remote monitoring and management tools and secure configuration management tools are other viable options. Bhatt says such tools “help the IT staff see which versions of software are running on their devices and help keep the same software baseline across the devices.”
Moreover, automated remote patching solutions allow security teams to “perform rolling updates where a small percentage of devices are updated first and the others follow after a certain time interval,” Bhatt continues. That way, “issues can be identified and fixed before the entire fleet of devices is updated.”
Bhatt recommends starting the process by applying the updates and patches “in a test environment that closely mirrors the actual production environment. Afterwards, you can apply the updates and patches to a small test bed of users in the production environment and ensure everything works properly for a certain period of time. Finally, you can take a staged approach where you start updating a small percentage and keep expanding until you’ve updated the entire production environment.”
But while some say “push,” others prefer a “pull” methodology, whereby clients initiate access to receive their updates.
“Agent-based solutions help with this, as the agent software resides locally and does not require opening inbound connections through firewalls and other controls, and can instigate requests to assess the vulnerability posture of the system or pull down fixes from a designated safe repository,” says Wenzler. “While it requires a bit more involvement to set up and get employees to install on their remote systems, a pulling strategy is a significantly safer and more reliable tactic for achieving visibility and delivering patches.”
For those concerned about the risk posed by employee devices connecting to corporate systems, Wenzler suggests Network Access Control (NAC) services, which he says can “serve as a gatekeeper for corporate networks when implemented at external connection points such as VPNs.”
“NAC can validate that a system attempting to connect to the corporate network meets basic security requirements in terms of patch levels, endpoint security controls and other factors deemed necessary by the internal security team. If a system does not meet the requirements, the user can be forwarded off to instructions on what they need to do in order to get their system healthy and secure enough to connect,” Wenzler continues.
Wenzler also advises that employee device connections can be better managed with more clearly communicated, stronger BYOD policies. “Ideally, these policies should require that any systems connecting to the corporate network or utilizing company resources of any type be patched, have endpoint security software installed and active, and are regularly kept up to date. While this may not eliminate the possibility of compromise, it will help to address the liability issues around trying to deliver patches to employee-owned devices from the corporate patch management tools,” he explains.
A more extreme measure, if enforceable (and therein lies the rub), is to banish BYOD devices altogether. “All work from home personnel should be using devices provided for them and managed by the company,” says Metcalf. “This puts the onus of the patching problem on the organization and not the person working from home. This is to not only protect the corporation, but to protect the network outside the corporation from attacks engendered by not applying patches.”
“Personal computers add additional risk to corporate networks and data, even when using VPNs, as many of these devices do not have endpoint security software installed or configured correctly,” says the electronics industry CISO. “If at all possible, supply remote workers with corporately configured and secured devices. Additional priority/attention should be given to the remote workers using VPN. Your VPN should be configured to allow only corporate devices with properly configured and updated security controls in place.”
But even if your remote connections are secure, a key question remains: What to patch first, especially with so many fires to put out?
Wenzler has a game plan in mind: “Right now, attackers will likely take advantage of the current chaos and will be looking for quick wins. This means leveraging known vulnerabilities that already have viable exploits readily available and going after as many exposed systems as possible that may not have been kept up to date as well as a typical corporate system would be.”
“In light of that, if security teams are looking to prioritize their remediation efforts, patching these known, exploitable vulnerabilities would be the single most important group to focus on first,” Wenzler continues. “That said, if you've set up your tools in such a way that you're able to easily deliver patches to remote systems, it's a good idea to err on the side of caution and patch as much as you can in order to close as many potential attack vectors as possible.”
