Automated remediation of cloud misconfigurations was a big theme in 2018, and here at DivvyCloud we expect the trend to continue through 2019. One of the significant challenges customers face is putting automation into action, instead of just talking about it.
When enterprises evaluate Cloud Security Posture Management (CSPM) solutions, automated remediation is frequently the end goal. As with any enterprise system, it is critical to learn, plan, and prototype your automation capabilities until the power is fully understood. Our challenge is to help those clients who are starting from a blank slate to take a “crawl, walk, run” approach. Running aggressive automated remediation from day 1 risks causing more issues than you’re solving. As a result, your team will most likely be averse to future automated remediation efforts. A poor initial implementation of remediation introduces a risk of organizational opposition to automation going forward.
Automation can range from basic notification and logging to fully automated remediation (the most advanced type of automation). You don’t need to start with 100% automated remediation from day 1. In fact, most organizations benefit greatly from working their way through the levels of automation to fully explore what approaches suit their environment best. In this paper, we’ll examine the different steps and levels of automation, and at the end of this document, you’ll be able to choose the appropriate level of automation for your environment.
First, let’s start by reviewing the benefits of automated remediation:
- Save time - humans no longer need to react and take action manually. The actions are performed automatically, allowing humans to work on higher value-add tasks. Especially at enterprise scale, the time savings can be significant.
- Improved security - vulnerabilities and problems are addressed immediately upon discovery, preventing bad actors from capitalizing on issues.
- Consistency - every action runs with the exact same workflow, and organizations can be sure that the prescribed procedures are always being followed correctly.
- Continuous compliance logging - provide proof of the results of real-time corrections to keep cloud environments compliant, rather than periodic audits.
Before Starting: Notification
Notification is the foundational building block of automated remediation, and something you will use on an ongoing basis. Because you’ll configure notifications to send reports of remediated events, this is a great place to start for testing.
During the initial rollout, using only notifications for the first tests is critical because it allows you to audit exactly what would be remediated without making changes. This is a great way to ensure any actions that would be invoked will do exactly what you want.
The old saying of “measure twice, cut once” applies here. Do not move out of the notification stage until you’re able to consistently validate that you are only receiving notifications for the defined resources.
When you’re ready, the steps are:
1. Decide which resource type you want to remediate and what check will trigger the automated remediation (exposed SSH, public buckets, etc.)
2. Perform several dry-runs with notification only, and make sure the reported results are exactly what you expect.
3. If the notifications and resources that were marked as non-compliant are aligned with expectations, then move on to level 2.
Tip: After initial testing and the first stage of remediation has been performed, it is important to keep notifications turned on so that the ongoing results of your remediation are logged for audit.
This is critical for several reasons. If things are breaking and being
fixed automatically, how will you know when something bad is happening? Also,
if there’s someone who is making unintentionally incorrect or insecure changes,
they won’t have any way of being notified that they need to change what they’re
doing.
It’s important that notifications don’t become “noise” or “spam” to the
recipients. To that end, notifications should include as much contextual
information as possible.
Sample notification, ticketing, and logging targets include:
• Email
• Splunk/Sumo
• Slack
• Service Now
• Post to API
Level 1: Ensure Visibility and Accountability through Logging
The next step in moving towards automated remediation should focus on locking down your account fundamentals. There are several initial configurations that every new cloud account should have, and most of them can be controlled with automation.
Sample automation for AWS can include:
- Cloudtrail - ensure there is one trail logging global services
- Cloudtrail - ensure that all regions are logging
- Cloudtrail - ensure all logs are being aggregated to a central bucket
- IAM - enforce a complex password policy
- S3 buckets - enable versioning
- S3 buckets - enable logging
- S3 buckets - enable server-side encryption
- VPC - ensure all VPCs have VPC Flow Logging enabled
- EC2 - Auto-tag instances that are spun up without an owner tag to populate the value with the username of whoever created it
- EBS - Ensure all volumes associated with an instance are tagged
These fundamentals will save valuable time and increase the security of your environment. None of these configurations should have a negative impact on your day to day operations or users.
Level 2: More Impactful Best Practices
The automation recommendations in level 1 and 2 line up closely with the AWS CIS benchmark. These are account fundamentals that any organization can employ to improve the security and overall hygiene of their cloud accounts.
The difference between level 1 and 2 is that in level 2 the automation for account fundamentals will take some planning to ensure they won’t have an impact on your users. The level 2 automation will be easier to roll out than the automation that provides remediation in levels 3 and 4.
Examples of AWS housekeeping automated remediation:
• Remove all unused security groups that start with launch-wizard*
• Delete the default rules on the default security groups
• Ensure that all custom AMIs are set to private
Level 3: Governance and Account Hygiene
Things get a bit more free-form in level 3. The goal here is to make this automation your own and add actions that bring your company the most value while affecting day to day operations as little as possible.
There are several use-cases that may be employed, and you’ll have to explore which of these best fits your company:
- Sandbox environment enforcement (clean up every X hours, or in line with your software release cycles)
- Notify / kill expensive instances that are spun up (either by cost, family type, or specs on the hardware)
- Cost control (clean up unused databases, old snapshots, orphaned resources, etc.)
- Port exposure cleanup (lock down SSH, RDP, etc.)
- Keep non-main regions unused (kill off or alert for assets outside of primary regions)
Level 4: “Classic” Automated Remediation
It’s not until level 4 that you’ll begin performing the kind of automated remediation most people think of when discussing the topic. That’s because while these are the most exciting form of automation, these are actions that give you the most control, and can also do the most damage. Additionally, when you start rolling out these kinds of actions, you need to make sure that the organization is completely on board. If these types of actions are being run in a vacuum, you’re going to have a lot of confused people and potentially a lot of broken systems.
Examples of level 4 automated remediations in AWS:
- Deactivating old API keys
- Deleting newly provisioned SG rules which do not have a description
- Killing instances that aren't tagged properly, or aren't running a golden AMI
- Lock down
public S3 buckets (zero out ACLs and bucket policy)
- Add in exceptions (by tag, name, etc.)
Other Considerations — Timing
It’s important to mention the concept of timing for the actions that are in levels 3 and 4. It’s appealing to initially roll out actions that have a lag between notification and remediation built into them, but that might not be the best approach.
For example, if you wanted to lock down SSH exposure in your development environment, you might design your remediation to send a Slack notification that an instance is out of compliance because it has SSH exposed and that it’ll be terminated in two hours if not fixed. Two hours later, the instance can be terminated.
It may seem counterintuitive, however, this is actually a more disruptive workflow than if the instance was turned off as soon as the issue was created. In the first scenario, if a developer spins up a non-compliant instance and then it goes away after two hours, they will have spent two hours of work on that instance. The code will have been loaded, the app might be running, and if it just goes away, that’s developer time that was wasted —and they’ll probably be upset! Instead, if the instance is torn down as soon as it’s seen, there wouldn't have been an opportunity for the developer to waste any time on the instance. It’ll go away a minute or two after it is created, and the supporting notifications will give them the context they need to avoid the same mistake again.
Different scenarios require different timing and you’ll always have to balance the risk of security exposure with the operational impact it will have on the organization.
To Sum Up…
Every organization will have a unique journey when implementing automation. For some, there’s no appetite for full automated remediation and just using automation for notifications will be enough. In other organizations, everything will be automated and completely locked down. Whatever level your organization strives to get to, by working through these 4 levels you can be successful in gradually rolling out automation to achieve fully automated remediation and get the most value out of the actions with the least amount of shock to the organization.
Chris DeRamus, CTO and co-founder, DivvyCloud
