We've all made one in our career, I'm talking about that blunder for which you thought you would be fired. My first blunder was rebooting all the campus router pairs at one time, not one by one, all at once. I had written a script to install a security update on all the routers and reboot them all one by one, I thought. Turns out my script had an error and didn't wait between routers. I thought for sure I was fired, but it turned out it was a great learning experience for everyone involved. We all learned a little about crisis management, I was introduced to change management, and my boss took a few hours to teach me how to verify the network was working properly as everything came back up online.
Sometimes, our blunders are not so instantly noticeable, and sometimes we make blunders that linger in our firewalls until they either cause an outage or an auditor calls us on them. One of my favorite activities is visiting and talking with firewall engineers around the world. Here are some common blunders I've seen and heard engineers talk about – maybe you'll recognize one or two of them yourself.
Creating firewall groups with no meaning
One of our engineers saw this client's configuration which had a certain network object in more than half the rules. It was named after a famous football star, let's call it “Joe_Montana” (after all I need to change the facts to protect the guilty). Whenever they need some access which is not clear, they add the IP address to this group, which is used in many permissive rules. The result is that the rulebase looks OK to an auditor (because there are no rules that contain ANY), but it's actually a huge security hole. The rules have become meaningless, and the cleanup of the rulebase is going to take many months of searching out what the actual businesses needs are.
Failing to upgrade your firewall software
We find many clients running old firewall software. I always hear the same story about keeping the version fixed for stability: The firewalls cannot be taken down for upgrades, etc. The fact is, your firewall vendor is upgrading its software for a reason. You don't need to be on the latest and greatest release, but when I see you on a version that is 15 or 20 releases ago and seven years old, I can't help but think you don't take security seriously.
Using the wrong technology
I've seen all sorts of ways in which the square peg is pounded into the round hole, but here is one of my favorite blunders. One client argued with auditors that because they have a firewall in front of their secure webservers it formed a second layer of authentication – and so they were using two-factor authentication, a password and a firewall. I like the effort, but a firewall (by itself) is not a two-factor authentication solution. Two-factor authentication requires your users have something, it's something they know and something they have, a token and password, for example.
The accidental outage
Many of us have caused an outage at one time or another. One of our engineers tells a story of being at a client site when one such outage happened. They were working on the production firewall server gathering some data for a support case (the server was Windows). The admin reached across the table and accidentally leaned on the mouse, which was over the start menu. As fate has it in for us network engineers at all times, the mouse activated the start menu, which happened to be over the shutdown menu item when it popped up. Yep, right there in the middle of production this financial corporation watched their production firewall shut down.
Poor documentation
All too often we talk to customers who are trying to understand what all their rules do. As we get in a hurry day after day and get lazy about documentation, we create a time bomb waiting to blow up. We often find we are working with a customer and they say, “I'm afraid to make changes on my firewall now, all the senior guys have left and we don't know what most of these names mean or what these rules do.”
Using excessive Drop rules
Often when we are in a hurry, we create rules with very excessive accesses, and place a rule just above it dropping the access we don't want to allow. We do this because we don't want to figure out how to write the correct rule. We see rules like, allow All DMZ devices to All Internal devices ACCEPT, with a rule right above it that says All DMZ devices to Secure Network device DROP. The two rules look OK at first, but it is really an ugly hack, because we didn't write out the business need in the first rule. When we operate like this, over time we have a rulebase with lots of the rule ‘pairs' and reordering the rulebase or editing rulebases is likely to expose more risk or block necessary traffic. Either way, we have a mess we must rewrite at some point.
Using routing as your security policy
I've seen many a firewall where rulebase changes need routing changes to accompany them. It's understandable when the change involves a new network on the firewall. There are two versions I see this blunder all the time. The first is the firewall with no default route. Every route is added to the firewall by hand and the smallest netmask possible is used so traffic will not reach unintended devices if the firewall has no policy. Wow, that sounds great, but it's totally unnecessary if you remove the policy on modern firewalls they revert to DENY ANY. This design becomes so hard to manage that the entire team begins to dread making changes. Soon, every change needs an engineer to examine the routing, thus every change takes too long and impacts servicing the business in a timely fashion – so there is no real value in added security.
The second version of this blunder is most often seen on Cisco devices where admins have ACLs between two interfaces that include the source or destination ANY. They don't actually mean ANY, they mean all the addresses behind this interface, but I'm too lazy to type the addresses in. This leads to a rulebase that can be understood only by knowing the routing table along with the firewall and trying to do the match in your head – way too complicated for a junior firewall admin to take over this firewall.
Using DNS objects in a rulebase
One of the options many firewalls provide is inserting a source or destination as a DNS object, like www.google.com. It sounds great because google.com can use so many IP addresses and this allows my firewall to always pass the traffic as google.com changes IP addresses. This blunder leads to many risks most organizations should consider unacceptable. First, your firewall is now more susceptible to denial-of-service attacks. What happens when it can't resolve google.com? Second, your firewall is going to waste CPU, memory and network IO on doing DNS lookups for every packet trying to decide if it might belong to google.com. Third, what happens if your DNS is poisoned and malicious addresses for the command and control of the attest botnet are returned with the google.com addresses? You now just allowed all the botnet command-and-control traffic though your firewall, and logged it as google.com.
Making changes in panic mode
Imagine that something goes wrong and you lose one of the RAID disks.
You replace it, and while the RAID is rebuilding, the service is slowed down – but you don't realize that it's because of the RAID. At this point, your customers have been denied service for 40 hours and you're losing a lot of money every minute, not to mention customers who are leaving you for alternative services.
You go into panic mode and start changing configurations: switches, routers, load balancers and firewalls – anything that you suspect may be causing the problem.
After another 24 hours, another sleepless night and many hours of expensive consultants, someone figures out the real cause of the problem. Now you want to revert all the changes you made on the switches, routers, load balancers and firewalls, but no one knows what they were because they were made in a rush without any documentation. So you spend another three days figuring out how to revert the system back until it's fully operational.
I hope you don't see any of these in your organization. But if you do, rest assured you are not alone. The best run organizations can find these blunders or others lurking in their firewall rules. The good news is that tools to automate the discovery and remediation of these blunders are now available. The not-so-good news is that as networks change, there will be new firewall risk vectors to consider, but the pace of innovation is moving quickly. Firewalls are a mainstay of network security and have been around for a long time. We are now at a point where any organization, whether they have one or 100 firewalls, has the capacity to ability to implement best practices for firewall management without breaking their budgets or losing their minds.
