Firewall Change Management: Changes That Require No Change

Creative Commons firewall image courtesy ecastro.

Firewall administrators are frequently asked to update rules for systems that don't work. Upon further research, the problem is frequently not with the firewall ruleset, but with the system itself. In fact, many problems magically solve themselves when they are looked at in the light of day - or at least, a tcpdump and a few minutes of actual review of the traffic between the host and its destination.

Thus today's security analyst quote: "The changes we do that require no change are always done without error".

A corollary is, "Those changes that require no change often require just as much time as those that do". This is because the research to find out why a request isn't necessary often involves just as much log review and traffic analysis as a properly specified request would, particularly if the firewall system does not allow arbitrary queries for existing rules.

There are a number of methods that you can use to help reduce the number of requests that turn out to be already allowed, or that are poorly specified. I'm going to take a moment today to outline a few of the most common reasons that I encounter poorly specified rules in change requests.

These poorly specified rules are often due to one of a small handful of issues:

1. Poorly understood architecture - the client/server relationships are unclear, either because the software is outside the norm - some software initiates connections on both sides, on the same port, and sometimes at very odd times, or because the architecture isn't understood and thus the requests don't account for the logical locations of the systems affected.
2. Lack of vendor documentation or poor vendor documentation - many vendors provide poor or little documentation for their applications. Some are particularly unclear, and simply specify a port, or a wide range of ports. Others don't document whether they mean TCP, UDP, or both. Yet another set don't document which system initiates the connection, and where it goes, or how long it stays open without keepalives. All of these can create a great deal of work for the firewall administrator if they aren't documented or if the request is poorly drafted.
3. Untested software or services -if documentation doesn't exist, administrators must test their systems to determine what they actually do. This can help determine actual ports and protocols, and can find those undocumented calls that can wreak havoc if missing from the ruleset.
   
4. Lack of communications and planning, or a total lack of a real design - when projects are conducted without review, designs can be created that don't fit the capabilities of the firewall, or which can put undue stress on it. Sometimes this results in data center wide system backups occurring through an already stressed firewall, or a reliance on broadcast UDP traffic between hosts that just won't work through a stateful firewall.

How does your organization handle rule requests? Perhaps more importantly, how do you handle the workload required to document the changes and to track them?

I'll talk about a few methods to help shape requests into a useful form in a post in the near future.

An Analysis of Firewall Rulebase (Mis)Management Practices

Mike Chapple, John D'Arcy, and Aaron Striegel, all from the University of Notre Dame recently published an article titled "An Analysis of Firewall Rulebase (Mis)Management Practices" (edit: link updated) in the ISSA Journal - described as the first widespread analysis of firewall management practices through a survey of firewall administrators and their supervisors.

According to the article, the survey was based on a series of open and closed ended questions which assessed the state of an organization's firewalls, their management practices, and the issues that were encountered with them. With 101 valid results, and 148 total respondents, this is a sizable survey of firewall administrators. In addition, they surveyed supervisors of firewall managers. In both cases, the distribution reached across industry.

Their findings are interesting -

• An average ruleset size double what it was in 2000.
• A high correlation between platform and ruleset size.
• A high turnover or churn rate for firewall changes.

Error rates were also a key part of the study - they found that 91% of administrators believed that least one error was introduced into their ruleset in the average month, and half of those administrators believe they go unfixed.

The article suggests that appropriate staffing levels and skillsets, as well as automated error detection are the best routes to fixing these issues. I would suggest a third step as well - a clear initial design and well defined rules for how rules are written, their syntax, and how they are tracked over time.

Overall, an interesting read for firewall administrators and supervisors or management of organizations with firewalls.