Incident Reponse - counterpoint

Richard Bejtlich, one of my favorite security bloggers posted a list of observations about incidents. I'm going to try to provide a little counterpoint - see his original points here.

1. Anti-Virus is (or should not be) an incident response tool.

In many cases in less advanced, lower funded, or simply smaller organizations, AV may be your first indicator that something is wrong. During a Hacker Defender outbreak I helped handle, many folks that I worked with found machines that kept getting the same AV hits, even after being cleaned. They were, in fact, Hacker Defender protected hacks, and the AV was just removing the tiny portion of the rootkit that it could see.

AV is an IR tool - if only because it can help you find portions of a compromise. Is it the best tool in your toolkit? I should hope not. Is it useful to see what AV thinks about that odd file you just found? Yes! Norman Sandbox or Virustotal are your friends, but so is a command line AV client.

2. Your default incident recovery strategy should be to rebuild from scratch.

Every IT person I know recommends rebuilding after a compromise. This is becoming increasingly easy with the growing use of VMs and imaging software. The gotcha now is that your last image probably has the same vulnerability that let the bad guy in last time - I've seen administrators put the same box up and act mystified when five minutes later it is owned the same way its identical twin was.

Your default incident recovery strategy should be to rebuild in a secure environment and to ensure that you have done your best to fix the problem that let the last event occur. Note that rebuild from scratch may carry overtones of "do it all over again the hard way". That's not the point - use that pre-built VM copy, your Ghost image, or your standard install - the key is learning from the last incident and following standards and best practices so you can repeat the build process effectively and securely.

I've been a fan of a simple solution - give your systems administrators and techs commodity broadband routers - a Linksys for example. Have them put machines they need secured for a short while behind the router (which, of course, has had its password changed) and tell them to rebuild there. Yes, it may slow down your link a bit, but the system won't get knocked over before they manage to install that patch that the standard image is missing. Best of all, the routers are cheap and light enough to carry around in a laptop bag or toolkit, meaning your administrator will have it with them. The security you have and use is the best security you have - and when you can pick up a router on Sundays for $10-20 at the local big box electronics retailer, it is one of the cheapest tools you can give your admins.

3.SPAN ports should not be the default traffic access option.

Span ports are realistically the best available option for smaller organizations that cannot afford a tap, or, more often do not have a tap available. This is why you'll find information security folks stashing old hubs in their toolkits - sometimes you have to work with what you have.

So, yes, span ports aren't what you should want, but they're often what you get. Plan ahead, and have options available. Know what is possible with network gear - and be ready to discuss it with the local network engineers (or lack of network engineers). Have a backup plan - that old hub may be your only chance to get inline with traffic. And yes, a gigabit span port makes the network guys a lot happier than a 100 meg hub sitting in front of your gigabit uplink.

4. A Linux live CD is not a substitute for a real network security monitoring platform.

Nor is a Linux live CD a substitute for a real forensic system. But I do like them.

The rise of inexpensive gigabit capable laptops and small form factor systems means that given even a small budget, a security professional can have a portable, reasonable performance network security monitoring platform that they understand and have configured for performance. Get a Mac Mini, a Shuttle XPC, or a low end laptop with gigabit onboard and have it ready for tasks like this (pay attention to your NIC chipset - and for laptops and the Mini, remember that you're writing to a laptop drive if you have a high traffic load.). If you have a pre-built portable system, you won't find yourself scrounging at the last minute, and you'll have a known good system that you have locked down and can trust. Even these systems aren't a true substitute for a dedicated platform for hardcore network security monitoring, but they're a lot closer than a LiveCD on a random old desktop you pulled from the junk closet on short notice - and normally, in an IR situation, you don't need to monitor for weeks on end at super high speeds.

This goes back to an earlier post - build your toolkit now, not during the incident. You're going to have to learn new tools during a lot of incidents anyway, so get as much as you can out of the way in advance. And while you're at it, burn some up to date copies of those Linux live CD IR distros.

5.When you are compromised, you are probably not facing a zero-day exploit unique to you and not capable of being prevented.

There's a good chance your management might think you are! Remember to keep your management in the loop, give only truths and not supposition - or, if you must theorize, make sure you're clear that you are, and keep it plausible and non-panic inducing, and provide clear, concise documentation that you want publicized. Most organizations leak information like a sieve so you need to get ahead of the rumor mill, provide the information that is appropriate, and make sure that what does get communicated is accurate.

And then my own:

6. Pretending an incident didn't happen doesn't help.

I consistently deal with and hear about organizations that attempt to conceal compromises from their constituents, customers, or management. While this may make it look like IT is competent, it also means that the threat environment that you face isn't being effectively represented, it may be contrary to the law (for example Indiana's SSN disclosure law), and it may violate your organization's policy.

Worse, if the incident is reported on later, it may cause distrust from your customers or your management. Security staff and systems administration both need the trust and cooperation of users and management to remain effective.

Tackle incidents head on - you don't have to be completely transparent, nor do you have to broadcast your problems, but you should notify responsibly and respond professionally.