Frontpage Lessons - UC Berkeley's Health Data Breach

UC Berkeley's recent announcement of a large scale health data breach serves as a great reminder of two basic basic security best practices - service separation and monitoring.

In this case, at least part of the flaw was a multi-use front facing system providing both web and database services with sensitive health data. Berkeley's announcement notes that "The attackers accessed a public Web site and subsequently bypassed additional secured databases stored on the same server." A multi-tier architecture with appropriate security precautions between each tier would likely have provided better security, and likely at a far lower cost than that of notifying 160,000 individuals of their potential exposure. The first lesson is that a strong architecture design for public facing systems can save a huge expense in both dollars and man hours later.

In addition to the multiple services on the single server, the length of time that the system was compromised indicates that it is likely that a range of other diagnostic and detection systems that could have been monitoring the server were not in use. According to the announcement, the server breach began on Oct. 9, 2008, and continued until April 9, 2009, and was only detected by routine server maintenance.

The second lesson here is that system monitoring is crucal - Tripwire, border flows to detect remote SSH sessions, and log auditing would have likely helped to find this compromise earlier.

Despite the flaws that led to the compromise, Berkeley appears to have done a good job after the fact of providing resources to affected individuals at datatheft.berkeley.edu and their process lists both activation of a CIRT and contact with the FBI. As any IT organization knows, some flaws will slip through, either old or new - Berkeley's response shows that they have a coherent plan, and this lesson should only serve to improve their overall security posture.

High value higher ed and compromise profiles

Dave G from Matasano Chargen posted about "Mac Punditry and the Office Paradox". My higher ed focused ears perked up when he asked "How are educational environments high value targets?".

Higher ed security folks know that we're major targets - and attractive targets too. Here's why:

1. Open networks - large public IP spaces, often with relatively loose border controls.
   
2. Open systems - frequently systems are not centrally supported, and there can be a wide separation between system security postures across the network.
   
3. High bandwidth - universities have bandwidth that many commercial entities and ISPs would be jealous of. Internet 2 connections are typically at least a gigabit, and some universities connect to things like the TeraGrid's 10 - 40 gig research backbone or other specialized high speed networks.
4. High value data - Social Security numbers, research data, personal data on students, faculty, staff, and donors, and credit card operations are all on the list for most higher ed institutions.
   

Historically, educational institutions have had relatively open borders. This has changed over the past few years with universities implementing border firewalls and other protections. Universities are still in a somewhat unique position - many have residential networks that act as ISPs for their students and have standards for academic and research freedoms that make a corporate style security architecture difficult, if not impossible.

Watching vendors and other security professionals react to statements along the lines of "well, no, we can't presume that it will be firewall protected" or "we have a class B, and everything we own is on a public IP" can be fun if you enjoy looks of sheer terror.

With attractive and relatively exposed systems - and a population that is often less formally controlled than those in the corporate world (at least in similarly sized institutions), compromises will occur. What do they look like?

Higher education security staff tend to see three attack profiles on systems - and I think that these three attack profiles show the three types of value that compromised systems have for attackers. They are:

1. Zombies - low value systems with no real useful data, smaller hard drives, and low to middling bandwidth. Often these are student machines on residential networks, or standard employee desktops. These systems tend to participate in botnets, and are valuable simply because of numbers and the fact that many are either not detected, or are not cleaned up properly if they are.

2. Storage and bandwidth - these systems are characterized by larger hard drives and bigger pipes. Universities often have relatively large pipes, either thanks to an Internet2 link, a research network link, or simply large commodity Internet upstream connections. Systems that are compromised by an attacker who actually cares about system profiles and that do have bigger drives or more bandwidth continue to be used as storage and distribution points.

3. Jumping off points - every organization has hosts that have sensitive data or that can be used to move through other systems. Your local IT support staff machines are a great jumping off point, and so is a dean's machine, or a business office system. Normally, you will want to focus your forensic efforts on these systems, as they are more likely to have sensitive data - or the keys to the kingdom. A single critical IT worker's machine can let attacks romp through your infrastructure. Most hacks of these machines tend to be detected via system monitoring software - AV and anti-spyware, via user notification, or via one of the methods above.

So how do you deal with these compromise profiles? Your detection and response tactics will likely vary due to your level of control and access.

Zombies can often be found by by monitoring outbound IRC connections or by using netflows and other monitoring technologies to keep track of known bad hosts on the outside. Most normal systems on campus won't be talking to your friendly neighborhood C&C. Since these systems are often outside of the normal support infrastructure for business or academic computing organizations, you have to work with your residential network support or enforce your AUP (you do have an acceptable use policy, don't you?)

Storage and bandwidth oriented hacks may not be reporting into a central C&C - although more and more do dial home. Use your border flows to check for hosts that have very high bandwidth usage - a good tactic is to check your top 10 or top 20 hosts on a daily basis, then filter out the known good hosts, check into the rest, rinse, and repeat. Yes, that public FTP mirror will be high, but why is a grad student's desktop machine in the top 10 for outbound traffic?

Jumping off points are the security analyst's bad day - follow your IR procedures, and make sure you understand what the system had on it, and what access the people who used the system have to other resources. The chain can be long, but following it can help ensure that further compromises don't occur.

The final analysis - at least from my perspective is that higher ed does have high value hosts. Educational institutions cover the spectrum of sensitive data from research data to SSNs and credit card data. In addition, most are highly concerned about their image, meaning that a data breach can cause significant damage to reputation, even if financial losses are smaller.

Where does that leave us? I'll write more about some of the directions that universities are moving in to handle both intrusion and extrusion detection in a coming post.

Incident Reponse - counterpoint

Richard Bejtlich, one of my favorite security bloggers posted a list of observations about incidents. I'm going to try to provide a little counterpoint - see his original points here.

1. Anti-Virus is (or should not be) an incident response tool.

In many cases in less advanced, lower funded, or simply smaller organizations, AV may be your first indicator that something is wrong. During a Hacker Defender outbreak I helped handle, many folks that I worked with found machines that kept getting the same AV hits, even after being cleaned. They were, in fact, Hacker Defender protected hacks, and the AV was just removing the tiny portion of the rootkit that it could see.

AV is an IR tool - if only because it can help you find portions of a compromise. Is it the best tool in your toolkit? I should hope not. Is it useful to see what AV thinks about that odd file you just found? Yes! Norman Sandbox or Virustotal are your friends, but so is a command line AV client.

2. Your default incident recovery strategy should be to rebuild from scratch.

Every IT person I know recommends rebuilding after a compromise. This is becoming increasingly easy with the growing use of VMs and imaging software. The gotcha now is that your last image probably has the same vulnerability that let the bad guy in last time - I've seen administrators put the same box up and act mystified when five minutes later it is owned the same way its identical twin was.

Your default incident recovery strategy should be to rebuild in a secure environment and to ensure that you have done your best to fix the problem that let the last event occur. Note that rebuild from scratch may carry overtones of "do it all over again the hard way". That's not the point - use that pre-built VM copy, your Ghost image, or your standard install - the key is learning from the last incident and following standards and best practices so you can repeat the build process effectively and securely.

I've been a fan of a simple solution - give your systems administrators and techs commodity broadband routers - a Linksys for example. Have them put machines they need secured for a short while behind the router (which, of course, has had its password changed) and tell them to rebuild there. Yes, it may slow down your link a bit, but the system won't get knocked over before they manage to install that patch that the standard image is missing. Best of all, the routers are cheap and light enough to carry around in a laptop bag or toolkit, meaning your administrator will have it with them. The security you have and use is the best security you have - and when you can pick up a router on Sundays for $10-20 at the local big box electronics retailer, it is one of the cheapest tools you can give your admins.

3.SPAN ports should not be the default traffic access option.

Span ports are realistically the best available option for smaller organizations that cannot afford a tap, or, more often do not have a tap available. This is why you'll find information security folks stashing old hubs in their toolkits - sometimes you have to work with what you have.

So, yes, span ports aren't what you should want, but they're often what you get. Plan ahead, and have options available. Know what is possible with network gear - and be ready to discuss it with the local network engineers (or lack of network engineers). Have a backup plan - that old hub may be your only chance to get inline with traffic. And yes, a gigabit span port makes the network guys a lot happier than a 100 meg hub sitting in front of your gigabit uplink.

4. A Linux live CD is not a substitute for a real network security monitoring platform.

Nor is a Linux live CD a substitute for a real forensic system. But I do like them.

The rise of inexpensive gigabit capable laptops and small form factor systems means that given even a small budget, a security professional can have a portable, reasonable performance network security monitoring platform that they understand and have configured for performance. Get a Mac Mini, a Shuttle XPC, or a low end laptop with gigabit onboard and have it ready for tasks like this (pay attention to your NIC chipset - and for laptops and the Mini, remember that you're writing to a laptop drive if you have a high traffic load.). If you have a pre-built portable system, you won't find yourself scrounging at the last minute, and you'll have a known good system that you have locked down and can trust. Even these systems aren't a true substitute for a dedicated platform for hardcore network security monitoring, but they're a lot closer than a LiveCD on a random old desktop you pulled from the junk closet on short notice - and normally, in an IR situation, you don't need to monitor for weeks on end at super high speeds.

This goes back to an earlier post - build your toolkit now, not during the incident. You're going to have to learn new tools during a lot of incidents anyway, so get as much as you can out of the way in advance. And while you're at it, burn some up to date copies of those Linux live CD IR distros.

5.When you are compromised, you are probably not facing a zero-day exploit unique to you and not capable of being prevented.

There's a good chance your management might think you are! Remember to keep your management in the loop, give only truths and not supposition - or, if you must theorize, make sure you're clear that you are, and keep it plausible and non-panic inducing, and provide clear, concise documentation that you want publicized. Most organizations leak information like a sieve so you need to get ahead of the rumor mill, provide the information that is appropriate, and make sure that what does get communicated is accurate.

And then my own:

6. Pretending an incident didn't happen doesn't help.

I consistently deal with and hear about organizations that attempt to conceal compromises from their constituents, customers, or management. While this may make it look like IT is competent, it also means that the threat environment that you face isn't being effectively represented, it may be contrary to the law (for example Indiana's SSN disclosure law), and it may violate your organization's policy.

Worse, if the incident is reported on later, it may cause distrust from your customers or your management. Security staff and systems administration both need the trust and cooperation of users and management to remain effective.

Tackle incidents head on - you don't have to be completely transparent, nor do you have to broadcast your problems, but you should notify responsibly and respond professionally.

Don't Panic!

Network World's recent article on keeping your senior management calm during an emergency rang true to me. In major security events in which I've been involved, communication with managers and senior staff was absolutely critical. In many cases, the CIO will become personally involved in an incident, and may request that you break your normal incident response practices to keep him informed. At other times, fighting rumors may take almost as much time as the incident itself does.

Dealing with incidents large and small led to a few basic observations that can help you weather the storm.

Things to do ahead of time:
1. Create a written process.
2. Establish clear lines of communication.
3. Make sure your CIO and others know when and why you will get them involved.
4. Establish who has a need to know, and how to do rumor control.
5. Make sure to have an emergency contact and communications plan.

During an incident:
1. Follow a written process.
2. Communicate early, clearly, and often.
3. Do not speculate - only communicate facts, even if the fact is "we do not know".
4. Avoid blame - the goal is to handle the incident, not to pin it on some person, department, or policy.
5. Make sure that those effected are communicated with appropriately. Observe "need to know" as appropriate, but realize that anything that you release is likely to travel far more widely than anticipated.

After an incident:
1. Write up a post-incident report. Note failures in the response plan, and work to fix them.
2. Communicate with those effected and those who may be effected by similar incidents.
3. Use data from the incident to assess other risks of a similar nature.

While you're thinking about handling incidents, check out the NIST guide to handling computer security incidents.