False Positive Forensics

Institutions often have a well defined incident response process, and may have either in-house or contracted forensic and incident analysis professionals. They should have a well defined process for handling incidents, and that well designed response process needs:

• Trained support for end users, servers, and the network that support must have
  
• An escalation process that leads to
  
• A central group or team responsible for investigating incidents who have
  
• A documented process using
  
• Standardized tools which helps shape
• A decision process from which
  
• A report is written in a common and understandable report format which
• Guides management and further responses

Communication of this is also important, as any step that isn't properly communicated can lead to a failure - either in the process, or in the implementation of the process.

Once the process is started however, there are three possible answers that can be found. This post focuses on the third answer, but first we need to discuss the worst case scenarios. In these three scenarios, I use two definitions: event - a documented occurrence that leads to an investigation, and incident - an event that when investigated turns out to be a compromise, exploit, or other actual compromise.

1. There is a real incident, and responses must handle the compromise, data loss, or other event. This may be part of an ongoing cycle, as some compromises lead to further investigation.
   
2. There may be a real incident, but insufficient data is available to assess the situation. In this case, response is typically to restore operation and to verify that the system or application fully meets current standards. This is typically accompanied by further monitoring of the system to ensure that a second compromise does not occur due to an unknown or undocumented problem.
3. Finally, the incident may be a false positive. In this case, the goal is to prove that the event was not an incident. This can be just as difficult, if not more so than proving that an incident occurred.
   

How can your organization handle forensics for these false positives? Often, the best route is to verify 5 things - you may pick and choose from these, depending on the system, and you might even add event appropriate checks beyond this list, but often you must check for these:

1. Verify that there is no sign of the reported event. If a system is reported to have acted like a system infected with a trojan, then verify that the system does not display any of the characteristics of a trojan infected system.
2. Check that the system or application matches its documented configuration. This is where Tripwire and other file integrity checkers may come in handy. In the some cases, simply checking MD5 sums and directory file lists from a matching system may help.
3. Check local logs for signs of compromise.
   
4. Check netflows, firewall logs, and other network traffic logs to determine if the system was inappropriately accessed or if it attempted to connect to remote sites.
5. Finally, look for a plausible cause for the behavior that caused the original alarm. In some cases, this is user error, or a misunderstanding, or a process that resulted in unexpected results.

In most cases, false positive forensics are a best effort task - proving beyond a shadow of a doubt that nothing happened is difficult on any but the best documented and monitored systems, and the great majority of workstations - and even servers don't fall into this category.

Once you have your documentation, two last tasks remain - documentation and communication. A report, even a simple "no sign of compromise" is required, and you should communicate to the affected user and support person. This is where appropriate thanks will pay off for future real events, and it can also serve as an excellent opportunity to sponsor awareness.

Digital Forensics: Data Carving With Foremost

If you're doing forensic work, or if you need to do data recovery, you'll likely run into deleted files that you need to match up with actual file types. This is where data carving, or file carving comes into play. Data carving involves searching an input (in this case, a dd image) for content, rather than metadata like filenames.

One of the easiest ways to do this is with an open source tool called foremost. Foremost recovers files using headers, footers, and standard data structures, allowing you to match files on a disk image. Usage is simple:

foremost -v -T -t (type) -i (file)

This enables verbose mode (-v), timestamps the output directory (-T), selects the type of files you want to search for (jpeg, gif, etc), and feeds in your dd'ed input image file (-i).

You can find previous DA posts about digital forensics here:

• An Inexpensive Forensic Toolkit
• Digital Forensics: When Users Put The Mass In Mass Storage
• Digital Forensics: A Few More Tips

Where are the law enforcment information security trainees?

Richard Bejtlich asked where the law enforcement trainees are in information security classes:

"When I teach, there are a lot of military people in my classes. The rest come from private companies. I do not see many law enforcement or other legal types. I'm guessing they do not have the funds or the interest?"

I've worked with cybercrime and computer forensic training programs in the past, and my former employer had a very close relationship with both state and local law enforcement. We saw many police officers and federal agents in forensics classes learning system forensics, and we often provided expertise for those who did not have it. What we see was many officers sent to network analysis or other broader information security classes - their jobs were focused on the investigation rather than threat prevention, or digital defense. Many of the classes spent a lot of time looking for predators online, which tends to be a high profile activity for departments when they do make an arrest.

With all of that said, forensic skills are becoming more common, and training for forensics is available from organizations like Purdue's CyberForensics lab and Eastern Michigan University's Staff and Command school. Even with these resources, network forensics and similar skillsets are typically not a focus at the local level, but do become more useful for state and federal agencies.

Does this mean that our law enforcement organizations are unprepared? In some cases, yes - either because the specialized training isn't available, or their budgets or time are restricted. In addition, many police departments continue to use antiquated IT infrastructure, and smaller police departments are reliant on external support, or no formal support at all. These departments are both more vulnerable and less likely to have access to the training and technology needed to do useful forensic analysis of systems. That's what regional forensic centers are seeking to help with.

I think that many security analysts would benefit from spending some time with their local police forensic analysts - perhaps by joining Infragard, or attending a local cyber forensics class. Those contacts can pay off in the future, and will help you understand what they're dealing with.

Forensic tools: WiebeTech HotPlug

Engadget has a short writeup of WiebeTech's HotPlug forensic system - in short, a tool for moving powered on systems, either by powering their power strip, or for injecting power into their power cord or outlet.

If you're doing police forensics, this looks like a tool to investigate!

An Inexpensive Forensic toolkit

What should you have in a basic homebuilt forensic toolkit? What can you do without specialized tools? We'll cover what you should have in a portable IR forensic toolkit if you're on a tight budget, with an emphasis on freebies and using inexpensive commodity tools and software. In our next article, we'll talk about what a well equipped forensic toolkit might contain if you aren't budget constrained.

A full kit composed only of commodity items many systems administrators have at hand:

• Screwdrivers (Phillips at the very least)
• USB 2.0/Firewire drive enclosure or a USB 2.0 -> IDE bridge device and 12v molex power adapter. Remember to check that your enclosure works with your Linux LiveCD.
  
• SATA->IDE Adapter OR USB 2.0/Firewire -> SATA enclosure - having the adapter is lighter than carrying two enclosures.
  
• 2.5" -> 3.5" laptop hard drive adapter
• Spare hard drive jumpers
• Known good ATA and SATA cables - in my experience, problem ATA cables are one of the most annoying issues when imaging drives.
  
• A capacious thumbdrive - a 1 GB drive is a good start, and one with write block switch is a good idea.
• A portable external USB/Firewire hard drive - larger is better - try to have at least the maximum size drive that your environment has deployed for a single desktop or server. You can cheat a bit and carry just one enclosure and a drive to put in it if you are attempting to travel light and don't anticipate needing to copy from a drive in the enclosure while you're using the big drive.
  
• IR and forensic LiveCDs.
• A notebook and pens - do not use pencil!
• A laptop with supported chipsets for your forensic CD of choice for USB/Firewire, NIC, and other critical hardware.
• A crossover network cable, in case you need to connect your laptop to the system you're imaging. If you don't have a crossover cable handy, a pocket hub or switch and some standard ethernet cables will work nicely.
  
• Anti-static bags for transporting drives. I like to carry the plastic clamshells that drives ship in to put drives in when I have to take them with me.
  
• Stick on labels to label drives and devices - something that will stay on, but peels off easily.
• A black fine tip permanent market for labeling devices
• Blank CDs and DVDs
• An extension cord and/or power strip with a long cord
  
• Business cards to leave so that you can be contacted
  

Total toolkit cost should be <$300 excluding the laptop, even if you have to buy all of this new. This toolkit provides the hardware you need to get a system open and to pull most commodity drives, and to walk away with a good image. If you're just doing forensics as part of an incident response investigation, and don't have to maintain evidenciary standards, this will get you where you need to go in most cases. This is also the sort of toolkit that many administrators could scrape together on short notice out of their standard administrator's toolkit, meaning that the tools you need are likely already at hand. In addition:

• If you frequently work with SCSI, you should have SCSI adapters - a SCA -> 68 pin adapter, a HD50 -> 68 pin adapter, and spare terminators. You'll also need a SCSI card or SCSI adapter. Sadly, I haven't run into a quality USB 2.0 -> SCSI bridge device yet.
  

What you don't have in this kit:

• Commercial analysis software like FTK or Encase
• A commercial disk duplicator with MD5 sum capabilities.
  
• Write blockers (although these can be had for less than $200 now)
  
• Large scale long term mass storage

We'll discuss a pro level forensic kit next time! Don't forget to check out our earlier posts on digital forensics tips.

Digital Forensics: A few more tips

I wanted to throw out a few more tips to add to those from the previous post, but first here's my own take on the SecurityFocus article.

First, of course storage is expanding rapidly. This has been the case for years and it can be a problem for forensics as you typically have to deal with full drive images. However, my belief is that law enforcement is still dealing primarily with individual's desktop systems and not large corporate servers. You still have to deal with large drives, but probably won't come across massive RAID arrays anytime soon. In these cases, you can always keep ahead of the desktop user if properly funded. To save on space though...

Tip 1
1) A number of forensic tools now allow for capture to a compressed image format such as AFF or Encase's Evidence File Format. If space is an issue, then use compressible formats for analysis and archival. You can always compress images for archival, but the formats above (and others) can be indexed and searched while still in a compressed state as well.


If you happen to be doing forensics in the corporate world, however, then dealing with servers and computer intrusions may be the norm. But there's still hope for you! If you're doing forensics primarily for business or incident response purposes rather than a legal matter then you might be able to get away with something less that a full drive image for analysis.

"But wait!", you cry, "can't all cases end in possible legal action." Certainly, but you just can't image everything and need to be able to take action rather than being frozen with fear of corrupting a hypothetical legal action.

Tip 2
The key is to have documented procedures for doing live forensics on a system. Even better is to capture those procedures through an automated first response process. Perform the initial live data collection from the system with a script. Some generic scripts are already available such as IRCR or FRUC.

Digital forensics: when users put the mass in mass storage

An article popped up across the SecurityFocus security news feed this week titled "Digital forensics plagued by expanding storage". This rang a bell, as I've conducted forensic analysis on arrays and drives that were impressively large.

There are a few basic things you can have in your toolkit to help out:

1. As much storage as you can afford - I like to have a large portable drive as well as an even larger network accessible drive that I can dd to. These days, it isn't unreasonable to carry a 500 GB drive as your portable forensic storage device, and 750 GB drives are starting to hit reasonable prices if you have to deal with large datasets. Remember, bitwise copies are the best route if you really need a deep analysis, but if you're forced to and having that level of accuracy is not required, you can use tools that copy only the live filesystem.

It is worth noting that vendors have begun to bring out enclosures with decent hardware RAID built in, meaning that you can have a terabyte or more of real space for images in a single reasonably portable box.

If you're imaging from the host, rather than a forensic workstation, a good USB/Firewire enclosure is a must, as well as a bootable Linux or Windows IR CD. Check out Helix or PenguinSleuth, or any of the host of other liveCDs out there.

2. A method of getting to arrays. In many forensic analysis situations, you need to copy from a RAID array, either on a dedicated controller or from an onboard host based controller. For Windows, BartPE is your friend once you locate the right drivers and load them. Some arrays may be hardware based - if they're not, you'll need to get the configuration and drivers loaded to deal with them in most cases.

3. A plan for how to handle oddball systems. Sometimes, getting an image is going to be impossible, and if analysis is your key issue, rather than preservation of a pristine image, you should have a plan for how to conduct live system analysis.

4. Last but probably most important - you need a way to take useful notes, and a plan of attack that you can follow. Standards will save you, and documentation and careful notes are critical.

Don't forget your screwdrivers!