Analyzing robots.txt: The White House and Search Engine Indexing

Jason Kottke's kottke.org has an interesting post about the difference in the robots.txt seen on the whitehouse.gov website before and after the inauguration. The previous administration used a 2400+ line robots.txt file, which prohibited automated indexing of a wide variety of pages. The new administration's robots.txt has two lines.

The Bush White House's site robots.txt included lines like this:

Disallow: /911/911day/text
Disallow: /911/heroes/text

While the text of those pages isn't sensitive, some sites may prefer that the data that they deny access to isn't generally indexed and searchable.

This is a great reminder to security minded administrators: relying on robots.txt to keep your content obscure only works with well mannered robots - those who read their logs well will note that many spiders do not actually heed robots.txt.

What would reconaissance of your site reveal if your robots.txt was reviewed, then a web spider tool was told to specifically ignore it and index your data? Would this provide more information than you might like? Some sites rely on the obscuring that not having their data indexed provides for a modicum of security or privacy. Do you?

Core Impact and Ed Skoudis: Penetration Testing Ninjitsu

Core Security is sponsoring Ed Skoudis's presentations on ethical hacking and penetration testing under the title "Penetration Testing Ninjitsu". Ed and Core will be doing a total of three webcasts - the first was focused on Windows command-line tricks, future webcasts are slated to include social engineering and other techniques. The next webcast will be on May 20th, 2008.

Ed is an excellent speaker, particularly for those who are unfamiliar with the techniques but who have a reasonable level of general technical knowledge. He's well worth listening to if you get a chance.

In the first presentation in the series Ed emphasized one of my favorite characteristics of an information security analyst right up front - the ability to think out of the box, and to use their innate creativity.

In his presentation, Ed talked about some of the basics of penetration testing which are worth repeating:

• You have to know the limitations - things like scope, time, access, methods, and the final truth: you won't find all of the vulnerabilities.
• Penetration testing can help find things that other approaches missed, and that unknown problems can be found. In addition, it often goes deeper than most audits.
• Penetration testing isn't the only approach you should use - reviews of configurations and architecture, automated tools, audits, and interviews with personnel. The key: a comprehensive security program.

You'll find more about how to deal with the risks that you find in a penetration test in my writeup on risk handling methods and denial.

Ed also covered a number of Windows command-line tips - many of these are covered in the GCIH training for Security 504 that SANS offers, as well as in his Windows Command-line Kung-Fu. If you've taken either class, today's presentation was largely review - ping, dns lookups, arp cache checking, SMB enumeration and shares, a huge amount of detail about for loops, and a few other tricks.

The amusing observation that I and others who watched the webcast had was that during a security presentation where users were unable to see each other's names to provide anonimity, questions showed the name of the submitter. If you want to remain a bit more anonymous, you can't ask questions...

Creative Commons licensed image credit Flickr user R'eyes.

Reconaissance: LinkedIn and social engineering

I attended Ed Skoudis's SANS 504 track in Las Vegas last week, and picked up a lot of useful tidbits. One of the more interesting offhand comments Ed made was about using LinkedIn to assess what vendors a given organization is buying from based on their recent link adds.

It makes for a fun exercise, and could potentially be useful when doing recon of an organization for penetration testing. A quick look at my own contacts lends some credence to the idea, and given a bit of other research, a LinkedIn survey seems like a clever method to get a few extra bits of information.

Does this mean that using professional social networking should be banned? Probably not, but it is a great reminder of the level of detail an intelligent aggressor can gather given a bit of cleverness and time.

Perils of Penetration Testing

Federal Computer Week is running an article discussing penetration testing - specifically they compare in house to outsourced penetration tests. It also discusses penetration testing software like Core Security Technologies’ Core Impact.

In general, insiders know the network, systems, and applications better than an outsider will, but outsiders often have the cachet necessary to sway management. With tools like the Metasploit framework commonly available, in house security folks have capabilities they have never had before.

So should you outsource your penetration testing? If your inside security folks have the skills, it all comes down to what you need out of the test and how their time and your money can best be spent. Arming them with the best tools you can afford? That's a no brainer...