Analyzing robots.txt: The White House and Search Engine Indexing

Jason Kottke's kottke.org has an interesting post about the difference in the robots.txt seen on the whitehouse.gov website before and after the inauguration. The previous administration used a 2400+ line robots.txt file, which prohibited automated indexing of a wide variety of pages. The new administration's robots.txt has two lines.

The Bush White House's site robots.txt included lines like this:

Disallow: /911/911day/text
Disallow: /911/heroes/text

While the text of those pages isn't sensitive, some sites may prefer that the data that they deny access to isn't generally indexed and searchable.

This is a great reminder to security minded administrators: relying on robots.txt to keep your content obscure only works with well mannered robots - those who read their logs well will note that many spiders do not actually heed robots.txt.

What would reconaissance of your site reveal if your robots.txt was reviewed, then a web spider tool was told to specifically ignore it and index your data? Would this provide more information than you might like? Some sites rely on the obscuring that not having their data indexed provides for a modicum of security or privacy. Do you?

Sentry Safe's Unsafe Survey

Sentry, maker of a wide range of safes, requests some interesting information on their new safe owner's registration form. The form requests a surprising amount of information for a warranty registration form, including a number of details that would concern any security minded individual.


A few of the more sensitive bits of information that you'd be mailing off are:

• Your address
• Who you purchased the safe for
• Where you will locate it (garage, basement, den, bedroom)
• How many guns you own
• How you stored your guns previously
• How much you make
• The other members of your household and their age

All of this is placed in a handy postcard sized fold out survey, ready to be mailed out. Individually, none of this information is particularly dangerous, however when taken as a whole, it provides a profile of a potential target - who, happily, gave you the model number and possibly the location of their safe!

Is this a huge threat? Probably not, but would new safe owners want to disclose the location of their safe, their income, and how many firearms they have? Probably not. Sometimes, security by obscurity isn't so bad.