"I suppose someone could write one to ignore that instruction."

Actually, you would have to write it to honor that instruction. It won't do it automatically.

I have a crawler and it ignores the robots.txt file. The program I have can also defeat hotlink lock outs. It's a really aggressive piece of software. Quite handy.

Brian