Exclude a section of your site from all spiders and bots
User-Agent: *
Disallow: /cgi-bin/