http://www.melbournechapter.net/wordpress/system-administration/rich/2006/04/27/robots-spiders-crawlerswhat-the/ web application development with popular technologies Fri, 29 Aug 2008 19:58:23 +0000 http://wordpress.org/?v=2.0.5 http://www.melbournechapter.net/wordpress/system-administration/rich/2006/04/27/robots-spiders-crawlerswhat-the/#comment-80 Thu, 27 Apr 2006 01:39:04 +0000 http://www.melbournechapter.net/wordpress/system-administration/rich/2006/04/27/robots-spiders-crawlerswhat-the/#comment-80 Careful not to list too specifically about private bits of information, such as Disallow: /myprivate/stuff.html or /newsletter/subscribers.csv as a hacker will be able to see your robots.txt and may hunt it for information about your site - not that you would leave something as open as that, but just remember to use .htaccess to make areas of your site forbidden. Just be conscious of the information you are making publicly available about your site. Careful not to list too specifically about private bits of information, such as Disallow: /myprivate/stuff.html or /newsletter/subscribers.csv as a hacker will be able to see your robots.txt and may hunt it for information about your site - not that you would leave something as open as that, but just remember to use .htaccess to make areas of your site forbidden. Just be conscious of the information you are making publicly available about your site.

]]>