Search engines make use of a file called robots.txt to determine which web pages (and other files) are excluded from indexing when they visit your site.

Not all search robots honour this file. Never keep any sensitive information in your public_html directory as it could be viewed by anyone or indexed by search engine robots.

Some services also index graphics on the web. If you don’t want your images to be accessed in this manner, include them in your robots.txt file.

If you are using software such as WordPress or Drupal, this is handled by the software and you will need to adjust any settings for it within your Admin section.

In case you wish to use a robots.txt file, here is a sample. Simply cut in-between the dashed lines and paste into a plain text document. Name the file ‘robots.txt’ and upload to your public_html folder.

---------
User-agent: *
Disallow: /images/
Disallow: /templates/
Disallow: /cgi-bin/
Disallow: *.jpg
Disallow: *.gif
---------