User-agent: * Disallow: /admin/ Disallow: /data/ Disallow: /4ek*/ Disallow: /fl09/ Disallow: /ran/ Disallow: /test/ Disallow: /zorf/ Disallow: /protected/ Disallow: /403.shtml Disallow: /404.shtml Disallow: /500.shtml Disallow: /gallery/ Disallow: /gallerypan/ Disallow: /pic/ Disallow: /picpro/ Disallow: /style/ Disallow: /wu/ Disallow: /js/ # Tells Scanning Robots Where They Are Not Welcome # --------------------- # User-agent: * applies to all robots, can also specify by name # Disallow: / disallow indexing of all pages. # If this matches first part of requested path: forget it! # NOTE: "Allow" tag is not officially endorsed # --------------------- # example # User-agent: infoseek # Disallow: /cgi-bin/ # --------------------- # http://www.searchengineworld.com/cgi-bin/robotcheck.cgi - Validation # http://www.jimtools.com/ go to Search Engine Stuff - Keywords Density