| commit | bcca8fdf1f042045be46068ed6398ea321677e4d | |
| author | Kyle J. McKay <mackyle@gmail.com> | |
| Sun, 26 Jan 2014 08:02:08 +0000 (26 00:02 -0800) | ||
| committer | Kyle J. McKay <mackyle@gmail.com> | |
| Sun, 26 Jan 2014 08:02:08 +0000 (26 00:02 -0800) | ||
| tree | d40978bde183a2479e246aac1baf310b1b953192 | treesnapshot (tar.gz zip) |
| parent | d2df4f8f027c9466dde0878dd7f592073fc29a73 | commitdiff |
robots.txt: add basic exclusions
Crawlers should not be visiting the blame, graphiclog or snapshot
links. Add specific exclusions to make it so.
Unfortunately the blame and snapshot links require wildcard
support to match which some robots may not support.
Crawlers should not be visiting the blame, graphiclog or snapshot
links. Add specific exclusions to make it so.
Unfortunately the blame and snapshot links require wildcard
support to match which some robots may not support.
| robots.txt | diffblobblamehistory |