| commit | 74b7f42c003942542f4c30ff06448aab91cef644 | |
| author | Kyle J. McKay <mackyle@gmail.com> | |
| Tue, 4 Feb 2014 09:23:19 +0000 (4 01:23 -0800) | ||
| committer | Kyle J. McKay <mackyle@gmail.com> | |
| Tue, 4 Feb 2014 09:23:19 +0000 (4 01:23 -0800) | ||
| tree | e30426bb97a9ec4c60de24917a3f0e3470536008 | treesnapshot (tar.gz zip) |
| parent | 87a0aa5984740d377b61d163563c5f52545133a3 | commitdiff |
robots.txt: add more bot exclusions
No reason for bots to be visiting any of the cgi pages other than
gitweb. Also no reason for bots to be fetching feeds as the same
information will be fetched via links on the gitweb pages. Also
disallow fetching the project list link off project pages as the
full project list will be retrieved via the project list link on
the home page.
No reason for bots to be visiting any of the cgi pages other than
gitweb. Also no reason for bots to be fetching feeds as the same
information will be fetched via links on the gitweb pages. Also
disallow fetching the project list link off project pages as the
full project list will be retrieved via the project list link on
the home page.
| robots.txt | diffblobblamehistory |