cgit/robots.txt, branch masterA hyperfast web frontend for git repositories written in C.https://git.zx2c4.com/cgit/atom/robots.txt?h=master2022-12-19T14:14:05Zui-tree,ui-blame: bail from blame if blob is binary2022-12-19T14:14:05ZJune McEnroejune@causal.agency2019-12-18T21:30:12Zurn:sha1:cc6d9cc7fc010db9be6c2d90fd054fb2d189d629
This avoids piping binary blobs through the source-filter. Also prevent
robots from crawling it, since it's expensive.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
robots.txt: disallow access to snapshots2013-08-12T19:14:10ZJason A. DonenfeldJason@zx2c4.com2013-05-28T12:17:00Zurn:sha1:23debef62104c70600be2b745ec3957538eeac6e
My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.
Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>