4f9720e76e
This runs gerrit in a container on review-dev01 using podman. Remove an unused web_server.py file that we found from copying it from puppet to ansible. Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
12 lines
206 B
Plaintext
12 lines
206 B
Plaintext
# Directions for web crawlers.
|
|
# See http://www.robotstxt.org/wc/norobots.html.
|
|
|
|
User-agent: HTTrack
|
|
User-agent: puf
|
|
User-agent: MSIECrawler
|
|
User-agent: Nutch
|
|
Disallow: /
|
|
|
|
User-agent: msnbot
|
|
Crawl-delay: 1
|