Add robots.txt to exclude crawlers from walking a 'staging' test publishing dir

Review Request #2072 — Created April 13, 2015 and submitted

zundel
pants
zundel/add-robots-txt
1416
b876188...
pants-reviews
jsirois, lahosken

Add robots.txt to exclude crawlers from walking a 'staging' test publishing dir

See https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961

$ ./build-support/bin/publish_docs.sh

robots.txt is copied to root of docs

./build-support/bin/publish_docs.sh -d zundel -p

Now publishes the site under the directory to staging/zundel where it will be excluded from crawlers that obey the robots.txt settings.

JS
  1. Ship It!
  2. 
      
LA
  1. Ship It!
  2. 
      
ZU
Review request changed

Status: Closed (submitted)

Change Summary:

Thanks for the reviews John & Larry. Commit 734bd04

Loading...