No OneTemporary
Actions

Size

1 KB

Referenced Files

None

Subscribers

None

View Options

	diff --git a/src/applications/system/controller/PhabricatorRobotsController.php b/src/applications/system/controller/PhabricatorRobotsController.php
	index e413984f5a..18be8862a8 100644
	--- a/src/applications/system/controller/PhabricatorRobotsController.php
	+++ b/src/applications/system/controller/PhabricatorRobotsController.php
	@@ -1,38 +1,39 @@
	<?php

	final class PhabricatorRobotsController extends PhabricatorController {

	public function shouldRequireLogin() {
	return false;
	}

	public function processRequest() {
	$out = array();

	// Prevent indexing of '/diffusion/', since the content is not generally
	// useful to index, web spiders get stuck scraping the history of every
	// file, and much of the content is Ajaxed in anyway so spiders won't even
	// see it. These pages are also relatively expensive to generate.

	// Note that this still allows commits (at '/rPxxxxx') to be indexed.
	// They're probably not hugely useful, but suffer fewer of the problems
	// Diffusion suffers and are hard to omit with 'robots.txt'.

	$out[] = 'User-Agent: *';
	$out[] = 'Disallow: /diffusion/';
	+ $out[] = 'Disallow: /source/';

	// Add a small crawl delay (number of seconds between requests) for spiders
	// which respect it. The intent here is to prevent spiders from affecting
	// performance for users. The possible cost is slower indexing, but that
	// seems like a reasonable tradeoff, since most Phabricator installs are
	// probably not hugely concerned about cutting-edge SEO.
	$out[] = 'Crawl-delay: 1';

	$content = implode("\n", $out)."\n";

	return id(new AphrontPlainTextResponse())
	->setContent($content)
	->setCacheDurationInSeconds(phutil_units('2 hours in seconds'))
	->setCanCDN(true);
	}
	}

File Metadata

Mime Type: text/x-diff
Expires: Fri, Mar 14, 6:27 AM (2 h, 40 m)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 71636
Default Alt Text: (1 KB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions