No OneTemporary
Actions

Size

70 KB

Referenced Files

None

Subscribers

None

View Options

	diff --git a/src/aphront/configuration/AphrontApplicationConfiguration.php b/src/aphront/configuration/AphrontApplicationConfiguration.php
	index b48ea3be13..dc32faedec 100644
	--- a/src/aphront/configuration/AphrontApplicationConfiguration.php
	+++ b/src/aphront/configuration/AphrontApplicationConfiguration.php
	@@ -1,642 +1,654 @@
	<?php

	/**
	* @task routing URI Routing
	* @task response Response Handling
	* @task exception Exception Handling
	*/
	abstract class AphrontApplicationConfiguration extends Phobject {

	private $request;
	private $host;
	private $path;
	private $console;

	abstract public function buildRequest();
	abstract public function build404Controller();
	abstract public function buildRedirectController($uri, $external);

	final public function setRequest(AphrontRequest $request) {
	$this->request = $request;
	return $this;
	}

	final public function getRequest() {
	return $this->request;
	}

	final public function getConsole() {
	return $this->console;
	}

	final public function setConsole($console) {
	$this->console = $console;
	return $this;
	}

	final public function setHost($host) {
	$this->host = $host;
	return $this;
	}

	final public function getHost() {
	return $this->host;
	}

	final public function setPath($path) {
	$this->path = $path;
	return $this;
	}

	final public function getPath() {
	return $this->path;
	}

	public function willBuildRequest() {}


	/**
	* @phutil-external-symbol class PhabricatorStartup
	*/
	public static function runHTTPRequest(AphrontHTTPSink $sink) {
	PhabricatorStartup::beginStartupPhase('multimeter');
	$multimeter = MultimeterControl::newInstance();
	$multimeter->setEventContext('<http-init>');
	$multimeter->setEventViewer('<none>');

	// Build a no-op write guard for the setup phase. We'll replace this with a
	// real write guard later on, but we need to survive setup and build a
	// request object first.
	$write_guard = new AphrontWriteGuard('id');

	PhabricatorStartup::beginStartupPhase('env.init');
	PhabricatorEnv::initializeWebEnvironment();

	$multimeter->setSampleRate(
	PhabricatorEnv::getEnvConfig('debug.sample-rate'));

	$debug_time_limit = PhabricatorEnv::getEnvConfig('debug.time-limit');
	if ($debug_time_limit) {
	PhabricatorStartup::setDebugTimeLimit($debug_time_limit);
	}

	// This is the earliest we can get away with this, we need env config first.
	PhabricatorStartup::beginStartupPhase('log.access');
	PhabricatorAccessLog::init();
	$access_log = PhabricatorAccessLog::getLog();
	PhabricatorStartup::setAccessLog($access_log);
	$access_log->setData(
	array(
	'R' => AphrontRequest::getHTTPHeader('Referer', '-'),
	'r' => idx($_SERVER, 'REMOTE_ADDR', '-'),
	'M' => idx($_SERVER, 'REQUEST_METHOD', '-'),
	));

	DarkConsoleXHProfPluginAPI::hookProfiler();

	// We just activated the profiler, so we don't need to keep track of
	// startup phases anymore: it can take over from here.
	PhabricatorStartup::beginStartupPhase('startup.done');

	DarkConsoleErrorLogPluginAPI::registerErrorHandler();

	$response = PhabricatorSetupCheck::willProcessRequest();
	if ($response) {
	PhabricatorStartup::endOutputCapture();
	$sink->writeResponse($response);
	return;
	}

	$host = AphrontRequest::getHTTPHeader('Host');
	$path = $_REQUEST['__path__'];

	switch ($host) {
	default:
	$config_key = 'aphront.default-application-configuration-class';
	$application = PhabricatorEnv::newObjectFromConfig($config_key);
	break;
	}

	$application->setHost($host);
	$application->setPath($path);
	$application->willBuildRequest();
	$request = $application->buildRequest();

	// Now that we have a request, convert the write guard into one which
	// actually checks CSRF tokens.
	$write_guard->dispose();
	$write_guard = new AphrontWriteGuard(array($request, 'validateCSRF'));

	// Build the server URI implied by the request headers. If an administrator
	// has not configured "phabricator.base-uri" yet, we'll use this to generate
	// links.

	$request_protocol = ($request->isHTTPS() ? 'https' : 'http');
	$request_base_uri = "{$request_protocol}://{$host}/";
	PhabricatorEnv::setRequestBaseURI($request_base_uri);

	$access_log->setData(
	array(
	'U' => (string)$request->getRequestURI()->getPath(),
	));

	$processing_exception = null;
	try {
	$response = $application->processRequest(
	$request,
	$access_log,
	$sink,
	$multimeter);
	$response_code = $response->getHTTPResponseCode();
	} catch (Exception $ex) {
	$processing_exception = $ex;
	$response_code = 500;
	}

	$write_guard->dispose();

	$access_log->setData(
	array(
	'c' => $response_code,
	'T' => PhabricatorStartup::getMicrosecondsSinceStart(),
	));

	$multimeter->newEvent(
	MultimeterEvent::TYPE_REQUEST_TIME,
	$multimeter->getEventContext(),
	PhabricatorStartup::getMicrosecondsSinceStart());

	$access_log->write();

	$multimeter->saveEvents();

	DarkConsoleXHProfPluginAPI::saveProfilerSample($access_log);

	// Add points to the rate limits for this request.
	if (isset($_SERVER['REMOTE_ADDR'])) {
	$user_ip = $_SERVER['REMOTE_ADDR'];

	// The base score for a request allows users to make 30 requests per
	// minute.
	$score = (1000 / 30);

	// If the user was logged in, let them make more requests.
	if ($request->getUser() && $request->getUser()->getPHID()) {
	$score = $score / 5;
	}

	PhabricatorStartup::addRateLimitScore($user_ip, $score);
	}

	if ($processing_exception) {
	throw $processing_exception;
	}
	}


	public function processRequest(
	AphrontRequest $request,
	PhutilDeferredLog $access_log,
	AphrontHTTPSink $sink,
	MultimeterControl $multimeter) {

	$this->setRequest($request);

	list($controller, $uri_data) = $this->buildController();

	$controller_class = get_class($controller);
	$access_log->setData(
	array(
	'C' => $controller_class,
	));
	$multimeter->setEventContext('web.'.$controller_class);

	$request->setController($controller);
	$request->setURIMap($uri_data);

	$controller->setRequest($request);

	// If execution throws an exception and then trying to render that
	// exception throws another exception, we want to show the original
	// exception, as it is likely the root cause of the rendering exception.
	$original_exception = null;
	try {
	$response = $controller->willBeginExecution();

	if ($request->getUser() && $request->getUser()->getPHID()) {
	$access_log->setData(
	array(
	'u' => $request->getUser()->getUserName(),
	'P' => $request->getUser()->getPHID(),
	));
	$multimeter->setEventViewer('user.'.$request->getUser()->getPHID());
	}

	if (!$response) {
	$controller->willProcessRequest($uri_data);
	$response = $controller->handleRequest($request);
	$this->validateControllerResponse($controller, $response);
	}
	} catch (Exception $ex) {
	$original_exception = $ex;
	$response = $this->handleException($ex);
	}

	try {
	$response = $this->produceResponse($request, $response);
	$response = $controller->willSendResponse($response);
	$response->setRequest($request);

	$unexpected_output = PhabricatorStartup::endOutputCapture();
	if ($unexpected_output) {
	$unexpected_output = pht(
	"Unexpected output:\n\n%s",
	$unexpected_output);

	phlog($unexpected_output);

	if ($response instanceof AphrontWebpageResponse) {
	echo phutil_tag(
	'div',
	array(
	'style' =>
	'background: #eeddff;'.
	'white-space: pre-wrap;'.
	'z-index: 200000;'.
	'position: relative;'.
	'padding: 8px;'.
	'font-family: monospace',
	),
	$unexpected_output);
	}
	}

	$sink->writeResponse($response);
	} catch (Exception $ex) {
	if ($original_exception) {
	throw $original_exception;
	}
	throw $ex;
	}

	return $response;
	}


	/* -( URI Routing )-------------------------------------------------------- */


	/**
	* Build a controller to respond to the request.
	*
	* @return pair<AphrontController,dict> Controller and dictionary of request
	* parameters.
	* @task routing
	*/
	final private function buildController() {
	$request = $this->getRequest();

	// If we're configured to operate in cluster mode, reject requests which
	// were not received on a cluster interface.
	//
	// For example, a host may have an internal address like "170.0.0.1", and
	// also have a public address like "51.23.95.16". Assuming the cluster
	// is configured on a range like "170.0.0.0/16", we want to reject the
	// requests received on the public interface.
	//
	// Ideally, nodes in a cluster should only be listening on internal
	// interfaces, but they may be configured in such a way that they also
	// listen on external interfaces, since this is easy to forget about or
	// get wrong. As a broad security measure, reject requests received on any
	// interfaces which aren't on the whitelist.

	$cluster_addresses = PhabricatorEnv::getEnvConfig('cluster.addresses');
	if ($cluster_addresses) {
	$server_addr = idx($_SERVER, 'SERVER_ADDR');
	if (!$server_addr) {
	if (php_sapi_name() == 'cli') {
	// This is a command line script (probably something like a unit
	// test) so it's fine that we don't have SERVER_ADDR defined.
	} else {
	throw new AphrontMalformedRequestException(
	pht('No %s', 'SERVER_ADDR'),
	pht(
	'Phabricator is configured to operate in cluster mode, but '.
	'%s is not defined in the request context. Your webserver '.
	'configuration needs to forward %s to PHP so Phabricator can '.
	'reject requests received on external interfaces.',
	'SERVER_ADDR',
	'SERVER_ADDR'));
	}
	} else {
	if (!PhabricatorEnv::isClusterAddress($server_addr)) {
	throw new AphrontMalformedRequestException(
	pht('External Interface'),
	pht(
	'Phabricator is configured in cluster mode and the address '.
	'this request was received on ("%s") is not whitelisted as '.
	'a cluster address.',
	$server_addr));
	}
	}
	}

	$site = $this->buildSiteForRequest($request);

	if ($site->shouldRequireHTTPS()) {
	if (!$request->isHTTPS()) {
	+
	+ // Don't redirect intracluster requests: doing so drops headers and
	+ // parameters, imposes a performance penalty, and indicates a
	+ // misconfiguration.
	+ if ($request->isProxiedClusterRequest()) {
	+ throw new AphrontMalformedRequestException(
	+ pht('HTTPS Required'),
	+ pht(
	+ 'This request reached a site which requires HTTPS, but the '.
	+ 'request is not marked as HTTPS.'));
	+ }
	+
	$https_uri = $request->getRequestURI();
	$https_uri->setDomain($request->getHost());
	$https_uri->setProtocol('https');

	// In this scenario, we'll be redirecting to HTTPS using an absolute
	// URI, so we need to permit an external redirect.
	return $this->buildRedirectController($https_uri, true);
	}
	}

	$maps = $site->getRoutingMaps();
	$path = $request->getPath();

	$result = $this->routePath($maps, $path);
	if ($result) {
	return $result;
	}

	// If we failed to match anything but don't have a trailing slash, try
	// to add a trailing slash and issue a redirect if that resolves.

	// NOTE: We only do this for GET, since redirects switch to GET and drop
	// data like POST parameters.
	if (!preg_match('@/$@', $path) && $request->isHTTPGet()) {
	$result = $this->routePath($maps, $path.'/');
	if ($result) {
	$slash_uri = $request->getRequestURI()->setPath($path.'/');

	// We need to restore URI encoding because the webserver has
	// interpreted it. For example, this allows us to redirect a path
	// like `/tag/aa%20bb` to `/tag/aa%20bb/`, which may eventually be
	// resolved meaningfully by an application.
	$slash_uri = phutil_escape_uri($slash_uri);

	$external = strlen($request->getRequestURI()->getDomain());
	return $this->buildRedirectController($slash_uri, $external);
	}
	}

	return $this->build404Controller();
	}

	/**
	* Map a specific path to the corresponding controller. For a description
	* of routing, see @{method:buildController}.
	*
	* @param list<AphrontRoutingMap> List of routing maps.
	* @param string Path to route.
	* @return pair<AphrontController,dict> Controller and dictionary of request
	* parameters.
	* @task routing
	*/
	private function routePath(array $maps, $path) {
	foreach ($maps as $map) {
	$result = $map->routePath($path);
	if ($result) {
	return array($result->getController(), $result->getURIData());
	}
	}
	}

	private function buildSiteForRequest(AphrontRequest $request) {
	$sites = PhabricatorSite::getAllSites();

	$site = null;
	foreach ($sites as $candidate) {
	$site = $candidate->newSiteForRequest($request);
	if ($site) {
	break;
	}
	}

	if (!$site) {
	$path = $request->getPath();
	$host = $request->getHost();
	throw new AphrontMalformedRequestException(
	pht('Site Not Found'),
	pht(
	'This request asked for "%s" on host "%s", but no site is '.
	'configured which can serve this request.',
	$path,
	$host),
	true);
	}

	$request->setSite($site);

	return $site;
	}


	/* -( Response Handling )-------------------------------------------------- */


	/**
	* Tests if a response is of a valid type.
	*
	* @param wild Supposedly valid response.
	* @return bool True if the object is of a valid type.
	* @task response
	*/
	private function isValidResponseObject($response) {
	if ($response instanceof AphrontResponse) {
	return true;
	}

	if ($response instanceof AphrontResponseProducerInterface) {
	return true;
	}

	return false;
	}


	/**
	* Verifies that the return value from an @{class:AphrontController} is
	* of an allowed type.
	*
	* @param AphrontController Controller which returned the response.
	* @param wild Supposedly valid response.
	* @return void
	* @task response
	*/
	private function validateControllerResponse(
	AphrontController $controller,
	$response) {

	if ($this->isValidResponseObject($response)) {
	return;
	}

	throw new Exception(
	pht(
	'Controller "%s" returned an invalid response from call to "%s". '.
	'This method must return an object of class "%s", or an object '.
	'which implements the "%s" interface.',
	get_class($controller),
	'handleRequest()',
	'AphrontResponse',
	'AphrontResponseProducerInterface'));
	}


	/**
	* Verifies that the return value from an
	* @{class:AphrontResponseProducerInterface} is of an allowed type.
	*
	* @param AphrontResponseProducerInterface Object which produced
	* this response.
	* @param wild Supposedly valid response.
	* @return void
	* @task response
	*/
	private function validateProducerResponse(
	AphrontResponseProducerInterface $producer,
	$response) {

	if ($this->isValidResponseObject($response)) {
	return;
	}

	throw new Exception(
	pht(
	'Producer "%s" returned an invalid response from call to "%s". '.
	'This method must return an object of class "%s", or an object '.
	'which implements the "%s" interface.',
	get_class($producer),
	'produceAphrontResponse()',
	'AphrontResponse',
	'AphrontResponseProducerInterface'));
	}


	/**
	* Verifies that the return value from an
	* @{class:AphrontRequestExceptionHandler} is of an allowed type.
	*
	* @param AphrontRequestExceptionHandler Object which produced this
	* response.
	* @param wild Supposedly valid response.
	* @return void
	* @task response
	*/
	private function validateErrorHandlerResponse(
	AphrontRequestExceptionHandler $handler,
	$response) {

	if ($this->isValidResponseObject($response)) {
	return;
	}

	throw new Exception(
	pht(
	'Exception handler "%s" returned an invalid response from call to '.
	'"%s". This method must return an object of class "%s", or an object '.
	'which implements the "%s" interface.',
	get_class($handler),
	'handleRequestException()',
	'AphrontResponse',
	'AphrontResponseProducerInterface'));
	}


	/**
	* Resolves a response object into an @{class:AphrontResponse}.
	*
	* Controllers are permitted to return actual responses of class
	* @{class:AphrontResponse}, or other objects which implement
	* @{interface:AphrontResponseProducerInterface} and can produce a response.
	*
	* If a controller returns a response producer, invoke it now and produce
	* the real response.
	*
	* @param AphrontRequest Request being handled.
	* @param AphrontResponse\|AphrontResponseProducerInterface Response, or
	* response producer.
	* @return AphrontResponse Response after any required production.
	* @task response
	*/
	private function produceResponse(AphrontRequest $request, $response) {
	$original = $response;

	// Detect cycles on the exact same objects. It's still possible to produce
	// infinite responses as long as they're all unique, but we can only
	// reasonably detect cycles, not guarantee that response production halts.

	$seen = array();
	while (true) {
	// NOTE: It is permissible for an object to be both a response and a
	// response producer. If so, being a producer is "stronger". This is
	// used by AphrontProxyResponse.

	// If this response is a valid response, hand over the request first.
	if ($response instanceof AphrontResponse) {
	$response->setRequest($request);
	}

	// If this isn't a producer, we're all done.
	if (!($response instanceof AphrontResponseProducerInterface)) {
	break;
	}

	$hash = spl_object_hash($response);
	if (isset($seen[$hash])) {
	throw new Exception(
	pht(
	'Failure while producing response for object of class "%s": '.
	'encountered production cycle (identical object, of class "%s", '.
	'was produced twice).',
	get_class($original),
	get_class($response)));
	}

	$seen[$hash] = true;

	$new_response = $response->produceAphrontResponse();
	$this->validateProducerResponse($response, $new_response);
	$response = $new_response;
	}

	return $response;
	}


	/* -( Error Handling )----------------------------------------------------- */


	/**
	* Convert an exception which has escaped the controller into a response.
	*
	* This method delegates exception handling to available subclasses of
	* @{class:AphrontRequestExceptionHandler}.
	*
	* @param Exception Exception which needs to be handled.
	* @return wild Response or response producer, or null if no available
	* handler can produce a response.
	* @task exception
	*/
	private function handleException(Exception $ex) {
	$handlers = AphrontRequestExceptionHandler::getAllHandlers();

	$request = $this->getRequest();
	foreach ($handlers as $handler) {
	if ($handler->canHandleRequestException($request, $ex)) {
	$response = $handler->handleRequestException($request, $ex);
	$this->validateErrorHandlerResponse($handler, $response);
	return $response;
	}
	}

	throw $ex;
	}


	}
	diff --git a/src/aphront/site/PhabricatorSite.php b/src/aphront/site/PhabricatorSite.php
	index 866ca178ca..95e0a538fa 100644
	--- a/src/aphront/site/PhabricatorSite.php
	+++ b/src/aphront/site/PhabricatorSite.php
	@@ -1,9 +1,18 @@
	<?php

	abstract class PhabricatorSite extends AphrontSite {

	public function shouldRequireHTTPS() {
	+ // If this is an intracluster request, it's okay for it to use HTTP even
	+ // if the site otherwise requires HTTPS. It is common to terminate SSL at
	+ // a load balancer and use plain HTTP from then on, and administrators are
	+ // usually not concerned about attackers observing traffic within a
	+ // datacenter.
	+ if (PhabricatorEnv::isClusterRemoteAddress()) {
	+ return false;
	+ }
	+
	return PhabricatorEnv::getEnvConfig('security.require-https');
	}

	}
	diff --git a/src/applications/config/option/PhabricatorClusterConfigOptions.php b/src/applications/config/option/PhabricatorClusterConfigOptions.php
	index bed97b15a5..bcf498c32b 100644
	--- a/src/applications/config/option/PhabricatorClusterConfigOptions.php
	+++ b/src/applications/config/option/PhabricatorClusterConfigOptions.php
	@@ -1,113 +1,120 @@
	<?php

	final class PhabricatorClusterConfigOptions
	extends PhabricatorApplicationConfigOptions {

	public function getName() {
	return pht('Cluster Setup');
	}

	public function getDescription() {
	return pht('Configure Phabricator to run on a cluster of hosts.');
	}

	public function getIcon() {
	return 'fa-sitemap';
	}

	public function getGroup() {
	return 'core';
	}

	public function getOptions() {
	$databases_type = 'custom:PhabricatorClusterDatabasesConfigOptionType';
	$databases_help = $this->deformat(pht(<<<EOTEXT
	WARNING: This is a prototype option and the description below is currently pure
	fantasy.

	This option allows you to make Phabricator aware of database read replicas so
	it can monitor database health, spread load, and degrade gracefully to
	read-only mode in the event of a failure on the primary host. For help with
	configuring cluster databases, see [[ %s \| %s ]] in the documentation.
	EOTEXT
	,
	PhabricatorEnv::getDoclink('Cluster: Databases'),
	pht('Cluster: Databases')));

	+
	+ $intro_href = PhabricatorEnv::getDoclink('Clustering Introduction');
	+ $intro_name = pht('Clustering Introduction');
	+
	return array(
	$this->newOption('cluster.addresses', 'list<string>', array())
	->setLocked(true)
	->setSummary(pht('Address ranges of cluster hosts.'))
	->setDescription(
	pht(
	- 'To allow Phabricator nodes to communicate with other nodes '.
	- 'in the cluster, provide an address whitelist of hosts that '.
	- 'are part of the cluster.'.
	+ 'Define a Phabricator cluster by providing a whitelist of host '.
	+ 'addresses that are part of the cluster.'.
	"\n\n".
	- 'Hosts on this whitelist are permitted to use special cluster '.
	- 'mechanisms to authenticate requests. By default, these '.
	- 'mechanisms are disabled.'.
	+ 'Hosts on this whitelist have special powers. These hosts are '.
	+ 'permitted to bend security rules, and misconfiguring this list '.
	+ 'can make your install less secure. For more information, '.
	+ 'see [[ %s \| %s ]].'.
	"\n\n".
	'Define a list of CIDR blocks which whitelist all hosts in the '.
	- 'cluster. See the examples below for details.',
	+ 'cluster and no additional hosts. See the examples below for '.
	+ 'details.'.
	"\n\n".
	'When cluster addresses are defined, Phabricator hosts will also '.
	- 'reject requests to interfaces which are not whitelisted.'))
	+ 'reject requests to interfaces which are not whitelisted.',
	+ $intro_href,
	+ $intro_name))
	->addExample(
	array(
	'23.24.25.80/32',
	'23.24.25.81/32',
	),
	pht('Whitelist Specific Addresses'))
	->addExample(
	array(
	'1.2.3.0/24',
	),
	pht('Whitelist 1.2.3.*'))
	->addExample(
	array(
	'1.2.0.0/16',
	),
	pht('Whitelist 1.2..'))
	->addExample(
	array(
	'0.0.0.0/0',
	),
	pht('Allow Any Host (Insecure!)')),
	$this->newOption('cluster.instance', 'string', null)
	->setLocked(true)
	->setSummary(pht('Instance identifier for multi-tenant clusters.'))
	->setDescription(
	pht(
	'WARNING: This is a very advanced option, and only useful for '.
	'hosting providers running multi-tenant clusters.'.
	"\n\n".
	'If you provide an instance identifier here (normally by '.
	'injecting it with a `%s`), Phabricator will pass it to '.
	'subprocesses and commit hooks in the `%s` environmental variable.',
	'PhabricatorConfigSiteSource',
	'PHABRICATOR_INSTANCE')),
	$this->newOption('cluster.read-only', 'bool', false)
	->setLocked(true)
	->setSummary(
	pht(
	'Activate read-only mode for maintenance or disaster recovery.'))
	->setDescription(
	pht(
	'WARNING: This is a prototype option and the description below '.
	'is currently pure fantasy.'.
	"\n\n".
	'Switch Phabricator to read-only mode. In this mode, users will '.
	'be unable to write new data. Normally, the cluster degrades '.
	'into this mode automatically when it detects that the database '.
	'master is unreachable, but you can activate it manually in '.
	'order to perform maintenance or test configuration.')),
	$this->newOption('cluster.databases', $databases_type, array())
	->setHidden(true)
	->setSummary(
	pht('Configure database read replicas.'))
	->setDescription($databases_help),
	);
	}

	}
	diff --git a/src/applications/config/option/PhabricatorSecurityConfigOptions.php b/src/applications/config/option/PhabricatorSecurityConfigOptions.php
	index 8c9907d736..311c2d30b6 100644
	--- a/src/applications/config/option/PhabricatorSecurityConfigOptions.php
	+++ b/src/applications/config/option/PhabricatorSecurityConfigOptions.php
	@@ -1,325 +1,328 @@
	<?php

	final class PhabricatorSecurityConfigOptions
	extends PhabricatorApplicationConfigOptions {

	public function getName() {
	return pht('Security');
	}

	public function getDescription() {
	return pht('Security options.');
	}

	public function getIcon() {
	return 'fa-lock';
	}

	public function getGroup() {
	return 'core';
	}

	public function getOptions() {
	$doc_href = PhabricatorEnv::getDoclink('Configuring a File Domain');
	$doc_name = pht('Configuration Guide: Configuring a File Domain');

	// This is all of the IANA special/reserved blocks in IPv4 space.
	$default_address_blacklist = array(
	'0.0.0.0/8',
	'10.0.0.0/8',
	'100.64.0.0/10',
	'127.0.0.0/8',
	'169.254.0.0/16',
	'172.16.0.0/12',
	'192.0.0.0/24',
	'192.0.2.0/24',
	'192.88.99.0/24',
	'192.168.0.0/16',
	'198.18.0.0/15',
	'198.51.100.0/24',
	'203.0.113.0/24',
	'224.0.0.0/4',
	'240.0.0.0/4',
	'255.255.255.255/32',
	);

	return array(
	$this->newOption('security.alternate-file-domain', 'string', null)
	->setLocked(true)
	->setSummary(pht('Alternate domain to serve files from.'))
	->setDescription(
	pht(
	'By default, Phabricator serves files from the same domain '.
	'the application is served from. This is convenient, but '.
	'presents a security risk.'.
	"\n\n".
	'You should configure a CDN or alternate file domain to mitigate '.
	'this risk. Configuring a CDN will also improve performance. See '.
	'[[ %s \| %s ]] for instructions.',
	$doc_href,
	$doc_name))
	->addExample('https://files.phabcdn.net/', pht('Valid Setting')),
	$this->newOption(
	'security.hmac-key',
	'string',
	'[D\t~Y7eNmnQGJ;rnH6aF;m2!vJ8@v8C=Cs:aQS\.Qw')
	->setHidden(true)
	->setSummary(
	pht('Key for HMAC digests.'))
	->setDescription(
	pht(
	'Default key for HMAC digests where the key is not important '.
	'(i.e., the hash itself is secret). You can change this if you '.
	'want (to any other string), but doing so will break existing '.
	'sessions and CSRF tokens.')),
	$this->newOption('security.require-https', 'bool', false)
	->setLocked(true)
	->setSummary(
	pht('Force users to connect via HTTPS instead of HTTP.'))
	->setDescription(
	pht(
	"If the web server responds to both HTTP and HTTPS requests but ".
	"you want users to connect with only HTTPS, you can set this ".
	- "to true to make Phabricator redirect HTTP requests to HTTPS.\n\n".
	-
	+ "to `true` to make Phabricator redirect HTTP requests to HTTPS.".
	+ "\n\n".
	"Normally, you should just configure your server not to accept ".
	"HTTP traffic, but this setting may be useful if you originally ".
	"used HTTP and have now switched to HTTPS but don't want to ".
	"break old links, or if your webserver sits behind a load ".
	"balancer which terminates HTTPS connections and you can not ".
	- "reasonably configure more granular behavior there.\n\n".
	-
	+ "reasonably configure more granular behavior there.".
	+ "\n\n".
	"IMPORTANT: Phabricator determines if a request is HTTPS or not ".
	"by examining the PHP `%s` variable. If you run ".
	"Apache/mod_php this will probably be set correctly for you ".
	"automatically, but if you run Phabricator as CGI/FCGI (e.g., ".
	"through nginx or lighttpd), you need to configure your web ".
	"server so that it passes the value correctly based on the ".
	- "connection type.",
	+ "connection type.".
	+ "\n\n".
	+ "If you configure Phabricator in cluster mode, note that this ".
	+ "setting is ignored by intracluster requests.",
	"\$_SERVER['HTTPS']"))
	->setBoolOptions(
	array(
	pht('Force HTTPS'),
	pht('Allow HTTP'),
	)),
	$this->newOption('security.require-multi-factor-auth', 'bool', false)
	->setLocked(true)
	->setSummary(
	pht('Require all users to configure multi-factor authentication.'))
	->setDescription(
	pht(
	'By default, Phabricator allows users to add multi-factor '.
	'authentication to their accounts, but does not require it. '.
	'By enabling this option, you can force all users to add '.
	'at least one authentication factor before they can use their '.
	'accounts.'))
	->setBoolOptions(
	array(
	pht('Multi-Factor Required'),
	pht('Multi-Factor Optional'),
	)),
	$this->newOption(
	'phabricator.csrf-key',
	'string',
	'0b7ec0592e0a2829d8b71df2fa269b2c6172eca3')
	->setHidden(true)
	->setSummary(
	pht('Hashed with other inputs to generate CSRF tokens.'))
	->setDescription(
	pht(
	'This is hashed with other inputs to generate CSRF tokens. If '.
	'you want, you can change it to some other string which is '.
	'unique to your install. This will make your install more secure '.
	'in a vague, mostly theoretical way. But it will take you like 3 '.
	'seconds of mashing on your keyboard to set it up so you might '.
	'as well.')),
	$this->newOption(
	'phabricator.mail-key',
	'string',
	'5ce3e7e8787f6e40dfae861da315a5cdf1018f12')
	->setHidden(true)
	->setSummary(
	pht('Hashed with other inputs to generate mail tokens.'))
	->setDescription(
	pht(
	"This is hashed with other inputs to generate mail tokens. If ".
	"you want, you can change it to some other string which is ".
	"unique to your install. In particular, you will want to do ".
	"this if you accidentally send a bunch of mail somewhere you ".
	"shouldn't have, to invalidate all old reply-to addresses.")),
	$this->newOption(
	'uri.allowed-protocols',
	'set',
	array(
	'http' => true,
	'https' => true,
	'mailto' => true,
	))
	->setSummary(
	pht('Determines which URI protocols are auto-linked.'))
	->setDescription(
	pht(
	"When users write comments which have URIs, they'll be ".
	"automatically linked if the protocol appears in this set. This ".
	"whitelist is primarily to prevent security issues like ".
	"%s URIs.",
	'javascript://'))
	->addExample("http\nhttps", pht('Valid Setting'))
	->setLocked(true),
	$this->newOption(
	'uri.allowed-editor-protocols',
	'set',
	array(
	'http' => true,
	'https' => true,

	// This handler is installed by Textmate.
	'txmt' => true,

	// This handler is for MacVim.
	'mvim' => true,

	// Unofficial handler for Vim.
	'vim' => true,

	// Unofficial handler for Sublime.
	'subl' => true,

	// Unofficial handler for Emacs.
	'emacs' => true,

	// This isn't a standard handler installed by an application, but
	// is a reasonable name for a user-installed handler.
	'editor' => true,
	))
	->setSummary(pht('Whitelists editor protocols for "Open in Editor".'))
	->setDescription(
	pht(
	'Users can configure a URI pattern to open files in a text '.
	'editor. The URI must use a protocol on this whitelist.'))
	->setLocked(true),
	$this->newOption(
	'celerity.resource-hash',
	'string',
	'd9455ea150622ee044f7931dabfa52aa')
	->setSummary(
	pht('An input to the hash function when building resource hashes.'))
	->setDescription(
	pht(
	'This value is an input to the hash function when building '.
	'resource hashes. It has no security value, but if you '.
	'accidentally poison user caches (by pushing a bad patch or '.
	'having something go wrong with a CDN, e.g.) you can change this '.
	'to something else and rebuild the Celerity map to break user '.
	'caches. Unless you are doing Celerity development, it is '.
	'exceptionally unlikely that you need to modify this.')),
	$this->newOption('remarkup.enable-embedded-youtube', 'bool', false)
	->setBoolOptions(
	array(
	pht('Embed YouTube videos'),
	pht("Don't embed YouTube videos"),
	))
	->setSummary(
	pht('Determines whether or not YouTube videos get embedded.'))
	->setDescription(
	pht(
	"If you enable this, linked YouTube videos will be embedded ".
	"inline. This has mild security implications (you'll leak ".
	"referrers to YouTube) and is pretty silly (but sort of ".
	"awesome).")),
	$this->newOption(
	'security.outbound-blacklist',
	'list<string>',
	$default_address_blacklist)
	->setLocked(true)
	->setSummary(
	pht(
	'Blacklist subnets to prevent user-initiated outbound '.
	'requests.'))
	->setDescription(
	pht(
	'Phabricator users can make requests to other services from '.
	'the Phabricator host in some circumstances (for example, by '.
	'creating a repository with a remote URL or having Phabricator '.
	'fetch an image from a remote server).'.
	"\n\n".
	'This may represent a security vulnerability if services on '.
	'the same subnet will accept commands or reveal private '.
	'information over unauthenticated HTTP GET, based on the source '.
	'IP address. In particular, all hosts in EC2 have access to '.
	'such a service.'.
	"\n\n".
	'This option defines a list of netblocks which Phabricator '.
	'will decline to connect to. Generally, you should list all '.
	'private IP space here.'))
	->addExample(array('0.0.0.0/0'), pht('No Outbound Requests')),
	$this->newOption('security.strict-transport-security', 'bool', false)
	->setLocked(true)
	->setBoolOptions(
	array(
	pht('Use HSTS'),
	pht('Do Not Use HSTS'),
	))
	->setSummary(pht('Enable HTTP Strict Transport Security (HSTS).'))
	->setDescription(
	pht(
	'HTTP Strict Transport Security (HSTS) sends a header which '.
	'instructs browsers that the site should only be accessed '.
	'over HTTPS, never HTTP. This defuses an attack where an '.
	'adversary gains access to your network, then proxies requests '.
	'through an unsecured link.'.
	"\n\n".
	'Do not enable this option if you serve (or plan to ever serve) '.
	'unsecured content over plain HTTP. It is very difficult to '.
	'undo this change once users\' browsers have accepted the '.
	'setting.')),
	);
	}

	protected function didValidateOption(
	PhabricatorConfigOption $option,
	$value) {

	$key = $option->getKey();
	if ($key == 'security.alternate-file-domain') {

	$uri = new PhutilURI($value);
	$protocol = $uri->getProtocol();
	if ($protocol !== 'http' && $protocol !== 'https') {
	throw new PhabricatorConfigValidationException(
	pht(
	"Config option '%s' is invalid. The URI must start with ".
	"'%s' or '%s'.",
	$key,
	'http://',
	'https://'));
	}

	$domain = $uri->getDomain();
	if (strpos($domain, '.') === false) {
	throw new PhabricatorConfigValidationException(
	pht(
	"Config option '%s' is invalid. The URI must contain a dot ('.'), ".
	"like '%s', not just a bare name like '%s'. ".
	"Some web browsers will not set cookies on domains with no TLD.",
	$key,
	'http://example.com/',
	'http://example/'));
	}

	$path = $uri->getPath();
	if ($path !== '' && $path !== '/') {
	throw new PhabricatorConfigValidationException(
	pht(
	"Config option '%s' is invalid. The URI must NOT have a path, ".
	"e.g. '%s' is OK, but '%s' is not. Phabricator must be installed ".
	"on an entire domain; it can not be installed on a path.",
	$key,
	'http://phabricator.example.com/',
	'http://example.com/phabricator/'));
	}
	}
	}


	}
	diff --git a/src/docs/user/cluster/cluster.diviner b/src/docs/user/cluster/cluster.diviner
	index 24f4261329..93800c81bb 100644
	--- a/src/docs/user/cluster/cluster.diviner
	+++ b/src/docs/user/cluster/cluster.diviner
	@@ -1,181 +1,255 @@
	@title Clustering Introduction
	@group cluster

	Guide to configuring Phabricator across multiple hosts for availability and
	performance.

	Overview
	========

	WARNING: This feature is a very early prototype; the features this document
	describes are mostly speculative fantasy.

	Phabricator can be configured to run on mulitple hosts with redundant services
	to improve its availability and scalability, and make disaster recovery much
	easier.

	Clustering is more complex to setup and maintain than running everything on a
	single host, but greatly reduces the cost of recovering from hardware and
	network failures.

	Each Phabricator service has an array of clustering options that can be
	configured independently. Configuring a cluster is inherently complex, and this
	is an advanced feature aimed at installs with large userbases and experienced
	operations personnel who need this high degree of flexibility.

	The remainder of this document summarizes how to add redundancy to each
	service and where your efforts are likely to have the greatest impact.

	For additional guidance on setting up a cluster, see "Overlaying Services"
	and "Cluster Recipes" at the bottom of this document.


	+Preparing for Clustering
	+========================
	+
	+To begin deploying Phabricator in cluster mode, set up `cluster.addresses`
	+in your configuration.
	+
	+This option should contain a list of network addess blocks which are considered
	+to be part of the cluster. Hosts in this list are allowed to bend (or even
	+break) some of the security and policy rules when they make requests to other
	+hosts in the cluster, so this list should be as small as possible. See "Cluster
	+Whitelist Security" below for discussion.
	+
	+If you are deploying hardware in EC2, a reasonable approach is to launch a
	+dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster,
	+and then deploy only Phabricator services into that VPC.
	+
	+If you have additional auxiliary hosts which run builds and tests via Drydock,
	+you should //not// include them in the cluster address definition. For more
	+detailed discussion of the Drydock security model, see @{Drydock User Guide:
	+Security}.
	+
	+Most other clustering features will not work until you define a cluster by
	+configuring `cluster.addresses`.
	+
	+
	+Cluster Whitelist Security
	+========================
	+
	+When you configure `cluster.addresses`, you should keep the list of trusted
	+cluster hosts as small as possible. Hosts on this list gain additional
	+capabilities, including these:
	+
	+Trusted HTTP Headers: Normally, Phabricator distrusts the load balancer
	+HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be
	+client-controlled and can be set to arbitrary values by an attacker if no load
	+balancer is deployed. In particular, clients can set `X-Forwarded-For` to any
	+value and spoof traffic from arbitrary remotes.
	+
	+These headers are trusted when they are received from a host on the cluster
	+address whitelist. This allows requests from cluster loadbalancers to be
	+interpreted correctly by default without requiring additional custom code or
	+configuration.
	+
	+Intracluster HTTP: Requests from cluster hosts are not required to use
	+HTTPS, even if `security.require-https` is enabled, because it is common to
	+terminate HTTPS on load balancers and use plain HTTP for requests within a
	+cluster.
	+
	+Special Authentication Mechanisms: Cluster hosts are allowed to connect to
	+other cluster hosts with "root credentials", and to impersonate any user
	+account.
	+
	+The use of root credentials is required because the daemons must be able to
	+bypass policies in order to function properly: they need to send mail about
	+private conversations and import commits in private repositories.
	+
	+The ability to impersonate users is required because SSH nodes must receive,
	+interpret, modify, and forward SSH traffic. They can not use the original
	+credentials to do this because SSH authentication is asymmetric and they do not
	+have the user's private key. Instead, they use root credentials and impersonate
	+the user within the cluster.
	+
	+These mechanisms are still authenticated (and use asymmetric keys, like SSH
	+does), so access to a host in the cluster address block does not mean that an
	+attacker can immediately compromise the cluster. However, an overbroad cluster
	+address whitelist may give an attacker who gains some access additional tools
	+to escalate access.
	+
	+Note that if an attacker gains access to an actual cluster host, these extra
	+powers are largely moot. Most cluster hosts must be able to connect to the
	+master database to function properly, so the attacker will just do that and
	+freely read or modify whatever data they want.
	+
	+
	Cluster: Databases
	=================

	Configuring multiple database hosts is moderately complex, but normally has the
	highest impact on availability and resistance to data loss. This is usually the
	most important service to make redundant if your focus is on availability and
	disaster recovery.

	Configuring replicas allows Phabricator to run in read-only mode if you lose
	-the master, and to quickly promote the replica as a replacement.
	+the master and to quickly promote the replica as a replacement.

	For details, see @{article:Cluster: Databases}.


	Cluster: Repositories
	=====================

	Configuring multiple repository hosts is complex, but is required before you
	can add multiple daemon or web hosts.

	Repository replicas are important for availability if you host repositories
	on Phabricator, but less important if you host repositories elsewhere
	(instead, you should focus on making that service more available).

	The distributed nature of Git and Mercurial tend to mean that they are
	naturally somewhat resistant to data loss: every clone of a repository includes
	the entire history.

	For details, see @{article:Cluster: Repositories}.


	Cluster: Daemons
	================

	Configuring multiple daemon hosts is straightforward, but you must configure
	repositories first.

	With daemons running on multiple hosts, you can transparently survive the loss
	of any subset of hosts without an interruption to daemon services, as long as
	at least one host remains alive. Daemons are stateless, so spreading daemons
	across multiple hosts provides no resistance to data loss.

	For details, see @{article:Cluster: Daemons}.


	Cluster: Web Servers
	====================

	Configuring multiple web hosts is straightforward, but you must configure
	repositories first.

	With multiple web hosts, you can transparently survive the loss of any subset
	of hosts as long as at least one host remains alive. Web hosts are stateless,
	so putting multiple hosts in service provides no resistance to data loss.

	For details, see @{article:Cluster: Web Servers}.


	Overlaying Services
	===================

	Although hosts can run a single dedicated service type, certain groups of
	services work well together. Phabricator clusters usually do not need to be
	very large, so deploying a small number of hosts with multiple services is a
	good place to start.

	In planning a cluster, consider these blended host types:

	Everything: Run HTTP, SSH, MySQL, repositories and daemons on a single
	host. This is the starting point for single-node setups, and usually also the
	best configuration when adding the second node.

	Everything Except Databases: Run HTTP, SSH, repositories and daemons on one
	host, and MySQL on a different host. MySQL uses many of the same resources that
	other services use. It's also simpler to separate than other services, and
	tends to benefit the most from dedicated hardware.

	Just Databases: Separating MySQL onto dedicated nodes

	Database nodes tend to benefit the most from

	Repositories and Daemons: Run repositories and daemons on the same host.
	Repository hosts //must// run daemons, and it normally makes sense to
	completely overlay repositories and daemons. These services tend to use
	different resources (repositories are heavier on I/O and lighter on CPU/RAM;
	daemons are heavier on CPU/RAM and lighter on I/O).

	Repositories and daemons are also both less latency sensitive than other
	service types, so there's a wider margin of error for underprovisioning them
	before performance is noticably affected.

	These nodes tend to use system resources in a balanced way. Individual nodes
	in this class do not need to be particularly powerful.

	Frontend Servers: Run HTTP and SSH on the same host. These are easy to set
	up, stateless, and you can scale the pool up or down easily to meet demand.
	Routing both types of ingress traffic through the same initial tier can
	simplify load balancing.

	These nodes tend to need relatively little RAM.


	Cluster Recipes
	===============

	This section provides some guidance on reasonable ways to scale up a cluster.

	The smallest possible cluster is two hosts. Run everything (web, ssh,
	database, repositories, and daemons) on each host. One host will serve as the
	master; the other will serve as a replica.

	Ideally, you should physically separate these hosts to reduce the chance that a
	natural disaster or infrastructure disruption could disable or destroy both
	hosts at the same time.

	From here, you can choose how you expand the cluster.

	To improve scalability and performance, separate loaded services onto
	dedicated hosts and then add more hosts of that type to increase capacity. If
	you have a two-node cluster, the best way to improve scalability by adding one
	host is likely to separate the master database onto its own host.

	Note that increasing scale may //decrease// availability by leaving you with
	too little capacity after a failure. If you have three hosts handling traffic
	and one datacenter fails, too much traffic may be sent to the single remaining
	host in the surviving datacenter. You can hedge against this by mirroring new
	hosts in other datacenters (for example, also separate the replica database
	onto its own host).

	After separating databases, separating repository + daemon nodes is likely
	the next step.

	To improve availability, add another copy of everything you run in one
	datacenter to a new datacenter. For example, if you have a two-node cluster,
	the best way to improve availability is to run everything on a third host in a
	third datacenter. If you have a 6-node cluster with a web node, a database node
	and a repo + daemon node in two datacenters, add 3 more nodes to create a copy
	of each node in a third datacenter.

	You can continue adding hosts until you run out of hosts.


	Next Steps
	==========

	Continue by:

	- learning how Phacility configures and operates a large, multi-tenant
	production cluster in ((cluster)).
	diff --git a/src/docs/user/cluster/cluster_databases.diviner b/src/docs/user/cluster/cluster_databases.diviner
	index 653b521ee0..03c9619c97 100644
	--- a/src/docs/user/cluster/cluster_databases.diviner
	+++ b/src/docs/user/cluster/cluster_databases.diviner
	@@ -1,310 +1,322 @@
	@title Cluster: Databases
	@group intro

	Configuring Phabricator to use multiple database hosts.

	Overview
	========

	WARNING: This feature is a very early prototype; the features this document
	describes are mostly speculative fantasy.

	You can deploy Phabricator with multiple database hosts, configured as a master
	and a set of replicas. The advantages of doing this are:

	- faster recovery from disasters by promoting a replica;
	- graceful degradation if the master fails;
	- reduced load on the master; and
	- some tools to help monitor and manage replica health.

	This configuration is complex, and many installs do not need to pursue it.

	Phabricator can not currently be configured into a multi-master mode, nor can
	it be configured to automatically promote a replica to become the new master.

	If you lose the master, Phabricator can degrade automatically into read-only
	mode and remain available, but can not fully recover without operational
	intervention unless the master recovers on its own.


	Setting up MySQL Replication
	============================

	TODO: Write this section.


	Configuring Replicas
	====================

	Once your replicas are in working order, tell Phabricator about them by
	configuring the `cluster.database` option. This option must be configured from
	the command line or in configuration files because Phabricator needs to read
	it //before// it can connect to databases.

	This option value will list all of the database hosts that you want Phabricator
	to interact with: your master and all your replicas. Each entry in the list
	should have these keys:

	- `host`: //Required string.// The database host name.
	- `role`: //Required string.// The cluster role of this host, one of
	`master` or `replica`.
	- `port`: //Optional int.// The port to connect to. If omitted, the default
	port from `mysql.port` will be used.
	- `user`: //Optional string.// The MySQL username to use to connect to this
	host. If omitted, the default from `mysql.user` will be used.
	- `pass`: //Optional string.// The password to use to connect to this host.
	If omitted, the default from `mysql.pass` will be used.
	- `disabled`: //Optional bool.// If set to `true`, Phabricator will not
	connect to this host. You can use this to temporarily take a host out
	of service.

	When `cluster.databases` is configured the `mysql.host` option is not used.
	The other MySQL connection configuration options (`mysql.port`, `mysql.user`,
	`mysql.pass`) are used only to provide defaults.

	Once you've configured this option, restart Phabricator for the changes to take
	effect, then continue to "Monitoring Replicas" to verify the configuration.


	Monitoring Replicas
	===================

	You can monitor replicas in {nav Config > Cluster Databases}. This interface
	shows you a quick overview of replicas and their health, and can detect some
	common issues with replication.

	The table on this page shows each database and current status.

	NOTE: This page runs its diagnostics //from the web server that is serving the
	request//. If you are recovering from a disaster, the view this page shows
	may be partial or misleading, and two requests served by different servers may
	see different views of the cluster.

	Connection: Phabricator tries to connect to each configured database, then
	shows the result in this column. If it fails, a brief diagnostic message with
	details about the error is shown. If it succeeds, the column shows a rough
	measurement of latency from the current webserver to the database.

	Replication: This is a summary of replication status on the database. If
	things are properly configured and stable, the replicas should be actively
	replicating and no more than a few seconds behind master, and the master
	should //not// be replicating from another database.

	To report this status, the user Phabricator is connecting as must have the
	`REPLICATION CLIENT` privilege (or the `SUPER` privilege) so it can run the
	`SHOW SLAVE STATUS` command. The `REPLICATION CLIENT` privilege only enables
	the user to run diagnostic commands so it should be reasonable to grant it in
	most cases, but it is not required. If you choose not to grant it, this page
	can not show any useful diagnostic information about replication status but
	everything else will still work.

	If a replica is more than a second behind master, this page will show the
	current replication delay. If the replication delay is more than 30 seconds,
	it will report "Slow Replication" with a warning icon.

	If replication is delayed, data is at risk: if you lose the master and can not
	later recover it (for example, because a meteor has obliterated the datacenter
	housing the physical host), data which did not make it to the replica will be
	lost forever.

	Beyond the risk of data loss, any read-only traffic sent to the replica will
	see an older view of the world which could be confusing for users: it may
	appear that their data has been lost, even if it is safe and just hasn't
	replicated yet.

	Phabricator will attempt to prevent clients from seeing out-of-date views, but
	sometimes sending traffic to a delayed replica is the best available option
	(for example, if the master can not be reached).

	Health: This column shows the result of recent health checks against the
	server. After several checks in a row fail, Phabricator will mark the server
	as unhealthy and stop sending traffic to it until several checks in a row
	later succeed.

	Note that each web server tracks database health independently, so if you have
	several servers they may have different views of database health. This is
	normal and not problematic.

	For more information on health checks, see "Unreachable Masters" below.

	Messages: This column has additional details about any errors shown in the
	other columns. These messages can help you understand or resolve problems.


	Testing Replicas
	================

	To test that your configuration can survive a disaster, turn off the master
	database. Do this with great ceremony, making a cool explosion sound as you
	run the `mysqld stop` command.

	If things have been set up properly, Phabricator should degrade to a temporary
	read-only mode immediately. After a brief period of unresponsiveness, it will
	degrade further into a longer-term read-only mode. For details on how this
	works interanlly, see "Unreachable Masters" below.

	Once satisfied, turn the master back on. After a brief delay, Phabricator
	should recognize that the master is healthy again and recover fully.

	Throughout this process, the {nav Cluster Databases} console will show a
	current view of the world from the perspective of the web server handling the
	request. You can use it to monitor state.

	You can perform a more narrow test by enabling `cluster.read-only` in
	configuration. This will put Phabricator into read-only mode immediately
	without turning off any databases.

	You can use this mode to understand which capabilities will and will not be
	available in read-only mode, and make sure any information you want to remain
	accessible in a disaster (like wiki pages or contact information) is really
	accessible.

	See the next section, "Degradation to Read Only Mode", for more details about
	when, why, and how Phabricator degrades.

	If you run custom code or extensions, they may not accommodate read-only mode
	properly. You should specifically test that they function correctly in
	read-only mode and do not prevent you from accessing important information.


	Degradation to Read-Only Mode
	=============================

	Phabricator will degrade to read-only mode when any of these conditions occur:

	- you turn it on explicitly;
	- you configure cluster mode, but don't set up any masters;
	- the master can not be reached while handling a request; or
	- recent attempts to connect to the master have consistently failed.

	When Phabricator is running in read-only mode, users can still read data and
	browse and clone repositories, but they can not edit, update, or push new
	changes. For example, users can still read disaster recovery information on
	the wiki or emergency contact information on user profiles.

	You can enable this mode explicitly by configuring `cluster.read-only`. Some
	reasons you might want to do this include:

	- to test that the mode works like you expect it to;
	- to make sure that information you need will be available;
	- to prevent new writes while performing database maintenance; or
	- to permanently archive a Phabricator install.

	You can also enable this mode implicitly by configuring `cluster.databases`
	but disabling the master, or by not specifying any host as a master. This may
	be more convenient than turning it on explicitly during the course of
	operations work.

	If Phabricator is unable to reach the master database, it will degrade into
	read-only mode automatically. See "Unreachable Masters" below for details on
	how this process works.

	If you end up in a situation where you have lost the master and can not get it
	back online (or can not restore it quickly) you can promote a replica to become
	the new master. See the next section, "Promoting a Replica", for details.


	Promoting a Replica
	===================

	TODO: Write this section.


	Unreachable Masters
	===================

	This section describes how Phabricator determines that a master has been lost,
	marks it unreachable, and degrades into read-only mode.

	Phabricator degrades into read-only mode automatically in two ways: very
	briefly in response to a single connection failure, or more permanently in
	response to a series of connection failures.

	In the first case, if a request needs to connect to the master but is not able
	to, Phabricator will temporarily degrade into read-only mode for the remainder
	of that request. The alternative is to fail abruptly, but Phabricator can
	sometimes degrade successfully and still respond to the user's request, so it
	makes an effort to finish serving the request from replicas.

	If the request was a write (like posting a comment) it will fail anyway, but
	if it was a read that did not actually need to use the master it may succeed.

	This temporary mode is intended to recover as gracefully as possible from brief
	interruptions in service (a few seconds), like a server being restarted, a
	network link becoming temporarily unavailable, or brief periods of load-related
	disruption. If the anomaly is temporary, Phabricator should recover immediately
	(on the next request once service is restored).

	This mode can be slow for users (they need to wait on connection attempts to
	the master which fail) and does not reduce load on the master (requests still
	attempt to connect to it).

	The second way Phabricator degrades is by running periodic health checks
	against databases, and marking them unhealthy if they fail over a longer period
	of time. This mechanism is very similar to the health checks that most HTTP
	load balancers perform against web servers.

	If a database fails several health checks in a row, Phabricator will mark it as
	unhealthy and stop sending all traffic (except for more health checks) to it.
	This improves performance during a service interruption and reduces load on the
	master, which may help it recover from load problems.

	You can monitor the status of health checks in the {nav Cluster Databases}
	console. The "Health" column shows how many checks have run recently and
	how many have succeeded.

	Health checks run every 3 seconds, and 5 checks in a row must fail or succeed
	before Phabricator marks the database as healthy or unhealthy, so it will
	generally take about 15 seconds for a database to change state after it goes
	down or comes up.

	If all of the recent checks fail, Phabricator will mark the database as
	unhealthy and stop sending traffic to it. If the master was the database that
	was marked as unhealthy, Phabricator will actively degrade into read-only mode
	until it recovers.

	This mode only attempts to connect to the unhealthy database once every few
	seconds to see if it is recovering, so performance will be better on average
	(users rarely need to wait for bad connections to fail or time out) and the
	datbase will receive less load.

	Once all of the recent checks succeed, Phabricator will mark the database as
	healthy again and continue sending traffic to it.

	Health checks are tracked individually for each web server, so some web servers
	may see a host as healthy while others see it as unhealthy. This is normal, and
	can accurately reflect the state of the world: for example, the link between
	datacenters may have been lost, so hosts in one datacenter can no longer see
	the master, while hosts in the other datacenter still have a healthy link to
	it.


	Backups
	======

	Even if you configure replication, you should still retain separate backup
	snapshots. Replicas protect you from data loss if you lose a host, but they do
	not let you recover from data mutation mistakes.

	If something issues `DELETE` or `UPDATE` statements and destroys data on the
	master, the mutation will propagate to the replicas almost immediately and the
	data will be gone forever. Normally, the only way to recover this data is from
	backup snapshots.

	Although you should still have a backup process, your backup process can
	safely pull dumps from a replica instead of the master. This operation can
	be slow, so offloading it to a replica can make the perforance of the master
	more consistent.

	To dump from a replica, wait for this TODO to be resolved and then do whatever
	it says to do:

	TODO: Make `bin/storage dump` replica-aware. See T10758.

	+With recent versions of MySQL, it is also possible to configure a //delayed//
	+replica which intentionally lags behind the master (say, by 12 hours). In the
	+event of a bad mutation, this could give you a larger window of time to
	+recognize the issue and recover the lost data from the delayed replica (which
	+might be quick) without needing to restore backups (which might be very slow).
	+
	+Delayed replication is outside the scope of this document, but may be worth
	+considering as an additional data security step on top of backup snapshots
	+depending on your resources and needs. If you configure a delayed replica, do
	+not add it to the `cluster.databases` configuration: Phabricator should never
	+send traffic to it, and does not need to know about it.
	+

	Next Steps
	==========

	Continue by:

	- returning to @{article:Clustering Introduction}.
	diff --git a/src/docs/user/cluster/cluster_repositories.diviner b/src/docs/user/cluster/cluster_repositories.diviner
	index d9e859fd42..c2750fbd78 100644
	--- a/src/docs/user/cluster/cluster_repositories.diviner
	+++ b/src/docs/user/cluster/cluster_repositories.diviner
	@@ -1,86 +1,112 @@
	@title Cluster: Repositories
	@group intro

	Configuring Phabricator to use multiple repository hosts.

	Overview
	========

	WARNING: This feature is a very early prototype; the features this document
	describes are mostly speculative fantasy.

	If you use Git or Mercurial, you can deploy Phabricator with multiple
	repository hosts, configured so that each host is readable and writable. The
	advantages of doing this are:

	- you can completely survive the loss of repository hosts;
	- reads and writes can scale across multiple machines; and
	- read and write performance across multiple geographic regions may improve.

	This configuration is complex, and many installs do not need to pursue it.

	This configuration is not currently supported with Subversion.


	Repository Hosts
	================

	Repository hosts must run a complete, fully configured copy of Phabricator,
	including a webserver. If you make repositories available over SSH, they must
	also run a properly configured `sshd`.

	Generally, these hosts will run the same set of services and configuration that
	web hosts run. If you prefer, you can overlay these services and put web and
	repository services on the same hosts.

	When a user requests information about a repository that can only be satisfied
	by examining a repository working copy, the webserver receiving the reqeust
	will make an HTTP service call to a repository server which hosts the
	repository to retrieve the data it needs. It will use the result of this query
	to respond to the user.


	How Reads and Writes Work
	=========================

	Phabricator repository replicas are multi-master: every node is readable and
	writable, and a cluster of nodes can (almost always) survive the loss of any
	arbitrary subset of nodes so long as at least one node is still alive.

	Phabricator maintains an internal version for each repository, and increments
	it when the repository is mutated.

	Before responding to a read, replicas make sure their version of the repository
	is up to date (no node in the cluster has a newer version of the repository).
	If it isn't, they block the read until they can complete a fetch.

	Before responding to a write, replicas obtain a global lock, perform the same
	version check and fetch if necessary, then allow the write to continue.


	+HTTP vs HTTPS
	+=============
	+
	+Intracluster requests (from the daemons to repository servers, or from
	+webservers to repository servers) are permitted to use HTTP, even if you have
	+set `security.require-https` in your configuration.
	+
	+It is common to terminate SSL at a load balancer and use plain HTTP beyond
	+that, and the `security.require-https` feature is primarily focused on making
	+client browser behavior more convenient for users, so it does not apply to
	+intracluster traffic.
	+
	+Using HTTP within the cluster leaves you vulnerable to attackers who can
	+observe traffic within a datacenter, or observe traffic between datacenters.
	+This is normally very difficult, but within reach for state-level adversaries
	+like the NSA.
	+
	+If you are concerned about these attackers, you can terminate HTTPS on
	+repository hosts and bind to them with the "https" protocol. Just be aware that
	+the `security.require-https` setting won't prevent you from making
	+configuration mistakes, as it doesn't cover intracluster traffic.
	+
	+Other mitigations are possible, but securing a network against the NSA and
	+similar agents of other rogue nations is beyond the scope of this document.
	+
	+
	Backups
	======

	Even if you configure clustering, you should still consider retaining separate
	backup snapshots. Replicas protect you from data loss if you lose a host, but
	they do not let you rewind time to recover from data mutation mistakes.

	If something issues a `--force` push that destroys branch heads, the mutation
	will propagate to the replicas.

	You may be able to manually restore the branches by using tools like the
	Phabricator push log or the Git reflog so it is less important to retain
	repository snapshots than database snapshots, but it is still possible for
	data to be lost permanently, especially if you don't notice the problem for
	some time.

	Retaining separate backup snapshots will improve your ability to recover more
	data more easily in a wider range of disaster situations.


	Next Steps
	==========

	Continue by:

	- returning to @{article:Clustering Introduction}.

File Metadata

Mime Type: text/x-diff
Expires: Mon, Mar 16, 11:36 PM (1 d, 12 h)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 963621
Default Alt Text: (70 KB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions