Page MenuHomestyx hydra
Files F83156

No OneTemporary
Actions

View Options

diff --git a/src/docs/user/cluster/cluster.diviner b/src/docs/user/cluster/cluster.diviner
index 15eed86eb4..30ad53efb8 100644
--- a/src/docs/user/cluster/cluster.diviner
+++ b/src/docs/user/cluster/cluster.diviner
@@ -1,354 +1,342 @@
@title Clustering Introduction
@group cluster
Guide to configuring Phabricator across multiple hosts for availability and
performance.
Overview
========
WARNING: This feature is a prototype. Installs should expect a challenging
adventure when deploying clusters. In the best of times, configuring a
cluster is complex and requires significant operations experience.
Phabricator can be configured to run on multiple hosts with redundant services
to improve its availability and scalability, and make disaster recovery much
easier.
Clustering is more complex to setup and maintain than running everything on a
single host, but greatly reduces the cost of recovering from hardware and
network failures.
Each Phabricator service has an array of clustering options that can be
configured somewhat independently. Configuring a cluster is inherently complex,
and this is an advanced feature aimed at installs with large userbases and
experienced operations personnel who need this high degree of flexibility.
The remainder of this document summarizes how to add redundancy to each
service and where your efforts are likely to have the greatest impact.
For additional guidance on setting up a cluster, see "Overlaying Services"
and "Cluster Recipes" at the bottom of this document.
Clusterable Services
====================
This table provides an overview of clusterable services, their setup
complexity, and the rough impact that converting them to run on multiple hosts
will have on availability, resistance to data loss, and scalability.
| Service | Setup | Availability | Loss Resistance | Scalability
|---------|-------|--------------|-----------|------------
| **Databases** | Moderate | **High** | **High** | Low
| **Repositories** | Complex | Moderate | **High** | Moderate
| **Daemons** | Minimal | Low | No Risk | Low
| **SSH Servers** | Minimal | Low | No Risk | Low
| **Web Servers** | Minimal | **High** | No Risk | Moderate
| **Notifications** | Minimal | Low | No Risk | Low
-| **Fulltext Search** | Moderate | **High** | Minimal Risk | Moderate
+| **Fulltext Search** | Minimal | Low | No Risk | Low
See below for a walkthrough of these services in greater detail.
Preparing for Clustering
========================
To begin deploying Phabricator in cluster mode, set up `cluster.addresses`
in your configuration.
This option should contain a list of network address blocks which are considered
to be part of the cluster. Hosts in this list are allowed to bend (or even
break) some of the security and policy rules when they make requests to other
hosts in the cluster, so this list should be as small as possible. See "Cluster
Whitelist Security" below for discussion.
If you are deploying hardware in EC2, a reasonable approach is to launch a
dedicated Phabricator VPC, whitelist the whole VPC as a Phabricator cluster,
and then deploy only Phabricator services into that VPC.
If you have additional auxiliary hosts which run builds and tests via Drydock,
you should //not// include them in the cluster address definition. For more
detailed discussion of the Drydock security model, see
@{article:Drydock User Guide: Security}.
Most other clustering features will not work until you define a cluster by
configuring `cluster.addresses`.
Cluster Whitelist Security
========================
When you configure `cluster.addresses`, you should keep the list of trusted
cluster hosts as small as possible. Hosts on this list gain additional
capabilities, including these:
**Trusted HTTP Headers**: Normally, Phabricator distrusts the load balancer
HTTP headers `X-Forwarded-For` and `X-Forwarded-Proto` because they may be
client-controlled and can be set to arbitrary values by an attacker if no load
balancer is deployed. In particular, clients can set `X-Forwarded-For` to any
value and spoof traffic from arbitrary remotes.
These headers are trusted when they are received from a host on the cluster
address whitelist. This allows requests from cluster loadbalancers to be
interpreted correctly by default without requiring additional custom code or
configuration.
**Intracluster HTTP**: Requests from cluster hosts are not required to use
HTTPS, even if `security.require-https` is enabled, because it is common to
terminate HTTPS on load balancers and use plain HTTP for requests within a
cluster.
**Special Authentication Mechanisms**: Cluster hosts are allowed to connect to
other cluster hosts with "root credentials", and to impersonate any user
account.
The use of root credentials is required because the daemons must be able to
bypass policies in order to function properly: they need to send mail about
private conversations and import commits in private repositories.
The ability to impersonate users is required because SSH nodes must receive,
interpret, modify, and forward SSH traffic. They can not use the original
credentials to do this because SSH authentication is asymmetric and they do not
have the user's private key. Instead, they use root credentials and impersonate
the user within the cluster.
These mechanisms are still authenticated (and use asymmetric keys, like SSH
does), so access to a host in the cluster address block does not mean that an
attacker can immediately compromise the cluster. However, an over-broad cluster
address whitelist may give an attacker who gains some access additional tools
to escalate access.
Note that if an attacker gains access to an actual cluster host, these extra
powers are largely moot. Most cluster hosts must be able to connect to the
master database to function properly, so the attacker will just do that and
freely read or modify whatever data they want.
Cluster: Databases
=================
Configuring multiple database hosts is moderately complex, but normally has the
highest impact on availability and resistance to data loss. This is usually the
most important service to make redundant if your focus is on availability and
disaster recovery.
Configuring replicas allows Phabricator to run in read-only mode if you lose
the master and to quickly promote the replica as a replacement.
For details, see @{article:Cluster: Databases}.
Cluster: Repositories
=====================
Configuring multiple repository hosts is complex, but is required before you
can add multiple daemon or web hosts.
Repository replicas are important for availability if you host repositories
on Phabricator, but less important if you host repositories elsewhere
(instead, you should focus on making that service more available).
The distributed nature of Git and Mercurial tend to mean that they are
naturally somewhat resistant to data loss: every clone of a repository includes
the entire history.
Repositories may become a scalability bottleneck, although this is rare unless
your install has an unusually heavy repository read volume. Slow clones/fetches
may hint at a repository capacity problem. Adding more repository hosts will
provide an approximately linear increase in capacity.
For details, see @{article:Cluster: Repositories}.
Cluster: Daemons
================
Configuring multiple daemon hosts is straightforward, but you must configure
repositories first.
With daemons running on multiple hosts you can transparently survive the loss
of any subset of hosts without an interruption to daemon services, as long as
at least one host remains alive. Daemons are stateless, so spreading daemons
across multiple hosts provides no resistance to data loss.
Daemons can become a bottleneck, particularly if your install sees a large
volume of write traffic to repositories. If the daemon task queue has a
backlog, that hints at a capacity problem. If existing hosts have unused
resources, increase `phd.taskmasters` until they are fully utilized. From
there, adding more daemon hosts will provide an approximately linear increase
in capacity.
For details, see @{article:Cluster: Daemons}.
Cluster: SSH Servers
====================
Configuring multiple SSH hosts is straightforward, but you must configure
repositories first.
With multiple SSH hosts you can transparently survive the loss of any subset
of hosts without interruption to repository services, as long as at last one
host remains alive. SSH services are stateless, so putting multiple hosts in
service provides no resistance to data loss because no data is at risk.
SSH hosts are very rarely a scalability bottleneck.
For details, see @{article:Cluster: SSH Servers}.
Cluster: Web Servers
====================
Configuring multiple web hosts is straightforward, but you must configure
repositories first.
With multiple web hosts you can transparently survive the loss of any subset
of hosts as long as at least one host remains alive. Web services are stateless,
so putting multiple hosts in service provides no resistance to data loss
because no data is at risk.
Web hosts can become a bottleneck, particularly if you have a workload that is
heavily focused on reads from the web UI (like a public install with many
anonymous users). Slow responses to web requests may hint at a web capacity
problem. Adding more hosts will provide an approximately linear increase in
capacity.
For details, see @{article:Cluster: Web Servers}.
Cluster: Notifications
======================
Configuring multiple notification hosts is simple and has no pre-requisites.
With multiple notification hosts, you can survive the loss of any subset of
hosts as long as at least one host remains alive. Service may be breifly
disrupted directly after the incident which destroys the other hosts.
Notifications are noncritical, so this normally has little practical impact
on service availability. Notifications are also stateless, so clustering this
service provides no resistance to data loss because no data is at risk.
Notification delivery normally requires very few resources, so adding more
hosts is unlikely to have much impact on scalability.
For details, see @{article:Cluster: Notifications}.
Cluster: Fulltext Search
========================
-At a certain scale, you may begin to bump up against the limitations of MySQL's
-built-in fulltext search capabilities. We have seen this with very large
-installations with several million objects in the database and very many
-simultaneous requests. At this point you may consider adding Elasticsearch
-hosts to your cluster to reduce the load on your MySQL hosts.
+Configuring search services is relatively simple and has no pre-requisites.
-Elasticsearch has the ability to spread the load across multiple hosts and can
-handle very large indexes by sharding.
+By default, Phabricator uses MySQL as a fulltext search engine, so deploying
+multiple database hosts will effectively also deploy multiple fulltext search
+hosts.
-Search does not involve any risk of data lost because it's always possible to
-rebuild the search index from the original database objects. This process can
-be very time consuming, however, especially when the database grows very large.
-
-With multiple Elasticsearch hosts, you can survive the loss of a single host
-with minimal disruption as Phabricator will detect the problem and direct
-queries to one of the remaining hosts.
-
-Phabricator supports writing to multiple indexing servers. This Simplifies
-Elasticsearch upgrades and makes it possible to recover more quickly from
-problems with the search index.
+Search indexes can be completely rebuilt from the database, so there is no
+risk of data loss no matter how fulltext search is configured.
For details, see @{article:Cluster: Search}.
Overlaying Services
===================
Although hosts can run a single dedicated service type, certain groups of
services work well together. Phabricator clusters usually do not need to be
very large, so deploying a small number of hosts with multiple services is a
good place to start.
In planning a cluster, consider these blended host types:
**Everything**: Run HTTP, SSH, MySQL, notifications, repositories and daemons
on a single host. This is the starting point for single-node setups, and
usually also the best configuration when adding the second node.
**Everything Except Databases**: Run HTTP, SSH, notifications, repositories and
daemons on one host, and MySQL on a different host. MySQL uses many of the same
resources that other services use. It's also simpler to separate than other
services, and tends to benefit the most from dedicated hardware.
**Repositories and Daemons**: Run repositories and daemons on the same host.
Repository hosts //must// run daemons, and it normally makes sense to
completely overlay repositories and daemons. These services tend to use
different resources (repositories are heavier on I/O and lighter on CPU/RAM;
daemons are heavier on CPU/RAM and lighter on I/O).
Repositories and daemons are also both less latency sensitive than other
service types, so there's a wider margin of error for under provisioning them
before performance is noticeably affected.
These nodes tend to use system resources in a balanced way. Individual nodes
in this class do not need to be particularly powerful.
**Frontend Servers**: Run HTTP and SSH on the same host. These are easy to set
up, stateless, and you can scale the pool up or down easily to meet demand.
Routing both types of ingress traffic through the same initial tier can
simplify load balancing.
These nodes tend to need relatively little RAM.
Cluster Recipes
===============
This section provides some guidance on reasonable ways to scale up a cluster.
The smallest possible cluster is **two hosts**. Run everything (web, ssh,
database, notifications, repositories, and daemons) on each host. One host will
serve as the master; the other will serve as a replica.
Ideally, you should physically separate these hosts to reduce the chance that a
natural disaster or infrastructure disruption could disable or destroy both
hosts at the same time.
From here, you can choose how you expand the cluster.
To improve **scalability and performance**, separate loaded services onto
dedicated hosts and then add more hosts of that type to increase capacity. If
you have a two-node cluster, the best way to improve scalability by adding one
host is likely to separate the master database onto its own host.
Note that increasing scale may //decrease// availability by leaving you with
too little capacity after a failure. If you have three hosts handling traffic
and one datacenter fails, too much traffic may be sent to the single remaining
host in the surviving datacenter. You can hedge against this by mirroring new
hosts in other datacenters (for example, also separate the replica database
onto its own host).
After separating databases, separating repository + daemon nodes is likely
the next step to consider.
To improve **availability**, add another copy of everything you run in one
datacenter to a new datacenter. For example, if you have a two-node cluster,
the best way to improve availability is to run everything on a third host in a
third datacenter. If you have a 6-node cluster with a web node, a database node
and a repo + daemon node in two datacenters, add 3 more nodes to create a copy
of each node in a third datacenter.
You can continue adding hosts until you run out of hosts.
Next Steps
==========
Continue by:
- learning how Phacility configures and operates a large, multi-tenant
production cluster in ((cluster)).
diff --git a/src/docs/user/cluster/cluster_search.diviner b/src/docs/user/cluster/cluster_search.diviner
index 662abecbc3..c658f50db4 100644
--- a/src/docs/user/cluster/cluster_search.diviner
+++ b/src/docs/user/cluster/cluster_search.diviner
@@ -1,76 +1,210 @@
@title Cluster: Search
@group cluster
Overview
========
-You can configure phabricator to connect to one or more fulltext search clusters
-running either Elasticsearch or MySQL. By default and without further
-configuration, Phabricator will use MySQL for fulltext search. This will be
-adequate for the vast majority of users. Installs with a very large number of
-objects or specialized search needs can consider enabling Elasticsearch for
-better scalability and potentially better search results.
+You can configure Phabricator to connect to one or more fulltext search
+services.
+
+By default, Phabricator will use MySQL for fulltext search. This is suitable
+for most installs. However, alternate engines are supported.
+
Configuring Search Services
===========================
-To configure an Elasticsearch service, use the `cluster.search` configuration
-option. A typical Elasticsearch configuration will probably look similar to
-the following example:
+To configure search services, adjust the `cluster.search` configuration
+option. This option contains a list of one or more fulltext search services,
+like this:
+
+```lang=json
+[
+ {
+ "type": "...",
+ "hosts": [
+ ...
+ ],
+ "roles": {
+ "read": true,
+ "write": true
+ }
+ }
+]
+```
+
+When a user makes a change to a document, Phabricator writes the updated
+document into every configured, writable fulltext service.
+
+When a user issues a query, Phabricator tries configured, readable services
+in order until it is able to execute the query successfully.
+
+These options are supported by all service types:
+
+| Key | Description |
+|---|---|
+| `type` | Constant identifying the service type, like `mysql`.
+| `roles` | Dictionary of role settings, for enabling reads and writes.
+| `hosts` | List of hosts for this service.
+
+Some service types support additional options.
+
+Available Service Types
+=======================
+
+These service types are supported:
+
+| Service | Key | Description |
+|---|---|---|
+| MySQL | `mysql` | Default MySQL fulltext index.
+| Elasticsearch | `elasticsearch` | Use an external Elasticsearch service
+
+
+Fulltext Service Roles
+======================
+
+These roles are supported:
+
+| Role | Key | Description
+|---|---|---|
+| Read | `read` | Allows the service to be queried when users search.
+| Write | `write` | Allows documents to be published to the service.
+
+
+Specifying Hosts
+================
+
+The `hosts` key should contain a list of dictionaries, each specifying the
+details of a host. A service should normally have one or more hosts.
+
+When an option is set at the service level, it serves as a default for all
+hosts. It may be overridden by changing the value for a particular host.
+
+
+Service Type: MySQL
+==============
+
+The `mysql` service type does not require any configuration, and does not
+need to have hosts specified. This service uses the builtin database to
+index and search documents.
+
+A typical `mysql` service configuration looks like this:
```lang=json
{
- "cluster.search": [
+ "type": "mysql"
+}
+```
+
+
+Service Type: Elasticsearch
+======================
+
+The `elasticsearch` sevice type supports these options:
+
+| Key | Description |
+|---|---|
+| `protocol` | Either `"http"` (default) or `"https"`.
+| `port` | Elasticsearch TCP port.
+| `version` | Elasticsearch version, either `2` or `5` (default).
+| `path` | Path for the index. Defaults to `/phabriator`. Advanced.
+
+A typical `elasticsearch` service configuration looks like this:
+
+```lang=json
+{
+ "type": "elasticsearch",
+ "hosts": [
{
- "type": "elasticsearch",
- "hosts": [
- {
- "host": "127.0.0.1",
- "roles": { "write": true, "read": true }
- }
- ],
- "port": 9200,
"protocol": "http",
- "path": "/phabricator",
- "version": 5
- },
- ],
+ "host": "127.0.0.1",
+ "port": 9200
+ }
+ ]
}
```
-Supported Options
------------------
-| Key | Type |Comments|
-|`type` | String |Engine type. Currently, 'elasticsearch' or 'mysql'|
-|`protocol`| String |Either 'http' or 'https'|
-|`port`| Int |The TCP port that Elasticsearch is bound to|
-|`path`| String |The path portion of the url for phabricator's index.|
-|`version`| Int |The version of Elasticsearch server. Supports either 2 or 5.|
-|`hosts`| List |A list of one or more Elasticsearch host names / addresses.|
-
-Host Configuration
-------------------
-Each search service must have one or more hosts associated with it. Each host
-entry consists of a `host` key, a dictionary of roles and can optionally
-override any of the options that are valid at the service level (see above).
-
-Currently supported roles are `read` and `write`. These can be individually
-enabled or disabled on a per-host basis. A typical setup might include two
-elasticsearch clusters in two separate datacenters. You can configure one
-cluster for reads and both for writes. When one cluster is down for maintenance
-you can simply swap the read role over to the backup cluster and then proceed
-with maintenance without any service interruption.
-
Monitoring Search Services
==========================
-You can monitor fulltext search in {nav Config > Search Servers}. This interface
-shows you a quick overview of services and their health.
+You can monitor fulltext search in {nav Config > Search Servers}. This
+interface shows you a quick overview of services and their health.
The table on this page shows some basic stats for each configured service,
followed by the configuration and current status of each host.
-NOTE: This page runs its diagnostics //from the web server that is serving the
-request//. If you are recovering from a disaster, the view this page shows
-may be partial or misleading, and two requests served by different servers may
-see different views of the cluster.
+
+Rebuilding Indexes
+==================
+
+After adding new search services, you will need to rebuild document indexes
+on them. To do this, first initialize the services:
+
+```
+phabricator/ $ ./bin/search init
+```
+
+This will perform index setup steps and other one-time configuration.
+
+To populate documents in all indexes, run this command:
+
+```
+phabricator/ $ ./bin/search index --force --background --type all
+```
+
+This initiates an exhaustive rebuild of the document indexes. To get a more
+detailed list of indexing options available, run:
+
+```
+phabricator/ $ ./bin/search help index
+```
+
+
+Advanced Example
+================
+
+This is a more advanced example which shows a configuration with multiple
+different services in different roles. In this example:
+
+ - Phabricator is using an Elasticsearch 2 service as its primary fulltext
+ service.
+ - An Elasticsearch 5 service is online, but only receiving writes.
+ - The MySQL service is serving as a backup if Elasticsearch fails.
+
+This particular configuration may not be very useful. It is primarily
+intended to show how to configure many different options.
+
+
+```lang=json
+[
+ {
+ "type": "elasticsearch",
+ "version": 2,
+ "hosts": [
+ {
+ "host": "elastic2.mycompany.com",
+ "port": 9200,
+ "protocol": "http"
+ }
+ ]
+ },
+ {
+ "type": "elasticsearch",
+ "version": 5,
+ "hosts": [
+ {
+ "host": "elastic5.mycompany.com",
+ "port": 9789,
+ "protocol": "https"
+ "roles": {
+ "read": false,
+ "write": true
+ }
+ }
+ ]
+ },
+ {
+ "type": "mysql"
+ }
+]
+```

File Metadata

Mime Type
text/x-diff
Expires
Fri, Mar 14, 8:18 AM (20 h, 1 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
71707
Default Alt Text
(23 KB)

Event Timeline

styx copyright 2020 sirocyl / ellie (kymkdd) / ncommander / styx contributors
Use of other names and brands is not intended to reflect a claim of, or right to, use the name or brand.
styx is an operating system using the Linux kernel and components, and "styx Linux" as a phrase does not reflect the inclusion of the Linux brand or trademark in the styx project.
Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
Terms and Conditions (TODO: fix this!)