R1:838f78128544

Add a robots.txt file to disallow /diffusion/

Summary:
Fixes T4610. Open to suggestions, etc., if there's anything I'm missing.

Also:

- Moves these "system" endpoints into a real application.
- Makes `isUnlisted()` work a little more consistently.

Test Plan: Accessed `/robots.txt`, `/status/` and `/debug/`.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: aran, epriestley

Maniphest Tasks: T4610

Differential Revision: https://secure.phabricator.com/D8532