Disallow webcrawlers to index Diffusion commits

Summary:
Phorge already sets Disallow: /diffusion/ and Disallow: /source/.
Thus consequently also disallow accessing specific commits via /r*.
See https://secure.phabricator.com/T4610 for previous discussions.

Closes T15670

Test Plan:
Go to /robots.txt in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.

Reviewers: O1 Blessed Committers, speck

Reviewed By: O1 Blessed Committers, speck

Subscribers: tobiaswiese, valerio.bozzolan, Matthew, Cigaryno

Maniphest Tasks: T15670

Differential Revision: https://we.phorge.it/D25474