It'd be nice to retire this old name. Obviously the re-writes will be "fun" (lots of puppet changes, I guess?).
Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krinkle | T289318 Move *.beta.wmflabs.org to *.beta.wmcloud.org | |||
| Resolved | BUG REPORT | • bd808 | T399349 Login broken by memcached ferm rules being bypassed by hiera configuration |
Event Timeline
Change #1165983 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] missing.php: Support beta suffix for auth.wikimedia error page
Change #1165983 merged by jenkins-bot:
[operations/mediawiki-config@master] missing.php: Support beta suffix for auth.wikimedia error page
Mentioned in SAL (#wikimedia-operations) [2025-07-02T20:35:16Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-02T20:37:25Z]
Change #1165989 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Include allowance for wmcloud.org in wgGraphAllowedDomains
Mentioned in SAL (#wikimedia-operations) [2025-07-02T21:05:10Z]
Change #1165998 had a related patch set uploaded (by Krinkle; author: Krinkle):
[mediawiki/extensions/WikimediaMaintenance@master] dumpInterwiki.php: Switch interwiki destination in Beta to wmcloud.org
Mentioned in SAL (#wikimedia-releng) [2025-07-02T21:38:26Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-02T21:41:25Z]
Change #1165999 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Change Beta wikidata canonical to beta.wmcloud.org
Change #1165989 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Include allowance for wmcloud.org in wgGraphAllowedDomains
Change #1165999 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Change Beta wikidata canonical to beta.wmcloud.org
Mentioned in SAL (#wikimedia-operations) [2025-07-04T18:57:20Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-04T18:59:17Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-04T20:32:13Z]
Change #1166438 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org
Change #1166438 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Change loginwiki/metawiki/auth canonical to beta.wmcloud.org
Mentioned in SAL (#wikimedia-operations) [2025-07-04T21:21:17Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-04T21:23:14Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-04T21:39:26Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-04T21:39:29Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-04T21:41:38Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-04T21:45:04Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-04T21:45:54Z]
Change #1165998 merged by jenkins-bot:
[mediawiki/extensions/WikimediaMaintenance@master] dumpInterwiki.php: Switch interwiki destination in Beta to wmcloud.org
Change #1166855 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] interwiki-labs.php: Regenerate interwiki map (switch to beta.wmcloud.org)
Change #1166922 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Move all non-wikipedia canonical to beta.wmcloud.org
Change #1166855 merged by jenkins-bot:
[operations/mediawiki-config@master] interwiki-labs.php: Regenerate interwiki map (switch to beta.wmcloud.org)
Change #1166922 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Move all non-wikipedia canonical to beta.wmcloud.org
Change #1167259 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Move beta wikipedia canonical to beta.wmcloud.org
Mentioned in SAL (#wikimedia-releng) [2025-07-08T16:27:18Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-08T16:28:30Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-08T16:36:24Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-08T16:38:57Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-08T17:16:55Z]
Change #1167266 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] varnish: Swap hardcoded upload.wm.o cond for upload_domain in path normalize
Change #1167268 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] beta: Add redirect for upload.wikimedia.beta.wmflabs.org
Mentioned in SAL (#wikimedia-releng) [2025-07-08T17:40:39Z]
Mentioned in SAL (#wikimedia-releng) [2025-07-08T18:03:10Z]
Change #1167273 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] beta: Change beta.wmcloud.org stub redirect to new Meta-Wiki canonical
Change #1167259 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Move beta wikipedia canonical to beta.wmcloud.org
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:22:31Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:24:31Z]
Change #1167985 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Change FileRepo zone URL to upload.wikimedia.beta.wmcloud.org
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:34:45Z]
Change #1167985 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Change FileRepo zone URL to upload.wikimedia.beta.wmcloud.org
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:43:46Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:45:42Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-11T00:55:16Z]
Change #1168038 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] varnish: Improve GeoIP to use cookie domain similar to prod
But https://en.wikipedia.beta.wmcloud.org/wiki/Special:Blankpage actually gives:
504 Gateway Time-out
The server didn't respond in time.
T399349: Login broken by memcached ferm rules being bypassed by hiera configuration feels related, but maybe login was already broken and I just didn't notice because I had a working session?
Change #1167266 merged by Ssingh:
[operations/puppet@production] varnish: Swap hardcoded upload.wm.o cond for upload_domain in path normalize
Hi, reported in Slack: https://stream.wikimedia.beta.wmcloud.org/ isn't working.
Since this routing now goes through the CDN, I would guess that some special configuration is needed to pass-thru/pipe the connection to the service, rather than attempting to cache it, like is done in prod.
The stream.wikimedia.beta.wmflabs.org handler was added by @Ottomata in 2020.
I added stream.wikimedia.beta.wmcloud.org as alias of stream.wikimedia.beta.wmflabs.org, and confirmed that the two /v2/ui and /?doc routes respond the same way at glance. They share the same CDN configuration.
I noticed some Wikitech pages mention an older stream-beta.wmflabs.org proxy, which presumably bypassed the CDN. I updated those Wikitech pages to mention the more prod-like endpoint instead for consistency.
I'm guessing the CDN configuration for stream.wikimedia.beta.* was already incorrect, but that my documentation update prompted people to actually use the subdomain that was added back in 2020, thus surfacing the issue.
You can edit the CDN configuration for the EventStreams domain in Horizon via "Puppet prefix: deployment-cache-text" (source).
diff --git a/deployment-prep/deployment-cache-text.yaml b/deployment-prep/deployment-cache-text.yaml index 30bc4fd..c20da00 100644 --- a/deployment-prep/deployment-cache-text.yaml +++ b/deployment-prep/deployment-cache-text.yaml @@ -7,6 +7,16 @@ caching: normal performance.wikimedia.beta.wmflabs.org: caching: normal + stream.wikimedia.beta.wmcloud.org: + caching: normal + subpaths: + ^/v2/stream/.+: + caching: pipe + stream.wikimedia.beta.wmflabs.org: + caching: normal + subpaths: + ^/v2/stream/.+: + caching: pipe cache::app_def_be_opts: between_bytes_timeout: 30s connect_timeout: 3s
Mentioned in SAL (#wikimedia-releng) [2025-07-14T21:55:42Z]
Hooray thank you both! Sorry I did not have time to follow up yesterday. Much appreciated!
Change #1153764 merged by Ssingh:
[operations/puppet@production] deployment-prep: Add Apache vhost aliases for *.beta.wmcloud.org
Change #1160441 merged by Ssingh:
[operations/puppet@production] beta: Document beta-specific "w.beta.wmcloud.org" handling
Change #1167268 merged by Ssingh:
[operations/puppet@production] beta: Add redirect for upload.wikimedia.beta.wmflabs.org
Change #1167273 merged by Ssingh:
[operations/puppet@production] beta: Change beta.wmcloud.org stub redirect to new Meta-Wiki canonical
Change #1170188 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/puppet@production] [WIP] beta: redirect misc *.beta.wmflabs.org to *.beta.wmcloud.org
Change #1170208 had a related patch set uploaded (by Krinkle; author: Krinkle):
[operations/mediawiki-config@master] beta: Remove routing for *.beta.wmflabs.org
Change #1170208 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Remove routing for *.beta.wmflabs.org
Mentioned in SAL (#wikimedia-operations) [2025-07-18T00:47:53Z]
Mentioned in SAL (#wikimedia-operations) [2025-07-18T00:49:48Z]
I think that's all for now. There are two non-MediaWiki services in the Beta Cluster that use .beta.wmflabs.org that I'm going to consider out of scope for this task and for someone else to tackle. In particular:
- Email/MX/DKIM/VERP (mail.beta.wmflabs.org) Codesearch. This should continue to work fine as there's no requirement for these domains to match wiki domains, in the same way that all production wikis use wiki@wikimedia.org. It would be nice to migrate, but it's not clear to me what order to perform these steps in, how to test each step, and what the impact of each change is.
- EventGate (intake-analytics.wikimedia.beta.wmflabs.org and intake-logging.wikimedia.beta.wmflabs.org) Codesearch. Idem. Not required to match, and not sure what levels of infra inspect their hostname, which ones support multiple hostnames, order of steps, what kind of redirect it can handle (if any). You'd presumably want to start by supporting both without redirects and then switching traffic.
Mentioned in SAL (#wikimedia-operations) [2025-07-18T01:10:06Z]
Change #1170188 merged by Ssingh:
[operations/puppet@production] beta: redirect misc *.beta.wmflabs.org to *.beta.wmcloud.org
Change #1168038 merged by BCornwall:
[operations/puppet@production] varnish: Improve GeoIP to use cookie domain similar to prod