Page MenuHomePhabricator

FY2025-26 WE 6.4.1: Move links tables of commons to a dedicated cluster
Closed, ResolvedPublic

Assigned To
Authored By
Ladsgroup
Jul 4 2025, 12:04 PM
Referenced Files
F102901490: grafik.png
Mon, Sep 14, 9:53 AM
F102323604: grafik.png
Sep 9 2026, 11:05 PM
F102321467: grafik.png
Sep 9 2026, 11:05 PM
F102321285: grafik.png
Sep 9 2026, 11:05 PM
F102321133: grafik.png
Sep 9 2026, 11:05 PM
F102320115: grafik.png
Sep 9 2026, 11:05 PM
F102319952: grafik.png
Sep 9 2026, 11:05 PM
F102319852: grafik.png
Sep 9 2026, 11:05 PM

Description

This is part of annual plan for FY 25-26. Commons database has grown to an extremely large size (2TB) and we have no choice but to split it. The most cost-benefit efficient way is to move the links tables to a dedicated cluster (x4) which would cut its size to half.

These tables must be moved to x4:

  • linktarget
  • externallinks
  • pagelinks
  • templatelinks
  • categorylinks
  • collation
  • imagelinks
  • globalimagelinks
  • iwlinks
  • existencelinks
  • langlinks

This can be done via introducing a new core virtual domain (virtual-links) and then migrating queries. Any query that join with page or other tables must be split to be two queries hitting different connections and if not possible, it should be done in a different way (e.g. T309738: Move MediaWiki QueryPages computation to Hadoop) or gets disabled in commons.

We need to set up back ups and wikireplicas too.

For now, we should assume page and redirect tables will exist in both databases. Depending on how many cases directly join and not easy to split, we could consider having those tables being shared.

Related Objects

StatusSubtypeAssignedTask
ResolvedLadsgroup
ResolvedZabe
ResolvedLadsgroup
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedBUG REPORTZabe
ResolvedBUG REPORTFunc
ResolvedLadsgroup
OpenZabe
ResolvedPRODUCTION ERROR• Scott_French
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
ResolvedZabe
OpenZabe
OpenZabe
DuplicateZabe
OpenZabe
OpenZabe
OpenZabe
DeclinedZabe
OpenZabe
ResolvedPRODUCTION ERRORZabe
ResolvedZabe
ResolvedLadsgroup
ResolvedZabe
ResolvedZabe
ResolvedAhoelzl
ResolvedZabe
ResolvedNone
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
OpenNone
ResolvedPRODUCTION ERRORZabe
Resolvedtaavi
Resolvedtaavi
ResolvedMarostegui
ResolvedMarostegui
ResolvedMarostegui
InvalidNone
InvalidNone
ResolvedMarostegui
ResolvedMarostegui
ResolvedPRODUCTION ERRORZabe
OpenLadsgroup
ResolvedMarostegui

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Doesn't this require communication towards the community similar to how the x3 change was announced for the Wikidata terms tables ?

Doesn't this require communication towards the community similar to how the x3 change was announced for the Wikidata terms tables ?

Yup but we are at least several months away from the split actually happening. There are a lot of complex issues that need to be solved e.g. how to replicate the page table from s4 to x4 (and the consolidation mechanism) and so on.

I noticed in a few commits that there is not just virtual-links but also a more narrow virtual-categorylinks. Is there more detail somehere on how that is meant to work exactly? The current plan describes a dedicated cluster for the link tables, which is easy to reason about and there's the task listing which tables are available to query/join there (with page+redirect earmarked as special TBD).

A domain like virtual-categorylinks suggests more decoupling, but I couldn't find any discussion or docs on what it is meant to be used for or what tables one we assume to exist there? Some of the queries using this join against linktarget, for example.

I noticed in a few commits that there is not just virtual-links but also a more narrow virtual-categorylinks. Is there more detail somehere on how that is meant to work exactly? The current plan describes a dedicated cluster for the link tables, which is easy to reason about and there's the task listing which tables are available to query/join there (with page+redirect earmarked as special TBD).

A domain like virtual-categorylinks suggests more decoupling, but I couldn't find any discussion or docs on what it is meant to be used for or what tables one we assume to exist there? Some of the queries using this join against linktarget, for example.

The idea was to mostly ease the migration of read queries and make sure they are correctly moved (specially for the test setup). Once we are done, all will be consolidated into virtual-links.

Change #1266343 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@master] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266343

Change #1266343 merged by jenkins-bot:

[mediawiki/core@master] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266343

Change #1266442 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@wmf/1.46.0-wmf.21] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266442

Change #1266443 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@wmf/1.46.0-wmf.22] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266443

Change #1266442 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.21] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266442

Change #1266443 merged by jenkins-bot:

[mediawiki/core@wmf/1.46.0-wmf.22] Deferred: Fix function to get virtual domain

https://gerrit.wikimedia.org/r/1266443

Mentioned in SAL (#wikimedia-operations) [2026-04-01T22:27:32Z] Started scap sync-world: Backport for [[gerrit:1266443|Deferred: Fix function to get virtual domain (T421914 T398709)]], [[gerrit:1266442|Deferred: Fix function to get virtual domain (T421914 T398709)]]

Mentioned in SAL (#wikimedia-operations) [2026-04-01T22:29:29Z] ladsgroup: Backport for [[gerrit:1266443|Deferred: Fix function to get virtual domain (T421914 T398709)]], [[gerrit:1266442|Deferred: Fix function to get virtual domain (T421914 T398709)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-04-01T22:34:08Z] Finished scap sync-world: Backport for [[gerrit:1266443|Deferred: Fix function to get virtual domain (T421914 T398709)]], [[gerrit:1266442|Deferred: Fix function to get virtual domain (T421914 T398709)]] (duration: 06m 37s)

Change #1267967 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/core@master] Simplify checks if link tables use virtual domains

https://gerrit.wikimedia.org/r/1267967

Change #1267967 merged by jenkins-bot:

[mediawiki/core@master] Simplify checks if link tables use virtual domains

https://gerrit.wikimedia.org/r/1267967

Zabe updated Other Assignee, added: Zabe; removed: Ahoelzl.Sep 6 2026, 11:42 AM

Change #1338142 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Move testcommonswiki links tables to x4

https://gerrit.wikimedia.org/r/1338142

Ladsgroup reassigned this task from Ladsgroup to Zabe.
Ladsgroup updated Other Assignee, added: Ladsgroup; removed: Zabe.

There is a lot of clean up and follow ups needed to be done but the main goal of the task which was split the database has been achieved. We have dropped around 1TB from each replica of commons

grafik.png (1,846×928 px, 48 KB)

And this has massively improved performance of these dbs by increasing reads from the memory and reducing reads on disk. It even reduced writes and made binlogs smaller

Some nice graphs:
On-x4 side:
https://grafana.wikimedia.org/d/000000273/mysql?from=2026-09-03T18:52:46.001Z&to=2026-09-09T22:38:01.645Z&timezone=utc&var-job=$__all&var-server=db1261&var-port=9104&refresh=1m&viewPanel=panel-13

grafik.png (1,847×908 px, 171 KB)

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db1261&var-datasource=000000026&var-cluster=mysql&from=2026-09-03T17:01:50.280Z&to=2026-09-09T22:39:09.456Z&timezone=utc&viewPanel=panel-35
grafik.png (1,851×941 px, 118 KB)

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db1261&var-datasource=000000026&var-cluster=mysql&from=2026-09-03T17:01:50.280Z&to=2026-09-09T22:39:09.456Z&timezone=utc&viewPanel=panel-35
grafik.png (1,852×936 px, 205 KB)

On s4-side:
https://grafana.wikimedia.org/d/000000273/mysql?from=2026-09-03T09:10:51.159Z&to=2026-09-09T22:47:24.454Z&timezone=utc&var-job=$__all&var-server=db1247&var-port=9104&refresh=1m&viewPanel=panel-13

grafik.png (1,843×931 px, 192 KB)

For example, while number of requests have increased (for various reasons):

grafik.png (1,895×932 px, 75 KB)

The reads from disk has gone down:
https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=db1247&var-datasource=000000026&var-cluster=mysql&from=2026-09-04T04:10:15.144Z&to=2026-09-09T22:39:09.456Z&timezone=utc&viewPanel=panel-35
grafik.png (1,864×923 px, 111 KB)

Writes were spikey so I made rolling average of 6 hours (in the past seven days):
grafik.png (1,735×450 px, 43 KB)

Great job @Zabe

Change #1338142 merged by jenkins-bot:

[operations/mediawiki-config@master] Move testcommonswiki links tables to x4

https://gerrit.wikimedia.org/r/1338142

Mentioned in SAL (#wikimedia-operations) [2026-09-09T23:11:39Z] Started scap sync-world: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]]

Mentioned in SAL (#wikimedia-operations) [2026-09-09T23:15:58Z] ladsgroup: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-09-09T23:22:54Z] Finished scap sync-world: Backport for [[gerrit:1338142|Move testcommonswiki links tables to x4 (T398709)]] (duration: 11m 15s)

The last one was actually on reads in the past six hours (I picked the wrong metric). The write is this, for the past 30 days:

grafik.png (1,715×413 px, 61 KB)

https://grafana-rw.wikimedia.org/goto/ssnmlk?orgId=default