Hi,

Thanks for reaching out. Roughly speaking, we start to throttle connections (for bots that respect maxlag) when the change propagation lag between wikidata.org and the WDQS secondary store is higher than 10 minute for a sustained amount of time. A typical reason for this lag increase is when WDQS is put under heavy read load, and writes can't keep up.

WDQS has been under heavy load starting Thursday, April 2. This had a cascading effect on lag and bot throttling (https://grafana.wikimedia.org/goto/dfhtdl5py8qv4f?orgId=1). We have alerting and operational processes in place to mitigate this issue. We've been tracking load since early alerts started to fire on Thursday. Unfortunately, we had several actors concurrently putting WDQS under strain for the past 5 days, that defied automated remediation we have in place and required manual intervention. This situation resulted in bot throttling kicking off more than we would have liked. We've seen spikes in load and lag also this morning CEST (https://grafana.wikimedia.org/goto/dfid0zv54dqm8e?orgId=1), as we keep monitoring the situation.

FWIW, if the maxlag is consistently high but some bots are still editing so fast that are keeping wdqs under pressure, it is a clear violation of bot policy and should be blocked.

The top editor yesterday and the day before was Mahir256 with 40K edits each day. The day before that was @Epidosis with 203K edits(!), the day before was Epìdosis again with 202K edits. I think that's causing issues. Epìdosis: Please respect maxlag.

@Ladsgroup both Epìdosis and I were using QuickStatements (he version 3.0 and I version 2.0); your complaint about tools not respecting maxlag should be directed at @Arcstur in the former case and @Magnus in the latter case.

Oops, I'll fix it in V2

Thanks!

Hi, this may be related to my import of data from GND into Wikidata via QS 3.0 which ran from April 1 to April 5 (https://w.wiki/KdP6). I thought QS 3.0 respected automatically maxlag; but maybe there is some kind of exception to be removed for users with admin flag. I have just reported at https://meta.wikimedia.org/wiki/Talk:QuickStatements_3.0#Maxlag_issues

V2 should be fixed now

For the record: The problems began on March 25th or 26th (see Grafana control panel), and it is still an issue currently because the minimum maxlag is 9 seconds for the last 1 hour. This blocks all those bots respecting the Bot Policy (like Pywikibot default settings).

In T421642#11785461, @Xqt wrote:

The problems began on March 25th:

Please (re)attach the file, so that it's visible if it's important (Phabricator/Help § File visibility).

In T421642#11797798, @matej_suchanek wrote:

Please (re)attach the file, so that it's visible if it's important (Phabricator/Help § File visibility).

done

In T421642#11785461, @Xqt wrote:

The problems began on March 25th:

Exact timestamp seems to be shortly after 2026-03-25 15:00 UTC, which apparently is pretty much exactly the moment when the northward datacenter switchover (March 2026 codfw to eqiad) took place T413974. Can anyone please check whether there is causality?

Thanks. I asked around to see if anyone would be willing to take a look.

Any update here?

The problem persists, my bots are regularly crashing due to persistent maxlag timeouts, and community members complain that my bots need a fix (when they simply obey with the maxlag policy).

As a note: I have started a new import through QS 3.0 a few hours ago - cf. https://www.wikidata.org/wiki/Property_talk:P227#Massive_import_of_data_from_GND_(May_2026) - but since the tool respects maxlag, it should not have a relevant impact.

@Epidosis Hi, what makes you say that QS 3.0 respects maxlag? I checked the source code and see no reference to maxlag.

Maxlag has been abnormally high for the past few days, and has exceeded five minutes at time of writing! Most of my queries are failing, important bots aren't working…

In T421642#11793464, @Mahir256 wrote:

...tools not respecting maxlag should be directed at @Arcstur in the former case...

Pinging @ACorrea-WMB just in case you no longer check your personal Phabricator account.

The issue has got progressively worse over the past several hours, and max lag is now at around 40 minutes:

I can't say with certainty that this is due to QS 3.0, though I do note that the majority of edits happening right now are through QS 3.0.

I wonder if someone did something; max lag seems to have returned to normal levels since around 10:00 UTC.

In T421642#11901977, @Yirba wrote:

I wonder if someone did something; max lag seems to have returned to normal levels since around 10:00 UTC.

I fear it's increasing again.

See also T425758.

Hello, everyone, I'll share here some info regarding QS3 so you can help me understand if we are or not respecting it... I'll split it into parts.

1. QS3 does not deal with maxlag directly because it does not use the Action API, only the Wikibase REST API.

2. QS3 will sometimes hit 429 in the REST API, when that happens, it starts an exponential backoff wait using the python urllib library. We are frequently hitting 90 edits per minute, which is the maximum allowed for the users. We are not editing more than that because that is not possible. The edits per minute can be checked in EditGroups for some batches.

3. In January a lot of users came to me warning that batches were stalling for hours. They were regularly being stalled for 1h+. After debugging I saw that the "Retry-After" header sent with the 429 had enormous delay times, up to 6 hours. So I modified QS3 to start ignoring that and use just the timeout with exponential backoff: https://github.com/wikimediabrasil/quickstatements3/issues/409

Without a good way to measure if maxlag is being respected or not is hard to tell. Maxlag is still a rather confusing term for me, since I'm most used to dealing with bare HTTP requests.

About (3), I can switch back for QS3 to use the Retry-After header received with 429's, but if that makes batches stall for 1h+, which is unreasonble since every user has 90 edits per minute per default.

I updated what I described above on Jan 24, seeing as the graph here shows the increase after 2026-03-25 15:00 UTC I wonder if they are really connected.

Maxlag dropped from 51 minutes (!) to a normal ca. 5 seconds at 10 UTC today, but already shortly after 11 UTC it restarted a significant growth and is now (17:30 UTC) slightly above 8 minutes. Considering the light editing today on QS 3.0, I guess we can say QS 3.0 is probably unrelated with the issue we are having.

Again the maxlag is increasing a lot:

@Xqt is it possible to check maxlag PER user-agent? I guess that coul help us sort thing out.

In T421642#11928938, @Arcstur wrote:

@Xqt is it possible to check maxlag PER user-agent? I guess that coul help us sort thing out.

I found this issue while distributing the new Pywikibot 11.3 stable release. The issue persisted for about 6 hours. The user agent was /unittest Pywikibot/11.3.0.dev0 (g2) Python/3.9.25.final.0 requests/2.32.5 or /pytest Pywikibot/11.3.0.dev0 (g2) Python/3.14.3.final.0 requests/2.32.5, because no user is logged in during Jenkins CI tests. The tests would have taken 1.5–4 hours while the maxlag issue was occurring, but they were aborted after 45 minutes.

It seems today is a bad day again for pywikibot trying to make only a single edit.

Maxlag is way too high:
https://grafana.wikimedia.org/d/TUJ0V-0Zk/wikidata-alerts?orgId=1&from=2026-08-10T05:20:04.599Z&to=2026-08-10T16:30:51.031Z&timezone=utc&refresh=30s&viewPanel=panel-12

This is breaking my tools too.

Possibly related: https://www.wikidata.org/wiki/Wikidata_talk:Requests_for_comment/Mass-editing_policy#Lag_impact_of_large_running_batches

@Epidosis: The impact is not solely for Wikidata.

As I wrote back in T244030 (which was T242081: Pywikibot fails to access Wikidata due to high maxlag lately back then):

In T244030#5842377, @JeanFred wrote:

I was under the impression that the purpose of Maxlag was to limit editing (which makes sense, as more edits just make the lag even worse) ; but here all I’m doing is read-access. Or am I missing something here?

I still don’t quite understand why read-access − or for that matter here, even Site object instantiation, ie we are not doing anything yet, gets throttled. Is this intended?

In T421642#12271821, @JeanFred wrote:

As I wrote back in T244030 (which was T242081: Pywikibot fails to access Wikidata due to high maxlag lately back then):

In T244030#5842377, @JeanFred wrote:

I still don’t quite understand why read-access − or for that matter here, even Site object instantiation, ie we are not doing anything yet, gets throttled. Is this intended?

• https://www.mediawiki.org/wiki/Manual_talk:Maxlag_parameter#Usage%20question
• https://wmds-archive.toolforge.org/t/should-read-only-api-requests-also-use-maxlag/593/

I guess that read requests do put load on the database, so I can see why they are subject to throttling as well. However, I’m not sure they should necessarily be treated exactly like writes.

A read does not create replication events itself, so it shouldn’t directly increase replication lag. It can of course contribute to the overall database load, which may indirectly make replication fall behind if the database is heavily loaded.

Personally, I wouldn’t be opposed to treating reads and writes differently—for example, allowing a higher maxlag threshold for reads. But I don’t think the current policy makes that distinction.

Here are some early and abandoned proposals for Pywikibot:
https://gerrit.wikimedia.org/r/q/is:watched+%22maxlag%22+is:abandoned

It depends on what kind of load and what is causing the lag to go up. e.g. If it's a gigantic write, reducing reads on replicas won't help on anything. If the whole site is under pressure and can't keep up, it could help (but not much since we switched to ssd ~nine years ago).

the longer term problem is that maxlag has turned into the catch-all metric of server load and it's clearly not designed for it. A bot can be easily overwhelming wikimedia infra without impacting the maxlag (so not seeing the impact and not backing down) but that's for the future.

If maxlag is an imperfect indicator of overall server load anyway, I think it would be reasonable to consider exempting certain read operations from throttling, or perhaps read operations in general could be given a higher maxlag threshold, especially where we know that they impose very little load. Also, I think some kind of priority aging would be more appropriate for bots, rather than keeping them waiting for days. Most bot operators are probably already familiar with the -maxlag parameter anyway.

Change #234723 had a related patch set uploaded (by Xqt; author: John Vandenberg):

[pywikibot/core@master] Disable maxlag for meta queries, paraminfo and help

https://gerrit.wikimedia.org/r/234723

Over the last days, I constantly get maxlag errors, mostly when attempting to get items using wikibaseintegrator, but also (to a lesser extent) for writing. I would be constantly reading and writing (not more than 2 API calls per second), but only a few hundred edits per day get through. This is the job (see bot contributions).

Change #1334890 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] Increase maxlag for read requests

https://gerrit.wikimedia.org/r/1334890

Mentioned in SAL (#wikimedia-operations) [2026-09-03T17:39:05Z] T421642 [WDQS] requestctl changes: 2026-09-03 16:23-17:33 UTC: added hard-deny pair cache-text/wdqs_futile_sparql_sep_2026_deny(+_bots); extended pattern ua/wdqs_heavy_sparql_bots_2026 and added default-scope twin wdqs_heavy_sparql_bots_jul_2026_ratelimit_default; added ipblock abuse/wdqs_sparql_scanners_sep_2026 + throttle wdqs_sparql_scanners_sep_2026_ratelimit (needed manual requestctl update-provenance-map)

11' now and now maxlag beneath 9'' for the last 24 hours.

18:42:40 VERBOSE  pywiki:logging.py:152 Sleeping for 0.1 seconds, 2026-09-19 16:28:53
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1016: 163.85 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:28:53
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1013: 144.26666666667 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:30:23
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1013: 144.98333333333 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:31:53
18:42:40 _________ [doctest] pywikibot.site._apisite.APISite.is_uploaddisabled __________
18:42:40 3168 Return True if upload is disabled on site.
18:42:40 3169 
18:42:40 3170         **Example:**
18:42:40 3171 
18:42:40 3172         >>> site = pywikibot.Site('commons')
18:42:40 3173         >>> site.is_uploaddisabled()
18:42:40 3174         False
18:42:40 3175         >>> site = pywikibot.Site('wikidata')
18:42:40 UNEXPECTED EXCEPTION: SkipTest('Maximum retries attempted due to maxlag on wikidata:wikidata without success.')
18:42:40 Traceback (most recent call last):
18:42:40   File "/opt/pyenv/versions/3.10.20/lib/python3.10/doctest.py", line 1350, in __run
18:42:40     exec(compile(example.source, filename, "single",
18:42:40   File "", line 1, in 
18:42:40   File "/src/pywikibot/__init__.py", line 279, in Site
18:42:40     _sites[key] = interface(code=code, fam=fam, user=user)
18:42:40   File "/src/pywikibot/site/_datasite.py", line 41, in __init__
18:42:40     super().__init__(*args, **kwargs)
18:42:40   File "/src/pywikibot/site/_apisite.py", line 141, in __init__
18:42:40     self.login(cookie_only=True)
18:42:40   File "/src/pywikibot/site/_apisite.py", line 397, in login
18:42:40     if self.userinfo['name'] == self.user():
18:42:40   File "/src/pywikibot/site/_apisite.py", line 675, in userinfo
18:42:40     uidata = uirequest.submit()
18:42:40   File "/src/pywikibot/data/api/_requests.py", line 1235, in submit
18:42:40     raise unittest.SkipTest(msg)
18:42:40 unittest.case.SkipTest: Maximum retries attempted due to maxlag on wikidata:wikidata without success.
18:42:40 /src/pywikibot/site/_apisite.py:3175: UnexpectedException
18:42:40 ----------------------------- Captured stderr call -----------------------------
18:42:40 Sleeping for 90.0 seconds, 2026-09-19 16:33:24
18:42:40 Sleeping for 90.0 seconds, 2026-09-19 16:34:55
18:42:40 Sleeping for 90.0 seconds, 2026-09-19 16:36:25
18:42:40 ------------------------------ Captured log call -------------------------------
18:42:40 VERBOSE  pywiki:logging.py:152 Found 1 wikidata:wikidata processes running, including this one.
18:42:40 VERBOSE  pywiki:logging.py:152 Sleeping for 0.1 seconds, 2026-09-19 16:33:24
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1013: 137.93333333333 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:33:24
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1016: 163.98333333333 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:34:55
18:42:40 VERBOSE  pywiki:logging.py:152 Pausing due to database lag: Waiting for wdqs1014: 111.46666666667 seconds lagged.
18:42:40 INFO     pywiki:logging.py:152 Sleeping for 90.0 seconds, 2026-09-19 16:36:25

Change #234723 merged by Xqt:

[pywikibot/core@master] Disable maxlag for meta queries, paraminfo and help

https://gerrit.wikimedia.org/r/234723

Change #1334890 merged by jenkins-bot:

[pywikibot/core@master] Use a separate maxlag threshold for read requests

https://gerrit.wikimedia.org/r/1334890