Still running:

In T439044#12381626, @mpopov wrote:

Under the hood, mentee questions are just regular edits (they're not special in any way, except they are made through their own interface and we guide newcomers to that interface). We (=MediaWiki Core) send a domain event for every single edit.

Oh, I actually didn't realize that – that's very helpful to know! I thought it was a private messaging system between mentees and mentors, but it's basically a specialized way to post on a user's talk page? (I should have reached out to Growth before filing this to learn more about the feature. Oops! 😅 )

Still running:

In T439651#12413900, @Ladsgroup wrote:

FWIW, most fatals and exception still trigger the mediawiki shutdown which triggers the unlock. I guess it called in a very brutal manner that didn't even get to run shutdown (which also has other side effects like not emitting some logs, etc.)

In T439651#12412763, @BPirkle wrote:

From a prioritization standpoint, do we have a sense of the production impact and/or frequency at which this occurs?

In T429626#12411090, @mahmoud.abdelsattar.wmde wrote:

@Urbanecm_WMF When we tested it, it worked (maybe it was caching, even when I cleared it), but now it doesn't for both versions.

I'm confused... I thought the patch worked when I shared it via the call? Are you saying the initial version and the final one behave differently on your local system?

Thanks for the ping! I removed them from project puppet, puppet still runs fine:

In T433373#12399066, @Nikerabbit wrote:

@Urbanecm_WMF I see only extension description in there. You want that already translated?

Tested with MU test 202610051018, it seems to work now. T440215 exists as a follow up.

In T440215#12397085, @Michael wrote:

While this task is mainly about CommunityConfiguration, the specific example is about the Mentorlist, which is one of the few cases that does not use CC's generic form builder, and so will likely have to separately implement showing the captcha.

In T439782#12397415, @Etonkovidova wrote:

Thank you, @Urbanecm_WMF! Hmm... I didn't think about a mentor being active. Zilant1b is indeed inactive according to criteria on https://test.wikipedia.org/wiki/Special:CommunityConfiguration/Mentorship, and, still, the status is Active. Is it another problem?

@mahmoud.abdelsattar.wmde I have finalised the patch that ensures GrowthExperiments considers any content namespace.

Thanks for double checking @Scott_French. From my perspective, this is not very urgent (if you need some time to widen the timeout globally, I don't think Growth would need a special rule in the interim; just let me know when I should look at this again). That said, I do think we should either widen the timeout globally, or shorten the MediaWiki-imposed one. Hope this makes sense.

@Scott_French I don't think it is fully fixed yet:

Thanks @Scott_French! Those jobs are (relatively) uncommon. I just scheduled one manually. Let's see what it will do.

In T439782#12395924, @Urbanecm_WMF wrote:

In principle, this should only affect users who do not have skipcaptcha?

FWIW I have no idea whether adding those values where I added them is a good long term fix. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040158 shows they possibly should be in puppet somewhere.

This appears to cause T440201: deployment-cp* puppet broken by recent puppet change

I added missing values to deployment-prep: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e03f2606f899152f20a6401452175f59a5619e10%5E%21/#F0

In principle, this should only affect users who do not have skipcaptcha? I was able to quit (and rejoin) using my staff account, but not using a brand new one (which doesn't yet have skipcaptcha).

Thank you, as always!

Yep, MW doesn't have the s3 DB configured for kaiwiki at all:

Thanks @Marostegui. Should we keep the empty DB there? AFAIK, kaiwiki shouldn't be on s3 to begin with (https://noc.wikimedia.org/db.php puts it to s5, where it does have the regular tables).

Script is still running:

since Cecil_Duncan was probably re-added before to Link recommendation set before the fix.

There are multiple writes per mentor, but we can certainly make them smaller. @Ladsgroup, are there any problems with transactions that are _too small_? Or is that not a concern at all.

I'm confused at what the goal/problem here is, as this is pretty much already the case. Under the hood, mentee questions are just regular edits (they're not special in any way, except they are made through their own interface and we guide newcomers to that interface). We (=MediaWiki Core) send a domain event for every single edit.

@Sdkb This is basically done (as in, new resignations/removals should work as intended); more details are available in the summary above. If you see an issue in resignations/removals not being cleaned up fully, please do let me know.

The reported symptom — a mentor quits and keeps their mentees — is not one bug but a chain of them. reassignMenteesJob processes one batch and reschedules itself; anything that stops the chain leaves the remaining mentees behind silently, because the job reports success in every one of those cases.

In T418194#12375821, @Urbanecm_WMF wrote:

I call this fixed from an user facing perspective. The remaining few issues are tracked separately now.

The script is still here:

@Michael Boldly moving this to current maintenance backlog, because this should now be unblocked and because it is Technical-Debt that hits us whenever we try to hit mentorship (see Caveats on T439638, which I run into when working on T418194).

I think this is done.

@Ladsgroup This was a mitigation you added, and I'm proposing to change it. Asking you for second opinion before I break anything :).

Why does this happen?

Pending mystery: Part of this issue is that enotifNotify jobs are getting scheduled for every single user, pushing our own follow up job further than a minute, thus breaking the lock handoff system we use.

I call this fixed from an user facing perspective. The remaining few issues are tracked separately now.

Why is MediaWiki continuing with the job when the client disconnects?

Let's try another user:

More questions: In changeprop, I see the DEFAULT_REQUEST_TIMEOUT for requests changeprop makes is set to 420 seconds. So even with the envoy timeout increased from 300s to something larger, we'd only increase the effective duration to 420s.

Adding DBA for advice on "MediaWiki versus databases". Is transaction this large a problem? Should/can we make it a smaller one? Or should we just ignore it altogether?

I discussed this with @Volans over Slack; their response was:

This is several bugs combined into one:

For my own self, I converted the logs into a timeline:

It has finished:

Preparing some users to tests this on:

FTR: Those tables were never accessed by MediaWiki.

How to access logs (for deployers):

For QA purposes, this is a log of the begining of the revalidation I have just started:

This is now running.

Revalidating everything 1790175600 and older should do the trick (matches Wed Sep 23 2026 15:00:00 GMT+0000). Deployed happened at 14:37, so anything 15:00 and older should be revalidated.