In T320436#12412554, @dancy wrote:

@Urbanecm Is this ticket a request for action (e.g., hiding the retry warnings), or just a note (in which case I will close it)?

During the DE4 Offsite, it was mentioned that linking Special:InvestigateBlock from SI would be useful as well.

Seems to be 1Passwords fault after all :/

Interestingly, I hit the login ratelimit too. I would not expect that to happen if I actually haven't finished anything.

In T439489#12373424, @Tgr wrote:

onMediaWikiServices is running well before onSetupAfterCache.

Related thought: Why do we even have the override in onSetupAfterCache? It seems to force using global to read config, which is..discouraged (to say the least). Reading the central DB should be probably routed via Utils::getCentralWiki() (or similar), which can replace false with the local wiki ID on the fly if so required.

In T434329#12231709, @matmarex wrote:

Done?

This is a block on the CDN's side:

@Aklapper: This would still need a fix to make it not happen in the future. Core Platform's review would be appreciated. I'm not sure if my change is sufficient (it seems to be some kind of race condition, so...maybe), and I'd like second eyes on that.

This is now done:

That being said, the script probably should fail more reasonably if stdin is not attached... Uploading a patch.

@Zabe: That is an invocation issue (the script needs an explicit confirmation => stdin needs to be attached => --attach must be used). I've executed the script properly, the page is now moving.

@Zabe The script wants you to type something. It should've been run with stdin attached (so --attach rather than -f). With that replacement, it works as intended :).

The script exists in puppet. It seems to be edited by @dcausse recently.

testcommonswiki was closed in T213295, but later reopened in T378127 and it was never re-closed. So, the wiki is presently opened, despite what the sitenotice says.

@SHB2000 You should now be able to login.

In any case, I believe this now has enough data to fix this in CentralAuth itself. Let's recover the account itself:

It seems like CentralAuth is treating "no row in user with this name" as a proof that its localnames and localuser records are incorrect, and it deletes them. However, during a global rename, this mismatch is deliberate, which might easily result in deleting correct data.

Interesting observation in CentralAuth itself:

The request ID of the job chain, a9ad97f0-4e44-47e1-ae17-59c24026f1cb, has quite a few exceptions logged. Posting few examples below, Logstash has full list:

The attachment happened on commonswiki, but something deleted it afterwards:

The account should be fixable by adding a row to localnames and merging, but before doing that, I'd like to understand why it disappeared in the first place.

Log entries:

Fun! And interesting, given https://meta.wikimedia.org/wiki/Special:CentralAuth/SHB doesn't even claim a commonswiki account exists, which means the underlying localnames table is probably wrong:

FWIW, this would also mean "Toolforge has credentials that can see any security ticket". Which, given the disclaimers the Cloud's ToU have (https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#9.2_Disclaimers) may or may not be advisable.

Traceback: P96434

No longer fails

This breaks since rEGBL2421bf90bc37: Partially update SpecialGlobalBlock for global account blocks (cc @Dreamy_Jazz).

FWIW, restricted is a subset of deployment in terms of permission (if you are in deployment, you can do whatever restricted can automatically).

In T437273#12297954, @pmiazga wrote:

That's a really good point, but looks like it doesn't work. If I set the readonly in MWDebug the Special:Version still works. The logging-in doesn't work.

Curiosity question...why have we decided to skip certificate verification altogether? Wouldn't it make sense to make the cert trusted, so that if someone is in the middle of the connection, it doesn't get compromised? I'm sure there are valid reasons for this decision, but I don't really understand it.

Now it works:

You cannot set individual wikis to read_only

This started with rECAU4954abed2545: Replace CentralAuthDatabaseManager with CentralAuthConnectionProvider (CC @SomeRandomDeveloper @Krinkle), which added CentralAuthConnectionProvider::getPrimaryDatabase(), which calls assertNotReadOnly(), and routes all CentralAuth DB needs through it. This would be fine, if...CentralAuthSessionManager::makeSessionKey() didn't use a primary DB connection to determine the global session key, which happens in absolutely any view. Prior version called IConnectionProvider::getPrimaryDatabase( 'virtual-centralauth' ), which doesn't assert read-write.

Ideally, we should also introduce automated testing that verifies MediaWiki as installed in production works when the primary DB is read only. Not sure how to do that though...

This task should be considered urgent and incident prevention. This essentially removes our ability to go read only at any time.

You can try adding this to your .my.cnf.

I see. Maybe you have already thought about this, but can we simply display "key also used by XX" in IDM? If we want to, we can even add an "Ack" button, forcing both accounts to explicitly acknowledge this is done intentionally.