Confirmed out of band that we are for now only returning the headers and not blocking traffic. We can do that in the same place when ready.
Once the haproxy patch is merged, headers will be returned for every path under /api/rest_v1/media/math/

@HCoplin-WMF We're only returning the header for now, and not a 403 response, right?

Looking at the new pod I created immediately fail a healthcheck, I decided to peek at p99 latency per pod:

Deleted coredns-85b4f68d95-96gz4 that was showing symptoms

In T431372#12406155, @Fabfur wrote:

Hello, yes we can deprecate it at HAProxy level, just let me know what exactly to use to match (all paths, specific hosts, and so on) and if we want to respond with some specific status code (usually 403 but we can also redirect or add some additional header)

In T390517#12306950, @MLechvien-WMF wrote:

I'm assuming this can be grouped with the clean up of RESTBase, which we're tracking with @MSantos . This will likely be Q2 work at this point, but @Clement_Goubert feel free to correct me if the dependency is wrong

In T437630#12379557, @Ottomata wrote:

Wow, sure why not! So deployment would look like helmfile apply -e eqiad --selector release=mylambda ?

Would the external URL (client-side) be /api/v1 or /api/rest_v1 like existing mobileapps routes? If it's the second, then they're fairly simple additions to the rest-gateway configuration (the mangling into /api/v1 is done in ATS by rb-mw-mangling.lua)
Tightening the query params would need adding something like this to the chosen routes.

In T437630#12374842, @Ottomata wrote:

Beautiful thank you!

It would be nice if the (a?) generic chart supported gRPC too, as I expect we will have more LAC lambdas to deploy in the future.

Apache patch is deployed and is ready for wgRestPath to be changed when needed

Host is being worked on but downtime expired.

We can allocate about 80GB more, but that's it, the vg would be full afterwards. I will allocate 50GB as a first step (so we keep some margin on the vg) and we'll go from there.

In T437209#12355339, @Krinkle wrote:

Both the place where we read Etcd data, and the place where we set PoolCounter configuration can stay unchanged.

Effective load order today:

• multiversion
• mediawiki/includes/WebStart.php
• mediawiki/LocalSettings.php
• wmf-config/InitialiseSettings.php
• wmf-config/CommonSettings.php
  • $wmgAllServices = ServiceConfig::getInstance()->getAllServices();
    • wmf-config/ProductionServices.php
  • $wmgLocalServices = $wmgAllServices[$wmgDatacenter];
  • $etcdConfig = wmfSetupEtcd( $wmgLocalServices['etcd'] );
  • ..
  • .. read stuff from Etcd ..
    • $wmgPCServers = $etcdConfig->get ..
    • $wmgMainStashServers = $etcdConfig->get ..
    • $wgReadOnly = $etcdConfig->get ..
    • $wmgMasterDatacenter = $etcdConfig->get ..
    • $wmgLocalDbConfig = $etcdConfig->get ..
  • ..
  • if wmgUsePoolCounter: wmf-config/PoolCounterSettings.php
    • $wgPoolCountClientConf['servers'] = $wmgLocalServices['poolcounter']

I suggest following the same pattern as other things we read from Etcd:

• Read it from Etcd into a variable, e.g. wmgPoolCounterServers, in the same place where other such data is read.
• Change $wgPoolCountClientConf['servers'] = $wmgLocalServices['poolcounter'] to $wgPoolCountClientConf['servers'] = $wmgPoolCounterServers
• Remove it from ProductionServices.php.

You'll also want need to decide how/where to set it for Beta Cluster (realm=labs). The simplest might be an inline conditional in CommonSettings.php, as $wmgMainStashServers does already.

It may be interesting for you to get these alerts during the early stages of your lambda so you can investigate possible memory leaks and be more autonomous in setting memory limits. I don't think you need more aggressive generic alerts, although you may be interested in the KubernetesContainerReachingMemoryLimit alert as well, which fires when containers use 95%+ of their memory limit over a 10 minute period.

IMO metrics that could be shared with other services should not include anything specific to the service (afc_something_metric) but just emit something_metric, and the prometheus scraper will add labels for app, release, etc.
If this is a service-specific metric (like instrumenting a specific operation, for example a parsing/transformation step that only this service does) then having a service-specific prefix can make sense.

@jasmine_ @Blake can one of you bump the replica count to 2 per dc to avoid paging when the host goes down? Thanks

I think that would be I/F now.

In T433924#12307540, @JMeybohm wrote:

In T433924#12303244, @Clement_Goubert wrote:

Agreed but I think an exception should be carved out for staging, as it would simplify the testing loop for developers by cutting out patching deployment-charts to bump the image version.

I can see the benefit, but I am not a huge fan and would advice against. Partly because it's a derivation from how production works, partly because it makes it harder to track which version is currently running and because it makes rollbacks pretty hard. There is also the "quirk" that changing an image tag to :latest does not update the default imagePullPolicy. E.g. a new deployment created with a :latest tag will have imagePullPolicy: Always set by default. Changing the tag for an image of an existing deployment to :latest does not update the imagePullPolicy, it will still be IfNotPresent.

In T437209#12310454, @Tgr wrote:

What exactly do you want to migrate? There's quite a bit of configuration.

This is a stopgap measure until T436223: Add backend information to webrequests in data lake can be implemented to avoid crumbling Druid through Turnilo when we need to separate requests that go to mw-web or rest-gateway

As far as complex queries go, T436223: Add backend information to webrequests in data lake would help, I also asked @JAllemandou to create a derived column for is_api_call in the meantime for

URI Host != 'thumb.wikimedia.org'
URI Path !~ (?i)^/(|w/api.php|w/rest.php|api/|core|service|feed)
Webrequest source = text

I'm gonna be bold and close this task. Invalid is the closest status I can think of, but it is not a reflection on the information or discussion that happened in this task.

Agreed but I think an exception should be carved out for staging, as it would simplify the testing loop for developers by cutting out patching deployment-charts to bump the image version.

I ran debmonitor manually and it's gone.

Updated, this should now be cleared but I'll check after the next debmonitor run

On it, this is mw-debug in staging which hasn't been updated since April because it's mostly sitting unused.

In T377497#12295133, @Urbanecm_WMF wrote:

In T377497#12295126, @MLechvien-WMF wrote:

this task came up as the last reason why we have PHP on deploy hosts.

Is this actually the last reason? That doesn't match what I remember. For example, sql enwiki executed on the deployment host also shells out to PHP internally. I'm not sure if that has a task though.

commonswiki Error: 2006 MySQL server has gone away

In T437033#12291001, @Mvolz wrote:

In T437033#12290640, @Scott_French wrote:

[...]
We'd also have to put our public key in some open repository somewhere, that might also be something service ops would do?

Deployed, but I can't seem to find the metric being reported by mw-debug's statsd exporters after triggering errors with curl -k -H 'Host: en.wikipedia.org' 'https://mwdebug.discovery.wmnet:4444/w/fatal-error.php?action=oom&password=XXXXXXX'

I'll try to sneak it in after T436781: Live-Test - September 2nd 2026

I'll be your scroll bearer, you can set up a meeting so we can go through the prologue detail.