Note: There's a discussion that @Lina_Farid_WMDE could be better served if this is a pipeline that generates a prep table. Effort for this would be similar to that of the one off investigation as the query for the investigation would just need to be instrumented 😊

@Arian_Bozorg and @Lydia_Pintscher: This task is now finalized and I've scheduled a check in call for us to look at WDQM late in the day on Tuesday :) You can see the results at wd_quality_metrics, with the current value for EntitySchema fulfillment being a proportion of 0.67 (~67%).

Happy to, @xcollazo! Just did the following edit: https://wikitech.wikimedia.org/w/index.php?title=Data_Platform%2FSystems%2FAirflow%2FDeveloper_guide&diff=2465731&oldid=2452021 :)

I took a moment just now to test this and can confirm that adding --set config.airflow.config.email.email_backend=airflow.utils.email.send_email_smtp does allow for the devenv to send an email. We have a MR above to add this to our docs, which resolves this 😊 Thanks @BTullis and @brouberol!

Subscribed @danshick-wmde as well as we'll be getting a copy review for this when we're done to make sure it's accessible to a wide audience. Exciting! 🎉

Note that @catherine.kelsey.wmde did the WD changes preference for September just now. Thanks, @catherine.kelsey.wmde!

Note: We'll merge this with https://phabricator.wikimedia.org/T411872

Same for here, @Lina_Farid_WMDE! Please feel free to add any further information that you feel is needed, but we can also sync when this task is picked up!

Just wrote a quick description here, @Lina_Farid_WMDE! Please feel free to add any further information that you feel is needed, but we can also sync when this task is picked up!

@awight: Could we sync with you on whether or not we're able to derive Wikidata QID citation usage in the coming days? Happy to set up a quick sync for us!

Let's similarly close this task on Thursday after our check in, @Lina_Farid_WMDE!

Note here, @Lina_Farid_WMDE: Let's be sure to confirm that this task is done in our meeting on Thursday :)

Hi @catherine.kelsey.wmde 👋 Looking at the tables, I'm not sure how useful wmf.webrequest will be, but maybe I'm misunderstanding the context of the task. I'd use wmf.webrequest if we know some specific user agents for WikiCite that we want to track the behavior of. For the presence of WikiCite related entities on Wikipedia pages, then we might need to look into wmf_raw.mediawiki_private_linktarget and other related tables where the connection between WD entities and Wikipedia pages are tracked.

@ItamarWMDE, tagging you here as it'd be nice to collaborate here and get your feedback on tmux. I've been kicking this down the road for some time. Goal here is documenting how to properly set up working productively in the stat servers, which you appear to have a lot of insights on. Let me know if it's of interest, and maybe we can schedule a call to work through things and document them!

Note that all stakeholders for this process have been included on the task now. In finalizing the metrics framework we have the goal to align the way we talk about data across the following areas of WMDE work:

Thanks, @brouberol! We'll give it a go in the next DAG testing and report back 😊

Thanks for tagging everyone and the intuition on where this might be coming from, @BTullis!

Based on a conversation with @mahmoud.abdelsattar.wmde, I wanted to comment to bring up a definite need for WikibaseQualityConstraint data being stored persistently. WMDE is using the proportion of mandatory constraints that are consistent as a part of the broader Wikidata Quality Metric (WDQM, epic is T427512). Work for this was originally based on a sample of such constraint statements and checking them via wbcheckconstraints and a bot flagged user (simplest way we could think to get initial results).

I'm not sure if I have approval rights for these sorts of requests, but as @Hany.elmokadem is an Engineering Manager for WMDE, this request should be approved.

@catherine.kelsey.wmde: One thing to note here is that we need to save those users that have this preference in a table and then recalculate at the next interval that it's ran for the churn. With this MariaDB table, we don't have a snapshot to go off of. It's just the current state of the table.

Note from our review planning: This task is prioritized with T435805.