Propagate rdf changes to the test eqiad nodes on wdqs1028, 1029, 1030, 1031, 1032.
AC
- rdf-streaming-consumer is deployed to eqiad test nodes, consuming from prod rdf topics
- Edits to wikidata are propagated in real-time to the secondary stores.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | gmodena | T414782 Hypothesis WE2.5.1: setup triple stores in eqiad | |||
| Resolved | gmodena | T414447 Propagate real-time wikidata changes to WDQS test nodes |
Event Timeline
Comment Actions
rdf-streaming-consumer has dependencies on Blazegraph and won't work with other databases out of the box. After an update (http POST) it parses the response to update metrics and chokes when trying to talk to QLever. It should be an easy fix but I was hoping to not have to touch code in the mono-repo, since things will get more manageable after we split the repo and streamline CI.
Till we roll out a fix we can fallback on the public stream consumer if we need to do quick experimentation. I set it up on wdqs1029 and it works fine. The only problem is
that it fetches data from the public internet (via https://stream.wikimedia.org), and it's not ideal to have this thing run 24/7 when we could consume from the internal Kafka instead.
Comment Actions
Change #1240793 had a related patch set uploaded (by Gmodena; author: Gmodena):
[wikidata/query/rdf@master] SUC: decouple response handler from blazegraph
Comment Actions
Change #1240797 had a related patch set uploaded (by Gmodena; author: Gmodena):
[wikidata/query/rdf@master] tools: add missing xsd PREFIX to SPARQL update templates
Comment Actions
Change #1240793 merged by jenkins-bot:
[wikidata/query/rdf@master] SUC: decouple response handler from blazegraph
Comment Actions
Change #1240797 merged by jenkins-bot:
[wikidata/query/rdf@master] tools: add missing xsd PREFIX to SPARQL update templates
Comment Actions
The updater consumer is available on test nodes at /srv/wdqs/tools//streaming-updater-consumer-0.3.162-SNAPSHOT-jar-with-dependencies.jar. We did not enable yet since we want to run some more validation on the static index, but the process is documented at: https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Migration/Development_Infrastructure#eqiad_test_nodes
The consumer consumes (urgh) from kafka-jumbo, the only parameters to be mindful of are:
- setting an appropriate consumer group separate from production nodes. To keep things simple, we can use the node hostname with a dev tag to avoid potential naming collisions (eg. wdqs1029-qlever-dev). D
- consuming from the update topic that matches graph split (main or scholarly).
rdf-streaming-consumer has dependencies on Blazegraph and won't work with other databases out of the box. After an update (http POST) it parses the response to update metrics and chokes when trying to talk to QLever. It should be an easy fix but I was hoping to not have to touch code in the mono-repo, since things will get more manageable after we split the repo and streamline CI.
This was addressed in
https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/1240793