View source for Wikidata:WikiProject Redundancy

{{shortcut|WD:WPR}}
<div style="box-shadow: 0 0 .3em #999999; border-radius: .2em; margin: 0 0 2em 0; padding: 1px;">
<div style="background: #339966; border-radius: .2em; color: #FFFFFF; padding: 1em .6em 1.1em; text-align: center; font-size:1.3em;">{{PAGENAME}}</div>
<div style="padding: 1em; border-{{dir|{{PAGELANGUAGE}}|left|right}}: 1px solid #BBBBBB">

<div style="font-size:1.1em;">

The primary aim of '''{{PAGENAME}}''' is to reduce the amount of Wikidata's ''data''—without reducing the amount of ''information'' in Wikidata!—for the well-being of Wikidata, its community, and its downstream users.

[[#Participants|Join us!]]
</div>
</div>
</div>

== Motivation ==
Wikidata's growth in recent years has sparked concerns about the likelihood of [[Wikidata:SPARQL query service/WDQS backend update/Blazegraph failure playbook|the collapse of its Query Service]] (WDQS) and the increasing inability to edit many of its larger items. Much of it stems from a considerably large amount of data within it being stored unnecessarily, both when this information is not actively used elsewhere on Wikidata and when the information represented by this data can be readily and reliably computed in other ways.

This WikiProject seeks to keep the amount of information on Wikidata constant while reducing the overall size of its data, both in terms of the lengths of item pages on wikidata.org and the number of RDF triples in WDQS. It will distinguish between several types of action that may be taken, including 1) what can in principle be done right now without affecting existing workflows, 2) what is also possible now but may require acceptable changes to queries for accommodation, and 3) what is not currently feasible since it necessitates software changes and possibly entirely new storage units. It is expected that some of the proposed actions may be controversial, but we hope to foster discussion about these taking into account Wikidata's site health, community health, and usability.

It is hoped that, depending on the types of action described, participants will be inspired to either take these actions directly or encourage those who develop Wikibase, its Lua interface, and WDQS to make appropriate changes and improvements so that those actions can later be taken.

== Data size aspects ==
There are two ways to measure Wikidata's size:
* the number of RDF triples (of relevance to WDQS); and
* the size of the Wikidata dump (whether in JSON or TTL; of relevance to external users).

The main difference between these is that adding a reference, quantity, time, or coordinate that exactly duplicates another elsewhere in Wikidata adds relatively more to the dump size than to the RDF triple count.

For statistics regarding the number of RDF triples in WDQS, cf. [https://wikitech.wikimedia.org/wiki/User:AKhatun/WDQS_Triples_Analysis User:AKhatun/WDQS Triples Analysis] (2021) and [[User:Mahir256/Triples]] (2022).

== Actions that can be taken ==
=== Editable data on Wikidata ===
==== Ongoing and uncontroversial ====
* Merge duplicate items - see [[Help:Merge]] and [[Wikidata:WikiProject Duplicates]]
* Removing references to other Wikimedia projects (i.e. containing {{P|143}} and/or {{P|4656}}) when third-party references are also present for the same statement
** Example: [[Special:Diff/2000166652|George Lopez's date of birth]]
** (Most references to Wikipedia should be substituted wherever they occur anyways!)<ref>Statements that are only supported by {{P|143}} are not considered sourced statements. If you encounter one of these statements, please replace "imported from Wikimedia project" with an actual reference. (from [[Help:Sources#Different_types_of_sources]])</ref>
* Removing {{P|973}} statements on (item) when the external ID corresponding to their value is also present
** Example: [[Special:Diff/2220644865|this P973 statement, which replicated the value of P9951]]
** QLever general query: https://qlever.cs.uni-freiburg.de/wikidata/b2y2UW
* Removing date values without references (or referenced only through {{P|143}}) if a more precise value, referenced without using {{P|143}}, is also present - apply only if the more precise value is compatible with the less precise value to be removed
** Example: [[Special:Diff/499417349|Alix Faucigny-Lucinge's date of birth]]
** cf. removals performed by [[User:MatSuBot]] on the basis of [[Wikidata:Requests for permissions/Bot/MatSuBot 7]] (2017; the [https://w.wiki/AuKr WDQS query] once used to find removable values times out as of 2024)
* Removing more generic {{P|106}} values if they don't have references or are referenced only through {{P|143}} (e.g. remove {{Q|Q13418253}} if {{Q|16267607}} is present)
** Example: [[Special:Diff/2226505675|Dorothy Cadzow's occupation]]
** If the more generic {{P|106}} is has references, how to manage it (keep with normal rank and set the more precise ones with preferred rank; keep with deprecated rank; just remove) has been discussed in [[Wikidata:Project chat/Archive/2023/10#Concept of bot edits]] (2023) with no clear conclusion
* Removing duplicate external IDs if the same value is present twice, the good one with qualifiers (typically {{P|1810}}) and the bad one without qualifiers
** Example: [[Special:DIff/2226511568|Rolf Mühlethaler's NKC without qualifiers]]
** WDQS query for {{P|691}}: https://w.wiki/AuLf

==== Will likely require consensus ====
* Removing {{P|1343}} statements on (item) if (item) also has the corrisponding external ID
** Discussions: [[Property talk:P3217#Described in source - is it necessary?]] (2020); [[Wikidata:Project chat/Archive/2020/09#Possible duplicated information]] (2020)
* Removing {{P|1343}} statements on (item) when their value has {{P|921}} (item)
** Example: [https://www.wikidata.org/wiki/Q334953#Q334953$16260392-F38B-40D3-8500-334DF6653FB6 the 4th Earl of Sandwich] and articles about him in DNB00 and EB1911
** Alternative: use such values as references on the item to substantiate that the 'source' has a 'description' of the item
* On references, removing {{P|248}} (item) where an identifier with {{P|9073}} (item) also present
** Example: [https://www.wikidata.org/wiki/Q5480#Q5480$2c9ca6ee-4638-6b84-362c-266eaae096a1 Bart Simpson's given names]
** ''This does not mean removing {{P|248}} claims where no such identifier property exists on the reference!''
* Adding multiple references only if they are based on independent sources, so removing references of lower quality/relevance if better references are already present 
** Discussion: [[Help talk:Sources#Add recommendation to avoid redundant references]]
* Removing certain inverse properties wherever they are used (e.g. 'student' or 'doctoral student')
** Example: [https://www.wikidata.org/wiki/Q92288#P185 Willi Jäger's doctoral students]
** ''This is meant to apply to properties where a single item more frequently has a lot of values for it!''
* Removing most occurrences of certain properties that may be inferred from a hierarchy (e.g. {{P|421}}, {{P|6379}})
** Example: {{Q|5599}}
** ''How long does it take for that item to load for you?''
* Deleting properties which use an exact subset of the IDs of another property
** Example: {{P|11686}} (subset of {{P|P9984}}), {{P|7902}} and others (subsets of {{P|227}})
* Removing IDs with ''novalue'', at least if not referenced
** Example: [https://www.wikidata.org/w/index.php?diff=2310579810&oldid=2171866404 Goldener Adler]
** Discussion (open): [[Wikidata talk:Identifiers#Novalue for missing IDs]]

==== Deprecated statements ====
* if a deprecated statement has always been wrong and is kept only because of the risk of it being reimported if removed, we should try to have it corrected in its source and then remove it safely → cf. '''[[Wikidata:Data round-tripping]]''' (about obtaining fixes in external websites when we find mistakes in them)
* if an external ID has been deprecated because it has been redirected or deleted, are we sure we should keep it (and how we should keep it)? - cf. '''[[Wikidata:Requests for comment/Handling of stored IDs after they've been deleted or redirected in the external database]]''' (2020-2025; consensus to keep it but was not approved a standard operative procedure)

==== Possibility of moving data to external databases ====
* [[m:WikiCite/Shared Citations|Move scientific articles to a separate database]] (as suggested by [[User:Wittylama]]). This would have many drawbacks and cost a lot of engineering effort because linking between Wikibases is not yet a trivial process. The graph split, to be finished during 2024 by WMF, is a middle ground where the items stay in Wikidata but are split in WDQS and accessed via federation by queries that need that data.
* Move astronomical data to a separate database (a question for [[User:Ghuron]], possibly). This would have many drawbacks and cost a lot of engineering effort because linking between Wikibases is not yet a trivial process. Another graph split is another option and a middle ground where the items stay in Wikidata but are split in WDQS and accessed via federation by queries that need that data.

=== Technical fixes needed ===
==== For Wikibase ====
* implement multilingual (mul) labels and aliases in order to reduce redundant labels and descriptions → [[phab:T285156|'''T285156: Add termbox language code mul to reduce redundancy in Wikidata labels and aliases''']]
* make it impossible to set the same string as label and alias for the same language → [[phab:T157774|'''T157774: Make it impossible to set the same content in the same language for label and alias''']]<ref name=W3>Mentioned in [[:meta:Community Wishlist Survey 2023/Wikidata/Prevent data duplication]].</ref>
* improve technically the management of inverse statements, e.g. storing the statement only in one item and showing its inverse by default in the other (e.g. not [[Property:P150|'wdt:P150']], but [[Property:P131|'!wdt:P131']], so this would allow deleting P150; see also [[MediaWiki:Gadget-relateditems.js]]) → [[phab:T209559|'''T209559: Inverse statements duplicate work, data, and may be out of sync''']]
* avoid dates with precision lower than month (10) being stored with two different encodes, which often causes duplicate statements → [[phab:T310981|'''T310981: Duplication of dates due to different encode''']]<ref name=BD>Mentioned in [[:meta:Community Wishlist/Wishes/Fix main bugs in Wikidata and WDQS handling of dates]].</ref><ref name=W3/>
** cf. deduplications performed ''a posteriori'' (only for {{P|569}} and {{P|570}}) by [[User:MatSuBot]] on the basis of [[Topic:Wxspna7q8jnn17u8]] (2021-)
* make it impossible to add exactly duplicate claims (but it could happen that two claims that aren't intended to be identical may be identical for a few seconds while they are being edited) → [[phab:T222274|'''T222274: Fix and prevent duplicate claims on Wikidata entities''']]

;References
* store reference data separately in the JSON rather than each time a reference is used (e.g. [https://www.wikidata.org/wiki/Special:EntityData/Q62513493.json the JSON representation] of [[Q62513493]]) → [[phab:T360224|'''T360224: Improve Wikidata handling of duplicate references in model and UI''']]<ref name=DR>Mentioned in [[:meta:Community Wishlist/Wishes/Improve Wikidata handling of duplicate references in model and UI]].</ref>
* make impossible to store exactly duplicate references → [[phab:T224333|'''T224333: It's possible to save a statement with duplicate references''']]<ref name=DR/><ref name=W3/>
* avoid storing references that differ only for the value {{P|813}} → [[phab:T270375|'''T270375: Saving identical references with different retrieval dates should be more difficult''']]<ref name=W3/>
* in general, avoid storing references that share most of their content, e.g. both having the same external ID, or one having the external ID X and the other having the corresponding {{P|854}} → no Phab yet
** cf. substitutions performed ''a posteriori'' by [[User:DifoolBot]] on the basis of [[Wikidata:Requests for permissions/Bot/DifoolBot 5]] ([https://github.com/DifoolBot/viaf/blob/main/splitrefs.py code on Github]) (2024-)
* avoid that merges create redundant statements, i.e. statement X with no qualifiers and references and statement X with 1+ qualifiers should be merged but aren't merge by [[Special:MergeItems]] as of now → [[phab:T334197|'''T334197: Improve Special:MergeItems for qualified and not-qualified statements''']]

==== For Wikidata ====
* automated descriptions on the basis of {{P|31}}, e.g. for {{Q|Q4167410}} and {{Q|Q4167836}}, or other properties → [[phab:T303677|'''T303677: Automatically generate descriptions for items based on their P31 (instance of) values''']]
* avoid items being created with the same sitelinks → [[phab:T44325|'''T44325: Prevent creation of items having the same sitelinks (duplicates)''']]<ref name=W3/>

==== For Commons ====
* move tabular data to the Data namespace of Wikimedia Commons → [[phab:T181319|'''T181319: Support external tabular datasets in WDQS''']]

== Participants ==

{{Participants}}

==References==
<references/>

==Related links==
;Other WikiProjects
* [[Wikidata:WikiProject Duplicates|WikiProject Duplicates]] (duplicate items are an obvious redundance that should be reduced inasmuch as possible, mostly – but not only – through manual merges)
* [[Wikidata:WikiProject Limits of Wikidata]]

;Presentations
* [[User:Mahir256|Mahir256]], [https://pretalx.com/wikidatacon2023/talk/ATCCHH/ ''Making Wikidata Smaller Without Reducing Information''] (online at WikidataCon 2023) - [https://www.youtube.com/watch?v=epBgPeWiI_M&t=22437s recording]

;Essays
* [[User:ASarabadani (WMF)|ASarabadani (WMF)]], [[User:ASarabadani (WMF)/Growth of databases of Wikidata|''Growth of databases of Wikidata'']]

;Suggested gadgets
* [[User:Difool/WikidataCleanup]]

[[Category:WikiProject Redundancy| ]]