Wikidata:Requests for comment/Anna's Archive

An editor has requested the community to provide input on "Anna's Archive" via the Requests for comment (RFC) process. This is the discussion page regarding the issue.

If you have an opinion regarding this issue, feel free to comment below. Thank you!

Background

(This issue was previously raised on project chat, but didn't receive much attention.)

There exists a well-known pirate library called Anna's Archive which is both (probably) the largest library and also hosts probably the largest ISBN metadata collection (more than 1 billion records), based on WorldCat records (though, obtained illegally), OpenLibrary records, and records from some smaller collections. There are info about all of sources and structure of the data (it includes ISBN, author, title, publisher, year and everything like that) available on their site: background, WorldCat dataset, OpenLibrary dataset, other datasets, their parsed metadata.

As you can see, it is huge — a few terabytes of data — and probably uploading all of it would considerably benefit Wikidata (similarly as data about research papers had), which is what I'm suggesting in this RFC.

Unfortunately I see several challenges we need to address to do that. Well very well (talk) 03:55, 2 February 2025 (UTC)

Challenges

How legal this is?

The most of the dataset (the WorldCat dataset) was obtained by AA illegally, though not exactly obvious in what exact way ― first OCLC claimed they were hacked, later they claimed "although its internal systems were not breached, it believes the site's actions legally constitute hacking" ― which could just be site scraping. According to AA themself, it was mostly just scraping plus exploiting some "security flaws" (this can mean anything from just broken rate limits to using some XSS to download databases — probably not, if they didn't get everything in that way). In my opinion, this is not so much more illegal than just scraping websites for data. I'm interested in what Wikidata community thinks about it. Well very well (talk) 03:55, 2 February 2025 (UTC)

Copyright concerns

I'm not sure about copyright concerns — can such data be copyrighted (by original author WorldCat) at all? Are we interested in copyright terms when scraping sites for data later used in Wikidata? Or is it of something like "One record could not be copyrighted because it's trivial, but the whole database could"?

Thinking more about that, is scraping of a site for data and then uploading that data to Wikidata (a common workflow, AFAIK) legal (in terms of copyright) by itself? And how much different this case it? Well very well (talk) 03:55, 2 February 2025 (UTC)

See m:Wikilegal/Database Rights. It's not super clear who would own database rights on AA metadata overall, but the oclc and isbndb datasets seem pretty clear-cut cases. Nemo 22:08, 17 February 2025 (UTC)

So does that mean that EEA databases are not OK to upload to Wikidata (unless they are explicitly released under a free license of course), while non-EEA databases are OK unless their selection rules have some originality? As I understand, in the current case OCLC and ISBNdb are both American organizations and their selection doesn't have originality (ISBNdb tries to collect all ISBN records, WorldCat tries to collect all bibliographical records from OCLC-participating libraries). Well very well (talk) 02:24, 18 February 2025 (UTC)

I would caution that even if the criteria for mere inclusion in the database impart no originality, the database also includes annotations, and it could be argued that WorldCat's annotations have some amount of creativity. That said, in OCLC's complaint linked from the Torrentfreak article, they describe the annotations as being things like "metadata that denotes associations between records, such as between an original work and a parodying work", which are basically pure facts, as opposed to more creative things like textual descriptions.

I also note that OCLC's complaint alleges a variety of counts (starting page 21) involving contract and tort law and computer hacking, but does not allege copyright infringement as a cause of action. That strongly suggests that OCLC knows they don't have any copyright interest in the database under US law. 108.255.194.88 02:55, 1 April 2025 (UTC)

Those annotations don't need to be imported. Thanks for these insights. If further data was to be imported I think it wouldn't be that difficult if a subset of the data fields are imported – one should also think about matching/identifiers when importing in part for that reason and because that allows importing items not yet imported from newer datasets. Prototyperspective (talk) 12:17, 1 April 2025 (UTC)

My understanding is that Wikidata is currently a Digital Public Good. According to the enwiki article, this source is a bit of a no-no. Do we need to be messing with it just for the frisson or are there other less controversial sources that might be prioritized? Maculosae tegmine lyncis (talk) 22:44, 1 January 2026 (UTC)

There isn't any other database like it and Wikidata is currently not really useful much in the real world outside of a few Wikipedia things and this won't change unless we get sizable fractions of data in some areas like books or studies etc. I don't see any ethical issues with that site, it's making knowledge, research and educational resources far more accessible and is appreciated across the world, especially in poorer countries. People don't need to download documents from there, the metadata is separate. Prototyperspective (talk) 00:45, 7 January 2026 (UTC)

their .org domain has just been suspended, perhaps because of their Spotify scrape. They are not a trusted organisation. And our existing book/edition situation is such a mess that we should be cautious of any big literary imports, let alone from here. Vicarage (talk) 06:57, 7 January 2026 (UTC)

That's irrelevant. That the current book/edition situation is problematic and there's scaling-related troubles with big data imports (if that's what you meant) is a very good point though. It may make sense to hold this off until that's sufficiently resolved. It doesn't make that much sense for users to import small batches of books metadata or even add them by hand then though. Prototyperspective (talk) 15:06, 7 January 2026 (UTC)

We should use all other legitimate sources of information before we even consider AA. And by then I hope we realise that their datasets are worse than ours. Vicarage (talk) 15:40, 7 January 2026 (UTC)

The dataset they have are the best humanity currently has on books so that's unlikely for the foreseeable future. They aren't really the source of information, they just compiled (aggregated, combined, etc) the metadata. Prototyperspective (talk) 17:56, 7 January 2026 (UTC)

How can we obtain the data?

We can either use their parsed dataset (which is openly available) or their full original datasets, which are not ― they state «If you are interested in mirroring this dataset for archival or LLM training purposes, please contact us» — the link for archival leads to here. I'm 99.99% sure that Wikidata is something which they could consider as commited to both preservation and access goals — so I'm pretty sure if we'd contact them they will agree to release the data to be stored in WD. Well very well (talk) 03:55, 2 February 2025 (UTC)

How we can upload and store the data?

I guess since Wikidata have splitted most of its data and research papers' data onto two different servers — probably a new server could be created to host bibliographical data as well? Well very well (talk) 03:55, 2 February 2025 (UTC)

Wikidata cannot handle 1 billion items with its current infrastructure. Perhaps a single item would have multiple ISBN's, but still this is a proposed dataset that is much larger than Wikidata is currently. On the other hand having a more complete selection of book items would indeed be very useful for Wikidata. Are there logical subsets that would make sense? Highly cited books for example? The m:WikiCite project ran into exactly this issue, though I don't think it ever quite settled how to handle it. ArthurPSmith (talk) 18:09, 18 February 2025 (UTC)

If a book item can have multiple ISBNs then it's not 1 billion records. I think all of it should be imported and only if that is done will things be useful and application be possible – such as inventaire.io (Q32193244). It's of no use it misses every second book you're reading and if other sites like Goodreads have so much more items. However, one could import it gradually in many steps where one could prioritize non-fiction books. Prototyperspective (talk) 01:01, 1 March 2025 (UTC)

The attitude of the book people on WD that both the work, and every edition of a work should have separate WD items will always cause problems. It might be good bibliographic practice, but will overwhelm our databases, and causes problems both for popular works with so many editions, and the long tail of single edition works. If importing datasets we should focus on single entries for works, and hold off entries for multiple editions. This means we can't record ISBNs or publishers well, for example, as they are edition based. Vicarage (talk) 09:49, 1 March 2025 (UTC)

Agree, I don't think anything speaks against considering creating additional items for each edition later on, e.g. once there's an item for most books and the database technologies have improved sufficiently. Until then, it would probably be wise – if we want Wikidata to succeed and be useful – to import books with one item per book.
Is there some page about the technical difficulties and proposals about how they could be addressed? (also @Well very well) There are lots of searchable databases that are larger so why would having a few hundred million items be an issue? ScienceOpen for example unlike WData has most or nearly all notable studies and 100 M items. I heard about the graph split but I don't yet understand why a larger database would be a problem. If it's about the large revision history and the server strain to create all these items maybe the import process could be modified for these. Prototyperspective (talk) 14:47, 1 March 2025 (UTC)

> Is there some page about the technical difficulties and proposals about how they could be addressed? (also @Well very well)
Probably this is something for Phabricator. Well very well (talk) 15:58, 1 March 2025 (UTC)

Yes, once it is known what the technical issues are or vice versa if some phab issue(s) about these already exist that somebody could link here. Prototyperspective (talk) 22:55, 1 March 2025 (UTC)

See Wikidata:WikiProject Limits of Wikidata, also Wikidata:Scaling Wikidata, and User:ASarabadani (WMF)/Growth of databases of Wikidata. The biggest problem is not so much the number of items, but the number of edits, and the editing rate. Most very large databases do not have the constant flow of changes that Wikidata has. ArthurPSmith (talk) 02:00, 2 March 2025 (UTC)

Thanks. I've seen two of those pages but didn't have the links anymore and haven't read them much yet. Maybe a solution could be just to create a new item with all the properties in one edit that writes all-at-once or even extending the database with an offline extension of data (on the server) and storing the revision history more separately. Most applications using data need a lot of data so scaling up seems like a bottleneck that needs to be addressed before WD becomes (more) used and useful. Maybe that needs more devs to work on this issue (e.g. a short video summarizing the current difficulties and solution ideas could help) and/or expanding the range of possible options or tech. Maybe not as much has to be in memory or quickly readable and I wonder how other databases solve this. Prototyperspective (talk) 02:25, 2 March 2025 (UTC)

The main problems are if I understood correctly:

• Policy related to API use, currently anyone can hit the endpoints even from commercial apps. OSM has a much lower read load because of clear policies about usage to keep the load down.
• Use of low memory relatively cheep commodity servers. WMDE has not published specs nor the annual budget for servers, but according to my investigations they have 40 replicas in s8 which is the main Wikidata database plus a master.

The combination of low memory, large database tables and very righ read load is bad. It is not sustainable long-term.

One solution is to reduce the read load by:

• Changing to a API and WDQS policy similar to OSM or require login to get 100 queries a day or the like. Disable or limit anonymous use.
• Change the server strategy to have a few fast, high memory machines like OSMF. Accept sponsored machines from partners like Google. Discontinue 90% of the replicas and tell people to setup their own infrastructure like OSM does.
• Publish full database backups so anyone can setup a Wikidata mirror easily.

So9q (talk) 18:09, 5 March 2025 (UTC)

> If a book item can have multiple ISBNs then it's not 1 billion records.
Yes, but it's a few hundred millions in any case. Well very well (talk) 14:30, 1 March 2025 (UTC)

If the data should be imported, it should be done right. Separate book editions with book itself. Don't create a huge conflation mess just to reduce number of items. The query services and SQL clusters are doubling, so we should have Q200000000 fine. Midleading (talk) 18:07, 22 March 2025 (UTC)

Wikidata only split the query service servers. The underlying UI and database is not split. However if you think a new server would be a good approach then this can be done as a completely separate Wikibase instance. That might be a good solution. See the Wikibase website for details. ArthurPSmith (talk) 18:17, 18 February 2025 (UTC)

Importing data into separate Wikibase rather than here means WMF projects can't use such data, so it's a bad idea. Midleading (talk) 11:23, 22 March 2025 (UTC)

Alternative approach: talking to OCLC

This dataset is great and it would be great to have it in WikiData. It would be even better to have current data in wikidata and a mutually-beneficial relationship with #OCLC as originator (apparently) of the data. I suggest that we reach out to OCLC with the intention of developing the kind of mutually-beneficial relationship that we have with the LoC. We have this dataset to fall back on if they say no or are unresponsive (and they would be aware of that) but I think a ISBN feedback loop like we have for LCCNs could be hugely useful. This would circumvent all the legal and copyright questions. OCLC will also be familiar with things like https://www.wikidata.org/wiki/Wikidata:ExLibris-Primo (ExLibris is a direct competitor with OCLC in some commercial spaces) and be wanting to not have things like that as killer features. I know that OCLC has a German office, maybe someone from Wikimedia Germany (Q8288) could reach out? Stuartyeates (talk) 09:24, 18 February 2025 (UTC)

That sounds like something we definitely should try — but are there really chances that they will give us their entire database (as @Hupaleju noted in the previous discussion, this is their most valuable data)?.. Either way, it definitely doesn't hurt us to try. Well very well (talk) 23:38, 19 February 2025 (UTC)

See my comment below. My wild guess is in fact that they wouldn't want to "just" give us their entire database since the overlap with their paid-for offerings targeting the library sector might then become quite significant, but it's not clear that we'd even want or care about that anyway. It may well be preferable to have richer bibliographic metadata (including perhaps e.g. broad cataloging, topics/subjects information etc.) about some limited subset of works than really basic metadata covering a vastly larger amount.

Consider that our highly unusual emphasis on data quality has led us to outright reject other very extensive sources of information in the past, e.g. the Freebase data was never imported in its entirety here. Hupaleju (talk) 00:33, 21 February 2025 (UTC)

IP concerns may be quite severe here

As I have mentioned when this proposal was first posted on Project Chat, this dataset could very likely be construed as misappropriated confidential/proprietary information, either by WorldCat themselves or any one of the member libraries or other organizations who have contributed to that dataset to begin with. It's worth noting that WorldCat has been pursuing legal action over this issue, see wikipedia:WorldCat#History. Please stay far away from misappropriated information if you're going to contribute to Wikidata as it is in fact regarded as a breach of IP rights and could put Wikidata itself at risk. See also Wikimedia's Terms of Use that appear to say as much.

OpenLibrary itself, by contrast, appears to be a non-problematic source for the kind of bibliographic data we would be interested in here at Wikidata.

(Of course, it may be possible to make use of some part of the data WorldCat has if we talk to OCLC first, as already mentioned above. It would most likely have to be a portion of the data that happens to be of more-or-less negligible commercial value from the point of view of the existing "market" involving WorldCat-related paid-for services, while nonetheless being unusually beneficial to us here at Wikidata. One especially obvious example might involve works that are old enough to be in the public domain, either in the U.S. -- this is comparatively easy to figure out based on the year when the work was first published -- or even perhaps worldwide, and could therefore be potentially hosted by us on Wikimedia Commons.) --Hupaleju (talk) 18:09, 18 February 2025 (UTC)

It's worth noting that most books that have aged out of copyright also lack ISBNs (unless they have been republished). Thinking about this for a couple of days, I'm leaning towards HathiTrust ID (P1844) being the key in this space. We have 14k items with a HathiTrust ID (P1844), about a thousandth of their collection of 18m books. Their policy seems sane. https://www.hathitrust.org/member-libraries/resources-for-librarians/metadata-in-the-digital-library/metadata-sharing-and-use-policy/ Stuartyeates (talk) 08:32, 19 February 2025 (UTC)

I'm quite aware that ISBN isn't really a thing wrt. older books, but I was mostly thinking about other kinds of bibliographic metadata here. I have not looked at the WorldCat-derived dataset so I have no idea what it might or might not have wrt. these older works. There are of course many unproblematic sources of bibliographic catalogue information that we could make use of, for example OpenLibrary ID which I pointed out above, is Open Library ID (P648). Hupaleju (talk) 19:19, 19 February 2025 (UTC)

I've checked, and there seems to be a bunch of old books (lacking ISBNs) there. Well very well (talk) 23:51, 19 February 2025 (UTC)

https://oai.hathitrust.org/?verb=ListRecords&metadataPrefix=marc21 is the Open Archives Initiative Protocol for Metadata Harvesting (Q2430433) endpoint for for HathiTrust MARC records (i.e. the indigenous format for the metadata on these items). There are 9,394,739 items (includes deleted items). These are works / editions. Every item I looked at had at least 3 identifiers, including an OCLC identifier. All of this is CC0. These link to ~18,000,000 scans of those (things scanned twice, things in multiple volumes, etc). Scans are not necessarily CC0. Stuartyeates (talk) 01:14, 20 February 2025 (UTC)

Looking at this data, there's quite a bit of data matching going to be needed. Does anyone know of a MARC to CSV converter that outputs something that can be put into OpenRefine? Stuartyeates (talk) 20:40, 22 February 2025 (UTC)

I have ~200,000 books downloaded and the bones of converted to a format the OpenRefine will understand. There's a lot of data here. Stuartyeates (talk) 08:28, 23 February 2025 (UTC)

I have ~200,000 books' metadata downloaded and a stylesheet to convert the parts we care about to csv for OpenRefine. I have an awk script to convert the csv back to MARCXML. Unfortunately I can't test them properly without also converting the bits we don't are about, so that will be my next step. Stuartyeates (talk) 19:32, 7 March 2025 (UTC)

The metadata dump is merely factual metadata...e.g. there is just one title and ISBN to a book edition, one can't copyright this metadata. There may be issues so some people should look into it, i.e. how to import data in a way that is fine. May be good to raise this issue at another venue where they can learn about this (e.g. I'll post this on the new subreddit /r/WData).
When I'm creating an item for a book I open a new tab, in that tab search for the book on goodreads or the publisher page, then copy and paste (ctrl+c & ctrl+v) information like the title and ISBN (I don't know them by heart of by just thinking hard about it). Many add data that way even if they don't disclose it. A machine is allowed to do the same. Moreover, people would prefer if the machine doesn't scrape sites but uses an API and a human could theoretically also use an API if they use some other client (e.g. let's say I have some goodreads app and instead check the books data in it). Furthermore, instead of doing it all anew, one could simply use the aggregated data somebody has already compiled. That is using the AA dataset. It would be good if some volunteers more knowledgable in these subjects would look into it but I think it would be fine. OpenLibrary does not contain nearly as much metadata nor is it sanitized as much. If there is an issue, it would be about how the data is read and imported and whether some fields like book description – if there is something like that – need to be excluded. Prototyperspective (talk) 19:17, 2 March 2025 (UTC)

Agreed. I think republishing any data from the AA database is too risky for Wikidata. That said, it could be interesting to compare our database to the AA database, and use it to measure our work and identify gaps in our data. But I feel like we'd want to have an opinion from WMF legal before doing even that – they might say that the mere act of downloading AA's data is illegal. –IagoQnsi (talk) 00:47, 8 October 2025 (UTC)

I don't see why it would be risky. Not doing it would be risky as Wikidata is fading in irrelevance as we don't have enough data for WD to be used and many other projects that do integrate the data. There is no available books database as large. I would suggest to not ask the WMF if you're acting as a volunteer and to not do AA things if you're a WMF employee. In regards to identifying gaps, it's not like there's almost all the data and some smaller or larger gaps here and there....instead it's like >90% of items missing and >95% of book items missing key data like the associated ISBNs. It's just metadata and only the unlicensable factual data like book title and page numbers would be used, not paragraphs of book description or the book cover. Prototyperspective (talk) 12:51, 8 October 2025 (UTC)

I'd agree that WD's book coverage is thin and muddled. But I think is more important for those with data skills to sort out the muddle here before embarking on a huge data import. And the idea that we'd encourage volunteers to do things that WMF employees could not legally do is very worrying. Vicarage (talk) 13:15, 8 October 2025 (UTC)

1. I didn't say it's muddled but in regards to that, it can also be cleaned/improved using AA data and I don't see why one should wait for volunteers to first waste time with more-manual cleanups when the same cleanups can be done using the data imports eventually (e.g. show data mismatches). 2. I personally encourage an import of Anna's Archive metadata but never said anything about actively encouraging anything as a group or community. 3. I don't see why that would be worrying if that was the case. 4. I didn't say they couldn't legally do that and I think it would be legal.

Basically, people who do care about Wikidata being successful and useful should just go ahead and import the metadata, maybe they don't even need to disclose the technical source of the metadata if they can prove that they are nearly entirely accurate. It shouldn't need any encouragement or that kind and if not WD will just fail due to bureaucracy and oldfashioned conservative ideas at times when Meta even downloaded all the books to train on, not the metadata. Prototyperspective (talk) 13:30, 8 October 2025 (UTC)

Legal judgements are going against AA I'm sure they will ignore them, but I don't think we should. Vicarage (talk) 12:21, 18 January 2026 (UTC)