The conversation

StableAlkyneHNSource author
Lessons learnedExperience excerpt · English

It's not clear which files you need, and the site itself is (or at least, was when I tried) …

… It's not clear which files you need, and the site itself is (or at least, was when I tried) "shipped" as some gigantic SQL scripts to rebuild the database with enough lines that the SQL servers I tried gave up reading them, requiring another script to split it up into chunks. Then when you finally do have the database, you don't have a local copy of Wikipedia. You're missing several more files, for example category information is in a separate dump. Also you need wiki software to use the dump and host the site. After a weekend of fucking around with SQL, this is the point where I gave up and just curled the 200 or so pages I was interested in. I'm pretty sure they want you to "just" download the database dump and go to town, but it's such a pain in the ass that I can see why someone else would just crawl it.

Original discussionCrawlers impact the operations of the Wikimedia projects66 source comments · at import
5

What was this responding to?

Parent context · original excerpt

IA
iansoHN

The dumbest part of this is that all Wikimedia projects already export a dump for bulk downloading: https://dumps.wikimedia.org/ So it's not like you need to crawl the sites to get content for training your models...

Full original

Community replies 0

No community replies yet. Add your perspective.

Replies from the source 5

Direct replies available in this import. Original-language excerpts.

JS
jsheardHN

> I tried using one of these dumps a year ago (wanted to play around and see what visualizations I could come up with based on text and the links between pages) and it was an incredibly unintuitive process. More recently they starting putting the data up on Kaggle in a format which is supposed to be easier to ingest. https://enterprise.wikimedia.com/blog/kaggle-dataset/

Full original
MJ
mjevansHN

What they need to do is have 'major edits' push out an updated static render physical file like old school processes would. Then either host those somewhere as is, or also in a compressed format. (E.G. compressed weekly snapshot retained for a year?) Also make a cname from bots.wikipedia.org to that site.

Full original
GU
GuinansEyebrowsHN

I think there are engineers working for crawler companies who are paid well enough to figure out how to do this without kneecapping the most well-known noncommercial projects still surviving on the capitalized internet.

Full original
NE
neetsHN

Have you tried any of the ZIM file exports? https://dumps.wikimedia.org/kiwix/zim/wikipedia/

Full original
PH
PhilpaxHN

Yeah, it's a bit confusing at first to navigate. Luckily, they offer XML dumps that aren't too bad to work with: 1. Go to https://dumps.wikimedia.org/enwiki/latest/ (or a date of your choice in /enwiki) 2. Download https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page... and https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page... . The first file is a bz2-multistream-compressed dump of a XML containing all of English Wikipedia's text, while the second file is an index to make it easier to find specific articles. 3. You can either: a. unpack the first file b. use the second file to locate specific articles within the first file; it maps page title -> file offset for the relevant bz2 stream c. use a streaming decoder to process the entire Wiki without ever decompressing it wholly 4. Once you have the XML, getting at the actual text isn't too difficult; you should use a stre…

Full original
Continue on Hacker News

Respect different perspectives. Share concrete experiences.

Sign in to post, reply, repost, and quote.