Downloading wiki corpus and aligning with multilingual wikis

wikidumps

wikipedia-parallel-titles

Use wikiextractor extract wikidumps corpus

Generating parallel sentence pairs based on Wikipedia corpus

Building a parallel corpus based on Wikipedia

wikidumps Corpus Download

wikidumps The website is en-wikidumps

wikidumps Processing of corpus

Handle wikidumps Corpus can be used wikiextractor Tools to extract . Implementation requires installation

pip install wikiextractor

Then there are two ways to use , One is to transfer the python Modules are used as scripts to run

python -m wikiextractor.WikiExtractor enwiki-latest-pages-articles.xml.bz2

Or enter the installed wikiextractor Directory operation WikiExtractor.py To deal with it wikidumps Expect

python WikiExtractor.py enwiki-latest-pages-articles.xml.bz2

Some common parameters

1.-b   File capacity   , for example ：-b 100M  When the output file reaches 100M when , Automatically add new files , Multiple files can be generated 
2.-o  The name of the output file , Can be preceded by a path , for example ：-o AA_yue  or -o /extract/AA_yue, The default output folder is text

The file format after processing is

<doc id="244" url="https://zh.wikipedia.org/wiki?curid=244" title=" Historians ">
 Historians 

 Historians are also called historians 、 Historian 、 Historians , It refers to writing historical works as a profession or the establishment of history 、 Intellectuals who have made great efforts to develop and apply . Historians include compilers of historical records and researchers of historical materials . People studying history must rely on the records left by their predecessors . Historians will study the events of the past and the authenticity of their records , And record their research . A historian can study someone's experience , A city 、 The development of a place or country . According to their different subjects , History can be divided into different categories , for example ：
 Personal history 
 Personal history , It's a study of what happened to someone in the past .
 Local history 
 Local history , It is a study of events that have occurred in a city or place .
...

</doc>
<doc id="256" url="https://zh.wikipedia.org/wiki?curid=256" title=" Open source ">
...
</doc>

wikidumps The corpus is based on title alignment

Online mode

Specific... Can be used online api, Such as wikipedia perhaps wikipediaapi, concrete wikipedia Use guide and Wikipedia API Use guide .

Offline mode

Offline mode requires us to download the aligned corpus first , Then use tools to deal with , Then write your own program to get the alignment information .

First of all, from the wikidumps Download the corpus required for alignment , Its naming format is

*-page.sql.gz

*-langlinks.sql.gz

among * Is a prefix , Generally, it contains language abbreviations and time information . And then use wikipedia-parallel-titles Tool to generate Title alignment information , The tool library has a build-corpus.sh Script , Run this script to get the title alignment file , The run command is ./build-corpus.sh en zhwiki-latest > titles.txt, This command is implemented according to zhwiki-latest Get and en Title alignment file titles.txt.