I started the second big scrape and while scraping for phonetic data from Albanian, Wikipron threw an error, the last line of which I'll reproduce below:

File ".../wikipron/config.py", line 73, in _parse_combining_modifiers last_char = chars.pop() IndexError: pop from empty list

The final line in the Albanian phonetic tsv is herë h ɛː ɾ meaning the scrape likely failed on this entry which contains what looks like word initial aspiration.

I guess for words like the one that caused this error we would want to combine with next char ʰi d r ɔ ɟ ɛ n?

-Adds tsv files in wikipron/langauges/wikipron/tsv_files -Adds a readme in wikpron/languages/wikipron

The tsv file names are formatted as such: iso693-2(B)code_phonetic/phonemic (If the language only has an iso639-3 code then that code is used. If a language doesn't have any phonetic entries on Wiktionary, then it will not have a phonetic file. Same goes for phonemic.)

The readme tsv file link links to the file (phonetic or phonemic) with more entries. I tried to determine whether or not to apply case-folding for each language, but may have gotten it wrong for a few languages. If you see any instances in the readme where I incorrectly applied or failed to apply case-folding then let me know and I can rerun those languages if need be.

I will add Russian once I have it, but it may take quite a long time to get it. I can also add all languages with more than 100 but less than 1000 entries in the same pull request as Russian if you'd like those files as well. I can submit a pull request for the code that generated all these files after that.

Hello

I can use the terminal version of WikiPron to scrape a small language like Amharic [amh], and to scrape a big one like French. But when I try to run it on Armenian [hye or arm], the code just stops running after an hour and outputs nothing -- there's not even any errors thrown. I suspect the code is finding a readtimeout error and then skipping it.

I suspect there's a readtimeout error because in the past, I used other wiktionary extractors to Wiktextract and that took 9-12 hours to scrape the Armenian words (just 17k words). I suspect that the Armenian entries are just oddly dispersed across Wiktionary that it takes a while for some scrapers to find them. Granted Wiktextract was using a wiktionary dump and that's how it managed to eventually work. Can WikiPron work over a Wiktionary dump or does it need to actively use an internet connection?

Wonderful resource.

There are some errors in the phone list for the Armenian dialects. A lot of the errors got cleaned up since you last scraped them.

The way that the Wiktionary contributors for Armenian work is that they do the following:

1. They take the orthographic form like գրպան (transliteration is <grban>, pronunciation is [gərbɑn])
2. They manually the orthographic and rewrite it with Armenian letters in order to apply any phonological rules like schwa insertion: գրպան to գըրպան <gərban>
3. They then use this script to convert from the rewritten form (2) into the IPA for Eastern and Western Armenian
4. This means that the Western entries are almost all redundant and automatically derived from the Eastern entries. A lot of words actually include both IPA entries together.

For Eastern (EA)

• You're right that the <ա> grapheme is /ɑ/. The script automatically converts the rewritten grapheme to [ɑ]. Some people had manually written the pronunciation entries and used non-IPA symbols like [a]. But they've mostly gotten cleaned up. A lot got cleaned before August. Then I personally scraped Wiktionary with another python package, found some overlooked [a]'s, and I cleaned them up too.

• Same issue with <ո> actually being [ɔ] but sometimes incorrectly written as [o].

• Armenian doesn't have phonemic geminates. What you see is that the orthography has a sequence of identical consonants like փթթել <pttel> [pəttel]. The script automatically converts a sequence of identical segments into a geminate with the [ː] diacritic. So if the rewritten form is փըթթել <pəttel>, then the outputted pronunciation is [pət:el]. If the rewritten form is փըթել <pətel>, then the automatic output is [pətel]. IMO, it would make more sense if the automatic transcription was with doubled segments [pəttel] because Armenian lacks phonemic length. So all the consonants have a possible geminated/lengthened/doubled form. It might just be an accidental gap in the Wiktionary data that you don't have geminate t͡s: and others.

• The back fricatives are in free variation between velar and uvular, with more tendency towards uvular. The automatic transcription uses χ, so any [x] that you see is just an error (which I think also got cleaned up in around summer).

• What's the issue with missing tie-bars?

  • TODO: for future cleanup, missing tie-bars on segments like: <ց> [tʃʰ] <չ> [tsʰ]

For Western (WA)

• Any [r] that you see is an error. The orthography has two rhotic graphemes ր ռ that are both pronounced the same as a flap. The trill is just someone manually writing it. That counts as an error and I think most of them got cleaned up on the WIktionary entries.
• Yup you're right about the mergers.
  • TODO: supposedly the following mergers have occurred in this dialect: aspirated > aspirated voiced > aspirated voiceless unaspirated > voiced
• The W dialect also developing a devoicing rule for clusters of orthographic voice+voiceless consonants. The automatic transcription covers that though.

This pull request is for #141 Negative flags in cli.py are renamed to positive statements. In order to accommodate this change,Wikipron/config.py and tests/test_wikipron/test_config.py are also edited accordingly.

• [x] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data.

Strangely enough some (but not all) Moksha pages are not standard. The pronunciations don't reside under the regular list item elements ("li"), but under the paragraph elements ("p"). For an example, see the page for ала

Please let me know whether you'd prefer a custom extractor for this rather than changing the default template.

• [x] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data.

The Wikipedia transcription guidelines say that the high front phoneme is /i/ but many transcriptions have /ɪ/ instead. We should fix this upstream (e.g., on Wiktionary itself) and rescrape.

See issue for more context.

(Sorry for the wall of text...) These changes are meant to address issues #66, #67, #68 as well as a few suggestions made in the comments of pull request #61.

Here are the larger changes introduced by this pull request:

• Separated the scraping and writing part of scrape.py (formerly scrape_and_write.py).

  • write.py now generates the README table by inspecting the contents of the tsv/ directory. In the process it also creates a tsv readme_tsv.tsv with similar information as is in the README table.

• Added no_stress, no_syllable_boundaries, and cut_off_date options to languages in languages.json.

  • Modified codes.py to specify default values for these options when adding new languages to languages.json and to copy over previously set values for these options.
  • cut_off_date should now be set in codes.py prior to running codes.py, I’ve updated the README in languages/wikipron with those instructions.

• Added dialect config option (and require_dialect_label option) to English, Spanish and Portuguese.

  • Restructured scrape.py to handle when one or more dialects are specified for a language. (Ran this new code on Portuguese, because it is a smaller language, to generate some sample data.)
  • README table now includes dialect information in Wiktionary language name column

The only small changes worth noting are:

• Logging in scrape.py will now also output to scraping.log which I’ve added to .gitignore. This way finding the languages that failed to be scraped is a bit easier (don’t need to scroll through the console). It also outputs the language dict from languages.json in the error message for languages that failed to be scraped within our set amount of retries, so it is a bit easier to build a temporary languages.json with the failed languages.
• scrape.py will now remove files with less than 100 entries. (TSVs with less than 100 entries have been removed.)

I have a few questions regarding dialects that I'd like your thoughts on:

• How dialects are handled in languages.json.

  • I added dialects to languages.json in the following way:

    "por": {
        ...
        "require_dialect_label": true,
        "dialect": {
            "_bz": "Brazil",
            "_po": "Portugal"
        },
        ...
    },
    

  • The keys (_bz, _po) in dialect serve as a sort of extension when naming the dialect tsv files (por_bz_phonetic.tsv, for example) and help with easy access to the dialect strings ("Brazil") in write.py. If you'd like me to change any of the keys because you'd like different extensions for certain dialects let me know. They can be longer than two letters. I'll provide links to the English, Spanish, and Portuguese entries in languages.json as a separate comment so you can review the keys and dialect strings I'm using.
  • Is there a process for finding which dialects are frequently used within a given Wiktionary language category? (Aside from just checking entries and seeing whether dialect information is specified.)

• How dialects are handled in scrape.py.

  • As written scrape.py will first scrape a language entirely and then scrape for dialects if any are specified. This means it will scrape por (Portuguese) with no dialect, then por with "Brazil" as the dialect and then por with "Portugal" as the dialect. Is there any reason to scrape for por with no dialect when we are specifying a dialect? Do we want to keep the tsvs generated from previously scraping por (or eng/spa) with no dialect?
  • Within scrape.py, I moved a lot of what was in main() to a separate function in order to handle dialects. There may well be a better way of handling dialects than the way I've tried to do it and I'm open to suggestions on how to improve it.

• [x] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data. This script automatically generates phone_summary, which has similar structure as language_summary. This script is written based on generate_summary.py .

The output of generate_phone_summary.py is a TSV file instead of README.md, since README.md is already used. I will change the output path after we discuss what would be the good path for the output.

• [x] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data.

Closes #389

I renamed the files with this script: link

I only changed the filenames, so I'm sure a lot of stuff is broken at the moment…

adds the json file with all unimorph files and wikipron lg names. Additionally uses download routine to grab data and logs statements to the console

• [ ] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data.

For Kapampangan(pam) the format of all pronunciation entries looks as follows:

Hyphenation: ba‧tia‧uan
IPA(key): /bəˈtjawən/, [bəˈtjäː.wən]

I suspect we can't parse this when both transcriptions are under the same heading. May be a duplicate.

Removed the statement casefold:true from the languages.json list. I rescraped hye and apw to confirm that the languages were still scrapped, but now with the original case marking from Wiktionary.

• [x] Updated Unreleased in CHANGELOG.md to reflect the changes in code or data.

For the word կարկանդակ, wikipron finds the correct pronunciation of [kɑɾkɑndɑk] but it also finds the IPA transcriptions of other words in the Usage Notes section like [pɛrɑʃˈki]. I'm not sure if this is an unavoidable glitch from Wikipron's side, or if it's a glitch that could be fixed from the Wiktionary side.

It seems that what's going on is that WikiPron is just finding any IPA transcription that's inside the Armenian entry, even if it's not associated with a dialect. E.g., if you run wikipron arm --dialect='ladygaga' --no-skip-parens --narrow > randos.tsv you get a handful of IPA transcriptions that aren't associated with the pre-defined dialects. These are either a) IPA transcriptions in the Usage notes or etymology, or b) IPA transcriptions for non-standard dialects. This isn't a problem for using Wikipron on a specific language (because the person can just filter those out manually). But I wonder if this glitch causes any other funny business for the other languages.

Side note: I wonder if there's been enough situations where people had to fix Wiktionary entries in order to optimize Wikipron's scraper (like on the various closed issues). If so, perhaps a tips and tricks page would be helpful down the line?

The commandline lets the user choose to apply casefolding so that entries like English can be changed to either English or english. But for the scraped data on the repo, it seems you apply casefolding by default. Would it be more useful if the online data didn't do casefolding? That way,

• If the user wanted to get the original data (with the correct cases), then they can just use the scraped data online instead of running WIkipron on the terminal
• If the user wanted to get the casefolded data, then they can take the un-casefolded data from the repo and then apply casefolding on their on their own machine (a simple fast Excel function).

Right now, if the user wants to get the original cases, then they have to run the terminal option (which takes a while).

Do you think there's a reasonable way to make an enhancement that will extract audio file URLs for Wiktionary words? At least for Armenian, the audio files are linked in the Pronunciation section.