I'm using the English tagger. If I run the same code as mentioned in https://ucrel.github.io/pymusas/usage/how_to/tag_text, it is throwing up an error. The error happens in the following line of the code:

english_tagger_pipeline = spacy.load('en_dual_none_contextual')

The error is: `/usr/local/lib/python3.7/dist-packages/en_dual_none_contextual/init.py in load(**overrides) 44 45 def load(**overrides): ---> 46 return load_model_from_init_py(file, **overrides)

TypeError: load_model_from_init_py() got an unexpected keyword argument 'enable'`

It was working perfectly fine 2 days ago. This error seems to be recent.

I'm facing trouble tagging English text. I'm using spaCy's 'en_core_web_sm' dataset for the pipeline. Apparently, the English dataset does not have 'token.pos', something that the 'usas_tagger' requires. The documentation says that the tagger should work even without 'token.pos', however when I go ahead and feed the english text to the tagger, it simply tags 'Z99' to all the words.

Would really appreciate any valuable inputs.

Below is the full code. Thanks!

!pip install pymusas
!python -m spacy download en_core_web_sm

import spacy
from pymusas.spacy_api.taggers import rule_based
from pymusas.pos_mapper import UPOS_TO_USAS_CORE

nlp = spacy.load('en_core_web_sm', exclude=['parser', 'ner'])

usas_tagger = nlp.add_pipe('usas_tagger')

_ = nlp.analyze_pipes(pretty=True)

usas_tagger.pos_mapper = UPOS_TO_USAS_CORE

input_text = "It was raining in London and the cat was missing"

tagged_text = nlp(input_text)

print(f'Text\tLemma\tPOS\tUSAS Tags')
for token in tagged_text:
    print(f'{token.text}\t{token.lemma_}\t{token.pos_}\t{token._.usas_tags}')

Adds the following:

• A mapping from the basic CorCenCC POS tagset to USAS core POS tagset.
• The usage documentation, for the "How-to Tag Text", has been updated so that it includes a Welsh example which does not use spaCy, instead uses the CyTag toolkit.

Creating a new release

I think once this has been added to the main branch we should create a new release, e.g. 0.2.0 as the new release will contain the following two POS mappings:

1. A mapping from the basic CorCenCC POS tagset to USAS core POS tagset.
2. A mapping from the Penn Chinese Treebank POS tagset to USAS core POS tagset.

What do you think @perayson ?

The Chinese spaCy model outputs POS tags that come from the Chinese treebank tagset rather than the Universal POS tagset. This therefore requires a mapping from the Chinese treebank tagset to the USAS core tagset to be able to use the POS tagger within the Chinese spaCy model for the USASRuleBasedTagger if we would like to make the most of the POS information within the Chinese USAS lexicon.

A solution to this is to take the mapping from the Universal POS (UPOS) tagset for mapping between the Chinese treebank tagset to the UPOS tagset, of which the mapping can be found here and swap the UPOS tags in that mapping to USAS core tagsets using the mapping we have current for UPOS to USAS core.

We want the Spacy version to allow for multiple languages, following their language-specific factory setup. As the rule based tagger requires a lexicon resource, we need to load in data, to do this we are going to follow option 2: Save data with the pipeline and load it in once on initialization. Option 2 was chosen as it will allow us to ship the models without the user having to specify where the data has come from, we can state in the shipped models where the data has come from with the license for the data which will reflect the license of the model.

With Option 1 we would either have to create download functions that save data to the users file system or we would have to ship the data with the python package, this would then require us to have different licenses for the code and data even though they are in the same repository.

Once the license has been decided this needs to be added to the setup.cfg file, for more information on how to add it to the setup.cfg see the setup.cfg guide.

Hi,

For the USAS tags like 'A11.1+', 'N3.2+', 'Z1mf', 'Z3c' etc, is there a lookup table to get the corresponding English name of the tags?

Thanks! Fahad

Added

• Roadmap added.
• Define the MWE template and it's syntax, this is stated in Notes -> Multi Word Expression Syntax in the Usage section of the documentation. This is the first task of issue #24.
• PEP 561 (Distributing and Packaging Type Information) compatible by adding py.typed file.
• Added srsly as a pip requirement, we use srsly to serialise components to bytes, for example the pymusas.lexicon_collection.LexiconCollection.to_bytes function uses srsly to serialise the LexiconCollection to bytes.
• An abstract class, pymusas.base.Serialise, that requires sub-classes to create two methods to_bytes and from_bytes so that the class can be serialised.
• pymusas.lexicon_collection.LexiconCollection has three new methods to_bytes, from_bytes, and __eq__. This allows the collection to be serialised and to be compared to other collections.
• A Lexicon Collection class for Multi Word Expression (MWE), pymusas.lexicon_collection.MWELexiconCollection, which allows a user to easily create and / or load in from a TSV file a MWE lexicon, like the MWE lexicons from the Multilingual USAS repository. In addition it contains the functionality to match a MWE template to templates stored in the MWELexiconCollection class following the MWE special syntax rules, this is all done through the mwe_match method. It also supports Part Of Speech mapping so that you can map from the lexicon's POS tagset to the tagset of your choice, in both a one-to-one and one-to-many mapping. Like the pymusas.lexicon_collection.LexiconCollection it contains to_bytes, from_bytes, and __eq__ methods for serialisation and comparisons.
• The rule based taggers have now been componentised so that they are based off a List of Rules and a Ranker whereby each Rule defines how a token(s) in a text can be matched to a semantic category. Given the matches from the Rules the for each token, a token can have zero or more matches, the Ranker ranks each match and finds the global best match for each token in the text. The taggers now support direct match and wildcard Multi Word Expressions. Due to this:
  • pymusas.taggers.rule_based.USASRuleBasedTagger has been changed and re-named to pymusas.taggers.rule_based.RuleBasedTagger and now only has a __call__ method.
  • pymusas.spacy_api.taggers.rule_based.USASRuleBasedTagger has been changed and re-named to pymusas.spacy_api.taggers.rule_based.RuleBasedTagger.
• A Rule system, of which all rules can be found in pymusas.taggers.rules:
  • pymusas.taggers.rules.rule.Rule an abstract class that describes how other sub-classes define the __call__ method and it's signature. This abstract class is sub-classed from pymusas.base.Serialise.
  • pymusas.taggers.rules.single_word.SingleWordRule a concrete sub-class of Rule for finding Single word lexicon entry matches.
  • pymusas.taggers.rules.mwe.MWERule a concrete sub-class of Rule for finding Multi Word Expression entry matches.
• A Ranking system, of which all of the components that are linked to ranking can be found in pymusas.rankers:
  • pymusas.rankers.ranking_meta_data.RankingMetaData describes a lexicon entry match, that are typically generated from pymusas.taggers.rules.rule.Rule classes being called. These matches indicate that some part of a text, one or more tokens, matches a lexicon entry whether that is a Multi Word Expression or single word lexicon.
  • pymusas.rankers.lexicon_entry.LexiconEntryRanker an abstract class that describes how other sub-classes should rank each token in the text and the expected output through the class's __call__ method. This abstract class is sub-classed from pymusas.base.Serialise.
  • pymusas.rankers.lexicon_entry.ContextualRuleBasedRanker a concrete sub-class of LexiconEntryRanker based off the ranking rules from Piao et al. 2003.
  • pymusas.rankers.lexical_match.LexicalMatch describes the lexical match within a pymusas.rankers.ranking_meta_data.RankingMetaData object.
• pymusas.utils.unique_pos_tags_in_lexicon_entry a function that given a lexicon entry, either Multi Word Expression or Single word, returns a Set[str] of unique POS tags in the lexicon entry.
• pymusas.utils.token_pos_tags_in_lexicon_entry a function that given a lexicon entry, either Multi Word Expression or Single word, yields a Tuple[str, str] of word and POS tag from the lexicon entry.
• A mapping from USAS core to Universal Part Of Speech (UPOS) tagset.
• A mapping from USAS core to basic CorCenCC POS tagset.
• A mapping from USAS core to Penn Chinese Treebank POS tagset tagset.
• pymusas.lexicon_collection.LexiconMetaData, object that contains all of the meta data about a single or Multi Word Expression lexicon entry.
• pymusas.lexicon_collection.LexiconType which describes the different types of single and Multi Word Expression (MWE) lexicon entires and templates that PyMUSAS uses or will use in the case of curly braces.
• The usage documentation, for the "How-to Tag Text", has been updated so that it includes an Indonesian example which does not use spaCy instead uses the Indonesian TreeTagger.
• spaCy registered functions for reading in a LexiconCollection or MWELexiconCollection from a TSV. These can be found in pymusas.spacy_api.lexicon_collection.
• spaCy registered functions for creating SingleWordRule and MWERule. These can be found in pymusas.spacy_api.taggers.rules.
• spaCy registered function for creating ContextualRuleBasedRanker. This can be found in pymusas.spacy_api.rankers.
• spaCy registered function for creating a List of Rules, this can be found here: pymusas.spacy_api.taggers.rules.rule_list.
• LexiconCollection and MWELexiconCollection open the TSV file downloaded through from_tsv method by default using utf-8 encoding.
• pymusas_rule_based_tagger is now a spacy registered factory by using an entry point.
• MWELexiconCollection warns users that it does not support curly braces MWE template expressions.
• All of the POS mappings can now be called through a spaCy registered function, all of these functions can be found in the pymusas.spacy_api.pos_mapper module.
• Updated the Introduction and How-to Tag Text usage documentation with the new updates that PyMUSAS now supports, e.g. MWE's. Also the How-to Tag Text is updated so that it uses the pre-configured spaCy components that have been created for each language, this spaCy components can be found and downloaded from the pymusas-models repository.

Removed

• pymusas.taggers.rule_based.USASRuleBasedTagger this is now replaced with pymusas.taggers.rule_based.RuleBasedTagger.
• pymusas.spacy_api.taggers.rule_based.USASRuleBasedTagger this is now replaced with pymusas.spacy_api.taggers.rule_based.RuleBasedTagger.
• Using PyMUSAS usage documentation page as it requires updating.

Adds the following:

• The usage documentation, for the "How-to Tag Text", has been updated so that it includes an Indonesian example which does not use spaCy instead uses the Indonesian TreeTagger.

Fixes #19

1. Adds a mapping from the Penn Chinese Treebank POS tagset to USAS core POS tagset.
2. In the documentation it clarifies that we used the Universal Dependencies Treebank version of the UPOS tagset rather than the original version from the paper by Petrov et al. 2012.

Some of the GitHub actions we have push content to the repository, of which these commits are not signed, it would be good if these could be signed.

Actions that require signatures:

1. ci.yml in the Update environment file step.
2. documentation.yml in the Release to GitHub Pages step.

I am Jaspreet and am using the code from URCEL under your repository for a project of mine. I was trying to implement the code for English language and the following error for 'en_dual_none_contextual" is coming repeatedly. I am not sure why? Could you please help out on this.

Kindly help out on this AEAP

Regards

Jaspreet Singh

pydoc-markdown has made a change since version 4.6.0 to use Novella as the render, this means the current way we create our Python API documentation needs to change if we want to use pydoc-markdown >= 4.6.0. The code that creates this Python API documentation is the ./scripts/py2md.py script.

This is not an urgent change as version 4.5.1 of pydoc-markdown works with Python versions >=3.7,<=3.10.

To support wildcard (*) syntax for single word lexicon files. This would also be useful for rules like all punctuation tokens, which should be labelled as the semantic category PUNCT, for punctuation.

The wildcard symbol in this syntax would mean that zero or more characters may appear after the word token and/or Part Of Speech (POS) tag. This syntax will therefore hold the same meaning between single word and Multi Word Expression files.

Example

Assuming the single word lexicon file:

lemma    pos    semantic_tags
*kg   num     N3.5
*    punc    PUNCT

In the first case it would allow tagging anything that ended with kg, e.g. 15kg to be tagged as a measurement, the N3.5 semantic tag. In the second case it would label all punctuation with the punctuation semantic tag, PUNCT.

To incorporate auxiliary verb rules into the USAS Rule Based Tagger.

Definition of auxiliary verb rules

All POS tags used here are from the CLAWS C7 tagset.

In English (at least in the C version of the semantic tagger) we use auxiliary verb rules for POS tags VB* (be), VD* (do), VH* (have), to determine the main and auxiliary verbs and therefore alter the semantic tag.

An auxiliary verb would normally be given the USAS semantic tag Z5 grammatical bin, whereas the main verb would be given a non Z5 tag. For example in the sentence (format is token_USAS semantic tag) below the auxiliary verb is have and the main verb is finished:

I_Z8mf have_Z5 finished_T2- my_Z8 lunch_F1 ._PUNC 

We have approximately 35 rules in place for amending the semantic tags on be, do, and have after the initial set of potential semantic tags are applied. An example rule for have is as follows:

VH*[Z5] (RR*n) (RT*n) (XX) (RR*n) (RT*n) V*N

If the sequence of POS tags matches a given context, VH* (POS tag for have) followed by V*N (POS tag for the word finished) with optional intervening adverbs (R* POS tags) or negation (XX POS tag), then the rule instructs the tagger to change the semantic tag on the auxiliary verb have to be Z5.

For semantic taggers in other languages (the Java versions), we do not have auxiliary/main verb rules in place.

How this rule maps to spaCy pipeline through UPOS tagset

In the UPOS tagset and therefore spaCy POS models we can use the AUX POS tag from the UPOS tagset, instead of VB* (be), VD* (do), VH* (have). Below is the code and output of running the small English spaCy model on the sentence I have finished my lunch.:

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('I have finished my lunch.')
print('Token\tPOS')
for token in doc:
    print(f'{token.text}\t{token.pos_}')

Output:

Token	POS
I	PRON
have	AUX
finished	VERB
my	PRON
lunch	NOUN
.	PUNCT

To incorporate Df tags from MWE templates to enhance the USAS Rule Based Tagger.

Definition of Df tags

A small number (currently 93) of English MWE templates have the tag Df, which stands for default. The Df tags refers to the first token starting with a wildcard (*) single word's semantic tag from the single word semantic lexicon, and the Df tag is replaced in the tagger output. Note that, in the C version of the semantic tagger, only the first part of any slash tag is copied across, and any gender markers (lower case letters) on the single word semantic tag are also removed during the replacement step.

Example 1

In the MWE template below, the semantic tag Df would be replaced with the semantic tags of the adjective (JJ) token by looking up that token's semantic tags in the single word semantic lexicon:

mwe_template    semantic_tags
*_JJ style_NN1    Df

To make this more concrete, given the text:

The acting style.

And the following single word lexicon

lemma    pos    semantic_tags
acting    JJ    A1/Z3

As well as the above MWE template, then the tokens acting style will be tagged as an MWE with the A1 semantic tag.

Example 2

Some MWE templates can include membership to more than one semantic category using the slash (/) notation, here is an example of how the Df tag is processed in these cases. Given the MWE template:

mwe_template    semantic_tags
*_JJ style_NN1    C1/Df

given the text:

The acting style.

And the following single word lexicon

lemma    pos    semantic_tags
acting    JJ    A1/Z3

Then the tokens acting style will be tagged as an MWE with the C1/A1 semantic tags.

Problems with the definition

As stated in the definition that:

The `Df` tags refers to the first token starting with a wildcard (`*`)

This means that an MWE with a Df tag must contain a word token element starting with a wildcard. If no such token exists in the template, then a warning should be issued.

To incorporate Multi Word Expressions (MWE) into the rule based tagger. The MWEs will come from MWE lexicons, of which examples of these can be found in the Multilingual USAS GitHub repository, e.g. the Spanish MWE lexicon.

Tasks

• [x] Define the MWE template. Update definition of MWE syntax can be found within the documentation notes.
• [x] Incorporate MWE templates that do not have a special syntax, e.g. all templates that do not use the wildcard (*) or curly brace syntax ({}).
• [x] Incorporate MWE templates that use the wildcard (*) special syntax.
• [ ] Incorporate MWE templates that use the curly brace ({}) special syntax, which cover discontinuous MWE's.