A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

Last update: Jan 02, 2023

Overview

A Deep Learning NLP/NLU library by Intel® AI Lab

Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing

NLP Architect is an open source Python library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing and Natural Language Understanding Neural Networks.

Overview

NLP Architect is an NLP library designed to be flexible, easy to extend, allow for easy and rapid integration of NLP models in applications and to showcase optimized models.

Features:

Core NLP models used in many NLP tasks and useful in many NLP applications
Novel NLU models showcasing novel topologies and techniques
Optimized NLP/NLU models showcasing different optimization algorithms on neural NLP/NLU models
Model-oriented design:
- Train and run models from command-line.
- API for using models for inference in python.
- Procedures to define custom processes for training, inference or anything related to processing.
- CLI sub-system for running procedures
Based on optimized Deep Learning frameworks:
- TensorFlow
- PyTorch
- Dynet
Essential utilities for working with NLP models - Text/String pre-processing, IO, data-manipulation, metrics, embeddings.

Installing NLP Architect

We recommend to install NLP Architect in a new python environment, to use python 3.6+ with up-to-date pip, setuptools and h5py.

Install using `pip`

Install core library only

pip install nlp-architect

Install from source (Github)

Includes core library, examples, solutions and tutorials:

git clone https://github.com/IntelLabs/nlp-architect.git
cd nlp-architect
pip install -e .  # install in developer mode

Running Examples and Solutions

To run provided examples and solutions please install the library with [all] flag which will install extra packages required. (requires installation from source)

pip install .[all]

Models

NLP models that provide best (or near) in class performance:

Natural Language Understanding (NLU) models that address semantic understanding:

Optimizing NLP/NLU models and misc. optimization techniques:

Solutions (End-to-end applications) using one or more models:

Term Set expansion - uses the included word chunker as a noun phrase extractor and NP2Vec to create semantic term sets
Topics and trend analysis - analyzing trending phrases in temporal corpora
Aspect Based Sentiment Analysis (ABSA)

Documentation

Full library documentation of NLP models, algorithms, solutions and instructions on how to run each model can be found on our website.

NLP Architect library design philosophy

NLP Architect is a model-oriented library designed to showcase novel and different neural network optimizations. The library contains NLP/NLU related models per task, different neural network topologies (which are used in models), procedures for simplifying workflows in the library, pre-defined data processors and dataset loaders and misc utilities. The library is designed to be a tool for model development: data pre-process, build model, train, validate, infer, save or load a model.

The main design guidelines are:

Deep Learning framework agnostic
NLP/NLU models per task
Different topologies used in models
Showcase End-to-End applications (Solutions) utilizing one or more NLP Architect model
Generic dataset loaders, textual data processing utilities, and miscellaneous utilities that support NLP model development (loaders, text processors, io, metrics, etc.)
Procedures for defining processes for training, inference, optimization or any kind of elaborate script.
Pythonic API for using models for inference
Extensive model documentation and tutorials

Note

NLP Architect is an active space of research and development; Throughout future releases new models, solutions, topologies and framework additions and changes will be made. We aim to make sure all models run with Python 3.6+. We encourage researchers and developers to contribute their work into the library.

Citing

If you use NLP Architect in your research, please use the following citation:

@misc{izsak_peter_2018_1477518,
  title        = {NLP Architect by Intel AI Lab},
  month        = nov,
  year         = 2018,
  doi          = {10.5281/zenodo.1477518},
  url          = {https://doi.org/10.5281/zenodo.1477518}
}

Disclaimer

The NLP Architect is released as reference code for research purposes. It is not an official Intel product, and the level of quality and support may not be as expected from an official product. NLP Architect is intended to be used locally and has not been designed, developed or evaluated for production usage or web-deployment. Additional algorithms and environments are planned to be added to the framework. Feedback and contributions from the open source and NLP research communities are more than welcome.

Contact

Contact the NLP Architect development team through Github issues or email: [email protected]

Comments

Runtime errors when running with the ABSA model

I am trying to run the ABSA solution on my local Ubuntu 18.04 even with the provided train reviews dataset but getting the following error (using the exact instructions + virtualenv): 'file_name': ['tripadvisor_co_uk-travel_restaurant_reviews_sample_2000_train.csv']}}], 'references': []}: OSError('Unable to load model. Filepath is not an hdf5 file (or h5py is not available) or SavedModel.',) When I am trying to load a bunch of reviews, I get the following error:

'references': []}: ParserError('Expected 2 fields in line 2, saw 3')

Please advise how to fix it.
bug

opened by stevesolun 10
Pip install failing from master

Target objective:

pip Install .

Steps to objective:

Successfully built nlp-architect spacy 2.0.18 has requirement numpy>=1.15.0, but you'll have numpy 1.14.5 which is incompatible. Installing collected packages: spacy, nlp-architect Killed

Pull-Request related:

opened by aribornstein 8

bug: Error exporting QuantizedBert to ONNX

When attempting to export a Q8BERT model to ONNX, TorchScript reports a runtime error.

Since the quantized forward methods embed Tensors with grads from layer instance attributes rather than input parameters, exporting the model has to be done with explicit torch script tracing; however that creates a traced model with lacks fidelity with the python model, as branching behavior is not captured.

However, attempting to create a script model with TorchScript, which should encode that behavior results in a missing config attribute on the BertModel superclass.

To Reproduce Steps to reproduce the behavior:

Train a Q8Bert model:

 nlp-train transformer_glue \
            --task_name mrpc \
            --model_name_or_path bert-base-uncased \
            --model_type quant_bert \
            --learning_rate 2e-5 \
            --output_dir $DATA_DIR \
            --data_dir $GLUE_DIR  \
            --evaluate_during_training \
            --do_lower_case \
            --per_gpu_train_batch_size 32 \
            --per_gpu_eval_batch_size 32 \
            --max_seq_length 128

Load the 8bit model for inference

import torch
from nlp_architect.models.transformers.quantized_bert import QuantizedBertForSequenceClassification

model = QuantizedBertForSequenceClassification.from_pretrained(configs.data_dir, from_8bit=True)
device = torch.device("cuda")
model.to(device)
model.eval()

Attempt to create the script model

script_model = torch.jit.script(model)

This produces the error:

Traceback (most recent call last):
  File "convert-torch-q8bert-via-onnx-to-tensorflow.py", line 105, in <module>
    script_model = torch.jit.script(model)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1203, in script
    return torch.jit.torch.jit._recursive.recursive_script(obj)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 173, in recursive_script
    return copy_to_script_module(mod, overload_stubs + stubs)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 95, in copy_to_script_module
    torch.jit._create_methods_from_stubs(script_module, stubs)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 195, in make_strong_submodule
    new_strong_submodule = recursive_script(module)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 173, in recursive_script
    return copy_to_script_module(mod, overload_stubs + stubs)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/_recursive.py", line 95, in copy_to_script_module
    torch.jit._create_methods_from_stubs(script_module, stubs)
  File "/home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
RuntimeError:
module has no attribute 'config':
at /home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/transformers/modeling_bert.py:675:15
        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        if attention_mask.dim() == 3:
            extended_attention_mask = attention_mask[:, None, :, :]

        # Provided a padding mask of dimensions [batch_size, seq_length]
        # - if the model is a decoder, apply a causal mask in addition to the padding mask
        # - if the model is an encoder, make the mask broadcastable to [batch_size, num_heads, seq_length, seq_length]
        if attention_mask.dim() == 2:
            if self.config.is_decoder:
               ~~~~~~~~~~~ <--- HERE
                batch_size, seq_length = input_shape
                seq_ids = torch.arange(seq_length, device=device)
                causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None]
                extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :]
            else:
                extended_attention_mask = attention_mask[:, None, None, :]

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
'__torch__.nlp_architect.models.transformers.quantized_bert.QuantizedBertModel.forward' is being compiled since it was called from '__torch__.nlp_architect.models.transformers.quantized_bert.QuantizedBertForSequenceClassification.forward'
at /home/krhys/.local/share/virtualenvs/q8bert-_S0V57VU/lib/python3.7/site-packages/transformers/modeling_bert.py:1014:8
    def forward(self, input_ids=None, attention_mask=None, token_type_ids=None,
                position_ids=None, head_mask=None, inputs_embeds=None, labels=None):

        outputs = self.bert(input_ids,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,
                            position_ids=position_ids,
                            head_mask=head_mask,
                            inputs_embeds=inputs_embeds)

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output)

Expected behavior

It should produce a script model without error

**Environment setup: **

OS (Linux/Mac OS): Ubuntu Xenial
Python version: 3.7

Additional context

It may be possible to create pure helper quantized forward functions with explicit arguments which have the @torch.jit.script annotations which are called by the quantized layer forward methods

Note that the HuggingFace transformers Bert model can be exported to ONNX.

bug Stale

opened by kirstin-rhys 7

Any readme for BERT example?

Target objective:

Want help to run https://github.com/NervanaSystems/nlp-architect/blob/master/nlp_architect/procedures/transformers/glue.py

Hi, I want to run the bert example newly added to the repo. Wondering is there any readme for running the example or any documentation for bert/quantized-bert related features? For instance, where to download model and 'labels.txt'?

Thanks so much for your help in advance!

opened by hkvision 7
nlp_architect command fails if dev req aren't installed
Target objective:

nlp_architect command fails if dev req aren't installed detect is libraries required are install and print a msg that states the error + how to install

Steps to objective:

[ ] fix cmd.py

[ ]

Pull-Request related:
bug
opened by peteriz 7
GPU training support

This is more of a question than issue but I wasn't able to find a forum to ask the question.

I see installation instruction on enabling GPU supports. Yet, many algorithms (e.g. ner) do not specifically have a parameter for GPU enabling. Is it an issue easy to get around with or is it going to be a feature supported in the future?
question

opened by DihuiLai 7

Problem running bist FileNotFoundError: [Errno 2] No such file or directory: (...) params.json

I tried to run bist in your interface using the command :

python3 server/serve.py --name bist

I get the following error :

25MB [00:28,  1.16s/MB]                                                                                                                                                                                     
Download Complete
Unzipping...
Done.
Traceback (most recent call last):
  File "serve.py", line 306, in <module>
    set_server_properties(app, args.name)
  File "serve.py", line 287, in set_server_properties
    service = Service(service_name)
  File "serve.py", line 143, in __init__
    self.service = self.load_service(service_name)
  File "serve.py", line 259, in load_service
    upload_service.load_model()
  File "/home/catalina/nlp-architect/nlp_architect/api/bist_parser_api.py", line 32, in load_model
    self.model = SpacyBISTParser()
  File "/home/catalina/nlp-architect/nlp_architect/pipelines/spacy_bist.py", line 48, in __init__
    self.bist_parser.load(bist_model if bist_model else SpacyBISTParser.pretrained)
  File "/home/catalina/nlp-architect/nlp_architect/models/bist_parser.py", line 117, in load
    with open(os.path.join(os.path.dirname(path), 'params.json'), 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/catalina/nlp-architect/nlp_architect/pipelines/bist-pretrained/params.json'

opened by Catadanna 7

How to improve bist model accuracy? :Bistmodel

I'm using spacybistparser with my own spacy and bist model trained using universal dependencies v2.0 dataset. My training follows the doc. I want to improve the accuracy of bistmodel. Currently, with default parameters, I train the bist model for 20iterations. The evaluation uas score is 87.8 percent. How can I improve it to 93.8 as mentioned in the docs? Is it possible with this english ud 2.0 dataset? Please help.
question

opened by ravishpankar 6
question about Quantize BERT?

I find that the excited codes about Quantize BERT, use int 8, but only loss a little accuracy, but after I train the mrpc with the example code, I find that the checkpoint file is 418mb, is it normal? I think it will be two times smaller.

is it the int 8 method describe similar with Vector Neural Network Instructions Enable Int8 AI Inference on Intel Architecture

And there is any example about Knowledge Distillation using Transformers?
question

opened by RyanHuangNLP 6
Question about Seq2SeqIntentModel

This is used for slot tagging, and in the example you provide, the dataset is SNIPS and it is used for NER. Wondering why it is placed under intent_extraction and come with the name IntentModel?
question Stale

opened by hkvision 6
Getting ValueError('Length must be less than 1000') error

I am trying to run ABSA with the a CSV and getting this error. I have sent the file to @danielkorat. It contains 1000 rows and it fails with the above error.

Please advise how can it be solved. Until now 500K reviews (in one file) have been sent and it worked.
bug

opened by stevesolun 5
Performance issue in the definition of _inference, examples/memn2n_dialogue/memn2n_dialogue.py(P1)

Hello, I found a performance issue in the difinition of_inference, examples/memn2n_dialogue/memn2n_dialogue.py, tf.nn.embedding_lookup will be calculated repeately during the program execution, resulting in reduced efficiency. So I think it should be created before the loop.

The same issue exist in tf.reduce_sum in line 187 and 200.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.
bug

opened by DLPerf 2

bug: ERROR: ResolutionImpossible

Describe the bug Installation error Model/procedure: $ pip install nlp-architect

To Reproduce Steps to reproduce the behavior: $ pip install nlp-architect

Expected behavior No Errors

**Environment setup: **

OS (Linux/Mac OS): GNU/Linux
Python version: Python 3.9.6
Backend: pip

Log:

Collecting nlp-architect
  Using cached nlp_architect-0.5.4-py3-none-any.whl (308 kB)
Collecting h5py
  Using cached h5py-3.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
Collecting requests
  Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 56 kB/s 
Collecting gensim
  Using cached gensim-4.0.1-cp39-cp39-linux_x86_64.whl
Collecting scipy
  Downloading scipy-1.7.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.4 MB)
     |████████████████████████████████| 28.4 MB 871 kB/s 
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting numpy==1.16.4
  Using cached numpy-1.16.4.zip (5.1 MB)
Collecting dynet==2.1
  Using cached dyNET-2.1.tar.gz (454 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/anks/STEALTH/ENV/absa_env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pj_foilz
         cwd: /tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-832t4ncx/dynet_ec03a5a10dc14ba6bb53d279dc810a9e/setup.py", line 17, in <module>
        from Cython.Distutils import build_ext as _build_ext
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/22/8e/148ecf0ec4ffce667e84abd6aa134bbff1f73cbb99c92c198149f122889e/dyNET-2.1.tar.gz#sha256=d0f58aaf3926da24baba6e3e76cb3d090c8b6d359196ce138b11faa291b2ec07 (from https://pypi.org/simple/dynet/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Collecting nlp-architect
  Using cached nlp_architect-0.5.3-py3-none-any.whl (301 kB)
  Using cached nlp_architect-0.5.2-py3-none-any.whl (978 kB)
Collecting tqdm
  Using cached tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
Collecting wordfreq
  Downloading wordfreq-2.5.0.tar.gz (56.8 MB)
     |████████████████████████████████| 56.8 MB 50 kB/s 
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 71 kB/s 
Collecting python-dateutil<2.8.1
  Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
     |████████████████████████████████| 226 kB 945 kB/s 
Collecting ftfy
  Downloading ftfy-6.0.3.tar.gz (64 kB)
     |████████████████████████████████| 64 kB 226 kB/s 
Collecting termcolor
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting numpy<1.17
  Using cached numpy-1.16.6.zip (5.1 MB)
Collecting hyperopt
  Downloading hyperopt-0.2.5-py2.py3-none-any.whl (965 kB)
     |████████████████████████████████| 965 kB 112 kB/s 
Collecting pywikibot
  Downloading pywikibot-6.4.0.tar.gz (506 kB)
     |████████████████████████████████| 506 kB 422 kB/s 
Collecting nltk
  Downloading nltk-3.6.2-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 696 kB/s 
Collecting tensorflow-hub
  Downloading tensorflow_hub-0.12.0-py2.py3-none-any.whl (108 kB)
     |████████████████████████████████| 108 kB 330 kB/s 
Collecting nlp-architect
  Using cached nlp_architect-0.5.1-py3-none-any.whl (1.6 MB)
Collecting pillow
  Downloading Pillow-8.3.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 182 kB/s 
Collecting regex==2018.02.21
  Downloading regex-2018.02.21.tar.gz (620 kB)
     |████████████████████████████████| 620 kB 475 kB/s 
Collecting nlp-architect
  Downloading nlp_architect-0.5-py3-none-any.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 107 kB/s 
  Downloading nlp_architect-0.4.post3-py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 102 kB/s 
Collecting spacy
  Downloading spacy-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB)
     |████████████████████████████████| 6.5 MB 857 kB/s 
Collecting nlp-architect
  Downloading nlp_architect-0.4.post2-py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 70 kB/s 
Collecting dynet==2.0.2
  Downloading dyNET-2.0.2.tar.gz (388 kB)
     |████████████████████████████████| 388 kB 557 kB/s 
    ERROR: Command errored out with exit status 1:
     command: /home/anks/STEALTH/ENV/absa_env/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py'"'"'; __file__='"'"'/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-etz16c1n
         cwd: /tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-832t4ncx/dynet_eb19eec4e56248fdb3fef59af0805ab5/setup.py", line 17, in <module>
        from Cython.Distutils import build_ext as _build_ext
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/01/10/cfc00197733dd7cf52d9e00c42017ebe7e818653980bccfe6f241ea5b79a/dyNET-2.0.2.tar.gz#sha256=d09f3799e3ad19572e9adec2784b0330a5d96bf6914a20858cd14b71e1482213 (from https://pypi.org/simple/dynet/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Collecting nlp-architect
  Downloading nlp_architect-0.4.post1-py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 611 kB/s 
  Downloading nlp_architect-0.4-py3-none-any.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 190 kB/s 
  Downloading nlp_architect-0.3.1-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 598 kB/s 
Collecting pytest-mock
  Downloading pytest_mock-3.6.1-py3-none-any.whl (12 kB)
ERROR: Cannot install nlp-architect==0.3.1, nlp-architect==0.4, nlp-architect==0.4.post1, nlp-architect==0.4.post2, nlp-architect==0.4.post3, nlp-architect==0.5, nlp-architect==0.5.1, nlp-architect==0.5.2, nlp-architect==0.5.3 and nlp-architect==0.5.4 because these package versions have conflicting dependencies.

The conflict is caused by:
    nlp-architect 0.5.4 depends on dynet==2.1
    nlp-architect 0.5.3 depends on tensorflow==1.15.0
    nlp-architect 0.5.2 depends on dynet==2.1
    nlp-architect 0.5.1 depends on dynet==2.1
    nlp-architect 0.5 depends on dynet==2.1
    nlp-architect 0.4.post3 depends on dynet==2.1
    nlp-architect 0.4.post2 depends on dynet==2.0.2
    nlp-architect 0.4.post1 depends on dynet==2.0.2
    nlp-architect 0.4 depends on dynet==2.0.2
    nlp-architect 0.3.1 depends on dynet==2.0.2

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

bug

opened by AnkS4 0

question: [Quantization] Which files to change to make inference faster for Q8BERT?

I know from previous issues it is mentioned that that Q8BERT was just an experiment to measure the accuracy of quantized BERT model. But, given that the accuracy is good, what changed would need to be made to torch.nn.quantization file to replace the FP32 operations by INT8 operations?

Replacing the FP32 Linear layers with the torch.nn.quantized.Linear should theoretically work since it will have optimized operations, but it doesn't. Same for other layers.

If someone could just point out how to improve the inference speed (hints, tips, directions, code, anything), it would be helpful since the model's accuracy is really good and I would like to use it for downstream tasks. I don't mind even creating a PR once those changes are done so that it merges with the main repo.

Thank you!
question

opened by sarthaklangde 1
question: [Q8Bert experiment Setting]

Hello, I read the Q8Bert paper and have tried to reproduce the experiment results. But, on some GLUE tasks ( e.g cola, mrpc ), the differences between the fp32 results and quantized ones are much larger than the differences reported in the paper. I tried sweeping initial learning rate but still the result was still far from the reported results.

So, I want to ask you if the experiment on Q8bert was done with default parameters set inside nlp-architect code as below.

If not, could you tell me the experiment setting.
question

opened by daumkh402 2
improvement: distillation for TransformerSequenceClassifier models for GLUE tasks
Hi,

I'm wondering if it would be easy to add support for knowledge distillation for the Transformers on GLUE tasks (i.e. the TransformerSequenceClassifier module).

I see that the distillation loss has been implemented, and it's an option for the NeuralTagger which uses the TransformerTokenClassifier. Would it be easy to add distillation support for the GLUE models?

Here's how I was envisioning implementing it, modeling off of the distillation implementation for the tagger models:

TransformerSequenceClassifier’s train calls the base transformer model’s _train(). I would need to add a distiller argument to this function. This function would then handle distillation by loading in the teacher and the relevant arguments, just like the NeuralTagger

in procedures/transformers/glue.py, i would need to add a do_kd_training function that adds distilation args. This function would need to create a teacher model from these args (loading in weights from the passed-in path), create a TeacherStudentDistill instance, and pass in this object as the distiller argument into the new _train() function.

Does this seem about right? Are there any roadblocks you'd envision / why wasn't distillation implemented for the sequence classifier models to begin with?
opened by rmovva 0

Releases(v0.5.5.1)

v0.5.5.1(Nov 17, 2020)

Source code(tar.gz)
Source code(zip)
v0.5.5(Oct 6, 2020)

Source code(tar.gz)
Source code(zip)
v0.5.4.1(Sep 16, 2020)

Source code(tar.gz)
Source code(zip)
v0.5.4(Apr 12, 2020)
Improved ID-CNN and LSTM sequence tagger models accuracy

Added support for BILOU format for tagging tasks

Updated Transformer token tagging and neural taggers to save best and support BILOU format

Refactor pseudo/distillation procedures to reproduce results according to our paper

Source code(tar.gz)
Source code(zip)
v0.5.3(Jan 9, 2020)

Source code(tar.gz)
Source code(zip)
v0.5.2(Dec 5, 2019)

Source code(tar.gz)
Source code(zip)
v0.5.1(Sep 9, 2019)

Source code(tar.gz)
Source code(zip)
v0.5(Aug 28, 2019)
General features

Added PyTorch as a backend

New nlp_architect CLI for training/running/process models and scripts

New nlp_architect.procedures package for creating procedures/scripts for doing scripted work with NLP architect.

S3 URL caching of pre-trained models

Many additions/refactors/base classes reflecting new API (WIP)

Transformer models

Integrated pytorch-transformers library into NLP Architect's model package.

Created a base transformer class for training transformer-based models

Added sequence/token classification models

Optimization related models

Added a base transformer model for training quantized (8bit) BERT models

Added distillation process from Transformer models into CNN/LSTM based token classification models

NLP related additions

Added GLUE benchmark tasks

Added CNN-LSTM and ID-CNN token classification models (pytorch backend)

Documentation

Updated documentation website to reflect 0.5 release

Updated website with logo and matching style

Source code(tar.gz)
Source code(zip)
v0.4(Apr 2, 2019)
Models

New Aspect Based Sentiment Analysis model and example

Solutions

New Aspect Based Sentiment Analysis solution

Improvements

Revised installation process (separate development dependencies)

Fixed NERCRF model bug with TF 1.12

Revised Documentation

Source code(tar.gz)
Source code(zip)
v0.3.1(Dec 17, 2018)
Improvements

Installation process refactored and supports pip install nlp-architect (much more simple)

Moved server and solutions into main package

Many Cross-doc coreference model fixes and additions

Updates and fixed lots of documentation files

Demos

Improved NLP Architect demos with a new design

Renamed nlp_architect demo to nlp_architect server

Source code(tar.gz)
Source code(zip)
v0.3(Nov 1, 2018)
New Solution

Topics and Trend Analysis - extract topics and compare two temporal versions a corpus, highlighting hot and cold trends.

New models

Sparse GNMT - A Tensorflow implementation of the GNMT model with sparsity and quantization operations integrated.

Semantic Relation Identification - Extract semantic relation types of two words or phrases using external resources.

Sieve-based Cross Document Coreference - A seive-based model for finding similar entities or events across different documents from the same domain.

Improvements

Reading comprehension - added inference mode.

Sequential models - updated NER, IE, Chunker models to use tf.keras and added CNN-character based feature extractors and improved accuracy of all models.

CRF Layer - added native Tensorflow based CRF layer.

Word Sense Disambiguation - model updated to use tf.keras.

Demo UI - updated demo UI using AngularJS.

Installation - improved installation process and added support for CPU/MKL/GPU backends for Tensorflow.

NLP Architect cmd - added nlp_architect - a simple command initiator to handle maintenance tasks, see nlp_architect -h for the list of commands.

Lots of bug fixes and refactoring.

Source code(tar.gz)
Source code(zip)
v0.2(Aug 15, 2018)
Release v0.2

New Solution

Term Set Expansion - the task of expanding a given partial set of terms into a more complete set of terms that belong to the same semantic class. This solution demonstrates the usage of NLP Architect models (Word Chunker and NP2Vec) used in an application solution.

New models

Unsupervised Crosslingual Embeddings model using a GAN to learn a mapping between languages - implemented in Tensorflow

Language Model (LM) using Temporal Convolution Network (TCN) - implemented in Tensorflow

Supervised Sentiment Classification - implemented in Keras

Model improvements

Reading comprehension - refactored to use Tensorflow

End-to-end Memory Network for Goal Oriented Dialogue - refactored to use Tensorflow

Word Chunker - refactored to use tf.keras and use state-of-art model

NP semantic segmentation - refactored to use tf.keras

Updated CONLL2000, Amazon_Review, PTB, Fasttext, Wikitext-103 and Wikipedia-dump dataset loaders.

New features

REST server refactored to use hug, new streamlined the UI and improved documentation. See updated documentation for further details.

Noun Phrase annotator plug-in for spaCy pipeline

Publications page with relevant material demonstrating the usage of NLP Architect

Tutorials page with a collection of Jupyter notebook tutorials using NLP Architect models

Source code(tar.gz)
Source code(zip)
0.1(May 29, 2018)
The current version of NLP Architect includes these features that we found interesting from both research perspectives and practical applications:

NLP core models that allow robust extraction of linguistic features for NLP workflow: for example, dependency parser (BIST) and NP chunker

NLU modules that provide best in class performance: for example, intent extraction (IE), name entity recognition (NER)

Modules that address semantic understanding: for example, colocations, most common word sense, NP embedding representation (e.g. NP2V)

Components instrumental for conversational AI: for example, ChatBot applications, including dialog system, sequence chunking and IE

End-to-end DL applications using new topologies: for example, Q&A, machine reading comprehension

Source code(tar.gz)
Source code(zip)

Owner

Intel Labs

GitHub Repository https://intellabs.github.io/nlp-architect

A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

Related tags

Overview

A Deep Learning NLP/NLU library by Intel® AI Lab

Overview | Models | Installation | Examples | Documentation | Tutorials | Contributing

Overview

Installing NLP Architect

Install using pip

Install from source (Github)

Running Examples and Solutions

Models

Documentation

NLP Architect library design philosophy

Note

Citing

Disclaimer

Contact

Comments

Target objective:

Steps to objective:

Pull-Request related:

Target objective:

Target objective:

Steps to objective:

Pull-Request related:

Releases(v0.5.5.1)

v0.5.5.1(Nov 17, 2020)

v0.5.5(Oct 6, 2020)

v0.5.4.1(Sep 16, 2020)

v0.5.4(Apr 12, 2020)

v0.5.3(Jan 9, 2020)

v0.5.2(Dec 5, 2019)

v0.5.1(Sep 9, 2019)

v0.5(Aug 28, 2019)

General features

Transformer models

Optimization related models

NLP related additions

Documentation

v0.4(Apr 2, 2019)

Models

Solutions

Improvements

v0.3.1(Dec 17, 2018)

Improvements

Demos

v0.3(Nov 1, 2018)

New Solution

New models

Improvements

v0.2(Aug 15, 2018)

Release v0.2

New Solution

New models

Model improvements

New features

0.1(May 29, 2018)

Owner

Intel Labs

DeepPavlov Tutorials

Repository to hold code for the cap-bot varient that is being presented at the SIIC Defence Hackathon 2021.

source code for paper: WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach.

A complete NLP guideline for enthusiasts

Malaya-Speech is a Speech-Toolkit library for bahasa Malaysia, powered by Deep Learning Tensorflow.

Protein Language Model

Open solution to the Toxic Comment Classification Challenge

Ceaser-Cipher - The Caesar Cipher technique is one of the earliest and simplest method of encryption technique

ZUNIT - Toward Zero-Shot Unsupervised Image-to-Image Translation

Official Stanford NLP Python Library for Many Human Languages

Implementation of TF-IDF algorithm to find documents similarity with cosine similarity

Code from the paper "High-Performance Brain-to-Text Communication via Handwriting"

Biterm Topic Model (BTM): modeling topics in short texts

DensePhrases provides answers to your natural language questions from the entire Wikipedia in real-time

Tools and data for measuring the popularity & growth of various programming languages.

EdiTTS: Score-based Editing for Controllable Text-to-Speech

Code and dataset for the EMNLP 2021 Finding paper "Can NLI Models Verify QA Systems’ Predictions?"

A Multi-modal Model Chinese Spell Checker Released on ACL2021.

Official PyTorch implementation of SegFormer

Syntax-aware Multi-spans Generation for Reading Comprehension (TASLP 2022)

Install using `pip`