👄 The most accurate natural language detection library for Python, suitable for long and short text alike

Hello, I try to use the module on google colab and I get this error during the installation:

ERROR: Could not find a version that satisfies the requirement lingua-language-detector (from versions: none)
ERROR: No matching distribution found for lingua-language-detector

What are the requirements of this module?

Hello.

When running this code with lingua_language_detector version 1.3.0.

with open('text.txt') as fh:
    text = fh.read()
    detector = LanguageDetectorBuilder.from_all_languages().build()
    print(text)
    result = detector.detect_language_of(text)
    print(result)

I get this error:

Traceback (most recent call last):
  File "/home/jordi/sc/crux-top-lists-catalan/bug.py", line 9, in <module>
    result = detector.detect_language_of(text)
  File "/home/jordi/.local/lib/python3.10/site-packages/lingua/detector.py", line 272, in detect_language_of
    confidence_values = self.compute_language_confidence_values(text)
  File "/home/jordi/.local/lib/python3.10/site-packages/lingua/detector.py", line 499, in compute_language_confidence_values
    normalized_probability = probability / denominator
ZeroDivisionError: float division by zero

I attached the text file that triggers the problem. It works fine with others texts. This happens often in a crawling application that I'm testing.

When loading the LanguageDetectorBuilder as recommended in the readme, I received the following error:

from lingua import LanguageDetectorBuilder ... ImportError: cannot import name 'LanguageDetectorBuilder' from 'lingua'

The following worked for me:

from lingua.builder import LanguageDetectorBuilder

Currently, for a given input string, only the most likely language is returned. However, if the input contains contiguous sections of multiple languages, it will be desirable to detect all of them and return an ordered sequence of items, where each item consists of a start index, an end index and the detected language.

Input: He turned around and asked: "Entschuldigen Sie, sprechen Sie Deutsch?"

Output:

[
  {"start": 0, "end": 27, "language": ENGLISH}, 
  {"start": 28, "end": 69, "language": GERMAN}
]

On occassion on longer texts I am getting this error. Steps to reproduce:

detector.detect_language_of(text)

Where text is

Flagged as potential abuser? No Retailer | Concept-store() Brand order:  placed on  Payout scheduled date: Not Scheduled Submission type: Lead How did you initially connected?: Sales rep When did you last reach out?:  (UTC) Did you add this person through ?: I don't know Additional information: Bonjour, Je travaille avec cette boutique depuis plusieurs années. C'est moi qui lui ai conseillé de passer par pour son réassort avec le lien direct que je lui avais transmis. Pourriez vous retirer la commission de 23% ? Je vous remercie. En lien pour preuve la dernière facture que je lui ai éditée et qui date du mois dernier. De plus, j'ai redirigé vers plusieurs autres boutiques avec qui j'ai l'habitude de travailler. Elles devraient passer commande prochainement: Ça m'ennuierai de me retrouver avec le même problème pour ces clients aussi. Merci d'avance pour votre aide ! Cordialement Click here to check out customer uploaded file Click here to approve / reject / flag as potential abuser

It's not an isolated example

Any help would be massively appreciated

Hi team, great library! Wanted to share an example I stumbled upon, when detecting the language of a very short basic Russian text. It comes out as Macedonian, even though as far as I can tell it's not actually correct Macedonian but is correct Russian. It is identified correctly by AWS Comprehend and other APIs:

detector = LanguageDetectorBuilder.from_all_languages().build()
detector.detect_language_of("как дела")
Language.MACEDONIAN

What do you think about passing results to softmax function instead min-max normalization? I think it's more clear way. Because, for example, you can have a threshold to filter-out unidentified languages.

Is there are some pitfalls that aren't clear for me? I've implemented this by slightly changing your code. I've also rounded results.

It passed black and mypy, but not tests. It's throwing me error like: INTERNALERROR> UnicodeEncodeError: 'charmap' codec can't encode characters in position 712-720: character maps to <undefined>

Hello

• "ITALIAN": 0.9900000000000001,
• "SPANISH": 0.8457074930316446,
• "ENGLISH": 0.6405700388041755,
• "FRENCH": 0.260556921899765,
• "GERMAN": 0.01,
• "CHINESE": 0,
• "RUSSIAN": 0

Bye

• "FRENCH": 0.9899999999999999,
• "ENGLISH": 0.9062076381164255,
• "GERMAN": 0.6259792361883574,
• "SPANISH": 0.46755135335558035,
• "ITALIAN": 0.01,
• "CHINESE": 0,
• "RUSSIAN": 0

Loss (not Löss)

• "GERMAN": 0.99,
• "ENGLISH": 0.9177028091362562,
• "ITALIAN": 0.9082690119891484,
• "FRENCH": 0.7091301303929289,
• "SPANISH": 0.01,
• "CHINESE": 0,
• "RUSSIAN": 0

Hi, I'm currently working on a project which requires me to filter all non-English text. It is comprised of mostly short texts, most of them in English. I thought of building the language detector with only Language.ENGLISH but got an error that at least two languages are required. I do not care about knowing what language each non-English text is actually in, only English / Non-English. What would be the correct way to go about it with lingua? I think it might be problematic if I set it to recognize all languages because it might just add unnecessary noise to the prediction, which should have a bias towards English in my case. Thanks!

On the test_case:

, Ресторан «ТИНАТИН»

Code fell down with an error:

Traceback (most recent call last):
  File "/home/essential/PycharmProjects/pythonProject/test_unnest.py", line 363, in <module>
    for lang, sentence in detector.detect_multiple_languages_of(text)
  File "/home/essential/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/lingua/detector.py", line 389, in detect_multiple_languages_of
    _merge_adjacent_results(results, mergeable_result_indices)
  File "/home/essential/PycharmProjects/pythonProject/venv/lib/python3.10/site-packages/lingua/detector.py", line 114, in _merge_adjacent_results
    end_index=results[i + 1].end_index,
IndexError: list index out of range

Code example:

languages = [Language.ENGLISH, Language.RUSSIAN, Language.UKRAINIAN]
detector = LanguageDetectorBuilder.from_languages(*languages).build()
text = ', Ресторан «ТИНАТИН»'
sentences = [(lang, sentence) for lang, sentence in detector.detect_multiple_languages_of(text)]

Hello, I need to detect language in user generated content, it's for a chat. I have tested this library but the library have strange result in short text, for exemple the word hello:

from lingua import Language, LanguageDetectorBuilder

languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
detector = LanguageDetectorBuilder.from_languages(*languages).build()

text = """
Hello
"""
confidence_values = detector.compute_language_confidence_values(text.strip())
for language, value in confidence_values:
    print(f"{language.name}: {value:.2f}")

return spanich (but the correct language is English)

SPANISH: 1.00
ENGLISH: 0.95
FRENCH: 0.87
GERMAN: 0.82

Do you know some tips to have better result for detecting language on user generated content?

Using version 1.3.1

Using a text that is in Catalan language only, that does not contain any fragments from other languages, and that it's very standard kind of text, detect_multiple_languages_of method detects: CATALAN, SOMALI, LATIN, FRENCH, SPANISH and PORTUGUESE. The expectation is that should report that the full text is CATALAN.

Code to reproduce the problem:

from lingua import Language, LanguageDetectorBuilder, IsoCode639_1

with open('text-catalan.txt') as fh:
    text = fh.read()

    detector = LanguageDetectorBuilder.from_all_languages().build()
    
    for result in detector.detect_multiple_languages_of(text):
        print(f"{result.language.name}")

Related to this problem also is that detect_language_of and detect_multiple_languages_of predict different languages over the same text. Below an example on the same input detect_language_of predicts Catalan and detect_multiple_languages_of predicts Tsonga.

My expectation is that both methods will predict the same given the same input.

Code sample:

from lingua import Language, LanguageDetectorBuilder, IsoCode639_1

with open('china.txt') as fh:
    text = fh.read()

    detector = LanguageDetectorBuilder.from_all_languages().build()
      
    result = detector.detect_language_of(text)
    print(f"detect_language_of prediction: {result}")
    
    for result in detector.detect_multiple_languages_of(text):
        print(f"detect_language_of prediction: {result.language.name}")

Using version 1.3.1

In a text that is 3.5K (31 lines) in my machine detect_multiple_languages_of takes 26.56 seconds while detect_language_of takes only 1.68 seconds.

26 seconds to analyse 3.5K of text (throughput of ~7 seconds per 1K) makes detect_multiple_languages_of method really not suitable for processing large corpus.

Code used for the benchmark:

from lingua import Language, LanguageDetectorBuilder, IsoCode639_1
import datetime


with open('text.txt') as fh:
    text = fh.read()

    detector = LanguageDetectorBuilder.from_all_languages().build()
    
    start_time = datetime.datetime.now()
    result = detector.detect_language_of(text)
    print('Time used for detect_language_of: {0}'.format(datetime.datetime.now() - start_time))
    print(result.iso_code_639_1)

    start_time = datetime.datetime.now()    
    results = detector.detect_multiple_languages_of(text)    
    print('Time used for detect_multiple_languages_of: {0}  '.format(datetime.datetime.now() - start_time))    
    for result in results:
        print(result)
        print(f"** {result.language.name}")

Hello! My understading is that this mapping:

https://github.com/pemistahl/lingua-py/blob/502bb9abef2a31b841c49e063f1a0bd7e47af86d/lingua/_constant.py#L34

It's used by the rule system to identity languages based on chars. Is my assumption correct?

Looking at this: https://github.com/pemistahl/lingua-py/blob/502bb9abef2a31b841c49e063f1a0bd7e47af86d/lingua/_constant.py#L191

Catalan language for example does NOT have "Áá" as valid chars (see reference https://en.wikipedia.org/wiki/Catalan_orthography#Alphabet).

Looking at the data I see other mappings that do not seem right.

May be the case that these mappings can be improved?

@pemistahl Peter, I think it would be beneficial for this library to have a separate method that will add probability prior (in a Bayesian way) to the mix.

Let's look into statistics: https://en.wikipedia.org/wiki/Languages_used_on_the_Internet

So if 57% of texts, that you see on the internet, are in English so, if you predicted "English" for any input you would be wrong only in 43%. It's like a stopped clock, but it is right every second probe.

For example: https://github.com/pemistahl/lingua-py/issues/100

Based on that premise, if we are using just plain character statistics "как дела" is more Macedonian than Russian. But overall, if we add language statistics to the mix, lingua-puy would be "wrong" less often.

There are more Russian-speaking users of this library, than Macedonians, just because there are more Russian-speaking people overall. And so when a random user writes "как дела" it's "more accurate" to predict "russian" than "macedonian", just because in general that is what is expected by these users.

So my proposition to add detector.detect_language_with_prior function and factorize it with prior: likelihood = probability X prior_probability

For example: https://github.com/pemistahl/lingua-py/issues/97

detector.detect_language_of("Hello")

"ITALIAN": 0.9900000000000001,
"SPANISH": 0.8457074930316446,
"ENGLISH": 0.6405700388041755,
"FRENCH": 0.260556921899765,
"GERMAN": 0.01,
"CHINESE": 0,
"RUSSIAN": 0

detector.detect_language_with_prior("Hello")

# Of course constants are for illustrative purposes only.
# Results should be normalized afterwords
"ENGLISH": 0.6405700388041755 * 0.577,
"SPANISH": 0.8457074930316446 * 0.045,
"ITALIAN": 0.9900000000000001 * 0.017,
"FRENCH": 0.260556921899765 * 0.039,

Linked issues:

• https://github.com/pemistahl/lingua-py/issues/94
• https://github.com/pemistahl/lingua-py/issues/100
• https://github.com/pemistahl/lingua-py/issues/97

It should be investigated if and how detection speed can be increased by compiling crucial parts of the library to native code, probably with the help of Cython or mypyc.