Python character encoding detector

Text in the hungarian language can't contain many english words inside detected text. For example xml files can have more english words because of tag names and others. This detector is based on the letter frequency. The second problem arises if the hungarian text has many sentences in uppercase.

This change helps pass three unit tests that were failing before. I have also tested the changes by comparing the output of this function with Mozilla's version over some large randomly generated byte strings and so far so good.

Most of the credit for this goes to @bsidhom. I just took his Python 3 port and added a bunch of __future__ imports and the occasional from io import open to make it backward compatible with 2.6 and 2.7.

I did some minor clean-up things like sorting imports and things like that. Oh and I added a .gitattributes file to ensure line endings are consistent.

@erikrose Are you still actively maintaining this? I notice there a few outstanding pull requests and I just want to make sure the version on PyPI is 2/3 compatible soon.

I am using chardet as part of a web crawler written in python3. I noticed that over time (many hours), the program consumes all memory. I narrowed down the problem to a single call of chardet.detect() method for certain web pages.

After some testing, it seems that chardet has problem with some special input and I managed to get a sample of such an input. It consumes on my machine about 220 MB of memory (however, the input is 2.5 MB) and takes about 1:22 minutes to process (in contrast to 43 ms when the file is truncated to about 2 MB). It seems not to be limited to python3, in python2 the memory consumption is even worse (312 MB).

Versions:

Fedora release 20 (Heisenbug) x86_64 chardet-2.2.1 (via pip) python3-3.3.2-11.fc20.x86_64 python-2.7.5-11.fc20.x86_64

How to reproduce:

I cannot attach any files to this issue so I uploaded them to my dropbox account: https://www.dropbox.com/sh/26dry8zj18cv0m1/sKgP_E44qx/chardet_test.zip Please let me know of a better place where to put it if necessary. Here is an overview of the content and the results:

setup='import chardet; html = open("mem_leak_html.txt", "rb").read()'
python3 -m timeit -s "$setup"  'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 43 ms per loop
python3 -m timeit -s "$setup"  'chardet.detect(html[:2543483])'
# produces: 1 loops, best of 3: 1min 22s per loop
python3 mem_leak_test.py
# produces:
# Good input left 2.65 MB of unfreed memory.
# Bad input left 220.16 MB of unfreed memory.

python -m timeit -s "$setup"  'chardet.detect(html[:2543482])'
# produces: 10 loops, best of 3: 41.7 ms per loop
python -m timeit -s "$setup"  'chardet.detect(html[:2543483])'
# produces: 10 loops, best of 3: 111 sec per loop
python mem_leak_test.py
# produces:
# Good input left 3.00 MB of unfreed memory.
# Bad input left 312.00 MB of unfreed memory.

mem_leak_test.py:

import resource
import chardet
import gc

mem_use = lambda: resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
html = open("mem_leak_html.txt", "rb").read()

def test(desc, instr):
    gc.collect()
    mem_start = mem_use()
    chardet.detect(instr)    
    gc.collect()
    mem_used = mem_use() - mem_start
    print('%s left %.2f MB of unfreed memory.' % (desc, mem_used))    

test('Good input', html[:2543482])
test('Bad input', html[:2543483])

This change adds heuristic detection of UTF-16 and UTF-32 files when they are missing their byte order marks.

At present we have no strategy for detecting the format of these files. Feel free to give feedback on the PR by the way, happy to have other eyes on it.

Note I report these files as UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE rather than UTF-16 and UTF-32. This is justified for the following reasons: it's quite material whether it is little endian or big endian - Python will not decode these files correctly when passed decode("utf-16") - Python assumes endian-ness. This is less important for the files with the BOMs, as UTF-aware readers will generally inspect this and auto-decode the file, but for things missing the BOM, it's really important to be told the endian-ness in hopes of reading the file.

We could change the reporting of of files with BOM to inform the endian-ness also, but I've avoided having an opinion on that for the moment.

This feels strange to be posing as a question, since I'm one of the co-maintainers, but @sigmavirus24 and @erikrose, do you know if it's okay/legal for us to change the license of chardet? Because it was started by Mark Pilgrim I feel like it's kind of a nebulous question, because he's not someone you can just email, and he has nothing to do with development anymore. I would really like to change the license to at least be MPL, since that's what the C++ version is, and our setup currently mirrors that code pretty closely.

I'm not a fan of the LGPL and feel weird having a project I work on use it.

MacRoman is not in particularly common use anymore, as it has been deprecated by Mac OS for over a decade. However, there are programs such as Microsoft Office for Mac that didn't get the memo, and will often output in MacRoman when they write plain text files.

This patch allows chardet to correctly detect MacRoman, instead of calling it something random and incorrect like ISO-8859-2. The MacRoman detector works similarly to the Latin-1 detector, but starts at a lower probability.

I hope this is the right way to do it. There is surprisingly little support in chardet for adding a new single-byte Latin encoding.

The concept here is pretty simple: This tries to test for the invariant that if a string comes from valid unicode and is encoded in one of the chardet supported encodings then chardet should detect some encoding.

More nuanced tests are possible (e.g. asserting that the string should be decodable from the detected encoding) but given that even this test is demonstrating a bunch of bugs this seemed to be a good starting point.

This is (more or less) the test that caught #65, #64 and #63. #62 had one extra line in it to try to reencode the data as the reported format.

Notes:

1. This test is currently failing. I'm pretty sure this is because of issues it's finding in the code, not issues with the test.
2. min_size=100 is to rule out bugs that come solely from the length, prompted by your saying that short strings aren't really supported. Anecdotally all of the bugs that have been found so far don't depend on the length and min_size=1 would have been fine (leaving min_size alone is also valid, but I assume '' having a None encoding is intended behaviour)

I have a page of text in ASCII with a single Microsoft-apostrophe chr(8217) detected as ISO-8859-2.

#1. Create problematic sample
>>> s = 'today' + chr(8217) + 's research'
>>> s
'today’s research'
>>> b = s.encode('windows-1252')
>>> b
b'today\x92s research'

#2. Attempt to decode it
>>> chardet.detect(b)
{'encoding': 'ISO-8859-2', 'confidence': 0.8060609643099236}
>>> b.decode('ISO-8859-2')
'today\x92s research'

#3. Now try the correct encoding
>>> b.decode('windows-1252')
'today’s research'

This text is very typical of anything created using a Microsoft editor. Furthermore, latest version of Firefox detects it correctly. I am using Python 3.3. Any help is appreciated.

This is very much a work in progress at the moment, and I'm just creating the PR to make it easier for me to keep track of Travis results.

I have a few goals for this branch:

1. Pull in changes from Mozilla's upstream code. There aren't as many as I had initially expected but there are some.
2. Improve PEP8 compliance all over the place. The previous maintainers tried to keep variable names identical to the C code, presumably to ease the comparison with the Mozilla code, but we're going to be diverging from upstream after pulling in the changes mentioned in 1. Basically, Mozilla seems very likely to abandon their character encoding detector in the near future and switch to using ICU, but ICU doesn't support all of the codecs we currently do, because it is more web-focused. If our goal here is to be a truly universal character encoding detector, we'll need to go our own way in the future in that respect.
3. Make the unit tests pass, or at the very least make it obvious that the tests are actually failing (instead of ignoring the failures like our current Travis build does).

So far, I've done a little bit of point 1 and updated the Travis testing setup to use nose and report test coverage via Coveralls.

If passed a string starting with \xff\xfe (low endian byte order mark) or \xfe\xff (big endian byte order mark) the encoding is detected as UTF-16LE, UTF-32LE, UTF-16BE or UTF-32BE respectively.

However, as the byte order mark is given in the string, the encoding should be simply UTF-16 or UTF-32. Otherwise bytes.decode() will fail or preserve the byte order mark:

s = 'foo'.encode('UTF-16')
encoding = chardet.detect(s)['encoding']  # "UTF-16LE"
s.decode(encoding)                        # "\ufefffoo"

s = codecs.BOM_BE + 'foo'.encode('UTF-16BE')
encoding = chardet.detect(s)['encoding']  # "UTF-16BE"
s.decode(encoding)                        # "\ufefffoo"

Hence code that uses chardet in order to detect the encoding to decode data, would need to wrap chardet.detect in following inconvenient and counter-intuitive way:

encoding = chardet.detect(enc)['encoding']
if encoding in ('UTF-16LE', 'UTF16BE'):
  dec = enc.decode('UTF-16')
elif encoding in ('UTF-32LE', 'UTF-32BE'):
  dec = enc.decode('UTF-32')
else:
  dec = enc.decode(encoding)

This PR changes the behavior to return simply UTF-16or UTF-32 respectively when a byte order mark were found, that the detected encoding can be passed unchanged to bytes.decode().

I want to be able to execute the chardet main script (packaged as an executable) by running python3 -m chardet .... Currently it doesn't work. Would be great if it did work.

Hi,

In the file, 'https://github.com/chardet/chardet/blob/main/tests/windows-1255-hebrew/hydepark.hevre.co.il.7957.xml', we have found the following license text:

" This copy is for your personal, non-commercial use only. To order presentation-ready copies for distribution to your colleagues, clients or customers, use the Order Reprints tool at the bottom of any article or visit: www.djreprints.com. " This may cause issues even for open source projects that allows commercial use.

Can you please let us know if there is an option to retain the file even for commercial use? Is it possible to remove the content that is only for non-commercial and personal use?

Regards, Rahul

Abstract

Current CP949 state machine has some false positives, and incorrectly marks valid CP949 texts as an error. This PR rewrites the state transition table, to comply the CP949 Specification.

Details

These are some cases, which a false-positive error can occur in the current implementation.

• 춉 (0xAD68) The first byte is classified as the class 8, as it is 0xAD. And in the START state, the class 8 makes an transition to the ERROR state. But this is a valid CP949.

• 힣 (0xC652) The first byte is classified as the class 9, and the second byte is classified as the class 5. In the START state, the class 9 makes an transition to the State 6, and in the State 6, the class 5 makes an transition to the ERROR state. But this is a valid CP949.

Test

I have tested the state machine (To-Be) for the all characters in the CP949 with following code, and it successfully returned Success. When I have tested it against the current implementation (As-Is), it shows Error! at byte 15479.

from chardet.codingstatemachine import CodingStateMachine
from chardet.mbcssm import CP949_SM_MODEL

sm = CodingStateMachine(CP949_SM_MODEL)

with open('./path/to/cp949-chars.txt', 'rb') as f:
    data = f.read()

for i, byte in enumerate(data):
    state = sm.next_state(byte)

    if state == 1:
        print("Error! at byte %d" % i)
        break

if state != 1:
  print("Success! :)")

I couldn't upload the cp949 characters to the test fixtures folder, as it will make the test fail because of the frequency-based probing, which will not successfully mark it as the CP949. (Because it is just a plain listing of the all possible characters of the CP949.)

Yesterday I encountered a strange CI failure for our Windows GitHub CI workflows which had been running fine until then. The Python 3.7 job passed fine but the Python 3.10 job failed.

https://github.com/deluge-torrent/deluge/actions/workflows/ci.yml?query=branch%3Adevelop

The only difference I could find from a diff of the logs was the new chardet 5.0.0 being pulled in. So I pinned chardet to 4.0.0 and CI is passing again.

GitHub Actions Environment:

Virtual Environment: windows-2022 (20220626.1)
Python 3.10.5

Just to note that I also tested same error occurs with windows-2019.

The traceback is rather cryptic since it comes from pytest but this is all there is from the job:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\_pytest\main.py", line 264, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\_pytest\config\__init__.py", line 995, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\pluggy\_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\hostedtoolcache\windows\Python\3.10.5\x64\lib\site-packages\_pytest\faulthandler.py", line 27, in pytest_configure
INTERNALERROR>     import faulthandler
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1024, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 171, in __enter__
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 123, in acquire
INTERNALERROR> KeyError: 1832

$ python3.11 --version
Python 3.11.0b3
$ python3.11 -m venv _e
$ . _e/bin/activate
(_e) $ pip install -e .
(_e) $ pip install -e pytest hypothesis
(_e) $ pytest

results in:

====================================================== FAILURES ======================================================
____________________________________ test_detect_all_and_detect_one_should_agree _____________________________________

txt = 'Ā𐀀', enc = 'utf-8', _ = HypothesisRandom(generated data)

    @given(
        st.text(min_size=1),
        st.sampled_from(
            [
                "ascii",
                "utf-8",
                "utf-16",
                "utf-32",
                "iso-8859-7",
                "iso-8859-8",
                "windows-1255",
            ]
        ),
        st.randoms(),
    )
    @settings(max_examples=200)
    def test_detect_all_and_detect_one_should_agree(txt, enc, _):
        try:
            data = txt.encode(enc)
        except UnicodeEncodeError:
            assume(False)
        try:
            result = chardet.detect(data)
            results = chardet.detect_all(data)
>           assert result["encoding"] == results[0]["encoding"]
E           AssertionError: assert None == 'utf-8'

test.py:183: AssertionError

The above exception was the direct cause of the following exception:

    @given(
>       st.text(min_size=1),
        st.sampled_from(
            [
                "ascii",
                "utf-8",
                "utf-16",
                "utf-32",
                "iso-8859-7",
                "iso-8859-8",
                "windows-1255",
            ]
        ),
        st.randoms(),
    )

test.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

txt = 'Ā𐀀', enc = 'utf-8', _ = HypothesisRandom(generated data)

    @given(
        st.text(min_size=1),
        st.sampled_from(
            [
                "ascii",
                "utf-8",
                "utf-16",
                "utf-32",
                "iso-8859-7",
                "iso-8859-8",
                "windows-1255",
            ]
        ),
        st.randoms(),
    )
    @settings(max_examples=200)
    def test_detect_all_and_detect_one_should_agree(txt, enc, _):
        try:
            data = txt.encode(enc)
        except UnicodeEncodeError:
            assume(False)
        try:
            result = chardet.detect(data)
            results = chardet.detect_all(data)
            assert result["encoding"] == results[0]["encoding"]
        except Exception as exc:
>           raise RuntimeError(f"{result} != {results}") from exc
E           RuntimeError: {'encoding': None, 'confidence': 0.0, 'language': None} != [{'encoding': 'utf-8', 'confidence': 0.505, 'language': ''}]

test.py:185: RuntimeError
----------------------------------------------------- Hypothesis -----------------------------------------------------
Falsifying example: test_detect_all_and_detect_one_should_agree(
    txt='Ā𐀀', enc='utf-8', _=HypothesisRandom(generated data),
)
============================================== short test summary info ===============================================
FAILED test.py::test_detect_all_and_detect_one_should_agree - RuntimeError: {'encoding': None, 'confidence': 0.0, '...
================================ 1 failed, 375 passed, 6 xfailed, 1 xpassed in 9.79s =================================

The same steps succeed with Python 3.10.4.