Bumps pyyaml from 3.11 to 5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

as reported in xonsh/xonsh#1569 some (maybe all) emojis are reported as being 2 char wide, while most (all?) terminals think they are 1 char wide:

The difference in position is because while editing a command xonsh is using wcwidth but to print the line it just prints and let the terminal position things

Hi, first off: Awesome project. This world is missing unicode-aware software.

I had a look at the code, and I found that it could easily be optimized by using more of the python standard library. The code as attached is about 10x faster on large strings (I tested it on 10MB of english text).

With the caching hack it's about 10x as fast (3.3s vs. 37s on my machine), without it's about 3x as fast (12.5s vs. 37s on my machine).

Tell me what you think of it and I'll format a more descriptive commit.

Regards, jaseg

import pandas as pd
from tabulate import tabulate
from io import StringIO
import wcwidth


csv_str = ''',0,1,2
0,a)  将净利润调节为经营活动现金流量,,
1,,2019 年度,2018 年度
2,净利润,"63,205,243","55,350,200"
3,加：资产减值损失,"(73,370)","10,465,899"
4,信用减值损失,"3,611,595",—
5,固定资产折旧,"6,543,253","6,530,713"
6,投资性房地产折旧,"1,718,108","1,514,560"
7,无形资产摊销,"447,821","440,444"
8,长期待摊费用摊销,"338,210","189,875"
9,处置固定资产、无形资产和其他长,,
10,期资产的收益,"(568,141)","(175,112)"
11,财务费用,"10,179,757","12,568,535"
12,公允价值变动损失,"484,752","368,343"
13,投资收益,"(4,212,538)","(5,646,311)"
14,递延所得税资产的增加,"(3,083,170)","(2,511,075)"
15,递延所得税负债的增加/(减少),"107,125","(644,386)"
16,存货的增加,"(70,420,830)","(96,125,732)"
17,受限资金的(增加)/减少,"(1,387,681)","280,759"
18,经营性应收项目的增加,"(125,133,902)","(108,365,569)"
19,经营性应付项目的增加,"83,217,173","135,881,219"
'''

csv_io = StringIO(csv_str)

df = pd.read_csv(csv_io, index_col=0)

print(df)

df_tabulated = tabulate(df, headers='keys', tablefmt='psql', showindex=False)

print(df_tabulated)

Some minor optimizations for wcwidth(). Should result in ~30% performance gain.

>>> from timeit import timeit
>>> timeit('wcwidth("a")', setup='from wcwidth import wcwidth', number=10000000)

Before optimizations: 7.065002638002625

After optimizations: 4.69557189499028

The main speedups come from using a set instead of a chain of boolean comparisons and passing the upper bound of the table into the binary search instead of calling length on the table for each run.

DerivedCombiningClass.txt contains the Canonical_Combining_Class field from UnicodeData.txt (see http://www.unicode.org/reports/tr44/#Canonical_Combining_Class_Values). This field is intended to be used for the collation algorithm.

wcwidth.py is currently assuming that characters are zero width combining characters if and only if they have a non-zero combining class. I think this is an invalid assumption. For example, characters that are enclosing marks (General Category = Me) all have a zero combining class, but they are also zero width combining characters.

I'm not sure what the standard way to determine zero width combining characters is. One possibility is to check for a General Category of Mn or Me, but I don't know if there are any exceptions to this. Also note that there are combining characters that do have a width (category Mc).

so that clicking them goes somewhere useful rather than opening the image in the browser.

Try it out at https://github.com/msabramo/wcwidth/tree/set_targets_on_badges

Support all versions of Unicode, using the UNICODE_VERSION environment variable, when defined, or, for non-shells, explicitly by passing argument unicode_version to the wcwidth family of functions.

A demonstration utility that determines the Terminal's Unicode Version is made available as a separate package, https://github.com/jquast/ucs-detect/ which contains a Problem and Solution statement, copied here:

Problem

Chinese, Japanese, Korean, and Emoticon characters are "double-wide", occupying 2 cells, instead of 1, and some other special characters are "zero-width".

Any terminal application that formats and displays these characters may have trouble determining how it will be displayed to the end-user.

This problem happens often, because the Unicode Consortium releases new versions of the Unicode Standard periodically, but the source code of libraries and applications are not updated at the same time, or at all!

Many languages and libraries continue to conform only to Unicode 5.0, which is the last version of wcwidth.c released by Markus Kuhn in 2007.

Solution

The most important factor is to determine: What version of unicode is the Terminal Emulator using?

This program, ucs-detect, is able to automatically detect the version of unicode that the connecting Terminal supports. The python wcwidth library supports all Unicode versions, 4.1.0 through 12.1.0 at time of this writing, and so it is able to select and match the correct return value for by using the given value of the UNICODE_VERSION environment variable.

We've updated created a new script for testing whether combining characters combine as expected, and found that many don't on our terminals. Should we consider the behavior on our terminals correct, or read more about all the offending classes of characters?

Any feedback on the code would be great too :)

On a fresh checkout:

$ python3 setup.py update
running update
data/ created.
retrieving http://www.unicode.org/Public/UNIDATA/EastAsianWidth.txt.
data/EastAsianWidth.txt saved.
parsing data/EastAsianWidth.txt ..
Traceback (most recent call last):
  File "setup.py", line 252, in <module>
    main()
  File "setup.py", line 248, in main
    cmdclass={'update': SetupUpdate},
  File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 64, in run
    self.do_east_asian()
  File "setup.py", line 72, in do_east_asian
    properties=(u'W', u'F',)
  File "setup.py", line 127, in _parse_east_asian
    uline = line.decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2: ordinal not in range(128)

The width for these are not correct:

• COMBINING CYRILLIC HUNDRED THOUSANDS SIGN
• COMBINING CYRILLIC HUNDRED MILLIONS SIGN

Test case:

import unicodedata
import math
import wcwidth

charnames = [u'COMBINING CYRILLIC HUNDRED THOUSANDS SIGN',
             u'COMBINING CYRILLIC HUNDRED MILLIONS SIGN',
             u'CIRCLED LATIN CAPITAL LETTER A',
             u'KANGXI RADICAL BIRD']

for charname in charnames:
    print(u'Character: {}'.format(charname))
    c = unicodedata.lookup(charname)
    cwidth = wcwidth.wcwidth(c)
    spacing = int(math.floor(2 - cwidth)) * u' '
    print(u"123456789")
    print(u"123{}{}6789".format(c, spacing))
    print(u"Char width: {}".format(cwidth, spacing))

This is printed with Python 2.7 and wcwidth 0.1.4:

Character: COMBINING CYRILLIC HUNDRED THOUSANDS SIGN
123456789
123҈ 6789
Char width: 1
Character: COMBINING CYRILLIC HUNDRED MILLIONS SIGN
123456789
123꙱ 6789
Char width: 1
Character: CIRCLED LATIN CAPITAL LETTER A
123456789
123Ⓐ 6789
Char width: 1
Character: KANGXI RADICAL BIRD
123456789
123⿃6789
Char width: 2

Related: https://github.com/dbcli/mycli/issues/149

Supersedes #59. I used Python 3.11.1 which only has unicodedata 14.0.0, so the name comments aren't perfect. Maybe someone could edit the script to use the comments from the input files directly.

Hi jquast/wcwidth!

This is a one-off automatically generated pull request from LGTM.com :robot:. You might have heard that we’ve integrated LGTM’s underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

pkg_resources is a little heavyweight, and has some rather glaring shortcomings for checking the metadata of an install Python package. importlib.metadata is contained in the standard library (as of 3.8), and provides a much nicer interface. Switch to using it, falling back to importlib_metadata if the Python version is too old.

• update tox.ini, requirements* files
  • use pip-compile with "requirements.in" files to generate "requirements.txt" files
  • use older testing tools for older python versions (requirements-tests38.in and requirements-tests39.in)
  • use python3.10 for most build steps
• bin/update-tables.py changes,
  • use https
  • use jinja2 (.j2) templating to generate code and documentation,
    • easier to support code generation of other languages
  • unreliable connection support using tenacity library to retry on connection timeout, etc.
  • optional --check-last-modified will compare last-modified for redownload
    • a unicode.org data file was only once updated for a botched release
• for wcwidth/table*.py files
  • uses new template system to generate new table data files for unicode version 14
  • when a description of unicode is single codepoint, describe only as 'A' instead of 'A..A'

Generated by python3.11 bin/update-tables.py. Using python 3.11 because only on python 3.11, unidecodedata.unidata_version is "14.0.0".

$ python3.11 -V
Python 3.11.0a5

Based on commit 53b0dd1.