As @hugovk mentioned in PR #310, we have a maintainer issue with tablib (similarly with records).

Facts:

• @kennethreitz is Development Lead (592 commits)
• @iurisilvio is Core Contributor (14 commits)
• There are 20 authors mentioned in AUTHORS.
  • @lbeltrame and @durden 13 commits
  • @f4nt, @claudep and @msabramo 5 commits or more
  • 10 other with 3 commits
  • 6 others with 2 commits
  • 47 other with 1 commit
• Issues: 107 closed, 68 open
  • last 25 open issues are younger than 12 months
• PR 135 closed, 18 open
  • 16 open PRs are younger than 12 months
  • good number of those PRs seem to fix open issues
  • last PR accepted is around 7 month old
• downloads/pip installs
  • 48 thousands in last 30 days (see table in PR #310)
• last version is 0.12.1 released on Sep 1, 2017

To summarize:

• There is clear interest on tablib from users as well as from contributors.
• Maintenance got somehow stuck since September last year.

Proposal for next actions:

• @kennethreitz , @iurisilvio are you willing to accept another maintainer or even hand over to someone?
• others: is there anyone feeling like becoming maintainer?

This issue tracks the implementation of the Jazzband guidelines for the project tablib

It was initiated by @vinayak-mehta who was automatically assigned in addition to the Jazzband roadies.

See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.

Feel free to ping a Jazzband roadie if you have any question.

TODOs

• [x] Fix all links in the docs (and README file etc) from old to new repo
• [x] Add the Jazzband badge to the README file
• [x] Add the Jazzband contributing guideline to the CONTRIBUTING.md or CONTRIBUTING.rst file
• [x] Check if continuous testing works (e.g. Travis-CI, CircleCI, AppVeyor, etc)
• [x] Check if test coverage services work (e.g. Coveralls, Codecov, etc)
• [x] Add jazzband account to PyPI project as maintainer role (e.g. URL: https://pypi.org/manage/project/tablib/collaboration/)
• [x] Add jazzband-bot as maintainer to the Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/tablib/users/)
• [x] Add incoming GitHub webhook integration to Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/tablib/integrations/)
• [x] Fix project URL in GitHub project description
• [x] Review project if other services are used and port them to Jazzband
• [x] Decide who is project lead for the project (if at all)
• [x] Set up CI for Jazzband project releases if needed and open ticket if yes
• [x] Remove the call for financial help that is currently included in the README
• [x] Figure out what to do with the develop branch that has some changes but is outdated. @claudep @hugovk

Project details

I tried to add data to the tablib Dataset. I use django import export to do so. Everything goes well with the data until it is in the dataset. Once it is added and I try to read the data I get the following error.

UnicodeEncodeError encountred while trying to read file: 'ascii' codec can't encode character u'\xc6' in position 241: ordinal not in range(128)

It is about the text Ævar. This name will fail when you try to read the data from the Dataset.

While tablib is MIT-licensed, there are several potential provenance and license issues with Oo, XLS and XLSX formats that tablib embeds. I have collected some of these potential issues here. This is at best ... byzantine.

It feels a tad overwhelming: I am just raising the issue and I would like to help to fix this.

There are some things to consider with regards to implementations of Excel formats read and write (in xlrt, xlwt and openpyxl) as there may be issues with components that support XLS or XLSX formats in general.

See the discussion on these:

• https://github.com/python-excel/xlrd/issues/32
• https://github.com/Niggler/js-xls/issues/1
• https://github.com/stephen-hardy/xlsx.js/issues/8
• https://github.com/stephen-hardy/xlsx.js/blob/master/LICENSE.txt

See also Apache guidelines for POI: http://poi.apache.org/guidelines.html

openpxyl

https://bitbucket.org/ericgazoni/openpyxl/ is reported as being derived from PHPExcel which is LGPL-licensed at https://github.com/PHPOffice/PHPExcel but openpyxl is not LGPL but MIT-licensed. If this is really derived then there is a possible issue as the license may be that of the original not of the derivative. It also contains code from other origins such as http://lethain.com/entry/2009/jan/22/handling-very-large-csv-and-xml-files-in-python/ PHPExcel in turns claims to contain code from these origins:

• PEAR::SpreadsheetExcelWriter (http://pear.php.net/package/Spreadsheet_Excel_Writer also LGPL
• JavaScript Excel Formula Parser (http://ewbi.blogs.com/develops/2007/03/excel_formula_p.html )
• PHP-ExcelReader (http://sourceforge.net/projects/phpexcelreader itself also based on http://www.andykhan.com/excelread/ or jexcelapi see below )
• FPDF (http://www.fpdf.org/) (probably not relevant here)
• TCPDF (http://www.tcpdf.org/) (probably not relevant here)
• PHP port of polyfit (http://www.phpmath.com/home?op=cat&cid=17) (probably not relevant here)
• PHP port of JAMA matrix library (http://www.phpmath.com/build03/JAMA/docs/) (probably not relevant here)

XLRD

Uses a combo of a BSD license and a BSD original license. XLRD itself is based on a BSD-original-licensed component written in C called xlreader at http://www.giffin.org/xlreader.php released circa fall 2001. This is turn is based on a LGPL 2.0-licensed component at http://www.andykhan.com/excelread/ released circa fall 2001 at http://web.archive.org/web/20011117133717/http://www.andykhan.com/excelread/ . ExcelRead later became jExcelAPI http://jexcelapi.sourceforge.net/, still LGPL-licensed.

The xlrd license history is confusing at best, because if the C xlreader is really a derivative of the Java LGPL-licensed excelread, then its BSD original license might be moot, as the license may be that of the original not of the derivative. If this was just used for ideas, then only the BSD original license issue may stand (with its extensive advertising clause being an issue and making it not GPL-compatible among other things and a pain with MIT or Apache-licensed code).

XLWT

Uses a combo of bsd-modified and bsd-original-license at https://github.com/python-excel/xlwt/blob/master/licences.py . It started as a fork of PyExcelerator by Roman V. Kiseliov that was also BSD-original at http://sourceforge.net/projects/pyexcelerator/ . Pyexcelerator in turn has this notice: http://ftp-master.metadata.debian.org/changelogs//main/p/pyexcelerator/pyexcelerator_0.6.4.1-1_copyright . And per https://github.com/python-excel/xlwt/commit/be268c8a01f1738c39aa3486802972ae6c7f34d1 and https://github.com/python-excel/xlwt/blob/master/licences.py ... this is a partial port of pyXLWriter http://pyxlwriter.sourceforge.net/ LGPL-2.0-licensed, but itself claiming to be a port of a Perl module by John McNamara (Perl Spreadsheet::WriteExcel which was Perl-licensed.)

The combo of BSD, BSD original, Perl and LGPL may not mesh well. The Perl issue may have been addressed here: https://launchpad.net/ubuntu/raring/+source/xlwt/+copyright

Other components:

• ODF/OpenOffice : this is LGPL-licensed. The license text is missing.
• omnijson : has code derived from simplejson and notice/license text is missing; it also includes ordereddict from Raymond Hettinger.
• unicodecsv (v0.8.0) from https://pypi.python.org/pypi/unicodecsv/ https://github.com/jdunck/python-unicodecsv is BSD-licensed and the notice is missing
• PyYAML : '3.09' from http://pyyaml.org/download/pyyaml/PyYAML-3.09.tar.gz is MIT licensed and the notice/license is missing
• markup is from http://markup.sourceforge.net/ is Public Domain

See also: http://ftp-master.metadata.debian.org/changelogs//main/p/python-tablib/python-tablib_0.9.11-1_copyright

Added import_set to formats/_xls.py using xlrd Added import_set and import_book to formats/_xlsx.py using openpyxl Also added a sheets() method to Databook, without that I couldn't find a way to access the sheets in a book.

I've removed the vendorized versions of xlwt, xlrd, openpyxl, ominjson, unicodecsv, and pyyaml and replaced them by installing them in setup.py.

If there's a reason this hasn't been done already, I'd love to know and see if there's anything we can do to update the vendorized versions to newer releases.

Edit: Ok, I now see it fails on python 3.2. Not sure how important that is, but if that's why this hasn't been done this can be closed I guess.

Trying to install tablib on python 3 fails:

Alexanders-MacBook-Pro:openstack-quantumclient alex_gaynor$ mktmpenv -p `which python3.3`
Running virtualenv with interpreter /usr/local/bin/python3.3
Using base prefix '/usr/local/Cellar/python3/3.3.2/Frameworks/Python.framework/Versions/3.3'
New python executable in fa1de240e7969a2b/bin/python3.3
Also creating executable in fa1de240e7969a2b/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
Downloading/unpacking wheel
  Using download cache from /Users/alex_gaynor/.pip/cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2Fw%2Fwheel%2Fwheel-0.21.0-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel
Cleaning up...
This is a temporary environment. It will be deleted when you run 'deactivate'.
(fa1de240e7969a2b)Alexanders-MacBook-Pro:fa1de240e7969a2b alex_gaynor$ pip install tablib
Downloading/unpacking tablib
  Using download cache from /Users/alex_gaynor/.pip/cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Ft%2Ftablib%2Ftablib-0.9.11.tar.gz
  Running setup.py egg_info for package tablib
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/Users/alex_gaynor/.virtualenvs/fa1de240e7969a2b/build/tablib/setup.py", line 7, in <module>
        import tablib
      File "./tablib/__init__.py", line 3, in <module>
        from tablib.core import (
      File "./tablib/core.py", line 15, in <module>
        from tablib import formats
      File "./tablib/formats/__init__.py", line 6, in <module>
        from . import _csv as csv
      File "./tablib/formats/_csv.py", line 6, in <module>
        from tablib.compat import is_py3, csv, StringIO
      File "./tablib/compat.py", line 25, in <module>
        import tablib.packages.xlwt3 as xlwt
      File "./tablib/packages/xlwt3/__init__.py", line 5, in <module>
        from .Row import Row
      File "./tablib/packages/xlwt3/Row.py", line 7, in <module>
        from . import ExcelFormula
      File "./tablib/packages/xlwt3/ExcelFormula.py", line 6, in <module>
        class Formula(object):
    ValueError: '__init__' in __slots__ conflicts with class variable
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/Users/alex_gaynor/.virtualenvs/fa1de240e7969a2b/build/tablib/setup.py", line 7, in <module>

    import tablib

  File "./tablib/__init__.py", line 3, in <module>

    from tablib.core import (

  File "./tablib/core.py", line 15, in <module>

    from tablib import formats

  File "./tablib/formats/__init__.py", line 6, in <module>

    from . import _csv as csv

  File "./tablib/formats/_csv.py", line 6, in <module>

    from tablib.compat import is_py3, csv, StringIO

  File "./tablib/compat.py", line 25, in <module>

    import tablib.packages.xlwt3 as xlwt

  File "./tablib/packages/xlwt3/__init__.py", line 5, in <module>

    from .Row import Row

  File "./tablib/packages/xlwt3/Row.py", line 7, in <module>

    from . import ExcelFormula

  File "./tablib/packages/xlwt3/ExcelFormula.py", line 6, in <module>

    class Formula(object):

ValueError: '__init__' in __slots__ conflicts with class variable

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /Users/alex_gaynor/.virtualenvs/fa1de240e7969a2b/build/tablib
Storing complete log in /Users/alex_gaynor/.pip/pip.log

Trying to determine the best way to handle this. Tablib is keeping them correctly. However, openpyxl needs an alternate syntax if you want to assign a literal.

An example is if I have a column of zip codes or social security numbers that I do not want to be interpreted as numeric.

Hi,

Concerning the issue #251 that I closed some minutes ago, I update the docs to clarify the usage of filters to generate filters type (or, and).

Regards,

Some dependencies need updating to support Python 3.7, including pandas.

There's no release of pandas which supports both 3.4 and 3.7.

So let's drop 3.4, it's very nearly EOL, on 2019-03-16.

https://en.wikipedia.org/wiki/CPython#Version_history

Right now, the list of formats is more or less hardcoded into the formats.available tuple, and it's not straightforward to extend (at least not to me).

It would be great if you could subclass an abstract Formatter class, and then either have it magically register (using some kind of metaclass magic, I suppose) with the available formats list or introduce a function and/or class decorator to explicitly register the new format.

Is that something you'd be interested in working on in the near term? I'd be happy to contribute some of my own time for it.

yaml file:

mylist:
  - 1
  - 2
  - 3

with open("config/aa.yaml","rb")as f:
     data.load(f,'yaml')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [21], in <module>
      1 with open("config/aa.yaml","rb")as f:
----> 2      data.load(f,'yaml')

File ~/anaconda3/envs/py3/lib/python3.9/site-packages/tablib/core.py:414, in Dataset.load(self, in_stream, format, **kwargs)
    411 if not import_set:
    412     raise UnsupportedFormat(f'Format {format} cannot be imported.')
--> 414 fmt.import_set(self, stream, **kwargs)
    415 return self

File ~/anaconda3/envs/py3/lib/python3.9/site-packages/tablib/formats/_yaml.py:32, in YAMLFormat.import_set(cls, dset, in_stream)
     29 """Returns dataset from YAML stream."""
     31 dset.wipe()
---> 32 dset.dict = yaml.safe_load(in_stream)

File ~/anaconda3/envs/py3/lib/python3.9/site-packages/tablib/core.py:341, in Dataset._set_dict(self, pickle)
    338     return
    340 # if list of rows
--> 341 if isinstance(pickle[0], list):
    342     self.wipe()
    343     for row in pickle:

KeyError: 0

I'm trying to load an xls file however it fails with the following error

with open("xls_file", "rb") as fh: 
    imported_data = Databook().load(fh, "xls")

Traceback (most recent call last):
  File ".\test.py", line 5, in <module>
    imported_data = Databook().load(fh, "xls")
  File "lib\site-packages\tablib\core.py", line 872, in load
    fmt.import_book(self, stream, **kwargs)
  File "lib\site-packages\tablib\formats\_xls.py", line 103, in import_book
    xls_book = xlrd.open_workbook(file_contents=in_stream)
  File "lib\site-packages\xlrd\__init__.py", line 166, in open_workbook
    file_format = inspect_format(filename, file_contents)
  File "lib\site-packages\xlrd\__init__.py", line 57, in inspect_format
    peek = content[:PEEK_SIZE]
TypeError: '_io.BufferedReader' object is not subscriptable

This PR allows user to specify column width in XLXS format. For it has a default value, which is not suitable for long values in dataset. As a result, it is not convenient to look these tables. Also, it eliminates questions like #318 in future

I didn't find tests for exporting, so omitted this. Let me know if they are required

A Dataset from an Excel file will store None for any empty cell. In CSV the convention is to not distinguish between ,, and ,"", and treat all empty cells as empty strings. It would be nice to add a feature like that in pandas.read_csv() where ,, is treated as NaN, but None here for symmetry with Excel. I know that toggling the na_filter argument in read_csv() is one way to govern this in pandas.

Anyways, thanks for the useful tool.

From the docs, I think this is working as designed, but wanted to check. There's a decent chance I'm doing something wrong.

File auto-detection becomes less useful to library consumer like me if I have to check the type of file I'm using and take different branches for it. It's still great to get a normalized representation of various formats . . . thanks for that.

If you open an Excel file (.xlsx) with just mode r, you get an error zipfile.BadZipFile: File is not a zip file. Doesn't matter if I specify the format in the second parameter to load() when calling open() myself. You must specify mode rb.

This in on Ubuntu 20.04, python3.9, openpyxl 3.0.9

# zipfile.BadZipFile: File is not a zip file
with open("mySheet.xlsx", "r") as fh:
    data.load(fh)

# OK
with open("mySheet.xlsx", "rb") as fh:
    data.load(fh)

If you pass the file path, it cannot determine the file type. If you specify the type, it doesn't open binary.

# zipfile.BadZipFile: File is not a zip file
data.load("mySheet.xlsx", "xlsx")

# tablib.exceptions.UnsupportedFormat: Tablib has no format 'None' or it is not registered.
data.load("mySheet.xlsx")

I think the issue in data.load("mySheet.xlsx", "xlsx") is because core.normalize_input() converts the path given as a str to a StringIO. And when openpyxl passes it to zipfile, zipfile would call open() with mode rb if it was still a string. I'm not sure zipfile is ready for StringIO at all. It hits this code block which doesn't seem to expect a StringIO: https://github.com/python/cpython/blob/main/Lib/zipfile.py#L1273

Now if I shift to CSV, I can't use mode rb. It does give instructions in the exception if you give it the file type hint. And it does get it right if you leave the call to open() to tablib and give it the format hint.

# tablib.exceptions.UnsupportedFormat: Tablib has no format 'None' or it is not registered.
with open("myCsv.csv", "rb") as fh:
    data.load(fh)

# _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
with open("myCsv.csv", "rb") as fh:
    data.load(fh, "csv")

# tablib.exceptions.UnsupportedFormat: Tablib has no format 'None' or it is not registered.
data.load("myCsv.csv")

# OK
data.load("myCsv.csv", "csv")

I'd like to be able to create a Dataset from plain python dicts. I read through just about everything in the docs and some of the code, and can't seem to find a way to do this directly.

What I'd like to be able to do is something like:

source_data = [
     {'column_1': 'value_1A',  'column_2':  'value_2A'},
     {'column_1': 'value_1B',  'column_2':  'value_2B'},
]
data = Dataset().load(*source_data)

Or maybe:

data = import_set(source_data, format='dict')

Current workarounds:

• json.dumps the data first before loading
• Register a custom format class
• Separately add headers and row values:

  data = DataSet()
  data.headers = source_data[0].keys()
  data.extend([item.values() for item in source_data])
  

So I guess my questions are:

• Is there currently a better way to do this?
• If not, would you be interested in a PR for this?