Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Texting service to receive current air quality conditions and maps, powered by AirNow, Twilio, and AWS

The Air Quality Bot is generally available by texting a zip code (and optionally the word "map") to (415) 212-4229. The bot will respond with the late

Alex Laird 8 Oct 16, 2022
This solution helps you deploy Data Lake Infrastructure on AWS using CDK Pipelines.

CDK Pipelines for Data Lake Infrastructure Deployment This solution helps you deploy data lake infrastructure on AWS using CDK Pipelines. This is base

AWS Samples 66 Nov 23, 2022
Simple Instagram Login Link Generator

instagram-account-login Simple Instagram Login Link Generator Info Program generates instagram login links and you may get into someone´s thought the

Kevin 5 Dec 03, 2022
ignorant allows you to check if a phone number is used on different sites like snapchat, instagram.

Ignorant For BTC Donations : 1FHDM49QfZX6pJmhjLE5tB2K6CaTLMZpXZ ignorant does not alert the target phone number ignorant allows you to check if a phon

Palenath 513 Dec 31, 2022
TypeRig is a Python library aimed at simplifying the current FontLab API

TypeRig TypeRig is a Python library aimed at simplifying the current FontLab API while offering some additional functionality that is heavily biased t

Vassil Kateliev 41 Nov 02, 2022
Asynchronous Python Wrapper for the Ufile API

Ufile.io Asynchronous Python Wrapper for the Ufile API (Unofficial).

Gautam Kumar 16 Aug 31, 2022
Discord-RAID-Tool - Hacks/tools

How to use Python must be installed run install-config If you dont have python installed, download python 3.7.6 and make sure you click on the 'ADD TO

1 Jan 01, 2022
Python Wrapper for aztro - The Astrology API | Get Daily Horoscope 💫

PyAztro PyAztro is a client library for aztro written in Python. aztro provides horoscope info for sun signs such as Lucky Number, Lucky Color, Mood,

Sameer Kumar 30 Jan 08, 2023
Petpy is an easy-to-use and convenient Python wrapper for the Petfinder API.

Petpy is an easy-to-use and convenient Python wrapper for the Petfinder API. Includes methods for parsing output JSON into pandas DataFrames for easier data analysis

Aaron Schlegel 27 Nov 19, 2022
Work with the AWS IP address ranges in native Python.

Amazon Web Services (AWS) publishes its current IP address ranges in JSON format. Python v3 provides an ipaddress module in the standard library that allows you to create, manipulate, and perform ope

AWS Samples 9 Aug 25, 2022
Python Tool To Get The Date That Your Account Joined Instagram

Date-Joined-Insta Python Tool To Get The Date That Your Account Joined Instagram You Dont Need To Login Just Enter The UserName If Id Did Not Work Ins

A B D U L L A H . 1 Dec 21, 2021
A Discord token grabber executing in a Microsoft Document.

🦊 Rage 🦊 Rage is a tool written in Python3 allowing you to inject a Python3 complete Discord token grabber (Riot) script in a Microsoft Document usi

Billy 73 Nov 03, 2022
Spotify Top Lists - get the current top lists of a user from the Spotify API and display them in a Flask app

Spotify Top Lists This is a simple script that will get the current top lists of a user from the Spotify API and display them in a Flask app. Requirem

Yasin 0 Oct 16, 2022
A telegram to pyrogram json bot

Pyrogram-Json-Bot A telegram to pyrogram json bot Please fork this repository don't import code Made with Python3 (C) @FayasNoushad Copyright permissi

Fayas Noushad 11 Dec 20, 2022
Python Twitter API

Python Twitter Tools The Minimalist Twitter API for Python is a Python API for Twitter, everyone's favorite Web 2.0 Facebook-style status updater for

2.9k Dec 29, 2022
Discord bot that displays the current Swatch Internet Time (.beat) as a status.

Internet-Time-Display Discord bot that displays the current Swatch Internet Time (.beat) as a status. Visit the website! Add the bot to your server! A

2 Mar 15, 2022
Spacecrypto-bot - SpaceCrypto Bot Auto Clicker

SpaceCrypto Auto Clicker Bot Também fiz um para Luna Rush ( https://github.com/w

Walter Discher Cechinel 5 Feb 22, 2022
Ap lokit lokit

🎵 FANDA PROJECT 🎵 HAI AKU FANDA! Requirements 📝 FFmpeg NodeJS nodesource.com Python 3.8 or higher PyTgCalls MongoDB Get STRING_SESSION from below:

Fatur 2 Nov 18, 2021
Bender: A Markov Babbler Slack Bot

See the Digital Ocean tutorial for instructions on how to get the basic bot structure in place. Once you have that, set the gunicorn command to run as

Andrew Howard 1 Dec 04, 2021
Código python para automatizar a junção de arquivos CSV's e salva-los em uma pasta final de destino.

merge_csv Código python para automatizar a junção de arquivos CSV's e salva-los em uma pasta final de destino. Esse projeto é usado pra unir alguns ar

Welder Fariles 1 Jan 12, 2022