Macro recording and metaprogramming in Python

Overview

macro-kit

macro-kit is a package for efficient macro recording and metaprogramming in Python using abstract syntax tree (AST).

The design of AST in this package is strongly inspired by Julia metaprogramming. Similar methods are also implemented in builtin ast module but macro-kit is more focused on the macro generation and customization.

Installation

  • use pip
pip install macro-kit
  • from source
pip install git+https://github.com/hanjinliu/macro-kit

Examples

  1. Define a macro-recordable function
from macrokit import Macro, Expr, Symbol
macro = Macro()

@macro.record
def str_add(a, b):
    return str(a) + str(b)

val0 = str_add(1, 2)
val1 = str_add(val0, "xyz")
macro
[Out]
var0x24fdc2d1530 = str_add(1, 2)
var0x24fdc211df0 = str_add(var0x24fdc2d1530, 'xyz')
# substitute identifiers of variables
# var0x24fdc2d1530 -> x
macro.format([(val0, "x")]) 
[Out]
x = str_add(1, 2)
var0x24fdc211df0 = str_add(x, 'xyz')
# substitute to _dict["key"], or _dict.__getitem__("key")
expr = Expr(head="getitem", args=[Symbol("_dict"), "key"])
macro.format([(val0, expr)])
[Out]
_dict['key'] = str_add(1, 2)
var0x24fdc211df0 = str_add(_dict['key'], 'xyz')
  1. Record class
macro = Macro()

@macro.record
class C:
    def __init__(self, val: int):
        self.value = val
    
    @property
    def value(self):
        return self._value
    
    @value.setter
    def value(self, new_value: int):
        if not isinstance(new_value, int):
            raise TypeError("new_value must be an integer.")
        self._value = new_value
    
    def show(self):
        print(self._value)

c = C(1)
c.value = 5
c.value = -10
c.show()
[Out]
-10
macro.format([(c, "ins")])
[Out]
ins = C(1)
ins.value = -10     # setattr (and setitem) will not be recorded in duplicate
var0x7ffed09d2cd8 = ins.show()
macro.eval({"C": C})
[Out]
-10
  1. Record module
import numpy as np
macro = Macro()
np = macro.record(np) # macro-recordable numpy

arr = np.random.random(30)
mean = np.mean(arr)

macro
[Out]
var0x2a0a2864090 = numpy.random.random(30)
var0x2a0a40daef0 = numpy.mean(var0x2a0a2864090)
from dask import array as da
dask_macro = macro.format([(np, "da")])
dask_macro
[Out]
var0x2a0a2864090 = da.random.random(30)
var0x2a0a40daef0 = da.mean(var0x2a0a2864090)
output = {}
dask_macro.eval({"da": da}, output)
output
[Out]
{:da: 
   
    ,
 :var0x2a0a2864090: dask.array
    
     ,
 :var0x2a0a40daef0: dask.array
     
      }

     
    
   
You might also like...
Find dependent python scripts of a python script in a project directory.

Find dependent python scripts of a python script in a project directory.

SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

An OData v4 query parser and transpiler for Python

odata-query is a library that parses OData v4 filter strings, and can convert them to other forms such as Django Queries, SQLAlchemy Queries, or just plain SQL.

Python program to do with percentages and chances, random generation.

Chances and Percentages Python program to do with percentages and chances, random generation. What is this? This small program will generate a list wi

A Python library for reading, writing and visualizing the OMEGA Format
A Python library for reading, writing and visualizing the OMEGA Format

A Python library for reading, writing and visualizing the OMEGA Format, targeted towards storing reference and perception data in the automotive context on an object list basis with a focus on an urban use case.

A simple and easy to use Spam Bot made in Python!

This is a simple spam bot made in python. You can use to to spam anyone with anything on any platform.

RapidFuzz is a fast string matching library for Python and C++

RapidFuzz is a fast string matching library for Python and C++, which is using the string similarity calculations from FuzzyWuzzy

pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.
pydsinternals - A Python native library containing necessary classes, functions and structures to interact with Windows Active Directory.

pydsinternals - Directory Services Internals Library A Python native library containing necessary classes, functions and structures to interact with W

A Python package implementing various colour checker detection algorithms and related utilities.
A Python package implementing various colour checker detection algorithms and related utilities.

A Python package implementing various colour checker detection algorithms and related utilities.

Releases(v0.3.8)
  • v0.3.8(Mar 16, 2022)

    Bug fixes

    • Length 1 tuple (1,) and length 0 set set() was not recorded correctly.
    • Import and __all__ did not match.
    • The __getitem__ was not correctly defined when a slice is used as indices.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.7(Jan 22, 2022)

  • v0.3.6(Jan 13, 2022)

    Changes

    • Mock object for easier Expr construction. Instead of writing like Expr("getattr", [Expr("getattr", ...), b]), you can use m = Mock("m") and create Expr object by m.a(0, "s").
    • Full support of Python 3.7, and add tox test.

    Bug Fixes

    • The __eq__ method will return False instead of raise Exception to make macro recording safer.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.4(Dec 12, 2021)

  • v0.3.3(Dec 10, 2021)

    New Features

    • New ast type supports: break, continue and raise.
    • Modules will be registered to Symbol whenever it is converted into a Symbol so that you'll not have to pass module object when you call eval. For instance, instead of macro.eval({"np", np}) you only have to call macro.eval().

    Changes

    • The "type" argument in Symbol is removed because it is never used.

    Bug Fixes

    • Hash of Symbol was not completely safe.
    • mModule did not return constant values correctly (such as np.pi).
    Source code(tar.gz)
    Source code(zip)
  • v0.3.2(Dec 8, 2021)

    New Features

    • register_type and Symbol.register_type can be used as a decorator.
    • Symbol.var("x") can create unique variable x.
    • Many missing AST types are supported, such as % (mod operation), += (in-place operations) and unary operations.
    • Validators are implemented in Expr to assert correct grammar for certain header (although they are not perfect). For instance, Expr("getattr", [Symbol("x"), "name"]) used to return :(x.'name') but now it returns :(x.name).
    • Many special methods, such as __int__ and __hash__, are correctly converted to int(X) in macro.
    • Macro can used as a field of a class. Unique Macro object will be created for different instances.

    Changes

    • Attribute _last_setval is moved to Macro.
    • Method type definition is changed to delayed definition.
    • Type mapping of symbol now uses dictionary as much as possible.
    • The type keyword argument in Symbol constructor is deleted because it was not used.

    Bug Fixes

    • Boolean operations (and and or) could not be parsed.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Dec 5, 2021)

    New Features

    • Support function definition, if, for, and many other expressions.
    • Macro.callbacks attribute for expression-added callbacks.
    • flags keyword argument to specify what kind of expressions will be recorded.

    Changes

    • Macro inherits Expr but head is fixed to Head.block to simplify module.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Oct 24, 2021)

    New Features

    • Support string parsing using ast.
    • optimize method of Macro can remove unused returned variables. It makes macro looks better.

    Changes

    • Head is different from that in v0.1.0, thus may be inconsistent.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Oct 23, 2021)

Owner
Department of Biological Sciences, School of Science, The University of Tokyo. My major is biophysics, especially microtubule and motor proteins.
Control-Alt-Delete - Help Tux Escape Beastie's Jail!

Control-Alt-Delete Help Tux escape Beastie's jail by completing the following challenges! Challenges Challenge 00: Drinks: Tux needs to drink less. Ch

NDLUG 8 Oct 31, 2021
This is a tool to calculate a resulting color of the alpha blending process.

blec: alpha blending calculator This is a tool to calculate a resulting color of the alpha blending process. A gamma correction is enabled and the def

Igor Mikushkin 12 Sep 07, 2022
Utility to add/remove licenses to/from source files

Utility to add/remove licenses to/from source files. Supports processing any combination of globs, files, and directories (recurse). Pruning options allow skipping non-licensing files.

Eduardo Ponce Mojica 2 Jan 29, 2022
A plugin to simplify creating multi-page Dash apps

Multi-Page Dash App Plugin A plugin to simplify creating multi-page Dash apps. This is a preview of functionality that will of Dash 2.1. Background Th

Plotly 19 Dec 09, 2022
Conveniently measures the time of your loops, contexts and functions.

Conveniently measures the time of your loops, contexts and functions.

Maciej J Mikulski 79 Nov 15, 2022
A thing to simplify listening for PG notifications with asyncpg

asyncpg-listen This library simplifies usage of listen/notify with asyncpg: Handles loss of a connection Simplifies notifications processing from mult

ANNA 18 Dec 23, 2022
.bvh to .mcfunction file converter.

bvh-to-mcf .bvh file to .mcfunction converter

Hanmin Kim 28 Nov 21, 2022
Edit SRT files to delay subtitle time-stamps.

subtitle-delay A program written in Python that directly edits SRT file to delay the subtitles. Features: Will throw an error if delaying with negativ

8 Jul 17, 2022
extract gene TSS/TES site form gencode/ensembl/gencode database GTF file and export bed format file.

GetTsite python Package extract gene TSS/TES site form gencode/ensembl/gencode database GTF file and export bed format file. Install $ pip install Get

laojunjun 7 Nov 21, 2022
a demo show how to dump lldb info to ida.

用一个demo来聊聊动态trace 这个仓库能做什么? 帮助理解动态trace的思想。仓库内的demo,可操作,可实践。 动态trace核心思想: 动态记录一个函数内每一条指令的执行中产生的信息,并导入IDA,用来弥补IDA等静态分析工具的不足。 反编译看一下 先clone仓库,把hellolldb

25 Nov 28, 2022
Aurin - A quick AUR installer for Arch Linux. Install packages from AUR website in a click.

Aurin - A quick AUR installer for Arch Linux. Install packages from AUR website in a click.

Suleman 51 Nov 04, 2022
VerSign: Easy Signature Verification in Python

VerSign: Easy Signature Verification in Python versign is a small Python package which can be used to perform verification of offline signatures. It a

Muhammad Saif Ullah Khan 3 Dec 01, 2022
Kanye West Lyrics Generator

aikanye Kanye West Lyrics Generator Python script for generating Kanye West lyrics Put kanye.txt in the same folder as the python script and run "pyth

4 Jan 21, 2022
Python tool to check a web applications compliance with OWASP HTTP response headers best practices

Check Your Head A quick and easy way to check a web applications response headers!

Zak 6 Nov 09, 2021
Simple collection of GTPS Flood in Python.

GTPS Flood Simple collection of GTPS Flood in Python. NOTE Give me credit if you use this source, don't trade/sell this tool, And USE AT YOUR OWN RISK

PhynX 6 Dec 07, 2021
ecowater-softner is a Python library for collecting information from Ecowater water softeners.

Ecowater Softner ecowater-softner is a Python library for collecting information from Ecowater water softeners. Installation Use the package manager p

6 Dec 08, 2022
Personal Toolbox Package

Jammy (Jam) A personal toolbox by Qsh.zh. Usage setup For core package, run pip install jammy To access functions in bin git clone https://gitlab.com/

5 Sep 16, 2022
Hot reloading for Python

Hot reloading for Python

Olivier Breuleux 769 Jan 03, 2023
A simple tool to move and rename Nvidia Share recordings to a more sensible format.

A simple tool to move and rename Nvidia Share recordings to a more sensible format.

Jasper Rebane 8 Dec 23, 2022
Dynamic key remapper for Wayland Window System, especially for Sway

wayremap Dynamic keyboard remapper for Wayland. It works on both X Window Manager and Wayland, but focused on Wayland as it intercepts evdev input and

Kay Gosho 50 Nov 29, 2022