There is currently an active effort for adding wasm as the build target to CPython in https://bugs.python.org/issue40280 by Christian Heimes. So it would be an ideal timing to review again our CPython patches and failing tests and re-evaluate what could be contributed upstream.

(Other things were already proposed upstream when possible).

I'm getting the following intermittent error running the develop version in Chrome, running from a web worker.

Chrome Version 89.0.4389.90 (Official Build) (64-bit) Windows 10 1909 build 18363.1379

pyodide.js:406 The cause of the fatal error was: RuntimeError: memory access out of bounds at PyArray_IntpFromIndexSequence (:wasm-function[991]:0x737a1) at PyArray_IntpConverter (:wasm-function[990]:0x73702) at byn$fpcast-emu$PyArray_IntpConverter (:wasm-function[3415]:0x15bfeb)

This is in accordance with discussion here: https://github.com/pyodide/matplotlib-pyodide/issues/6

Summary: The wasm_backend is refactored into 3 files. The browser_backend.py file contains code that is common to both wasm_backend and html5_canvas_backend. Both these backends inherit from the common functionality present in browser_backend.py and implement only the required differences.

Testing Infrastructure is also added for this new html5_canvas_backend.

Elevator Pitch

Enable file system persistence with IndexedDB (IDBFS), Node.js (NODEFS), blobs (WORKERFS), proxy (PROXYFS).

Motivation

Continuing the discussion from #328, this would allow a pyodide instance to work more naturally with persistent files.

Some external use cases:

• persisting files between user sessions
• sharing files between concurrent browser tabs

Some internal use cases:

• caching downloaded wheels and API responses?

Changes

• [x] add -l*fs.js to Makefile to restore all backends
  • ~~considered other backends, but as this has some impact on build time and size without a clear use case, I have not included them for now, but an eventual API should probably plan for supporting them explicitly~~
    • [x] #1689 enabled node, and i've already though about uses cases for WORKERFS
• [x] #1692 ~~hoist FS.mount, FS.mkdir and FS.sync and IDBFS up to the pyodide API?~~
  • rather than doing something fancy, this has the highest likelihood of working with other language runtimes
  • as discussed below, the preferred approach is to hoist FS to the pyodide namespace, pending a better name...
• [x] ~~add a convenience API in e.g. loadPyodide({mount: "/home/user1"})?~~
  • a lot of use cases would be covered by "simple" persistence
  • could also add it to sys.prefix
  • moved to https://github.com/pyodide/pyodide/issues/1715
• [x] tests?
  • haven't looked into what this will take...
• [x] docs?
  • depends on what features we end up landing...
  • focus on documenting notional API https://github.com/pyodide/pyodide/issues/1715

Findings

• you can make as many mount points as you want
• they all get synced into the MEMFS (which must exist) with FS.sync(true, (err)=>{...}) and likewise back out with FS.sync(false, (err) => {...})
  • the example guidance suggests sync-in at the start of the process and sync-out before exiting, but this is pretty limiting!

Dead-ends

The image above was gotten by hacking main.c, emulating a naive/broken approach for polling. A better path is being investigated!

  // Create and mount userfs immediately.
	EM_ASM({
		FS.mkdir('/home/jo');
		FS.mount(IDBFS, {}, '/home/jo');
    FS.syncfs(true, function (err) {
      err && console.error('error syncing FROM IndexedDB', err);
    });
    setInterval(function() {
        FS.syncfs(false, function (err) {
          err && console.error('error syncing TO IndexedDB', err);
      });
    }, 5000);
	});

What it says. Emscripten 2.0.9 upstream (non-fastcomp, compiles straight to wasm), plus patches for binaryen and emscripten so that dynamic linking and passing of function pointers between side modules and main modules works.

I'm part way through pull requests for binaryen and emscripten so that the patches there aren't needed.

All other patches to emscripten and binaryen have been dropped - I'm not sure if they were all things that are fixed in the new backend - none of them worked with new backend anyway

Closes #531 Closes #476

This is a first attempt at removing fpcast from the build. Quite a lot of tests pass now, including numpy, scipy.

Closes https://github.com/pyodide/pyodide/issues/1577

Add a pyodide.mountPackage("name", object) api that mounts object at name, so that import "name" imports object, and from name import field imports object.field as field.

For example, the current js import would then be installed by saying pyodide.mountPackage("js", globalThis) though we would keep the current behavior as the default. We could get rid of import js by saying pyodide.mountPackage("js", undefined).

relative to #955 , i want a sandboxed python runtime, i face two question is that :

1. how to limit the import function ? have any offical way todo this work ? i want limit the import with a denyList or a allowList, like the js i will deny it in a special pyodide.runPythonAsync call.
2. how to pip out the stdio ( redirect the print output from python ) ? now i use a way is make a patch in the sys.stdout from here, but seems like it will failure after i import the numpy package , so strange .

about the question (1) , i was try 2 way to do. ( sorry i never read the code of the pyodide , all the follow patch work are try on runtime console and then write to code . ) Summary: i run the pyodide in WebWorker to use the javascript sandboxie runtime to have a additional protection .

1. let the debuger hard to direct find the pyodide ref from root ( Worker's self or globalThis )

• in one side, i wrapper all the thing into a Typescript class , i named it class SClass ( mean : secure class)

use follow code to hide it

class SClass {
    constructor() {
        workerSelf.onmessage = this.onmessage.bind(this);
    }

    // .......

const workerSelf: typeof WorkerSelf = self as any;
workerSelf.aaaaaa = new SClass();
workerSelf.aaaaaa.init();
// https://github.com/Microsoft/TypeScript/issues/24587#issuecomment-412287117
// we have the `workerSelf.onmessage` on the ref link, the `onmessage` are `onmessage.bind(this)`,
//   so, the GC couldn't delete our object,
//   so, we can safe delete the direct ref from root object,
//   then, we implement the hidden all the ref target,
//   no one can access our object from debug console.
// @ts-ignore
// workerSelf[Symbol() as any] = workerSelf.aaaaaa;
delete workerSelf.aaaaaa;

now , no one can access the SClass from self .

• in other side, because the pyodide cannot be delete from globalThis, (it be set the non-configable flag), i cloned all the field and delete them from pyodide, only recover the pyodide._module to let it can use import js

    init() {
        return workerSelf.languagePluginLoader.then(async () => {
            // await workerSelf.pyodide.loadPackage(['numpy', 'pytz']);

            // delete root ref from self.root
            this.pyodide = clone(workerSelf.pyodide);
            // console.log(this.pyodide);
            // console.log(this.pyodide._module);
            // console.log(Object.getOwnPropertyNames(workerSelf.pyodide));
            Object.keys(workerSelf.pyodide).forEach(T => delete workerSelf.pyodide[T]);

            // delete packages lookup-table from pyodide
            // console.log('packages', workerSelf.pyodide._module.packages);
            // console.log('packages', this.pyodide._module.packages);
            workerSelf.pyodide._module = {};
            workerSelf.pyodide._module.packages = this.pyodide._module.packages;
            workerSelf.pyodide._module.packages = {
                dependencies: {},
                import_name_to_package_name: {},
            };
            // console.log('packages', workerSelf.pyodide._module.packages);

        })

now, other than the js package cannot be import, all the try will throw error .

.

but if i only want a allowList or denyList, i cannot comlete it use follow code to recover the pyodide._module.packages . it's not work.

            // recover selected packages to pyodide lookup-table
            const recoverPack = (pName: string) => {
                workerSelf.pyodide._module.packages.dependencies[pName] =
                    this.pyodide._module.packages.dependencies[pName];
                workerSelf.pyodide._module.packages.import_name_to_package_name[pName] =
                    this.pyodide._module.packages.import_name_to_package_name[pName];
            };
            const allowList = [
                'numpy',
            ];
            allowList.forEach(T => recoverPack(T));
            Object.getOwnPropertyNames(this.pyodide._module.packages.dependencies).forEach(T => recoverPack(T));
            // dont remember to recover the prototype
            if (allowList.length > 0) {
                // console.log('this.pyodide._module.packages.prototype',
                //     Object.getPrototypeOf(this.pyodide._module.packages));
                Object.setPrototypeOf(workerSelf.pyodide._module.packages,
                    Object.getPrototypeOf(this.pyodide._module.packages));
                Object.setPrototypeOf(workerSelf.pyodide._module.packages.dependencies,
                    Object.getPrototypeOf(this.pyodide._module.packages.dependencies));
                Object.setPrototypeOf(workerSelf.pyodide._module.packages.import_name_to_package_name,
                    Object.getPrototypeOf(this.pyodide._module.packages.import_name_to_package_name));
            }
            console.log('packages', workerSelf.pyodide._module.packages);

seems like some other important thing i deleted from the workerSelf.pyodide.

in the end, i remove above the attempt that delete thing from pyodide.

2. use pyodide.find_imports to check the code before run it . and simple not run it if find something not allowed.

see pyodide.find_imports

thsi is a easy way, but in the document i see some dangerous function can bypass this way. like pyodide.eval_code or the pyodide.open_url , and i not sure not have any other way can load code from remote or eval a string to code in runtime.

BTW: the cursor params of pyodide.get_completions API seems like not receive {row, col} as params, i dont known how to use it in a large section code. so, now i run a second WebWorker to run the Jedi to do the code complete task. but seems like the jedi cannot get information from a imported package , like numpy .

sorry for my knowledge, i'm good at C++/Typescript and software security/software architecture, the Python is in my weakness area.

remotePath-feature for Pyodide

This pull request implements the remotePath-feature. This feature makes Pyodide capable to fetch Python modules remotely from a set of given URLs. Both single Python py-files as well as Python-modules in sub-folders using __init__.py are supported.

In the demonstrated example below, there is a module called xyz located at https://phorward.info/tmp/pyodide/xyz/, which in turn imports the modules x and y.

Python console:

JavaScript console:

Any remotely fetched module is parsed for further imports which are imported as well, including pre-build packages from Pyodide. Due to this entirely changed behavior on how imports are resolved, packages from the pyodide packages.json take a higher precedence than remotely loaded packages.

The feature is already tested well in a re-implementation of a Python-based HTML5-web-app that previously was compiled into JavaScript using PyJS, but now can directly be interpreted (further details here)

[INITIAL MESSAGE OF THIS PULL REQUEST] Hi there!

This is a work-in-progress draft to resolve my problem I reported in #481 to allow importing Python files directly, which are not pre-packaged with pyodide, but directly served. So this pull request now makes it possible to make a

import a

and a.py is fetched from the same source as pyodide is served from (baseURL). In case a.py imports further modules, let it be b.py and c.py, these are also fetched. This is done until all required packages are fetched, including imports to available packages, as it is the case before.

Python files fetched from the server are stored into the browser cache using FS.writeFile, and then imported by Python by setting PYTHONPATH to /. I hope this won't be a security problem.

A demo of this feature is available here: https://phorward.info/tmp/pyodide/

• abc.html does the above import of a.py, b.py, c.py and the package html5
• game.html just does an import game, loads game.py and html5 and executes.
• console.html can also be used: just type import game or import a there to see what's happening (take a look into the JavaScript console for output)

This is just a first draft, I want to enhance it also to import module directories served. Is this useful also for others? Does anyone have suggestions for improvements?

I'm messing around with adding sqlite3, as mentioned in #345 .

It's not working. For some reason, makesetup seems to be choking on the added sqlite3 line in Setup.local and then it emits a garbage line in the generated Makefile, resulting in an error:

[email protected]:/src/cpython# make
cp Setup.local /src/cpython/build/3.7.0/Python-3.7.0/Modules/
cat pyconfig.undefs.h >> /src/cpython/build/3.7.0/Python-3.7.0/pyconfig.h
( \
	cp build/3.7.0/host/lib/python3.7/`/src/cpython/build/3.7.0/host/bin/python3 -c "import sysconfig; print(sysconfig._get_sysconfigdata_name())"`.py build/3.7.0/Python-3.7.0/Lib/_sysconfigdata__emscripten_.py; \
	cd /src/cpython/build/3.7.0/Python-3.7.0; \
	emmake make HOSTPYTHON=/src/cpython/build/3.7.0/host/bin/python3 HOSTPGEN=/src/cpython/build/3.7.0/host/bin/pgen CROSS_COMPILE=yes libpython3.7.a \
)
make[1]: Entering directory '/src/cpython/build/3.7.0/Python-3.7.0'
Makefile:271: *** missing separator.  Stop.
make[1]: Leaving directory '/src/cpython/build/3.7.0/Python-3.7.0'
Makefile:91: recipe for target '/src/cpython/build/3.7.0/Python-3.7.0/libpython3.7.a' failed
make: *** [/src/cpython/build/3.7.0/Python-3.7.0/libpython3.7.a] Error 2

Description

This adds boost-histogram, one of three key compiled libraries in the Scikit-HEP ecosystem (and like all three, useful beyond HEP). It was featured at SciPy 2020. The other two packages (iMinuit and Awkward) use setuptools CMake in the build process, so might be more challenging to add.

I wasn't able to run the build again after updating to 1.3.0, since on updating pyodide I now get:

Error building pyparsing. Printing build logs.
/src/.venv/bin/python: No module named wheel

when trying to build. ~~Pretty sure I didn't do that.~~ Actually, I did, as I made a venv to put pyodide-build in.

See #2167.

Checklists

• [x] Add a CHANGELOG entry
• [x] Add / update tests
• [x] Add new / update outdated documentation

This adds a mechanism to hide deprecated or private arguments from the documentation and applies it to errorCallbackDeprecated.

Also, the positional callback deprecation was not working correctly so I fixed it.

• [ ] Add / update tests

🐛 Bug

Chrome webview is crashing when statsmodels package is loaded in webworker on Android.

To Reproduce

1. Create webworker and Load pyodide and statsmodels package in your webapp
2. Run any python code in the webworker
3. Interact with the app
4. Chrome webview will crash randomly (not crashing at all when statsmodels is not loaded) - seems like a memory leak or issue with threading

Expected behavior

Should not crash

Environment

• Pyodide Version: 0.21.3
• Browser version: Chrome 108.0.5359.128 on Android 12
• Any other relevant information: Not happening on Chrome on Windows

Close: #3340

Checklists

• [x] Update version strings
• [x] Check which files are included in JS package:

DRY_RUN=1 ./deploy_to_npm.sh
Dry run: npm publish --tag 0.22.0
npm notice 
npm notice 📦  [email protected]
npm notice === Tarball Contents === 
npm notice 2.7kB   README.md       
npm notice 7.4kB   console.html    
npm notice 3.3kB   package.json    
npm notice 153.6kB pyodide_py.tar  
npm notice 5.0MB   pyodide.asm.data
npm notice 1.5MB   pyodide.asm.js  
npm notice 7.5MB   pyodide.asm.wasm
npm notice 24.7kB  pyodide.d.ts    
npm notice 21.4kB  pyodide.js      
npm notice 77.9kB  pyodide.js.map  
npm notice 20.9kB  pyodide.mjs     
npm notice 77.7kB  pyodide.mjs.map 
npm notice 8.2kB   repodata.json   
npm notice === Tarball Details === 
npm notice name:          pyodide                                 
npm notice version:       0.22.0                                  
npm notice filename:      pyodide-0.22.0.tgz                      
npm notice package size:  6.1 MB                                  
npm notice unpacked size: 14.5 MB                                 
npm notice shasum:        77794bd4faec626cac866380f3a04bd93f1ba026
npm notice integrity:     sha512-grdqjaUl3Lpq1[...]rR0cA7p4ALg5g==
npm notice total files:   13                                      
npm notice 
npm WARN This command requires you to be logged in to https://registry.npmjs.org/ (dry-run)
npm notice Publishing to https://registry.npmjs.org/ (dry-run)
+ [email protected]

• [x] Cleanup Changelog
• [x] Add list of contributors
• [x] Check documentation warnings #3391

/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:94: WARNING: more than one target found for 'any' cross-reference 'JsBuffer': could be :py:class:`pyodide.ffi.JsBuffer` or :py:class:`pyodide.JsBuffer`
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:94: WARNING: more than one target found for 'any' cross-reference 'JsPromise': could be :py:class:`pyodide.ffi.JsPromise` or :py:class:`pyodide.JsPromise`
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:124: WARNING: 'any' reference target not found: shorten
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:142: WARNING: 'any' reference target not found: captureThis
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:416: WARNING: 'any' reference target not found: loadPyodide
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:912: WARNING: 'any' reference target not found: JsProxy.to_string
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:912: WARNING: 'any' reference target not found: JsProxy.to_bytes
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:912: WARNING: 'any' reference target not found: JsProxy.to_memoryview
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:916: WARNING: 'any' reference target not found: JsProxy.to_file
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:916: WARNING: 'any' reference target not found: JsProxy.from_file
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/changelog.md:1661: WARNING: Failed to create a cross reference. A title or caption not found: micropip
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/release-notes/v0.17.0.md:105: WARNING: 'any' reference target not found: JsProxy.then
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/release-notes/v0.17.0.md:105: WARNING: 'any' reference target not found: JsProxy.catch
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/release-notes/v0.17.0.md:105: WARNING: 'any' reference target not found: JsProxy.finally_
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/release-notes/v0.17.0.md:148: WARNING: 'any' reference target not found: JsProxy.assign
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/project/release-notes/v0.17.0.md:148: WARNING: 'any' reference target not found: JsProxy.assign_to
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/sphinx_pyodide/README.md:18: WARNING: Could not lex literal_block as "pyodide". Highlighting skipped.
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/usage/type-conversions.md:470: WARNING: 'any' reference target not found: JsProxy.assign
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/usage/type-conversions.md:481: WARNING: 'any' reference target not found: JsProxy.assign
/home/docs/checkouts/readthedocs.org/user_builds/pyodide/checkouts/3389/docs/usage/type-conversions.md:481: WARNING: 'any' reference target not found: JsProxy.assign_to

• [ ] Release notes https://github.com/pyodide/pyodide-blog/pull/34
• [x] Fix cli tests #3390

Problem

Currently there seems to be no way to read parquet files through pandas with read_pandas as the engine needs to be specified. It can be either fastparquet or pyarrow. I tried to install it with pip, micropip or piplite but I'm getting an error that none of these packages has wheels.

Proposed Solution

Please bundle one of these engines directly to pyodide so we can import not only .csv, but also the much lighter on RAM files like parquet. Thanks

This is an attempt to resolve #3380. There seems to be three problems:

1. The usual business with fortran enums, string arguments and ftnlen for the transa parameter.
2. The aprod argument is declared to be a subroutine, which should have return type i32 but it in fact has return type f32, so we'd better say it's a function with return type real
3. For reasons I cannot comprehend, when I say the return type is real, f2c generates a declaration that says it has return type double. It feels like a bug in f2c but it is so major and obvious that it's hard to believe anyone could have missed it. Anyways I hack around it by adding a little bit more nonsense to f2cfixes.

This probably isn't quite right yet since running _svdp doesn't crash but it prints a very large number of the following warning:

Warning: call-back function cb_daprod_in___user__routines did not provide return value (index=0, type=float)

Checklists

• [ ] Add a CHANGELOG entry
• [x] Add / update tests

🐛 Bug

To Reproduce

import numpy as np
from scipy.sparse.linalg._svdp import _svdp

np.random.seed(0)
n, k = 70, 10
A = np.random.random((n, n))
_svdp(A, k, kmax=5*k)

Stack-trace with debug symbols:

Uncaught RuntimeError: null function or function signature mismatch
    at dgetu0_ (00055cfe:0x9d7e)
    at dlanbpro_ (00055cfe:0xd2b2)
    at dlansvd_irl__ (00055cfe:0xa19e)
    at dynCall (pyodide.asm.js:16146:40)
    at pyodide.asm.js:16154:18
    at stubs.<computed> (pyodide.asm.js:16567:33)
    at f2py_rout__dpropack_dlansvd_irl (00055cfe:0x75a7)
    at fortran_call (0003e3ba:0x6642)
    at _PyObject_Call (pyodide.asm.wasm:0x131064)
    at PyObject_Call (pyodide.asm.wasm:0x13109d)

Debugging with Chromium, the function is called with 7 arguments with a i32 return type:

call_indirect (param i32 i32 i32 i32 i32 i32 i32) (result i32)

but wasm functions has 8 arguments and returns f32

(func $cb_daprod_in___user__routines (;111;) (param $var0 i32) (param $var1 i32) (param $var2 i32) (param $var3 i32) (param $var4 i32) (param $var5 i32) (param $var6 i32) (param $var7 i32) (result f32)

Not sure where this comes from ...

Expected behavior

No error

Environment

• Pyodide Version: both stable (0.21.3) and latest
• Browser version: not relevant
• Any other relevant information:

Additional context

Bumped into this when looking at scipy.sparse test suite issues in https://github.com/lesteve/scipy-tests-pyodide/issues/3