This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers. Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue. I'll open a PR within a few hours, please be patient!

Thanks again for this package!

I'm currently trying to build my recently released package Anneal.jl and this error is showing up during precompilation. One must run ] precompile to see the message below:

ERROR: LoadError: InitError: failed process: Process(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'C:\Users\pedroxavier\.julia\environments\v1.7\.CondaPkg\env' --override-channels --no-channel-priority pip 'python >=3.5,<4' 'python >=3.7,<4' -c anaconda -c conda-forge`, ProcessExited(3221226505)) [3221226505]

A way to fix this is to plug the command between backticks in a Julia run call.

run(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'C:\Users\pedroxavier\.julia\environments\v1.7\.CondaPkg\env' --override-channels --no-channel-priority pip 'python >=3.5,<4' 'python >=3.7,<4' -c anaconda -c conda-forge`)

This way, micro mamba gets installed and everything works fine. The problem is that my package won't precompile directly without this trick. I guess it has something to do with privileges, chmod or something like that.

To reproduce this issue, it is enough to run

] add Anneal
] precompile

Anneal is available in the General Registry, so the first line will error after a little while and ] precompile will reveal the message above.

I'm trying to implement a Python class using PythonCall.pyclass.

The following works as expected:

using PythonCall

Person = pyclass("Person", (); print_random = print ∘ rand)
@pyexec (Person = Person) => """
p1 = Person()
p1.print_random() 
"""
# prints a random number

But any attempt to include methods that accept arguments referencing the instance of the class itself:

Person = pyclass("Person", (); __init__ = (self) -> (self.state = 0))
@pyexec (Person = Person) => """
p1 = Person()
"""

fails:

ERROR: Python: TypeError: Julia: MethodError: no method matching (::var"#5#6")()
Closest candidates are:
  (::var"#5#6")(!Matched::Any) at REPL[15]:1
Python stacktrace:
 [1] __call__
   @ /home/yakir/.julia/packages/PythonCall/Z6DIG/src/jlwrap/any.jl:167:30
 [2] <module>
   @ REPL[16]:1:1
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/Z6DIG/src/err.jl:10 [inlined]
 [3] pycallargs(f::Py, args::Py)
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:153
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:171
 [5] pycall
   @ ~/.julia/packages/PythonCall/Z6DIG/src/abstract/object.jl:161 [inlined]
 [6] #_#11
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [7] Py
   @ ~/.julia/packages/PythonCall/Z6DIG/src/Py.jl:330 [inlined]
 [8] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::NamedTuple{(:Person,), Tuple{Py}})
   @ PythonCall ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:98
 [9] top-level scope
   @ ~/.julia/packages/PythonCall/Z6DIG/src/concrete/code.jl:146

More context

I'm trying to use the picamera Python module and process each frame in Julia. The following MWE however segfaults:

ENV["JULIA_PYTHONCALL_EXE"] = "/usr/bin/python3" # I have to include this due to https://github.com/cjdoris/PythonCall.jl/issues/120

using PythonCall

SideEffect = pyclass("SideEffect", (), 
                   __init__ = () -> nothing,
                   write = s -> println(rand()),
                   flush = () -> print("done")
                  )

@pyexec (SideEffect = SideEffect) => """
import picamera
with picamera.PiCamera() as camera:
    camera.start_recording(SideEffect(), format='h264') # start_recording calls SideEffect.write for each frame and SideEffect.flush at the end
    camera.wait_recording(5)
    camera.stop_recording()
"""

It would be useful to have an example of "Fast non-copying conversion of numeric arrays" somewhere in the documentation. For example, I've never been able to figure out how to do this. I imagine some others have not either.

More specifically, I'd like to pass a numpy array from Python to a Julia function where the Julia function receives the data as an Array with no copying.

Hi, I can reproduce a hang (no cpu activity) with the following code on 1.8rc1, 1.8rc3 & 1.9dev. I'm running on Ubuntu 21.10, but I can also reproduce it on Apple M1.

Curiously, but only in 1.8rc1, the hang does not occur if I uncomment the println(42). I could not reproduce the hang directly on julia. On my real code I tried merging the 2 seval calls, but it still freezes with 1.

It's very important to enable parallelism setting env vars JULIA_NUM_THREADS=6

from juliacall import Main as jl

jl.seval(
    """
    function worker()
            for i in 1:typemax(Int64)
                a = Float64[]
                push!(a, 0.42)
                i % 1000 == 0 && println(i)
            end
    end
"""
)
jl.seval(
    """
begin
#println(42) #this fixes hang only on 1.8rc1
t = [email protected] worker()
println("waiting")
wait(t)
end
"""
)

On my computer it hangs after printing 30200

....
301000
302000

The code runs fine also with a single eval

from juliacall import Main as jl

jl.seval(
    """
begin
    function worker()
            for i in 1:typemax(Int64)
                a = Float64[]
                push!(a, 0.42)
                i % 1000 == 0 && println(i)
            end
    end

#println(4) #this fixes hang only on 1.8rc1
t = [email protected] worker()
println("waiting")
wait(t)
end
"""
)

thanks

Hello, I would be very gratefull to understand what is the problem. I followed the instructions:

julia > ENV["JULIA_CONDAPKG_BACKEND"] = "System"
julia > ENV["JULIA_CONDAPKG_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python3.9"

Then I install PythonCall.

(RiosDelCosmos) pkg> st
      Status `~/casa/work/2022/RiosDelCosmos/Project.toml`
  [6099a3de] PythonCall v0.9.6

When I try to use it I get an error (I do have astropy installed in the cosmos environment):

julia> using PythonCall
julia> ast = pyimport("astropy")
ERROR: Python: ModuleNotFoundError: No module named 'astropy'
Python stacktrace: none
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/DqZCE/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/DqZCE/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/DqZCE/src/concrete/import.jl:11
 [4] top-level scope
   @ REPL[5]:1

I also tried with julia > ENV["JULIA_CONDAPKG_EXE"] = "/home/mmestre/.conda/envs/cosmos/bin/python" obtaining the same result. Thank you very much in advance.

Consider a dummy function inside my custom package EOS:

function dummy(x::AbstractVector)
    @show typeof(x)
end

From Python, I call it as follows

jl.EOS.dummy(np.array([1.1, 2.3]))

and the following error occurs

TypeError: Julia: MethodError: no method matching dummy(::PythonCall.PyIterable{PythonCall.Py})
Closest candidates are:
  dummy(!Matched::AbstractVector{T} where T) at /home/shuhua/Work/opt/EOS/src/EOS.jl:26

According to Conversion to Julia, I expect that a numpy array is converted to a Julia array automatically. Did I miss anything?

Hi, thank you for PythonCall.jl/juliacall! I hope it's ok to just open an issue here. When trying to import juliacall on Apple M1 I am getting the following error:

Python 3.9.12 (main, Apr  5 2022, 01:52:34) 
[Clang 12.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import juliacall
fatal: error thrown and no exception handler available.
ErrorException("could not load symbol "jl_n_threads":
dlsym(RTLD_DEFAULT, jl_n_threads): symbol not found")
ijl_errorf at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_dlsym at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_load_and_lookup at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
nthreads at ./threadingconstructs.jl:23 [inlined]
__preinit_threads__ at ./task.jl:706
jfptr___preinit_threads___40244 at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_finish_julia_init at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
julia_init at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_init at /Applications/Julia-1.8.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.8.dylib (unknown line)

I would be really happy about any help with this. Thanks again! :)

Hi,

Two users of my Julia pkg via JuliaCall have encountered the same issue when multithreading on the Python side was involved. They were mainly Python users who haven't used Julia before. After they installed JuliaCall via pip install juliacall, they had no issue calling Julia in a serial program. However, when they used multithreading on the Python side, they told me Julia tried to repeatly install PythonCall everytime with lengthy error messages.

The easy fix would be to manually install PythonCall in Julia. I haven't encountered this issue myself previously because I have all the pkgs already installed on the Julia side.

Is this behaviour expected? Or should we mention this in the doc?

If I do:

using Python
using PyCall

the latter segfaults (at least on 1.6-rc1). For the reverse order I do get an error.

I don't personally need both to work, just curious if they could (probably not). That's not really the end-goal, but some packages do have PyCall as a dependency, e.g. PyPlot.jl and Pandas.jl. Is there a good way to support such, as it seems this package has a similar API as PyCall?

Thanks for the excellent package! I recently ran into a situation where I tried to install a Python package using CondaPkg.jl that depended upon libstdcxx-ng v12 and was unable to install it due to incompatible dependencies. This resulted in some error about GLIBCXX_3.4.30 not being found. I think I could in principle resolve this by building Julia from source with a more recent version of libstdc++, but because of the lines below I don't think it would work even then. Would it be a reasonable idea to actually run the strings command and find the highest GLIBCXX version string in PythonCall?

https://github.com/cjdoris/PythonCall.jl/blob/77ee463d149b418ed1e6b34036f50172429ae46e/src/cpython/context.jl#L67-L80

Also, the link in the comment on L70 doesn't contain GLIBCXX_3.4.30, however I do see it mentioned here: https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

I'm on Red Hat Enterprise Linux 8.6, running Python 3.9.7. I haven't installed Julia, but juliapkg appears to download and install Julia 1.8.4 just fine. I've created and activated a virtual environment, then installed matplotlib and juliacall:

$ virtualenv -p `which python3.9` venv
$ source ./venv/bin/activate
$ pip install matplotlib juliacall

In my working directory I have one Julia file:

$ cat Foo.jl
struct Foo end
$

and two Python files. The first Python file includes the Foo.jl file:

$ cat foo.py
import os

# Create a new Julia module that will hold all the Julia code imported into this Python module.
import juliacall; jl = juliacall.newmodule("FooModule")

# Get the directory this file is in, then include the `Foo.jl` Julia source code.
d = os.path.dirname(os.path.abspath(__file__))
jl.include(os.path.join(d, "Foo.jl"))

Foo = jl.Foo
$

And the second one imports the first foo.py file:

$ cat run.py
import matplotlib
from foo import Foo
$

When I run the run.py file, I get this error:

$ python run.py
ERROR: `ccall` requires the compilerTraceback (most recent call last):
  File "/home/dingraha/projects/pythoncall_import_error/run.py", line 2, in <module>
    from foo import Foo
  File "/home/dingraha/projects/pythoncall_import_error/foo.py", line 4, in <module>
    import juliacall; jl = juliacall.newmodule("FooModule")
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 218, in <module>
    init()
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 214, in init
    raise Exception('PythonCall.jl did not start properly')
Exception: PythonCall.jl did not start properly
$

Now, if I switch the order of the two import lines in run.py, there's no error. Also if I import juliacall first, i.e.:

$ cat run-no_error.py
import juliacall
import matplotlib
from foo import Foo
$

the error goes away.

Any ideas?

When converting to PyArray some numpy arrays with string elements, I end up with strings of type PythonCall.Utils.StaticString{UInt32, 44}.

It seems that this type is not well supported though, and I get the following errors when indexing and computing the length:

julia> s = PythonCall.Utils.StaticString{UInt32, 44}("ababababb")
"ababababb"

julia> s[1]
ERROR: MethodError: no method matching isvalid(::PythonCall.Utils.StaticString{UInt32, 44}, ::Int64)
Closest candidates are:
  isvalid(::AbstractString, ::Integer) at strings/basic.jl:143
  isvalid(::SubstitutionString, ::Integer) at regex.jl:566
  isvalid(::Type{T}, ::Integer) where T<:AbstractChar at strings/unicode.jl:58
  ...
Stacktrace:
 [1] isvalid
   @ ./strings/basic.jl:143 [inlined]
 [2] getindex(s::PythonCall.Utils.StaticString{UInt32, 44}, i::Int64)
   @ Base ./strings/basic.jl:186
 [3] top-level scope
   @ REPL[5]:1

julia> length(s)
ERROR: MethodError: no method matching isvalid(::PythonCall.Utils.StaticString{UInt32, 44}, ::Int64)
Closest candidates are:
  isvalid(::AbstractString, ::Integer) at strings/basic.jl:143
  isvalid(::SubstitutionString, ::Integer) at regex.jl:566
  isvalid(::Type{T}, ::Integer) where T<:AbstractChar at strings/unicode.jl:58
  ...
Stacktrace:
 [1] isvalid
   @ ./strings/basic.jl:143 [inlined]
 [2] length
   @ ./strings/basic.jl:396 [inlined]
 [3] length(s::PythonCall.Utils.StaticString{UInt32, 44})
   @ Base ./strings/basic.jl:387
 [4] top-level scope
   @ REPL[6]:1

julia> s isa AbstractString # in principle the type should implement the whole interface
true

I can confirm new PythonPlot.jl (which uses matplotlib) works.

Then having support here, even minimal seems redundant. IF it's slowing startup (a big if), then consider just dropping it and update major version (possible to drop more?).

PythonPlot.jl should have downloaded matplotlib automatically. I guess I got an error because of non-clean environment, at least I could easily install it.

Whatever is used for plotting (from Python or not), needs not (should not?) be in the package. And for users who rely on, can use older version of your package.

If you don't want to drop it (or even if) then docs could be improved. I thought about improving them, or manage to follow it, then fix, but will not if it's redundant. At a minimum you may want to point to PythonPlot (not PyPlot).

[It's ok to wait a few days to see if new PR on master helps with startup, before investigating this or other ways for improved startup.]

Off-topic here (only related to new major version if you so choose):

If you do consider a new breaking version change then may too:

version = ">=3.11,<4"

since you download automatically for users. It may not be needed since you get latest anyway 3.11 now. Maybe there's a downside to that, blocking use of some Python packages? At least if/when you users can use 3.11, it's faster and your users will be grateful, and being able to rely on new features:

PEP 654 – Exception Groups and except*

Is that something you need to support somehow from the Julia side?

3.12 is still a year away but 3.12 alpha 2 is out with better error messages. It would be nice if you could easily opt into prereleases but I think not up to you, rather Conda, and I know I can install my own, bypassing Conda. Possibly documenting that such is possible (for e.g. better error messages) is helpful, though I guess mostly if you're developing on the Python side...

Is there something you rely on or want to rely on (or want dropped) from Python?

I see some modules have been dropped in the alpha, and:

The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623.

It's to pave way for UTF-8 internally in Python (I believe not yet there). I don't think the removal affects you (only Python extension developers), but it would be great if/when Julia and Python can share strings, send back and forth, keeping same encoding. At least then, likely you want a new version number.

First of all I'd like to express my appreciation for all the great work being done here. I think this package is a great facilitator for moving Julia out of it's initial niche towards integration into real, practical applications.

In that line, I've been trying to call Julia from a Python application that is then made into a (Windows) executable using pyinstaller. This did not work "out of the box". After a lot of digging, I discovered the bug was in the init_consts function, in particular in the evaluation of

:(pycopy!(pybuiltins.help, pybuiltinsmodule.help)), 

Commenting the :help in src/constants/consts.jl:66 gave a working application, much to my joy and that of my colleagues, but I don't know exactly why this helped, and if there's any downside to doing this. Even if my immediate problem has been resolved/avoided, I thought I should share what I found for the benefit of the community.

This is possible related to GLIBC, but I don't know for sure. I cannot reliably reproduce the error, but here are the problematic steps:

pkg> conda channel_add conda-forge
pkg> conda channel_add huggingface
pkg> conda add datasets

Sometimes this works, sometimes works but when I load my package importing datasets I get

ERROR: InitError: Python: ImportError: /home/lucibello/.julia/juliaup/julia-1.8.2+0.x64/bin/../lib/julia/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/lucibello/.julia/dev/HuggingFaceDatasets/.CondaPkg/env/lib/python3.10/site-packages/pyarrow/../../../libarrow.so.600)

and sometimes the installation itself fails with

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
Encountered problems while solving:
  - package datasets-2.7.0-py_0 requires pyarrow >=6.0.0, but none of the providers can be installed

The environment can't be solved, aborting the operation
error    libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError
ERROR: LoadError: InitError: failed process: Process(`/home/runner/.julia/artifacts/8d5103d84a46e89c60d007a7d30b926037514616/bin/micromamba -r /home/runner/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root create -y -p /tmp/jl_dCVni6/.CondaPkg/env --override-channels --no-channel-priority "datasets[version='>=2.7, <3']" "libstdcxx-ng[version='>=3.4,<11.4',channel='conda-forge']" "numpy[version='>=1.23, <2']" "pillow[version='>=9.2, <10']" "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" -c conda-forge -c huggingface`, ProcessExited(1)) [1]

in this CI run.

Sorry but the behavior is very erratic and I cannot offer a stable way to trigger the issue.

I fail to install the PyPy-package ImpedanceFitter. I also tried in Julia RC v.1.8.3. Here my commands:

import Pkg; Pkg.add("CondaPkg")
import Pkg; Pkg.add("PythonCall")
impfit = PythonCall.pyimport("impedancefitter")

and here is the error output on my machine:

The environment can't be solved, aborting the operation
error    libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError
ERROR: failed process: Process(`'C:\Users\stefanpofahl\.julia\artifacts\7950c161b9c76ecc6d58b2b6734696fee7d8e4a0\Library\bin\micromamba.exe' -r 'C:\Users\stefanpofahl\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'C:\Users\stefanpofahl\.julia\environments\v1.9\.CondaPkg\env' --override-channels --no-channel-priority "impedancefitter[version='*']" "python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']" -c conda-forge`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ .\process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base .\process.jl:480
  [3] run(::Cmd)
    @ Base .\process.jl:477
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\resolve.jl:301
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\resolve.jl:202
  [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\resolve.jl:452
  [7] resolve()
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\resolve.jl:315
  [8] add(pkgs::AbstractVector; resolve::Any, file::Any, temp::Any)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:244
  [9] add(pkgs::AbstractVector)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:231
 [10] add(pkg::Union{CondaPkg.ChannelSpec, CondaPkg.PipPkgSpec, CondaPkg.PkgSpec}; kw::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:248
 [11] add(pkg::Union{CondaPkg.ChannelSpec, CondaPkg.PipPkgSpec, CondaPkg.PkgSpec})
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:248
 [12] add(pkg::AbstractString; version::Any, channel::Any, build::Any, kw::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:375
 [13] add(pkg::AbstractString)
    @ CondaPkg C:\Users\stefanpofahl\.julia\packages\CondaPkg\RuSdR\src\deps.jl:375
 [14] top-level scope
    @ REPL[6]:1

Do you have asuggestion?