BadTrial issue during ContractionTree search in Julia interface

[andreadegirolamo99]

I am using cotengra by calling Python from Julia using PyCall. I have created an apposite interface to make it work, however with newer cotengra updates it starts giving me problems related to parallelism when calling opt.search(inputs, output, size_dict).
In particular, I get a segmentation fault which originates from the fact that, when concurrent.futures opens new processes to look for the contraction tree, a series of julia processes, not python, is spawned, and, interestingly enough, they are not closed after execution. An easy way around this would of course be to deactivate parallelism fully in the heuristic phase of the algorithm, but I want to exploit cotengra for very huge contraction tasks, and therefore it would be nice to continue using parallelism.
Another way would be to manually kill all spawned julia processes after the search is done, but I don't know how that could work unless I kill the processes individually, which apparently does not work either. Do you have a workaround in mind? Maybe changing the parallel backend could help?

Following, the full error log:

signal (11): Segmentation fault
in expression starting at <my_code>
unknown function (ip: 0x7f0aae95b06f)
unknown function (ip: 0x7f0aae9419b5)
unknown function (ip: 0x7f0aae91fa9d)
unknown function (ip: 0x7f0aae980d20)
cfunction_call at /usr/local/src/conda/python-3.9.13/Objects/methodobject.c:543
_PyObject_MakeTpCall at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3489
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
method_vectorcall at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
method_vectorcall at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
_PyObject_FastCallDictTstate at /usr/local/src/conda/python-3.9.13/Objects/call.c:129
_PyObject_Call_Prepend at /usr/local/src/conda/python-3.9.13/Objects/call.c:489
slot_tp_call at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:6731
_PyObject_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:281
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
_PyObject_FastCallDictTstate at /usr/local/src/conda/python-3.9.13/Objects/call.c:129
_PyObject_Call_Prepend at /usr/local/src/conda/python-3.9.13/Objects/call.c:489
slot_tp_call at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:6731
_PyObject_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:281
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
method_vectorcall at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3537
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:367 [inlined]
_PyObject_FastCallDictTstate at /usr/local/src/conda/python-3.9.13/Objects/call.c:118
_PyObject_Call_Prepend at /usr/local/src/conda/python-3.9.13/Objects/call.c:489
slot_tp_init at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:6971
type_call at /usr/local/src/conda/python-3.9.13/Objects/typeobject.c:1028 [inlined]
_PyObject_MakeTpCall at /usr/local/src/conda/python-3.9.13/Objects/call.c:191
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:116 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:103 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3520
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3489
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396 [inlined]
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
method_vectorcall at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:53
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
PyVectorcall_Call at /usr/local/src/conda/python-3.9.13/Objects/call.c:243
do_call_core at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5125 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3582
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
gen_send_ex at /usr/local/src/conda/python-3.9.13/Objects/genobject.c:215 [inlined]
gen_iternext at /usr/local/src/conda/python-3.9.13/Objects/genobject.c:549
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3308
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
_PyEval_EvalCode at /usr/local/src/conda/python-3.9.13/Python/ceval.c:4329
_PyFunction_Vectorcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:396
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
PyObject_Vectorcall at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:127 [inlined]
call_function at /usr/local/src/conda/python-3.9.13/Python/ceval.c:5077 [inlined]
_PyEval_EvalFrameDefault at /usr/local/src/conda/python-3.9.13/Python/ceval.c:3506
_PyEval_EvalFrame at /usr/local/src/conda/python-3.9.13/Include/internal/pycore_ceval.h:40 [inlined]
function_code_fastcall at /usr/local/src/conda/python-3.9.13/Objects/call.c:330
_PyObject_VectorcallTstate at /usr/local/src/conda/python-3.9.13/Include/cpython/abstract.h:118 [inlined]
method_vectorcall at /usr/local/src/conda/python-3.9.13/Objects/classobject.c:83
macro expansion at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/exception.jl:108 [inlined]
#107 at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:43 [inlined]
disable_sigint at ./c.jl:458 [inlined]
__pycall! at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:42 [inlined]
_pycall! at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:29
_pycall! at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:11
unknown function (ip: 0x7f0b1775c773)
jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
##114 at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:86
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:713
PyObject at /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:86
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
ContractionTree at <my_code>
unknown function (ip: 0x7f0b1775a9e4)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
ctg_heuristics at <my_code>
ctg_contractor at <my_code>
##core#276 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:489
##sample#277 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:495
unknown function (ip: 0x7f0b177419e1)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
#_run#48 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:99
_run##kw at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:93
unknown function (ip: 0x7f0b177300e8)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:757
#invokelatest#2 at ./essentials.jl:718 [inlined]
invokelatest##kw at ./essentials.jl:714 [inlined]
#run_result#45 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:34 [inlined]
run_result##kw at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:34 [inlined]
#run#49 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:117
run##kw at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:117 [inlined]
run##kw at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:117 [inlined]
#warmup#54 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:169 [inlined]
warmup##kw at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:169 [inlined]
#tune!#58 at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:250
tune! at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:250 [inlined]
tune! at /home/ubuntu/.julia/packages/BenchmarkTools/0owsb/src/execution.jl:250
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:126
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:215
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:166 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:587
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:731
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:885
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:830
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:944
eval at ./boot.jl:373 [inlined]
include_string at ./loading.jl:1196
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
_include at ./loading.jl:1253
include at ./Base.jl:418
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
exec_options at ./client.jl:292
_start at ./client.jl:495
jfptr__start_40531.clone_1 at /home/ubuntu/Andrea/julia-1.7.1/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
true_main at /buildworker/worker/package_linux64/build/src/jlapi.c:559
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:701
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x400808)
Allocations: 27838854 (Pool: 27829714; Big: 9140); GC: 31
ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /home/ubuntu/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'concurrent.futures.process.BrokenProcessPool'>
BrokenProcessPool('A child process terminated abruptly, the process pool is not usable anymore')
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/cotengra/hyperoptimizers/hyper.py", line 652, in search
self._search(
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/cotengra/hyperoptimizers/hyper.py", line 623, in _search
for trial in trials:
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/cotengra/hyperoptimizers/hyper.py", line 552, in _gen_results_parallel
future = submit(
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/cotengra/parallel.py", line 156, in submit
return pool.submit(fn, *args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.9/concurrent/futures/process.py", line 707, in submit
raise BrokenProcessPool(self._broken)

And also an example of the julia spawned processes from top:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
#main task
153395 - 20 0 3871684 810540 142864 R 47.7 0.2 11:14.46 julia
#other processes
154141 - 20 0 0 0 0 Z 0.0 0.0 0:04.53 julia
154142 - 20 0 0 0 0 Z 0.0 0.0 0:04.46 julia
154143 - 20 0 0 0 0 Z 0.0 0.0 0:04.50 julia
154144 - 20 0 0 0 0 Z 0.0 0.0 0:04.49 julia
154145 - 20 0 0 0 0 Z 0.0 0.0 0:04.48 julia
154146 - 20 0 0 0 0 Z 0.0 0.0 0:04.49 julia
154147 - 20 0 0 0 0 Z 0.0 0.0 0:04.49 julia
154149 - 20 0 0 0 0 Z 0.0 0.0 0:04.48 julia
154150 - 20 0 0 0 0 Z 0.0 0.0 0:04.48 julia
154151 - 20 0 0 0 0 Z 0.0 0.0 0:04.46 julia
154152 - 20 0 0 0 0 Z 0.0 0.0 0:04.48 julia
154153 - 20 0 0 0 0 Z 0.0 0.0 0:04.46 julia
154154 - 20 0 0 0 0 Z 0.0 0.0 0:04.46 julia
154155 - 20 0 0 0 0 Z 0.0 0.0 0:04.09 julia
154175 - 20 0 0 0 0 Z 0.0 0.0 0:04.36 julia
154176 - 20 0 0 0 0 Z 0.0 0.0 0:04.48 julia
154177 - 20 0 0 0 0 Z 0.0 0.0 0:04.47 julia
154178 - 20 0 0 0 0 Z 0.0 0.0 0:04.37 julia

[jcmgray]

Mmm not sure about the julia and concurrent.futures interactions! But you could try using the parallel='ray' or parallel='dask' backends, they are a bit heavier but more robust, and can connect to an existing dask.distributed or ray cluster that you can start independently.

[andreadegirolamo99]

I also forgot to mention: it is a strict requirement for me to control the number of workers that my program is using, so I need to insert, say, parallel=16 if I want to use 16 cores. However, in theory this prevents me from inserting an option for a different backend, since it's controlled by the same variable parallel. Would it be possible to insert the backend name and the number of workers to use as two separate options?

[jcmgray]

Take a look at the docs for starting dask or ray clusters - you can specify the number of workers for both, then supply parallel={backend} and it will connect to that pool of workers.

[andreadegirolamo99]

Hi again :)
I managed to implement ray in the pipeline and it seems that using both cotengra heuristics and cotengra contractor works fine if no slicing is performed. However, whenever I introduce slicing_opts or slicing_reconf_opts, and in particular a total_size option, I get a similar error as in Issue #15. Following the error log (not a lot of info to be honest):

ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /u/halle/dea/home_at/.julia/packages/PyCall/ilqDX/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'ray.exceptions.RayTaskError(ValueError)'>
RayTaskError(ValueError)(ValueError('math domain error'))
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 649, in search
    self._search(
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 620, in _search
    for trial in trials:
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 560, in _gen_results_parallel
    yield self._get_and_report_next_future()
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 536, in _get_and_report_next_future
    trial = future.result()
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/parallel.py", line 336, in result
    return get_ray().get(self._obj, timeout=timeout)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/ray/_private/worker.py", line 2547, in get
    raise value.as_instanceof_cause()

This is the HyperOptimizer I am using:

HyperOptimizer(methods="greedy", parallel="ray", 
                             slicing_opts={"target_size": 2**max_qubits},
                             slicing_reconf_opts={"target_size": 2**max_qubits}, 
                             reconf_opts={})

I translated the input to Python even though I'm running from Julia.
The variable max_qubits is just the maximum number of qubits that the system can store exactly. For instance, in an 8GB RAM system, it would be 28.

In particular, I am simulating a 30-qubit, 100-layer random quantum circuit. When I changed the parallel input in the HyperOptimizer from above to False, I obtained the following error (which gives more info on where the error actually comes up):

ValueError('math domain error')
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 649, in search
    self._search(
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 620, in _search
    for trial in trials:
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 518, in _gen_results
    trial = trial_fn(
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 257, in __call__
    trial = self.fn(*args, **kwargs)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 172, in __call__
    trial = self.trial_fn(*args, **kwargs)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 201, in __call__
    trial = self.trial_fn(*args, **kwargs)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/hyperoptimizers/hyper.py", line 158, in __call__
    tree.slice_(**self.opts)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/core.py", line 1745, in slice
    ix_sl, _ = sf.search(max_repeats)
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/slicer.py", line 420, in search
    self.trial(
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/slicer.py", line 371, in trial
    ix = max(
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/slicer.py", line 375, in <lambda>
    self.minimize.score_slice_index(cost, ix) -
  File "/u/halle/dea/home_at/.local/lib/python3.10/site-packages/cotengra/scoring.py", line 91, in score_slice_index
    return math.log(

I don't think this error is linked to Julia anymore, it just comes up only when I insert a total_size parameter to either slicing_opts or slicing_reconf_opts for a large enough circuit. Let me know if you need more details.

[jcmgray]

I think that means the cost of that particular trial has gone out of range for a python float. One might need to modify the relevant driver e.g. greedy to raise BadTrial in that case.

It might just be a very expensive (looks deep!) circuit especially to slice.

Note also that setting the same target size for slicing and reconfed slicing is probably not what you want, since it means that the contraction is already at the target size from normal slicing before you turn on the reconf version.