Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Thread sanitization issues with garbage collection #52690

Closed
MilesCranmer opened this issue Jan 1, 2024 · 7 comments
Closed

Thread sanitization issues with garbage collection #52690

MilesCranmer opened this issue Jan 1, 2024 · 7 comments

Comments

@MilesCranmer
Copy link
Member

MilesCranmer commented Jan 1, 2024 •
edited
Loading

I tried to build Julia 1.10 with thread sanitizer support but I just get an onslaught of data race warnings about the Julia runtime itself, in particular from the garbage collection. Are these warnings real? Here are some of them (300 lines out of 39,701 - just from opening the REPL):

(expand) 
==================
WARNING: ThreadSanitizer: data race (pid=756256)
  Write of size 8 at 0x7ff6d34cc790 by thread T3:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0xb95c6)
    #6 jl_init_root_task /home/mc2473/juliasanitizer/julia/src/task.c:1637:33 (libjulia-internal-debug.so.1.10+0xb9a0d)
    #7 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:180:21 (libjulia-internal-debug.so.1.10+0x1257d3)
 
  Previous write of size 8 at 0x7ff6d34cc790 by thread T2:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0xb95c6)
    #6 jl_init_root_task /home/mc2473/juliasanitizer/julia/src/task.c:1637:33 (libjulia-internal-debug.so.1.10+0xb9a0d)
    #7 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:180:21 (libjulia-internal-debug.so.1.10+0x1257d3)
 
  Location is global 'jl_gc_debug_env' of size 152 at 0x7ff6d34cc758 (libjulia-internal-debug.so.1.10+0x653790)
 
  Thread T3 (tid=756262, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
  Thread T2 (tid=756261, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
SUMMARY: ThreadSanitizer: data race /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 in gc_debug_alloc_check
==================
==================
WARNING: ThreadSanitizer: data race (pid=756256)
  Write of size 8 at 0x7ff6d34cc790 by main thread:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 ijl_alloc_string /home/mc2473/juliasanitizer/julia/src/array.c:505:13 (libjulia-internal-debug.so.1.10+0xbe6f4)
    #6 ijl_pchar_to_string /home/mc2473/juliasanitizer/julia/src/array.c:521:21 (libjulia-internal-debug.so.1.10+0xbe591)
    #7 ijl_cstr_to_string /home/mc2473/juliasanitizer/julia/src/array.c:529:12 (libjulia-internal-debug.so.1.10+0xbe999)
    #8 ijl_get_cpu_name /home/mc2473/juliasanitizer/julia/src/./processor_x86.cpp:1051:12 (libjulia-internal-debug.so.1.10+0x1b1975)
    #9 julia___init___35106 sysinfo.jl:118 (sys-debug.so+0x220cf1)
    #10 jfptr___init___35107 <null> (sys-debug.so+0xbc1a6b)
    #11 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #12 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #13 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xebf1c)
    #14 jl_module_run_initializer /home/mc2473/juliasanitizer/julia/src/toplevel.c:76:9 (libjulia-internal-debug.so.1.10+0xebd6d)
    #15 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:901:13 (libjulia-internal-debug.so.1.10+0xb475c)
    #16 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #17 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #18 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #19 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
  Previous write of size 8 at 0x7ff6d34cc790 by thread T3:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0xf9666)
    #6 ijl_box_uint64 /home/mc2473/juliasanitizer/julia/src/datatype.c:1330:1 (libjulia-internal-debug.so.1.10+0x102141)
    #7 jl_type_infer /home/mc2473/juliasanitizer/julia/src/gf.c:368:16 (libjulia-internal-debug.so.1.10+0x55d3a)
    #8 jl_generate_fptr_impl /home/mc2473/juliasanitizer/julia/src/jitlayers.cpp:502:19 (libjulia-codegen-debug.so.1.10+0x18ea1f)
    #9 jl_compile_method_internal /home/mc2473/juliasanitizer/julia/src/gf.c:2480:16 (libjulia-internal-debug.so.1.10+0x63611)
    #10 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2886:16 (libjulia-internal-debug.so.1.10+0x694c0)
    #11 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #12 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xb650c)
    #13 jl_finish_task /home/mc2473/juliasanitizer/julia/src/task.c:320:13 (libjulia-internal-debug.so.1.10+0xb6442)
    #14 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:191:5 (libjulia-internal-debug.so.1.10+0x125822)
 
  Location is global 'jl_gc_debug_env' of size 152 at 0x7ff6d34cc758 (libjulia-internal-debug.so.1.10+0x653790)
 
  Thread T3 (tid=756262, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
SUMMARY: ThreadSanitizer: data race /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 in gc_debug_alloc_check
==================
==================
WARNING: ThreadSanitizer: data race (pid=756256)
  Write of size 8 at 0x7ff6d34cc790 by thread T3:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0x12cfd6)
    #6 jl_gc_alloc /home/mc2473/juliasanitizer/julia/src/gc.c:3570:12 (libjulia-internal-debug.so.1.10+0x137f55)
    #7 _new_array_ /home/mc2473/juliasanitizer/julia/src/array.c:134:26 (libjulia-internal-debug.so.1.10+0xbb50b)
    #8 _new_array /home/mc2473/juliasanitizer/julia/src/array.c:198:12 (libjulia-internal-debug.so.1.10+0xbdff1)
    #9 ijl_alloc_array_1d /home/mc2473/juliasanitizer/julia/src/array.c:436:12 (libjulia-internal-debug.so.1.10+0xbe053)
    #10 Array boot.jl:477 (sys-debug.so+0xb20b46)
    #11 IdDict iddict.jl:30 (sys-debug.so+0xb20b46)
    #12 CachedMethodTable compiler/methodtable.jl:61 (sys-debug.so+0xb20b46)
    #13 julia_YY.NativeInterpreterYY.287_19687 compiler/types.jl:362 (sys-debug.so+0xb20b46)
    #14 NativeInterpreter compiler/types.jl:349 (sys-debug.so+0x9e3a39)
    #15 julia_typeinf_ext_toplevel_24549 compiler/typeinfer.jl:1078 (sys-debug.so+0x9e3a39)
    #16 jfptr_typeinf_ext_toplevel_24550 <null> (sys-debug.so+0x2d1928)
    #17 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #18 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #19 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0x561ec)
    #20 jl_type_infer /home/mc2473/juliasanitizer/julia/src/gf.c:394:32 (libjulia-internal-debug.so.1.10+0x55e8e)
    #21 jl_generate_fptr_impl /home/mc2473/juliasanitizer/julia/src/jitlayers.cpp:502:19 (libjulia-codegen-debug.so.1.10+0x18ea1f)
    #22 jl_compile_method_internal /home/mc2473/juliasanitizer/julia/src/gf.c:2480:16 (libjulia-internal-debug.so.1.10+0x63611)
    #23 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2886:16 (libjulia-internal-debug.so.1.10+0x694c0)
    #24 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #25 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xb650c)
    #26 jl_finish_task /home/mc2473/juliasanitizer/julia/src/task.c:320:13 (libjulia-internal-debug.so.1.10+0xb6442)
    #27 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:191:5 (libjulia-internal-debug.so.1.10+0x125822)
 
  Previous write of size 8 at 0x7ff6d34cc790 by main thread:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 ijl_alloc_string /home/mc2473/juliasanitizer/julia/src/array.c:505:13 (libjulia-internal-debug.so.1.10+0xbe6f4)
    #6 ijl_pchar_to_string /home/mc2473/juliasanitizer/julia/src/array.c:521:21 (libjulia-internal-debug.so.1.10+0xbe591)
    #7 ijl_cstr_to_string /home/mc2473/juliasanitizer/julia/src/array.c:529:12 (libjulia-internal-debug.so.1.10+0xbe999)
    #8 ijl_get_julia_bindir /home/mc2473/juliasanitizer/julia/src/jlapi.c:353:12 (libjulia-internal-debug.so.1.10+0x15d2f0)
    #9 julia___init_build_35133 sysinfo.jl:126 (sys-debug.so+0xc3a787)
    #10 julia___init___35106 sysinfo.jl:120 (sys-debug.so+0x220da8)
    #11 jfptr___init___35107 <null> (sys-debug.so+0xbc1a6b)
    #12 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #13 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #14 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xebf1c)
    #15 jl_module_run_initializer /home/mc2473/juliasanitizer/julia/src/toplevel.c:76:9 (libjulia-internal-debug.so.1.10+0xebd6d)
    #16 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:901:13 (libjulia-internal-debug.so.1.10+0xb475c)
    #17 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #18 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #19 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #20 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
  Location is global 'jl_gc_debug_env' of size 152 at 0x7ff6d34cc758 (libjulia-internal-debug.so.1.10+0x653790)
 
  Thread T3 (tid=756262, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
SUMMARY: ThreadSanitizer: data race /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 in gc_debug_alloc_check
==================
==================
WARNING: ThreadSanitizer: data race (pid=756256)
  Write of size 8 at 0x7ff6d34cc790 by main thread:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 ijl_alloc_string /home/mc2473/juliasanitizer/julia/src/array.c:505:13 (libjulia-internal-debug.so.1.10+0xbe6f4)
    #6 _string_n strings/string.jl:90 (sys-debug.so+0x7df82d)
    #7 StringVector iobuffer.jl:32 (sys-debug.so+0x7df82d)
    #8 julia_YY.IOBufferYY.469_43624 iobuffer.jl:115 (sys-debug.so+0x7df82d)
    #9 GenericIOBuffer iobuffer.jl:106 (sys-debug.so+0xbc4e5e)
    #10 japi1_print_to_string_44441 strings/io.jl:146 (sys-debug.so+0xbc4e5e)
    #11 string strings/io.jl:189 (sys-debug.so+0xc3a877)
    #12 julia___init_build_35133 sysinfo.jl:127 (sys-debug.so+0xc3a877)
    #13 julia___init___35106 sysinfo.jl:120 (sys-debug.so+0x220da8)
    #14 jfptr___init___35107 <null> (sys-debug.so+0xbc1a6b)
    #15 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #16 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #17 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xebf1c)
    #18 jl_module_run_initializer /home/mc2473/juliasanitizer/julia/src/toplevel.c:76:9 (libjulia-internal-debug.so.1.10+0xebd6d)
    #19 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:901:13 (libjulia-internal-debug.so.1.10+0xb475c)
    #20 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #21 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #22 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #23 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
  Previous write of size 8 at 0x7ff6d34cc790 by thread T3:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 ijl_gc_pool_alloc /home/mc2473/juliasanitizer/julia/src/gc.c:1341:23 (libjulia-internal-debug.so.1.10+0x12e10d)
    #5 IdDict iddict.jl:30 (sys-debug.so+0xb20b8d)
    #6 CachedMethodTable compiler/methodtable.jl:61 (sys-debug.so+0xb20b8d)
    #7 julia_YY.NativeInterpreterYY.287_19687 compiler/types.jl:362 (sys-debug.so+0xb20b8d)
    #8 NativeInterpreter compiler/types.jl:349 (sys-debug.so+0x9e3a39)
    #9 julia_typeinf_ext_toplevel_24549 compiler/typeinfer.jl:1078 (sys-debug.so+0x9e3a39)
    #10 jfptr_typeinf_ext_toplevel_24550 <null> (sys-debug.so+0x2d1928)
    #11 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #12 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #13 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0x561ec)
    #14 jl_type_infer /home/mc2473/juliasanitizer/julia/src/gf.c:394:32 (libjulia-internal-debug.so.1.10+0x55e8e)
    #15 jl_generate_fptr_impl /home/mc2473/juliasanitizer/julia/src/jitlayers.cpp:502:19 (libjulia-codegen-debug.so.1.10+0x18ea1f)
    #16 jl_compile_method_internal /home/mc2473/juliasanitizer/julia/src/gf.c:2480:16 (libjulia-internal-debug.so.1.10+0x63611)
    #17 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2886:16 (libjulia-internal-debug.so.1.10+0x694c0)
    #18 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #19 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xb650c)
    #20 jl_finish_task /home/mc2473/juliasanitizer/julia/src/task.c:320:13 (libjulia-internal-debug.so.1.10+0xb6442)
    #21 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:191:5 (libjulia-internal-debug.so.1.10+0x125822)
 
  Location is global 'jl_gc_debug_env' of size 152 at 0x7ff6d34cc758 (libjulia-internal-debug.so.1.10+0x653790)
 
  Thread T3 (tid=756262, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
SUMMARY: ThreadSanitizer: data race /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 in gc_debug_alloc_check
==================
==================
WARNING: ThreadSanitizer: data race (pid=756256)
  Write of size 8 at 0x7ff6d34cc790 by main thread:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0x12cfd6)
    #6 jl_gc_alloc /home/mc2473/juliasanitizer/julia/src/gc.c:3570:12 (libjulia-internal-debug.so.1.10+0x137f55)
    #7 ijl_string_to_array /home/mc2473/juliasanitizer/julia/src/array.c:289:22 (libjulia-internal-debug.so.1.10+0xbc86a)
    #8 unsafe_wrap strings/string.jl:100 (sys-debug.so+0x7df847)
    #9 StringVector iobuffer.jl:32 (sys-debug.so+0x7df847)
    #10 julia_YY.IOBufferYY.469_43624 iobuffer.jl:115 (sys-debug.so+0x7df847)
    #11 GenericIOBuffer iobuffer.jl:106 (sys-debug.so+0xbc4e5e)
    #12 japi1_print_to_string_44441 strings/io.jl:146 (sys-debug.so+0xbc4e5e)
    #13 string strings/io.jl:189 (sys-debug.so+0xc3a877)
    #14 julia___init_build_35133 sysinfo.jl:127 (sys-debug.so+0xc3a877)
    #15 julia___init___35106 sysinfo.jl:120 (sys-debug.so+0x220da8)
    #16 jfptr___init___35107 <null> (sys-debug.so+0xbc1a6b)
    #17 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #18 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #19 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xebf1c)
    #20 jl_module_run_initializer /home/mc2473/juliasanitizer/julia/src/toplevel.c:76:9 (libjulia-internal-debug.so.1.10+0xebd6d)
    #21 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:901:13 (libjulia-internal-debug.so.1.10+0xb475c)
    #22 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #23 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #24 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #25 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)
 
  Previous write of size 8 at 0x7ff6d34cc790 by thread T3:
    #0 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 (libjulia-internal-debug.so.1.10+0x146d64)
    #1 jl_gc_debug_check_other /home/mc2473/juliasanitizer/julia/src/gc-debug.c:494:12 (libjulia-internal-debug.so.1.10+0x146df0)
    #2 maybe_collect /home/mc2473/juliasanitizer/julia/src/gc.c:936:62 (libjulia-internal-debug.so.1.10+0x1389c4)
    #3 jl_gc_pool_alloc_inner /home/mc2473/juliasanitizer/julia/src/gc.c:1293:5 (libjulia-internal-debug.so.1.10+0x12e1f2)
    #4 jl_gc_pool_alloc_noinline /home/mc2473/juliasanitizer/julia/src/gc.c:1350:12 (libjulia-internal-debug.so.1.10+0x12e68d)
    #5 jl_gc_alloc_ /home/mc2473/juliasanitizer/julia/src/./julia_internal.h:477:13 (libjulia-internal-debug.so.1.10+0x12cfd6)
    #6 jl_gc_alloc /home/mc2473/juliasanitizer/julia/src/gc.c:3570:12 (libjulia-internal-debug.so.1.10+0x137f55)
    #7 _new_array_ /home/mc2473/juliasanitizer/julia/src/array.c:134:26 (libjulia-internal-debug.so.1.10+0xbb50b)
    #8 _new_array /home/mc2473/juliasanitizer/julia/src/array.c:198:12 (libjulia-internal-debug.so.1.10+0xbdff1)
    #9 ijl_alloc_array_1d /home/mc2473/juliasanitizer/julia/src/array.c:436:12 (libjulia-internal-debug.so.1.10+0xbe053)
    #10 Array boot.jl:477 (sys-debug.so+0xb20c1e)
    #11 Array boot.jl:496 (sys-debug.so+0xb20c1e)
    #12 julia_YY.NativeInterpreterYY.287_19687 compiler/types.jl:364 (sys-debug.so+0xb20c1e)
    #13 NativeInterpreter compiler/types.jl:349 (sys-debug.so+0x9e3a39)
    #14 julia_typeinf_ext_toplevel_24549 compiler/typeinfer.jl:1078 (sys-debug.so+0x9e3a39)
    #15 jfptr_typeinf_ext_toplevel_24550 <null> (sys-debug.so+0x2d1928)
    #16 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2875:35 (libjulia-internal-debug.so.1.10+0x693f5)
    #17 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #18 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0x561ec)
    #19 jl_type_infer /home/mc2473/juliasanitizer/julia/src/gf.c:394:32 (libjulia-internal-debug.so.1.10+0x55e8e)
    #20 jl_generate_fptr_impl /home/mc2473/juliasanitizer/julia/src/jitlayers.cpp:502:19 (libjulia-codegen-debug.so.1.10+0x18ea1f)
    #21 jl_compile_method_internal /home/mc2473/juliasanitizer/julia/src/gf.c:2480:16 (libjulia-internal-debug.so.1.10+0x63611)
    #22 _jl_invoke /home/mc2473/juliasanitizer/julia/src/gf.c:2886:16 (libjulia-internal-debug.so.1.10+0x694c0)
    #23 ijl_apply_generic /home/mc2473/juliasanitizer/julia/src/gf.c:3076:12 (libjulia-internal-debug.so.1.10+0x69649)
    #24 jl_apply /home/mc2473/juliasanitizer/julia/src/./julia.h:1982:12 (libjulia-internal-debug.so.1.10+0xb650c)
    #25 jl_finish_task /home/mc2473/juliasanitizer/julia/src/task.c:320:13 (libjulia-internal-debug.so.1.10+0xb6442)
    #26 jl_threadfun /home/mc2473/juliasanitizer/julia/src/partr.c:191:5 (libjulia-internal-debug.so.1.10+0x125822)
 
  Location is global 'jl_gc_debug_env' of size 152 at 0x7ff6d34cc758 (libjulia-internal-debug.so.1.10+0x653790)
 
  Thread T3 (tid=756262, running) created by main thread at:
    #0 pthread_create /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1022 (julia-debug+0x694b2)
    #1 uv_thread_create_ex /workspace/srcdir/libuv/src/unix/thread.c:270 (libjulia-internal-debug.so.1.10+0x265de6)
    #2 _finish_julia_init /home/mc2473/juliasanitizer/julia/src/init.c:890:5 (libjulia-internal-debug.so.1.10+0xb45c5)
    #3 julia_init /home/mc2473/juliasanitizer/julia/src/init.c:844:5 (libjulia-internal-debug.so.1.10+0xb3fd6)
    #4 jl_repl_entrypoint /home/mc2473/juliasanitizer/julia/src/jlapi.c:725:5 (libjulia-internal-debug.so.1.10+0x15e082)
    #5 jl_load_repl /home/mc2473/juliasanitizer/julia/cli/loader_lib.c:568:12 (libjulia-debug.so.1.10+0x148b3)
    #6 main /home/mc2473/juliasanitizer/julia/cli/loader_exe.c:58:15 (julia-debug+0xecfed)

If there aren't real, do you have any tips for hiding them so I can debug my Julia code?

@vchuravy @yuyichao looks like you saw the same issue on #27173. Do you have tips for hiding these ones?

(Or are these data races in the GC the actual reason for my segfaults (JuliaPy/PyCall.jl#1072)...?)
@MilesCranmer

This comment was marked as outdated.

Sign in to view
@MilesCranmer
Copy link
Member Author

MilesCranmer commented Jan 1, 2024 •
edited
Loading

Summary of all the TSAN issues just from loading a library, quitting when the log reached 1GB:

Categories of sanitizer warnings:

  • Signal-unsafe call: 332,536
  • Data race: 2,010
  • Lock-order-inversion: 2

Here are all the data races (again, just the first 1GB of starting up Julia) – just the last frame

arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:49:22
arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:50:20
arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:60:22
arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:61:20
arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:64:12
arraylist_push /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:70:26
gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 arraylist_grow /home/mc2473/juliasanitizer/julia/src/support/arraylist.c:42:21
gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9
gc_debug_alloc_check /home/mc2473/juliasanitizer/julia/src/gc-debug.c:481:9 length essentials.jl:10
gc_mark_obj8 /home/mc2473/juliasanitizer/julia/src/gc.c:1862:36
gc_mark_objarray /home/mc2473/juliasanitizer/julia/src/gc.c:1955:32
gc_mark_outrefs /home/mc2473/juliasanitizer/julia/src/gc.c:2375:29
gc_mark_outrefs /home/mc2473/juliasanitizer/julia/src/gc.c:2509:17
gc_setmark_pool_ /home/mc2473/juliasanitizer/julia/src/gc.c:888:22
gc_try_claim_and_push /home/mc2473/juliasanitizer/julia/src/gc.c:1834:20
gc_try_setmark_tag /home/mc2473/juliasanitizer/julia/src/gc.c:837:11
isassigned array.jl:270
jl_array_nbytes /home/mc2473/juliasanitizer/julia/src/gc.c:1184:23
jl_svecref /home/mc2473/juliasanitizer/julia/src/./julia.h:1072:5
memcpy /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:899
realloc /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:700

(note to self: made with NUM_BT=1 && cat tsan_load_log.txt | grep -A $((1+$NUM_BT)) 'WARNING: ThreadSanitizer: data race' | vims -e 'WARNING' 'd_' '^ \S' 'd_' '^--\s*$' 'd_' | awk '{print $2" "$3}' | vims -e '^gc_debug' "${NUM_BT}J" | uniq | sort | uniq > summary.txt)

Edit: seems like https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions is good for suppressing the signal errors

@gbaraldi
Copy link
Member

gbaraldi commented Jan 1, 2024

I'm not sure if the gc debug was made thread safe, I'm doubtful it has.

@vtjnash
Copy link
Member

vtjnash commented Jan 2, 2024

Yes it is a real error, in that we should forcibly disable gc threads when gc debug mode is enabled

@MilesCranmer
Copy link
Member Author

Ah i see. That would also explain the segfaults which I tried to fix with #48210

@Liozou
Copy link
Member

Liozou commented Jan 2, 2024

That looks like the same root issue as in #52256. See also this comment:

A lot of these seem to be coming from gc-debug.c functions which don't seem to be well-suited to be used with multiple GC threads.
For instance, gc_scrub_task uses the thread's current task, but GC threads have no tasks associated with them.
gc_scrub_record_task pushes into an array-list, which is not a thread-safe operation.
We should probably audit which of them can actually be used with the multi-threaded GC.

Originally posted by @d-netto in #52256 (comment)

@MilesCranmer MilesCranmer mentioned this issue Mar 25, 2024
Closed
@vtjnash
Copy link
Member

vtjnash commented Aug 15, 2024

Disabled by #48600

@vtjnash vtjnash closed this as completed Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vtjnash @MilesCranmer @Liozou @gbaraldi