LLVM error when using PackageCompiler

[korbinian90]

When I compile my project with PackageCompiler.jl in Ubuntu or Windows, it fails with the following error related to LoopVectorization. It compiles in MacOS though.

- PackageCompiler: compiling nonincremental system image
LLVM ERROR: Cannot select: 0xa6f3080: i32 = X86ISD::BZHI Constant:i32<-1>, 0xba734e0, /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:338 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:338 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:357 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ] ]
  0xa6f2ad0: i32 = Constant<-1>
  0xba734e0: i32 = X86ISD::CMOV 0x821ffb38, Constant:i32<8>, TargetConstant:i8<4>, 0x82208c30, /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:356 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ]
    0x821ffb38: i32 = and 0x8280c048, Constant:i32<7>, int.jl:347 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:325 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:325 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ] ] ]
      0x8280c048: i32 = truncate 0x82866580, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
        0x82866580: i64 = AssertZext 0x4a636a10, ValueType:ch:i63, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
          0x4a636a10: i64,ch = CopyFromReg 0x6db47518, Register:i64 %0, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
            0x82a10170: i64 = Register %0
      0x9e97018: i32 = Constant<7>
    0x8286e430: i32 = Constant<8>
    0xba73270: i8 = TargetConstant<4>
    0x82208c30: i32 = X86ISD::CMP 0x821ffb38, Constant:i32<0>, promotion.jl:499 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:356 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
      0x821ffb38: i32 = and 0x8280c048, Constant:i32<7>, int.jl:347 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:325 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:325 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ] ] ]
        0x8280c048: i32 = truncate 0x82866580, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
          0x82866580: i64 = AssertZext 0x4a636a10, ValueType:ch:i63, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
            0x4a636a10: i64,ch = CopyFromReg 0x6db47518, Register:i64 %0, int.jl:518 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:355 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:350 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:436 @[ /home/runner/.julia/packages/VectorizationBase/z4vAE/src/llvm_intrin/masks.jl:467 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/reconstruct_loopset.jl:955 @[ /home/runner/.julia/packages/LoopVectorization/lCUVJ/src/simdfunctionals/mapreduce.jl:149 ] ] ] ] ] ] ] ]
              0x82a10170: i64 = Register %0
        0x9e97018: i32 = Constant<7>
      0x6b175cf8: i32 = Constant<0>
In function: julia__vreduce_73944

[3036] signal (6): Aborted
in expression starting at none:0
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN4llvm18report_fatal_errorERKNS_5TwineEb at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel15CannotYetSelectEPNS_6SDNodeE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel16SelectCodeCommonEPNS_6SDNodeEPKhj at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_115X86DAGToDAGISel6SelectEPN4llvm6SDNodeE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel22DoInstructionSelectionEv at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel17CodeGenAndEmitDAGEv at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel20SelectAllBasicBlocksERKNS_8FunctionE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm16SelectionDAGISel20runOnMachineFunctionERNS_15MachineFunctionE.part.902 at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_115X86DAGToDAGISel20runOnMachineFunctionERN4llvm15MachineFunctionE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/hostedtoolcache/julia/nightly/x64/bin/../lib/julia/libLLVM-14jl.so (unknown line)
operator() at /cache/build/default-amdci5-5/julialang/julia-master/src/aotcompile.cpp:611
jl_dump_native_impl at /cache/build/default-amdci5-5/julialang/julia-master/src/aotcompile.cpp:623
jl_write_compiler_output at /cache/build/default-amdci5-5/julialang/julia-master/src/precompile.c:94
ijl_atexit_hook at /cache/build/default-amdci5-5/julialang/julia-master/src/init.c:208
jl_repl_entrypoint at /cache/build/default-amdci5-5/julialang/julia-master/src/jlapi.c:712
main at /cache/build/default-amdci5-5/julialang/julia-master/cli/loader_exe.c:59
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 368420323 (Pool: 368319092; Big: 101231); GC: 482
Compile Test: Error During Test at /home/runner/work/CompileMRI.jl/CompileMRI.jl/test/runtests.jl:5
  Got exception outside of a @test
  LoadError: failed process: Process(`/opt/hostedtoolcache/julia/nightly/x64/bin/julia --color=yes --startup-file=no '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' -O3 --sysimage=/tmp/jl_ehOsh3/sys.ji --project=/home/runner/work/CompileMRI.jl/CompileMRI.jl/App --output-o=/tmp/jl_JC7CpMPqjM.o /tmp/jl_RkDsrKh4PI`, ProcessSignaled(6)) [0]

I'm automatically compiling my fairly large project using github actions for Linux, Window and MacOS. The automatic Linux and Windows compilations started to crash some months ago. Unfortunately, I didn't save the Manifest.toml files of successful runs, because I'm suspecting some dependency updated and is causing this problem.
I tried Ubuntu 20.04 and 22.04, Windows 2019, Windows 2022 and local Windows 10, each with the Julia versions 1.6, 1.7, 1.8 and nightly and all crashed.

I don't use LoopVectorization (which seems to be related to the error) directly in my fairly large project, but through a dependency.

Maybe someone can tell from the error message what the problem might be. Otherwise, in case it would help, I can try to provide a minimal example that produces this error.

[chriselrod]

It's probably compiling on a system that supports bzhi, targeting a system that does not.

[chriselrod]

You could try

using HostCPUFeatures, Static
HostCPUFeatures.has_feature(::Val{:x86_64_bmi}) = static(false)

before using any other packages that may load/use LoopVectorization, and see if that helps or at least changes the error message.

[chriselrod]

LoopVectorization doesn't really support/work with package compiler, though.
A rewrite that avoids the fundamental problems with the current version is ongoing, but I only have so much time I can dedicate to it, so I wouldn't expect that for the next year or so.

[korbinian90]

Thank you for the answers!
I tried deactivating the CPU feature with the command, but it didn't change the error message. I might have placed it at the wrong point though.

I solved the issue for me by removing the LoopVectorization dependency for now.

(I will keep it open, in case other people have similar problems)

[Tobriand]

Playing with a minimal project, a couple of observations:

• It is sufficient to ] add LoopVectorization in a minimal project (no usings, nothing) to make PackageCompiler throw this error
• By targeting generic;sandybridge,-xsaveopt,clone_all instead of generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1) for the create_app(...; cpu_target="...") argument means that, at least on an azure build agent running windows-latest, you get an output. It seems like the key issue is including haswell as a build target. I'm not sufficiently aware of what impact this will have though on how much LoopVectorization is able to do its job.

[chriselrod]

Do you need to distribute the image?
What happens if you do cpu_target="native"?

Haswell supports BZHI, but Sandybridge does not (and obviously generic doesn't either).

Does the project get precompiled as part of building the image? Odds are that's when the error is encountered, then.

Does package compiler set any flags I can check?

[chriselrod]

@Tobriand mind upgrading to HostCPUFeatures 0.1.11 once it is released (should be in about 15 minutes), and letting me know if it fixes the problem? JuliaRegistries/General#72190

[Tobriand]

Hi @chriselrod - tried adding [email protected] and I'm still getting the same result with haswell in the cpu compile target (when compiling on an azure build agent, anyway).

I'm a Julia newbie, so worth confirming the correct syntax is ] add [email protected] from my project environment? I.e. that'll add the package into my scope, so the version supplied to anything that takes it as a dependency will be 0.1.11 rather than whatever they have in their dependency list?

Could also add that since this is a dummy project, I could probably push it to Github if that would be helpful? I guess the key thing is that you'd probably need an azure build agent image too in order to actually test it (which obviously I don't have)

[chriselrod]

https://julialang.slack.com/archives/C6A044SQH/p1668053535015429