QSC encoder library build loses cross-compilation settings

[distorted-mdw]

/Suppose I configure the OQS provider for cross-compilation with

cmake \
                -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 \
                -DCMAKE_CROSSCOMPILING_EMULATOR=qemu-aarch64 \
                -DCMAKE_C_COMPILER=.../bin/aarch64-linux-gnu \
                ..

My build fails with

$ make VERBOSE=1
[...]
[ 33%] Linking C shared library ../lib/oqsprovider.so
cd BUILDDIR/oqsprov && /usr/bin/cmake -E cmake_link_script CMakeFiles/oqsprovider.dir/link.txt --verbose=1
TOOLCHAIN/bin/aarch64-linux-gnu-gcc -fPIC -shared -Wl,-soname,oqsprovider.so.1 -o ../lib/oqsprovider.so.0.5.0-dev CMakeFiles/oqsprovider.dir/oqsprov.c.o CMakeFiles/oqsprovider.dir/oqsprov_capabilities.c.o CMakeFiles/oqsprovider.dir/oqsprov_keys.c.o CMakeFiles/oqsprovider.dir/oqs_kmgmt.c.o CMakeFiles/oqsprovider.dir/oqs_sig.c.o CMakeFiles/oqsprovider.dir/oqs_kem.c.o CMakeFiles/oqsprovider.dir/oqs_encode_key2any.c.o CMakeFiles/oqsprovider.dir/oqs_endecoder_common.c.o CMakeFiles/oqsprovider.dir/oqs_decode_der2key.c.o CMakeFiles/oqsprovider.dir/oqsprov_bio.c.o  -Wl,-rpath,PREFIX/lib: PREFIX/lib/liboqs.a PREFIX/lib/libcrypto.so ../install/lib/libqsc_key_encoder.a
TOOLCHAIN/aarch64-linux-gnu/bin/ld: ../install/lib/libqsc_key_encoder.a(encoding_qsc.c.o): Relocations in generic ELF (EM: 62)
TOOLCHAIN/aarch64-linux-gnu/bin/ld: ../install/lib/libqsc_key_encoder.a(encoding_qsc.c.o): Relocations in generic ELF (EM: 62)
TOOLCHAIN/aarch64-linux-gnu/bin/ld: ../install/lib/libqsc_key_encoder.a(encoding_qsc.c.o): Relocations in generic ELF (EM: 62)
../install/lib/libqsc_key_encoder.a: error adding symbols: File in wrong format
collect2: error: ld returned 1 exit status
make[2]: *** [oqsprov/CMakeFiles/oqsprovider.dir/build.make:242: lib/oqsprovider.so.0.5.0-dev] Error 1
make[2]: Leaving directory 'BUILDDIR'
make[1]: *** [CMakeFiles/Makefile2:138: oqsprov/CMakeFiles/oqsprovider.dir/all] Error 2
make[1]: Leaving directory 'BUILDDIR'
make: *** [Makefile:182: all] Error 2

The problem is that

$ objdump -f oqsprov/CMakeFiles/oqsprovider.dir/oqsprov.c.o | head

oqsprov/CMakeFiles/oqsprovider.dir/oqsprov.c.o:     file format elf64-littleaarch64
architecture: aarch64, flags 0x00000011:
HAS_RELOC, HAS_SYMS
start address 0x0000000000000000

$ objdump -f install/lib/libqsc_key_encoder.a | head
In archive install/lib/libqsc_key_encoder.a:

encoding_qsc.c.o:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000011:
HAS_RELOC, HAS_SYMS
start address 0x0000000000000000


encoding_kyber.c.o:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000011:

The provider code has been built for ARM64, but the QSC encoder library is built for the host AMD64 architecture.

I don't need this functionality, so I've just set -DUSE_ENCODING_LIB=OFF, but I think it's still a bug.

[baentsch]

Thanks for the bug report. We have disabled QSC encoder library by default now, so this should not hit other users.

[baentsch]

Same question as in #201 (review): Is it worthwhile keeping tracking this issue? What's your take as to the utility/acceptance of the encoding library, @bhess?

[baentsch]

Closing as #460 removed the faulting code