Three '_threaded_reallocs' tests fail (Subprocess aborted): 0 == memcmp(alloc, &thread_data->thread_idx, 1) on Arch s390x

[wombelix]

Describe the bug

When building aws-c-common for Fedora on Arch s390x, three tests fail.

99% tests passed, 3 tests failed out of 443
Total Test time (real) =   8.73 sec
The following tests FAILED:
	265 - sba_threaded_reallocs (Subprocess aborted)
	268 - default_threaded_reallocs (Subprocess aborted)
	270 - aligned_threaded_reallocs (Subprocess aborted)
Errors while running CTest

The tests pass without issues on x86_64, aarch64 and ppc64le.

Expected Behavior

Tests pass similar as on the three other Architectures.

Current Behavior

Relevant log output:

263/443 Test #265: sba_threaded_reallocs ................................................Subprocess aborted***Exception:   0.11 sec
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
################################################################################
Stack trace:
################################################################################
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_backtrace_print+0x6c) [0x3ffba6235ec]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_fatal_assert+0x4a) [0x3ffba61c06a]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/tests/aws-c-common-tests(+0x18e56) [0x2aa38218e56]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(+0x2704a) [0x3ffba62704a]
/lib64/libc.so.6(+0xac36a) [0x3ffba42c36a]
/lib64/libc.so.6(+0x12c2a0) [0x3ffba4ac2a0]
################################################################################
Stack trace:
################################################################################
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_backtrace_print+0x6c) [0x3ffba6235ec]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_fatal_assert+0x4a) [0x3ffba61c06a]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/tests/aws-c-common-tests(+0x18e56) [0x2aa38218e56]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(+0x2704a) [0x3ffba62704a]
/lib64/libc.so.6(+0xac36a) [0x3ffba42c36a]
/lib64/libc.so.6(+0x12c2a0) [0x3ffba4ac2a0]

        Start 268: default_threaded_reallocs
266/443 Test #268: default_threaded_reallocs ............................................Subprocess aborted***Exception:   0.11 sec
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
################################################################################
Stack trace:
################################################################################
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_backtrace_print+0x6c) [0x3ff9a1235ec]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_fatal_assert+0x4a) [0x3ff9a11c06a]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/tests/aws-c-common-tests(+0x18e56) [0x2aa21498e56]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(+0x2704a) [0x3ff9a12704a]
/lib64/libc.so.6(+0xac36a) [0x3ff99f2c36a]
/lib64/libc.so.6(+0x12c2a0) [0x3ff99fac2a0]

        Start 270: aligned_threaded_reallocs
268/443 Test #270: aligned_threaded_reallocs ............................................Subprocess aborted***Exception:   0.10 sec
Fatal error condition occurred in /builddir/build/BUILD/aws-c-common-0.9.17/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
Exiting Application
################################################################################
Stack trace:
################################################################################
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_backtrace_print+0x6c) [0x3ff9c0235ec]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(aws_fatal_assert+0x4a) [0x3ff9c01c06a]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/tests/aws-c-common-tests(+0x18e56) [0x2aa22498e56]
/builddir/build/BUILD/aws-c-common-0.9.17/redhat-linux-build/libaws-c-common.so.1(+0x2704a) [0x3ff9c02704a]
/lib64/libc.so.6(+0xac36a) [0x3ff9be2c36a]
/lib64/libc.so.6(+0x12c2a0) [0x3ff9beac2a0]

Full build log attached for reference: aws-c-common_build_s390x.log

Reproduction Steps

/usr/bin/cmake -S . -B redhat-linux-build -DCMAKE_C_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_Fortran_FLAGS_RELEASE:STRING=-DNDEBUG -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_DO_STRIP:BOOL=OFF -DCMAKE_INSTALL_PREFIX:PATH=/usr -DINCLUDE_INSTALL_DIR:PATH=/usr/include -DLIB_INSTALL_DIR:PATH=/usr/lib64 -DSYSCONF_INSTALL_DIR:PATH=/etc -DSHARE_INSTALL_PREFIX:PATH=/usr/share -DLIB_SUFFIX=64 -DBUILD_SHARED_LIBS:BOOL=ON -DBUILD_SHARED_LIBS=ON

Possible Solution

No response

Additional Information/Context

Similar Issue reported once in October 2022 (staled): #710

aws-c-common version used

0.9.17

Compiler and version used

cmake (3.28.3-4.fc41), gcc (14.0.1-0.16.fc41)

Operating System and version

Fedora Rawhide (F41)

[sharkcz]

If access to a s390x system would help, please let me know.

[graebm]

Are you just trying to get aws-c-common working on the s390x? Or are you trying to get aws-crt-python (and therefore all the other aws-c-*** libraries) working as well? Have you run the other libraries' tests?

We don't support any big-endian architectures at the moment. I'm just wondering if this is the first of N problems you're going to encounter...

[wombelix]

Are you just trying to get aws-c-common working on the s390x? Or are you trying to get aws-crt-python (and therefore all the other aws-c-*** libraries) working as well? Have you run the other libraries' tests?

I'm packaging all AWS C libs for Fedora and EPEL. This will lead at some point to aws-crt-php and other aws-crt-* too. By default, the guidelines expect that a package builds on all supported platforms. This includes s390x. I started with aws-c-common and work through them based on dependencies. I didn't test e.g. aws-c-cal on s390x yet.

We don't support any big-endian architectures at the moment. I'm just wondering if this is the first of N problems you're going to encounter...

Then I'm pretty certain tests in other libs will fail too. Sounds like that I should then exclude s390x as build architecture for all of them.

[graebm]

Sounds like that I should then exclude s390x as build architecture for all of them.

That seems like the safe bet

[wombelix]

Thanks for your Feedback, I will go ahead and exclude s390x as build platform from all AWS C lib related packages for now.