segfault in igzip_rand_test stateless test

[ColinIanKing]

I noticed this was failing on hppa and i386 builds when packaging isa-l for Debian.

Built with:
make -f Makefile.unx arch=32 host_cpu=base_aliases check -j 8

Debugging on i386 with valgrind I see the stack trace as follows:

valgrind ./igzip_rand_test 
==19502== Memcheck, a memory error detector
==19502== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==19502== Using Valgrind-3.20.0 and LibVEX; rerun with -h for copyright info
==19502== Command: ./igzip_rand_test
==19502== 
igzip_rand_test stateless:              ==19502== Use of uninitialised value of size 4
==19502==    at 0x113633: set_long_icf_fg_base (igzip_icf_body.c:56)
==19502==    by 0x11F3D3: set_long_icf_fg (igzip_base_aliases.c:173)
==19502==    by 0x113B88: icf_body_hash1_fillgreedy_lazy (igzip_icf_body.c:276)
==19502==    by 0x11F1C7: isal_deflate_icf_body_lvl3 (igzip_base_aliases.c:95)
==19502==    by 0x113D38: isal_deflate_icf_body (igzip_icf_body.c:337)
==19502==    by 0x1109A3: isal_deflate_icf_pass (igzip.c:502)
==19502==    by 0x111EC2: isal_deflate_int_stateless (igzip.c:760)
==19502==    by 0x111EC2: isal_deflate_stateless (igzip.c:1432)
==19502==    by 0x10D0AB: compress_stateless (igzip_rand_test.c:1557)
==19502==    by 0x10DE38: test_compress_stateless (igzip_rand_test.c:2022)
==19502==    by 0x1094D5: main (igzip_rand_test.c:2826)
==19502== 
==19502== Use of uninitialised value of size 4
==19502==    at 0x113664: set_long_icf_fg_base (igzip_icf_body.c:50)
==19502==    by 0x11F3D3: set_long_icf_fg (igzip_base_aliases.c:173)
==19502==    by 0x113B88: icf_body_hash1_fillgreedy_lazy (igzip_icf_body.c:276)
==19502==    by 0x11F1C7: isal_deflate_icf_body_lvl3 (igzip_base_aliases.c:95)
==19502==    by 0x113D38: isal_deflate_icf_body (igzip_icf_body.c:337)
==19502==    by 0x1109A3: isal_deflate_icf_pass (igzip.c:502)
==19502==    by 0x111EC2: isal_deflate_int_stateless (igzip.c:760)
==19502==    by 0x111EC2: isal_deflate_stateless (igzip.c:1432)
==19502==    by 0x10D0AB: compress_stateless (igzip_rand_test.c:1557)
==19502==    by 0x10DE38: test_compress_stateless (igzip_rand_test.c:2022)
==19502==    by 0x1094D5: main (igzip_rand_test.c:2826)
==19502== 
==19502== Invalid read of size 4
==19502==    at 0x1136C5: compare (huffman.h:331)
==19502==    by 0x1136C5: set_long_icf_fg_base (igzip_icf_body.c:56)
==19502==    by 0x11F3D3: set_long_icf_fg (igzip_base_aliases.c:173)
==19502==    by 0x113B88: icf_body_hash1_fillgreedy_lazy (igzip_icf_body.c:276)
==19502==    by 0x11F1C7: isal_deflate_icf_body_lvl3 (igzip_base_aliases.c:95)
==19502==    by 0x113D38: isal_deflate_icf_body (igzip_icf_body.c:337)
==19502==    by 0x1109A3: isal_deflate_icf_pass (igzip.c:502)
==19502==    by 0x111EC2: isal_deflate_int_stateless (igzip.c:760)
==19502==    by 0x111EC2: isal_deflate_stateless (igzip.c:1432)
==19502==    by 0x10D0AB: compress_stateless (igzip_rand_test.c:1557)
==19502==    by 0x10DE38: test_compress_stateless (igzip_rand_test.c:2022)
==19502==    by 0x1094D5: main (igzip_rand_test.c:2826)
==19502==  Address 0xca334987 is not stack'd, malloc'd or (recently) free'd
==19502== 
==19502== 
==19502== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==19502==  Access not within mapped region at address 0xCA334987
==19502==    at 0x1136C5: compare (huffman.h:331)
==19502==    by 0x1136C5: set_long_icf_fg_base (igzip_icf_body.c:56)
==19502==    by 0x11F3D3: set_long_icf_fg (igzip_base_aliases.c:173)
==19502==    by 0x113B88: icf_body_hash1_fillgreedy_lazy (igzip_icf_body.c:276)
==19502==    by 0x11F1C7: isal_deflate_icf_body_lvl3 (igzip_base_aliases.c:95)
==19502==    by 0x113D38: isal_deflate_icf_body (igzip_icf_body.c:337)
==19502==    by 0x1109A3: isal_deflate_icf_pass (igzip.c:502)
==19502==    by 0x111EC2: isal_deflate_int_stateless (igzip.c:760)
==19502==    by 0x111EC2: isal_deflate_stateless (igzip.c:1432)
==19502==    by 0x10D0AB: compress_stateless (igzip_rand_test.c:1557)
==19502==    by 0x10DE38: test_compress_stateless (igzip_rand_test.c:2022)
==19502==    by 0x1094D5: main (igzip_rand_test.c:2826)
==19502==  If you believe this happened as a result of a stack
==19502==  overflow in your program's main thread (unlikely but
==19502==  possible), you can try to increase the size of the
==19502==  main thread stack using the --main-stacksize= flag.
==19502==  The main thread stack size used in this run was 8388608.

Debugging with gdb I get:

(gdb) run
Starting program: /home/cking/isa-l/igzip_rand_test 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
igzip_rand_test stateless:              
Program received signal SIGSEGV, Segmentation fault.
compare (str1=0xc5a0a197 <error: Cannot access memory at address 0xc5a0a197>, 
    str2=0x43fb58 "@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030\030V\177N\362\225\301\221"..., 
    max_length=<optimized out>) at igzip/huffman.h:332
332	                test = load_le_u64(str1);
(gdb) where
#0  compare (str1=0xc5a0a197 <error: Cannot access memory at address 0xc5a0a197>, 
    str2=0x43fb58 "@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030\030V\177N\362\225\301\221"..., 
    max_length=<optimized out>) at igzip/huffman.h:332
#1  set_long_icf_fg_base (
    next_in=0x43fb50 "\357Ś\026˰\346A@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030"..., processed=3808, 
    input_size=70364449214464, match_lookup=0x4173b6 <set_long_icf_fg+6>) at igzip/igzip_icf_body.c:56
#2  0x004173d4 in set_long_icf_fg (
    next_in=0x43fb50 "\357Ś\026˰\346A@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030"..., 
    end_in=0xee0 <error: Cannot access memory at address 0xee0>, match_lookup=0x0, level_buf=0x1000) at igzip/igzip_base_aliases.c:173
#3  0x0040bb89 in icf_body_hash1_fillgreedy_lazy (stream=0xbffea050) at igzip/igzip_icf_body.c:276
#4  0x004171c8 in isal_deflate_icf_body_lvl3 (stream=0xbffea050) at igzip/igzip_base_aliases.c:95
#5  0x0040bd39 in isal_deflate_icf_body (stream=0xbffea050) at igzip/igzip_icf_body.c:337
#6  0x004089a4 in isal_deflate_icf_pass (stream=stream@entry=0xbffea050, inbuf_start=<optimized out>) at igzip/igzip.c:502
#7  0x00409ec3 in isal_deflate_int_stateless (stream=<optimized out>) at igzip/igzip.c:760
#8  isal_deflate_stateless (stream=<optimized out>) at igzip/igzip.c:1432
#9  0x004050ac in compress_stateless (
    data=0x43fb50 "\357Ś\026˰\346A@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030"..., data_size=13512, 
    compressed_buf=0x443020 '\270' <repeats 200 times>..., compressed_size=0xbfffe26c, flush_type=0, gzip_flag=0, level=3, 
    hist_bits=0) at igzip/igzip_rand_test.c:1557
#10 0x00405e39 in test_compress_stateless (
    in_data=0xb7f5fcd7 "\357Ś\026˰\346A@1\240\026\223\220\231/\232\233\023\242\001\206\252x\023", '\030' <repeats 118 times>, "V\177N\362\225\301\221", '\020' <repeats 28 times>, "\216|\017ϧPYe\233\001\3526 \373\030\030\030\030\030\030\030\030"..., in_size=13512, 
    flush_type=0) at igzip/igzip_rand_test.c:2022
#11 0x004014d6 in main (argc=1, argv=0xbffff524) at igzip/igzip_rand_test.c:2826
(gdb) print $str1
$1 = void
(gdb) print str1
$2 = (uint8_t *) 0xc5a0a197 <error: Cannot access memory at address 0xc5a0a197>
(gdb) 

[ColinIanKing]

I bisected this issue down, the following commit trips the segfault, e.g.:

./igzip_rand_test
Window Size: 32 K
Test Seed  : 4660
Randoms    : 64
igzip_rand_test stateless:              .make: *** [make.inc:169: igzip_rand_test.run] Segmentation fault (core dumped)

The commit in question that causes the issue is:

1fdc5941a37f2d2780339b55ff052f272e7566be is the first bad commit
commit 1fdc5941a37f2d2780339b55ff052f272e7566be
Author: Roy Oursler <[email protected]>
Date:   Tue Oct 2 14:01:57 2018 -0700

    igzip: Modify set_long_icf to handle small end_in
    
    Change-Id: I24c3420df5d9e84d27fe28eff96155e5fcd51760
    Signed-off-by: Roy Oursler <[email protected]>

 igzip/igzip_icf_body.c             | 24 +++++++++++++-----------
 igzip/igzip_set_long_icf_fg_04.asm | 42 ++++++++++++++++++++++++++----------------
 igzip/igzip_set_long_icf_fg_06.asm | 42 ++++++++++++++++++++++++++----------------
 3 files changed, 65 insertions(+), 43 deletions(-)

[pablodelara]

Thanks Colin. We are not validating the 32-bit implementation, as said in the Release Notes. We are planning to remove this 32-bit support in the next release.

[ColinIanKing]

Just to confirm, is that all 32 bit support to be removed for all architectures?

[pablodelara]

Yes, ARM and PowerPC have 64-bit versions only.

[pablodelara]

Closing down this issue, thanks!