Stack smashing detected and SIGABRT during finding of best embedding

[eribertomota]

This bug was taken from Debian[1].

[1] https://bugs.debian.org/1037481

---

From: Björn Wiberg [email protected]
To: Debian Bug Tracking System [email protected]
Subject: outguess: Stack smashing detected and SIGABRT during finding of best embedding
Date: Tue, 13 Jun 2023 12:30:40 +0200

Package: outguess
Version: 1:0.4-2
Severity: important

Hello,

When trying to run OutGuess on Debian 12, I get a "stack smashing detected" error message and the program is aborted (SIGABRT).
This appears to happen for all JPEG images, i.e. it is not triggered by a certain image.

Steps to reproduce:

$ wget -q https://upload.wikimedia.org/wikipedia/commons/3/3f/JPEG_example_flower.jpg
$ echo msg1 > msg1.txt
$ echo msg2 > msg2.txt
$ outguess -k "key1" -d msg1.txt -E -K "key2" -D msg2.txt -p 100 JPEG_example_flower.jpg JPEG_example_flower.steg.jpg
Initialize encoding/decoding tables
Reading JPEG_example_flower.jpg....
JPEG compression quality set to 100
Extracting usable bits:   70325 bits
Correctable message size: 17434 bits, 24.79%
Encoded 'msg1.txt': 40 bits, 5 bytes
Finding best embedding...
    0:    33(45.8%)[82.5%], bias    28(0.85), saved:    -1, total:  0.05%
    1:    28(38.9%)[70.0%], bias    25(0.89), saved:    -1, total:  0.04%
    6:    30(42.3%)[75.0%], bias    19(0.63), saved:    -1, total:  0.04%
   11:    28(38.9%)[70.0%], bias    13(0.46), saved:    -1, total:  0.04%
11, 41: Embedding data: 40 in 70325
Bits embedded: 72, changed: 28(38.9%)[70.0%], bias: 13, tot: 68673, skip: 68601
Encoded 'msg2.txt' with ECC: 96 bits, 12 bytes
Finding best embedding...
*** stack smashing detected ***: terminated
Aborted

A GDB session shows the following (nothing new):

$ gdb --args outguess -k "key1" -d msg1.txt -E -K "key2" -D msg2.txt -p 100 JPEG_example_flower.jpg JPEG_example_flower.steg.jpg
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from outguess...
(No debugging symbols found in outguess)
(gdb) run
Starting program: /usr/bin/outguess -k key1 -d msg1.txt -E -K key2 -D msg2.txt -p 100 JPEG_example_flower.jpg JPEG_example_flower.steg.jpg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Initialize encoding/decoding tables
Reading JPEG_example_flower.jpg....
JPEG compression quality set to 100
Extracting usable bits:   70325 bits
Correctable message size: 17434 bits, 24.79%
Encoded 'msg1.txt': 40 bits, 5 bytes
Finding best embedding...
    0:    33(45.8%)[82.5%], bias    28(0.85), saved:    -1, total:  0.05%
    1:    28(38.9%)[70.0%], bias    25(0.89), saved:    -1, total:  0.04%
    6:    30(42.3%)[75.0%], bias    19(0.63), saved:    -1, total:  0.04%
   11:    28(38.9%)[70.0%], bias    13(0.46), saved:    -1, total:  0.04%
11, 41: Embedding data: 40 in 70325
Bits embedded: 72, changed: 28(38.9%)[70.0%], bias: 13, tot: 68673, skip: 68601
Encoded 'msg2.txt' with ECC: 96 bits, 12 bytes
Finding best embedding...
*** stack smashing detected ***: terminated

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt full
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
        tid = <optimized out>
        ret = 0
        pd = <optimized out>
        old_mask = {__val = {0}}
        ret = <optimized out>
#1  0x00007ffff7d83d2f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
No locals.
#2  0x00007ffff7d34ef2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#3  0x00007ffff7d1f472 in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {140737351587994, 17179869190, 8589934656, 140737488344528, 8589939592, 6848, 93824992358358, 1431883968, 1, 1706640, 0, 93824992485456, 93824992754952, 140737488344528, 93824992485384, 93824992485744}}, sa_flags = 1431738664, sa_restorer = 0x8}
#4  0x00007ffff7d782d0 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7e92210 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155
        ap = {{gp_offset = 24, fp_offset = 247, overflow_arg_area = 0x7fffffffd050, reg_save_area = 0x7fffffffcfe0}}
        fd = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
#5  0x00007ffff7e10e82 in __GI___fortify_fail (msg=msg@entry=0x7ffff7e921f8 "stack smashing detected") at ./debug/fortify_fail.c:26
No locals.
#6  0x00007ffff7e10e60 in __stack_chk_fail () at ./debug/stack_chk_fail.c:24
No locals.
#7  0x0000555555557422 in ?? ()
No symbol table info available.
#8  0x00005555555574ba in ?? ()
No symbol table info available.
#9  0x0000555555557aa8 in ?? ()
No symbol table info available.
#10 0x0000555555557e44 in ?? ()
No symbol table info available.
#11 0x0000555555558a04 in ?? ()
No symbol table info available.
#12 0x0000555555556d12 in ?? ()
No symbol table info available.
#13 0x00007ffff7d2018a in __libc_start_call_main (main=main@entry=0x5555555562e0, argc=argc@entry=14, argv=argv@entry=0x7fffffffe158) at ../sysdeps/nptl/libc_start_call_main.h:58
        self = <optimized out>
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737488347480, 4746325038488689852, 0, 140737488347600, 93824992401496, 140737354125344, -4746325037838689092, -4746307070948467524}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7fffffffe158, 0x7fffffffe158}, data = {prev = 0x0, cleanup = 0x0, canceltype = -7848}}}
        not_first_call = <optimized out>
#14 0x00007ffff7d20245 in __libc_start_main_impl (main=0x5555555562e0, argc=14, argv=0x7fffffffe158, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe148) at ../csu/libc-start.c:381
No locals.
#15 0x0000555555556f81 in ?? ()
No symbol table info available.
(gdb) quit
A debugging session is active.

        Inferior 1 [process 187740] will be killed.

Quit anyway? (y or n) y
$

Best regards
Björn

[bernhardu]

I tried to take a closer look to the debian bug and the stack canary gets overwritten here:

outguess/src/outguess.c

Lines 164 to 167 in 24810e1

	if (n < j - 1) {
	memmove(detect + n + 1, detect + n,
	(j - n) * sizeof(int));
	memmove(priority + n + 1, priority + n,

The destination of the memmove is always at least detect + 1, so just 8 bytes are allowed to be written.
But the size parameter to memmove is 12.

I attached some debugging attempts to the debian bug.