Unresolved __chkstk call

[2Trepidatious]

When the size of all local variables for a function (combined) is greater than the stack page size (4k bytes for x86, 8k bytes for x64) a compiler routine adds a call to __chkstk to the code. This causes a linking error where __chkstk isn't resolved.

[hasherezade]

Hi @2Trepidatious ! Please share the code snippet, it will be easier for me to test. I will check it soon.

[2Trepidatious]

You should be able to replicate it by taking any testing code you already have and just add a char test[10000]; to one of the functions.

The solution is likely to just dynamically allocate something that large instead of putting it on the stack, but I figured it was worth documenting since it happens when the TOTAL size of a function's variables reach the page size and not just a single variable.

The __chkstk documentation says it's added for both x86 and x64, but I've only tried to replicate it with x64. Another potential solution might be that if __chkstk gets added it could just be replaced with the code for __chkstk.

[2Trepidatious]

Actually, the above may not be fully true. Here is a better way to replicate the issue:

#include <stdio.h>
#include "peb_lookup.h"

#define CHAR_NUM 10000

int main(void) {
    LPVOID kernel32_handle = get_module_by_name((const LPWSTR)L"kernel32.dll");
    LPVOID load_lib = get_func_by_name((HMODULE)kernel32_handle, (LPSTR)"LoadLibraryA");
    LPVOID get_proc = get_func_by_name((HMODULE)kernel32_handle, (LPSTR)"GetProcAddress");
    auto _LoadLibraryA = reinterpret_cast<decltype(&LoadLibraryA)>(load_lib);
    auto _GetProcAddress = reinterpret_cast<decltype(&GetProcAddress)>(get_proc);
    HMODULE msvcrt_handle = _LoadLibraryA("msvcrt.dll");
    decltype(&printf) _printf = reinterpret_cast<decltype(&printf)>(_GetProcAddress((HMODULE)msvcrt_handle, "printf"));

    char test[CHAR_NUM];
    
    for (int i =0; i < CHAR_NUM - 1; i++) {
        test[i] = 'A';
    }

    test[CHAR_NUM - 1] = '\0';

    _printf(test);
}

[mstjazhkin]

/GS- /Gs100000000 /STACK:0x100000,0x100000