Some hooks cause Java programs to crash or hang

[psalire]

Current Behavior

.jars and Java based .exes crash unless I use exclude-apis=RtlDispatchException:NtProtectVirtualMemory

Steps to Reproduce

To make sure it wasn't an issue specific with the samples that I'm using, I compiled a simple jar file that just prints "hello world"

1. Submit the helloworld.jar file to CAPE without any options
2. Observe that it crashes
3. Submit the helloworld.jar file with option exclude-apis=RtlDispatchException:NtProtectVirtualMemory
4. Observe that it successfully runs

Individually, I noticed that:

• RtlDispatchException causes Java to crash
• NtProtectVirtualMemory causes Java to hang

Context

Question	Answer
Git commit (CAPE)	2391d5ad343f5f307dee4c0b053da64d3c1e9452
OS version	Ubuntu 20.04 host, Windows 10 64-bit guest
Java version	11.0.11

[kevoreilly]

Thanks for testing/supplying the hooks that cause the issues - this helps a lot. I'll attempt to get to the bottom of this.

[psalire]

Great, thanks
Minor correction on my part - helloworld.jar run successfully for me with just excluding RtlDispatchException. However, if I convert the jar into an exe with launch4j, both hooks are needed (RtlDispatchException:NtProtectVirtualMemory).

[ethhart]

Any progress on this? Some additional info that may be of use: Running on NT6.3, NtProtectVirtualMemory needs exclusion for jar and 64 bit exe's to work on my CAPE instance. It doesn't have this issue when run on NT6.1.

[kevoreilly]

by NT6.3 do you mean Windows 8.1?!

[ethhart]

Yes, and Server 2012 R2. Same behavior on Windows 10. Just thought the Win 8.1 and 2012 R2 tests would help trace the issue.

[kevoreilly]

Ah ok thanks for the clarification. I will test with Windows 10. Finding the relevant Java installer is proving non-trivial.

[ethhart]

I've been using this installer for my CAPE env. https://download.oracle.com/java/17/archive/jdk-17.0.1_windows-x64_bin.exe
Got the same results for v18 as well. If you run through CAPE's submission, make sure you add a path to this Java installation in the jar package. I've replicated this by just feeding a config.ini and the capemon_x64.dll into the loader_64.exe though.

[kevoreilly]

Sorry to let this slide for so long - looking again now I see that I could avoid the need to set up Java in advance by testing against one of your launch4j binaries - any chance you could share one please?