WiX Toolset-based MSI with custom actions detected as ransomware by Windows Defender on Windows 10

[asimmon]

• Which version of WiX are you building with?

3.11.2.4516

• Which version of .NET are you building with?

4.5

• If the problem occurs when installing your packages built with WiX, what is the version of Windows the package is running on?

Windows 10

• Describe the problem and the steps to reproduce it.

We've been using WiX Toolset v3.x at ShareGate Desktop as our installer software for several years.

A few weeks ago, an ASR (Attack Surface Reduction) rule of Windows Defender started to block the installation of our tool. The rule name is "Use advanced protection against ransomware". We haven't changed any code / custom action related to our installer for a long time.

It seems to be related to how custom actions are executed in a WiX Toolset-based installer: a new process of rundll32.exe that interacts with a DLL as a tmp file - Windows Defender deletes the tmp file.

We came to this conclusion by looking at our Windows Defender traces:

Process: [1784] (Valid-OsVendor) rundll32.exe "C:\Users\username\AppData\Local\Temp\MSI55D8.tmp",zzzzInvokeManagedCustomActionOutOfProc SfxCA_1857093 1 Sharegate.InstallerWix.CustomActions!Sharegate.InstallerWix.CustomActions.CustomActions.AnyOfOurCustomActionMethod

Then we searched for zzzzInvokeManagedCustomActionOutOfProc in your GitHub repository and we found how any custom is executed:

https://github.com/wixtoolset/wix3/blob/master/src/DTF/Tools/SfxCA/SfxCA.cpp

const wchar_t* entry = L"zzzzInvokeManagedCustomActionOutOfProc";
wchar_t szCommandLine[1024] = {0};
swprintf_s(szCommandLine, 1024, L"%s \"%s\",%s %s %d %s", rundll32, szModule, entry, szSessionName, hSession, szEntryPoint)

Have you experienced this kind of issue recently and/or do you have an idea to fix it? Can WiX Toolset v4 solve this issue? Is client support required for you to help us?

[barnson]

Is there any other detail from Defender (e.g., in audit mode)? If your custom action isn't Authenticode signed, try that, as it has a history of calming aggressive antimalware.

[asimmon]

Thanks @barnson , we will try to sign the custom action DLLs - we only do that for the executables and the whole MSI. I will also contact our IT folks for Windows Defender in audit mode.

[BMurri]

@asimmon I always Authenticode sign the managed code assembly that contains the custom action code by signing @(IntermediateAssembly) using a target that is BeforeTargets="PackCustomAction" and then I sign $(IntermediateOutputPath)$(TargetCAFileName) in a separate target that is AfterTargets="PackCustomAction".

[asimmon]

Signing the custom actions DLLs solved the issue 🎉, Windows Defender does no longer block the installation. Thank you very much @barnson !

However, downgrading WiX Toolset from 3.11.2.4516 to 3.11.1.2318 was required, because signing the DLL using signtool.exe with 3.11.2.4516 always returns the following error:

SignTool Error: SignedCode::Sign returned error: 0x800700C1
For more information, please see http://aka.ms/badexeformat

It doesn't seem related to the Windows 10 version or the Windows 10 SDK version as I tried to sign the DLL built on different agents/computers with different specs.

[rseanhall]

#6089