MSBuild does not log "fatal" error messages from Clang as MSbuild errors

[sejonyv]

Issue Description

For certain classes of errors (like missing files) the Clang compiler logs a "fatal" error, ex)
fatal.c:1:10: fatal error: 'nosuchfile.h' file not found

When run through MSBuild, using for example the Exec task this error message is not logged as an error through the MSbuild log system and is instead logged as normal text output, most likely because the word "fatal" breaks the detection of canonical errors.

Steps to Reproduce

fatal.zip

With the LLVM toolset support installed for Visual Studio and running from a Visual Studio developer prompt:

• To reproduce an example of a fatal error message:
  clang.exe fatal.c
  fatal.c:1:10: fatal error: 'nosuchfile.h' file not found
  1 | #include <nosuchfile.h>
  | ^~~~~~~~~~~~~~
  1 error generated.

• To reproduce the behaviour from MSBuild:
  MSbuild fatal.proj

Expected Behavior

MSBuild should detect the message from Clang as an error and log an MSBuild error message along the lines of:
fatal.c(1,10): error ABCDEF: nosuchfile.h: No such file or directory [fatal.proj]

Actual Behavior

The output from Clang is logged as normal message instead of an error

Analysis

The regexes in src\Shared\CanonicalError.cs does not account for the extra word "fatal" in these kind of error messages and therefore fails to detect them as actual errors

Versions & Configurations

msbuild -version
MSBuild version 17.9.8+b34f75857 for .NET Framework
17.9.8.16306

clang -v
clang version 17.0.3
Target: i686-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\Llvm\bin

[baronfel]

The canonical error format is specifically for MSBuild errors and is provided as a convenience. If a tool wants to accurately report errors from the Exec task, it should use the available knobs (like CustomErrorRegularExpression, CustomWarningRegularExpression, IgnoreStandardErrorWarningFormat from the docs) to massage the output into MSBuild's model. We cannot ship knowledge of every possible tool that can be invoked via Exec.

[sejonyv]

"We cannot ship knowledge of every possible tool that can be invoked via Exec."

While I agree that this is a completely fair statement in general, MSBuild already has built in custom support specifically for the error/warning formats of Clang and GCC as evidenced by the "s_originCategoryCodeTextExpression2" regex in CanonicalError and also specific unittests for Clang and GCC messages so to me it would make sense to have complete support also for the "fatal" form.
But if not I guess this is mostly an inconvenience, the failure can still be detected via the exit code but you may need to dig through logfiles to find the actual root cause instead of being able to tell directly from the build summary.

[baronfel]

This is a strong point - I didn't realize we already had some level of compatibility for these error types specifically. If we can add support for detecting "fatal error" then that seems reasonable - though I think it would only be "error" level severity in MSBuild still.

[rainersigwald]

I wish we didn't have that in there. Looks like it went in in the first mono-support commit.

Despite it being there I'm a bit concerned about extending the regex, since that could theoretically take passing builds to failing. Do we know if these compilers reliably return a failure code after emitting a fatal message? If they do that's a pretty good mitigation and taking this would only map failing-with-only-message builds to failing-with-clear-error builds.

[yuehuang010]

FYI. In VC, the clang task has a custom error parser.