Build error: Build failure on a Japanese environment Windows.

[mikunimaru]

Description

When the system language on Windows is set to Japanese, Mozc build fails.

Commit-id

f79e092

Build target

3. Windows build
   • https://github.com/google/mozc/blob/master/docs/build_mozc_in_windows.md

CI build status

1. passing

Environment

• OS: Windows
• Python version (optional): 3.11.4
• Qt5 version (optional): 6.5.2
• Compiler version (optional): MSVC 2022

Build commands

Steps of command lines to reproduce your error.

1. git clone https://github.com/google/mozc.git
2. cd mozc\src
3. python build_tools/update_deps.py
4. python build_tools/build_qt.py --release --confirm_license
5. python build_mozc.py gyp
6. python build_mozc.py build -c Release package

Error logs

Take 1

メモ: インクルード ファイル:   C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include\atlstr.h
メモ: インクルード ファイル:    C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include\cstringt.h
メモ: インクルード ファイル:     C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include\atlsimpstr.h
メモ: インクルード ファイル:      C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include\atlmem.h
メモ: インクルード ファイル:       C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\ATLMFC\include\ATLComMem.h
メモ: インクルード ファイル:  C:\work\mozc\src\third_party\wil\include\wil/resource.h
メモ: インクルード ファイル:  C:\work\mozc\src\base/system_util.h
メモ: インクルード ファイル:  C:\work\mozc\src\third_party\abseil-cpp\absl/types/span.h
メモ: インクルード ファイル:   C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\include\cassert
メモ: インクルード ファイル:    C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt\assert.h
メモ: インクルード ファイル:   C:\work\mozc\src\third_party\abseil-cpp\absl/types/internal/span.h
メモ: インクルード ファイル:    C:\work\mozc\src\third_party\abseil-cpp\absl/algorithm/algorithm.h
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\work\mozc\src\build_mozc.py", line 900, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\work\mozc\src\build_mozc.py", line 885, in main
    BuildMain(cmd_opts, cmd_args)
  File "C:\work\mozc\src\build_mozc.py", line 648, in BuildMain
    BuildOnWindows(targets)
  File "C:\work\mozc\src\build_mozc.py", line 630, in BuildOnWindows
    RunOrDie([ninja, '-C', build_arg, target_name])
  File "C:\work\mozc\src\build_tools\util.py", line 140, in RunOrDie
    raise RunOrDieError('\n'.join(['',
build_tools.util.RunOrDieError:
==========
 ERROR: C:\work\mozc\src\third_party\ninja\ninja.exe -C out_win/Release mozc_win32_build32
==========

Take 2

メモ: インクルード ファイル:    C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared\pshpack8.h
メモ: インクルード ファイル:    C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared\poppack.h
メモ: インクルード ファイル:   C:\work\mozc\src\base/win32/hresultor.h
メモ: インクルード ファイル:    C:\work\mozc\src\base/win32/hresult.h
メモ: インクルード ファイル:    C:\work\mozc\src\base/win32/hresultor_internal.h
メモ: インクルード ファイル:  C:\work\mozc\src\base/win32/wide_char.h
メモ: インクルード ファイル:  C:\work\mozc\src\win32/base/display_name_resource.h
メモ: インクルード ファイル:  C:\work\mozc\src\win32/base/tsf_profile.h
[19/145] LINK_EMBED protoc.exe
FAILED: protoc.exe protoc.exe.pdb
C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-with-manifests environment.x86 True protoc.exe "C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-wrapper environment.x86 False link.exe /nologo /OUT:protoc.exe @protoc.exe.rsp" 1 mt.exe rc.exe "obj\protobuf\protoc.protoc.exe.intermediate.manifest" obj\protobuf\protoc.protoc.exe.generated.manifest
Traceback (most recent call last):
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 332, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 31, in main
    exit_code = executor.Dispatch(args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 73, in Dispatch
    return getattr(self, method)(*args[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 136, in ExecLinkWrapper
    out, _ = link.communicate()
             ^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\scoop\apps\python\current\Lib\subprocess.py", line 1196, in communicate
    stdout = self.stdout.read()
             ^^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
Traceback (most recent call last):
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 332, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 31, in main
    exit_code = executor.Dispatch(args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 73, in Dispatch
    return getattr(self, method)(*args[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 182, in ExecLinkWithManifests
    subprocess.check_call(ldcmd + add_to_ld)
  File "C:\Users\user\scoop\apps\python\current\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-wrapper environment.x86 False link.exe /nologo /OUT:protoc.exe @protoc.exe.rsp protoc.exe.manifest.res' returned non-zero exit status 1.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\work\mozc\src\build_mozc.py", line 900, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\work\mozc\src\build_mozc.py", line 885, in main
    BuildMain(cmd_opts, cmd_args)
  File "C:\work\mozc\src\build_mozc.py", line 648, in BuildMain
    BuildOnWindows(targets)
  File "C:\work\mozc\src\build_mozc.py", line 630, in BuildOnWindows
    RunOrDie([ninja, '-C', build_arg, target_name])
  File "C:\work\mozc\src\build_tools\util.py", line 140, in RunOrDie
    raise RunOrDieError('\n'.join(['',
build_tools.util.RunOrDieError:
==========
 ERROR: C:\work\mozc\src\third_party\ninja\ninja.exe -C out_win/Release mozc_win32_build32
==========

Take 3

PS C:\work\mozc\src> python build_mozc.py build -c Release package
INFO: Running: C:\work\mozc\src\third_party\ninja\ninja.exe -C out_win/Release mozc_win32_build32
ninja: Entering directory `out_win/Release'
[1/127] LINK_EMBED protoc.exe
FAILED: protoc.exe protoc.exe.pdb
C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-with-manifests environment.x86 True protoc.exe "C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-wrapper environment.x86 False link.exe /nologo /OUT:protoc.exe @protoc.exe.rsp" 1 mt.exe rc.exe "obj\protobuf\protoc.protoc.exe.intermediate.manifest" obj\protobuf\protoc.protoc.exe.generated.manifest
Traceback (most recent call last):
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 332, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 31, in main
    exit_code = executor.Dispatch(args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 73, in Dispatch
    return getattr(self, method)(*args[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 136, in ExecLinkWrapper
    out, _ = link.communicate()
             ^^^^^^^^^^^^^^^^^^
  File "C:\Users\user\scoop\apps\python\current\Lib\subprocess.py", line 1196, in communicate
    stdout = self.stdout.read()
             ^^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
Traceback (most recent call last):
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 332, in <module>
    sys.exit(main(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 31, in main
    exit_code = executor.Dispatch(args)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 73, in Dispatch
    return getattr(self, method)(*args[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\work\mozc\src\out_win\Release\gyp-win-tool", line 182, in ExecLinkWithManifests
    subprocess.check_call(ldcmd + add_to_ld)
  File "C:\Users\user\scoop\apps\python\current\Lib\subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'C:\Users\user\scoop\apps\python\current\python.exe gyp-win-tool link-wrapper environment.x86 False link.exe /nologo /OUT:protoc.exe @protoc.exe.rsp protoc.exe.manifest.res' returned non-zero exit status 1.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\work\mozc\src\build_mozc.py", line 900, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\work\mozc\src\build_mozc.py", line 885, in main
    BuildMain(cmd_opts, cmd_args)
  File "C:\work\mozc\src\build_mozc.py", line 648, in BuildMain
    BuildOnWindows(targets)
  File "C:\work\mozc\src\build_mozc.py", line 630, in BuildOnWindows
    RunOrDie([ninja, '-C', build_arg, target_name])
  File "C:\work\mozc\src\build_tools\util.py", line 140, in RunOrDie
    raise RunOrDieError('\n'.join(['',
build_tools.util.RunOrDieError:
==========
 ERROR: C:\work\mozc\src\third_party\ninja\ninja.exe -C out_win/Release mozc_win32_build32
==========

Additional context

The content of the following discussion might also be due to the same underlying cause.
#706

Here's the language environment output from PowerShell

PS C:\Program Files\Microsoft Visual Studio\2022\Community> Get-WinSystemLocale

LCID             Name             DisplayName
----             ----             -----------
1041             ja-JP            日本語 (日本)


PS C:\Program Files\Microsoft Visual Studio\2022\Community> Get-Culture

LCID             Name             DisplayName
----             ----             -----------
1041             ja-JP            日本語 (日本)

I've successfully built it multiple times when the system language is set to English.

[yukawa]

I tried to reproduce the issue but so far could not succeed to reproduce the reported issue so converting this to discussion. Let's see if someone can figure out what is the missing condition to reproduce the issue and/or how we can avoid the problem.

[yukawa]

メモ: インクルード ファイル:    C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared\pshpack8.h
メモ: インクルード ファイル:    C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\shared\poppack.h
メモ: インクルード ファイル:   C:\work\mozc\src\base/win32/hresultor.h
メモ: インクルード ファイル:    C:\work\mozc\src\base/win32/hresult.h
メモ: インクルード ファイル:    C:\work\mozc\src\base/win32/hresultor_internal.h
メモ: インクルード ファイル:  C:\work\mozc\src\base/win32/wide_char.h
メモ: インクルード ファイル:  C:\work\mozc\src\win32/base/display_name_resource.h
メモ: インクルード ファイル:  C:\work\mozc\src\win32/base/tsf_profile.h
[19/145] LINK_EMBED protoc.exe
FAILED: protoc.exe protoc.exe.pdb

Seeing Japanese message (メモ: インクルード ファイル) basically means that 64d2c28 isn't working as expected in your environment for some unknown reasons.

Ninja expects messages from cl.exe for this scenario start with Note: including file: rather than メモ: インクルード ファイル.

https://github.com/ninja-build/ninja/blob/ce700488e01af33bc478bc986e261e306180b993/src/clparser.cc#L44-L58

What I can confirm is that without 64d2c28, cl.exe does show メモ: インクルード ファイル: in my local environment. However, with 64d2c28, cl.exe starts showing Note: including file: in my environment.

[mikunimaru]

https://stackoverflow.com/questions/2286216/how-to-change-msbuild-error-message-language

I followed the advice in the link above and entered "chcp 437" into cmd before building Mozc, the build was successful even in a Japanese environment.
However, in the case of PowerShell, the build fails.

I also found out how to reproduce it. It seems to occur if the English language pack is not enable in Visual Studio Installer.

[yukawa]

Thank you for figuring out the missing condition to reproduce the issue. I also found similar comments in ninja-build/ninja#613, where the fundamental underlying problem has been discussed.

ninja-build/ninja#613 (comment)

What we learned: always install EN version of VS, then install language pack if needed :(

ninja-build/ninja#613 (comment)

After some experimenting, we ended up matching all the languages rather than using VSLANG=1033, because the latter doesn't work if the user doesn't have the English language pack installed. (Seems to work otherwise, though.)

I'm planning to add a bit more caution to build_mozc_in_windows.md, but other than that I don't have any immediate solution on the Mozc side beyond setting VSLANG=1033.

This limitation should disappear once we complete the migration from GYP/Ninja to Bazel in Windows anyway.