Build failing, unable to find Python3 on Windows 10

[C3pa]

Hello!

While trying to build the project on Windows 10, I followed the instructions provided at: https://github.com/SFTtech/openage/blob/master/doc/build_instructions/windows_msvc.md#procedure-for-microsoft-windows-users.

I managed to get all the needed dependencies, I got to the following step: https://github.com/SFTtech/openage/blob/master/doc/build_instructions/windows_msvc.md#building-openage

cmake -DCMAKE_TOOLCHAIN_FILE=<vcpkg directory>\scripts\buildsystems\vcpkg.cmake ..

This step is failing with:

-- Could NOT find Python3 (missing: Python3_LIBRARIES Python3_NumPy_INCLUDE_DIRS Development NumPy Development.Module Development.Embed) (found suitable exact version "3.11.6")
CMake Error at C:/Users/Vlasnik/openage/build/CMakeFiles/CMakeTmp/CMakeLists.txt:32 (target_link_libraries):
  Target "cmTC_ff525" links to:

    Python3::Python

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



CMake Error at buildsystem/modules/FindPython.cmake:81 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  C:/vcpkg/scripts/buildsystems/vcpkg.cmake:859 (_find_package)
  buildsystem/HandlePythonOptions.cmake:6 (find_package)
  CMakeLists.txt:176 (include)


-- Configuring incomplete, errors occurred!

Do you have any ideas about what I have missed?

[heinezen]

Did you install the development version of Python3?

[C3pa]

I installed Python 3.12.3 with these checks ticked in the installer:

Then I ran:

pip install cython numpy lz4 toml pillow pygments pyreadline3 mako

[heinezen]

Do you have a Python 3.11 installation? The error above says

(found suitable exact version "3.11.6")

[C3pa]

OK, I uninstalled Python 3.12 and installed the latest Python 3.11, with the needed Python dependencies. I tried building again with the same error. This time I helped it with the -DPython3_ROOT_DIR (as mentioned in the build script) and I managed to successfully build the project.

Thanks for the suggestion to use Python 3.11!

Now, I was following Running openage (in devmode)

I got to the step:

python -m openage game

I got:

openage game: error: the following arguments are required: --modpacks

I haven't found the docs on flags that openage game accepts. Can you point me to where I can read about those?

[heinezen]

@C3pa Ah, I should probably fix the docs for that.

You should first try to run

python -m openage convert --jobs 1

This will convert the gamedata of the other game files to openage modpacks.

After that, you can run

python -m openage main

or

python -m openage game --modpacks <your-converted-modpack>

to start the engine. The name of the modpack depends on which version you converted.

[C3pa]

I'm so sorry about my stream of errors. After executing:

python -m openage convert --jobs 1

I got:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 188, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\convert\main.py", line 247, in main
    from ..cppinterface.setup import setup
  File "C:\Users\Vlasnik\openage\build\openage\cppinterface\setup.py", line 11, in <module>
    from ..log.log_cpp import enable_log_translation
ImportError: DLL load failed while importing log_cpp: The specified module could not be found.

[heinezen]

Could you try adding --add-dll-search-path <openage-build-folder-path> to the invocation?

[C3pa]

Sorry, after calling it with (where the <openage-build-folder-path> is C:\Users\Vlasnik\openage\build):

python -m openage --add-dll-search-path C:\Users\Vlasnik\openage\build convert --jobs 1

I'm still getting:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 188, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\convert\main.py", line 247, in main
    from ..cppinterface.setup import setup
  File "C:\Users\Vlasnik\openage\build\openage\cppinterface\setup.py", line 11, in <module>
    from ..log.log_cpp import enable_log_translation
ImportError: DLL load failed while importing log_cpp: The specified module could not be found.

Do you have a suggestion for some debug logging lines I can add to a certain file to make tracing down which import paths are being checked easier?

[heinezen]

I think the problem might be that --add-dll-search-path is not recursive, but that's a wild guess.

Could you try adding <build-folder>/openage/log explicitly and see if that changes anything? This directory should contain the compiled cython module. You can recognize them by the .pyd extension, so if log_cpp is missing, then it's looking for a filename similar to log_cpp.<plattform-string>.pyd.

[C3pa]

Calling it like this:

python -m openage --add-dll-search-path C:\Users\Vlasnik\openage\build\openage\log convert --jobs 1

Still gives the error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 188, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Vlasnik\openage\build\openage\convert\main.py", line 247, in main
    from ..cppinterface.setup import setup
  File "C:\Users\Vlasnik\openage\build\openage\cppinterface\setup.py", line 11, in <module>
    from ..log.log_cpp import enable_log_translation
ImportError: DLL load failed while importing log_cpp: The specified module could not be found.

Here I provide an image of the contents of the <build-folder>/openage/log folder if it helps:

[heinezen]

Different strategy maybe to see if it works at all. Can you download the packaged build from https://github.com/SFTtech/openage/actions/runs/9043283783/artifacts/1493550110 . This should have the dlls in a separate folder when you unpack it. You may still have to use --add-dll-search-path dll/ but it might work without it.

[C3pa]

All right, I tried running the run.exe. First, it complained it needed Python 3.9, so I grabbed that first. Then I tried a series of invocations:

run --add-dll-search-path C:\Users\Vlasnik\Downloads\package-files convert --jobs 1
run --add-dll-search-path C:\Users\Vlasnik\Downloads\package-files\openage convert --jobs 1
run --add-dll-search-path C:\Users\Vlasnik\Downloads\package-files\openage\log convert --jobs 1

Where all of them gave the same error as above:

Traceback (most recent call last):
  File "run.py", line 20, in init run
    main()
  File "C:\Users\Vlasnik\Downloads\package-files\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
  File "C:\Users\Vlasnik\Downloads\package-files\openage\convert\main.py", line 247, in main
    from ..cppinterface.setup import setup
  File "C:\Users\Vlasnik\Downloads\package-files\openage\cppinterface\setup.py", line 11, in <module>
    from ..log.log_cpp import enable_log_translation
ImportError: DLL load failed while importing log_cpp: The specified module could not be found.

[wu1274704958]

I encountered the same problem.
python -m openage --add-dll-search-path D:\code\openage\build\openage convert --jobs 1

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\code\openage\build\openage\__main__.py", line 188, in <module>
    sys.exit(main())
             ^^^^^^
  File "D:\code\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\code\openage\build\openage\convert\main.py", line 247, in main
    from ..cppinterface.setup import setup
  File "D:\code\openage\build\openage\cppinterface\setup.py", line 11, in <module>
    from ..log.log_cpp import enable_log_translation
ImportError: DLL load failed while importing log_cpp: The specified module could not be found.```

[heinezen]

I think I tracked the issue down yesterday. By looking at the .pyd dependency tree, I noticed that the python311.dll can also not be found by default which is a dependency of all .pyd files. The .pyd files don't load properly if their dependencies are missing, even if the files themselves exist. The error message just doesn't communicate this properly.

The required .dll files that Python can't find by default seem to be:

• openage.dll (i.e. the openage library)
• nyan.dll (i.e. our own nyan dpendency)
• python311.dll (i.e. the Python dll for the Python version that openage was built with)

So you actually need to add 3 paths to --add-dll-search-path:

• openage.dll: Should be in <build-folder>/libopenage/RelWithDebInfo
• nyan.dll: I build nyan separately, o for me it's in <nyan-build-folder>/nyan/RelWithDebInfo . But if you let the build script download it, the dll should be somewhere inside the openage build folder
• python311.dll: Was in my user folder under AppData/Local/Programs/Python/Python311

[C3pa]

A success. Thanks for digging into this issue! For reference, if somebody else gets stuck here, this worked for me:

python -m openage --add-dll-search-path C:\Users\Vlasnik\openage\build\libopenage\RelWithDebInfo --add-dll-search-path C:\Users\Vlasnik\nyan\build\nyan\RelWithDebInfo  --add-dll-search-path C:\Users\Vlasnik\AppData\Local\Programs\Python\Python311 convert --jobs 1

Continuing my stream of errors, I tried to start the game:

python -m openage --add-dll-search-path C:\Users\Vlasnik\openage\build\libopenage\RelWithDebInfo --add-dll-search-path C:\Users\Vlasnik\nyan\build\nyan\RelWithDebInfo  --add-dll-search-path C:\Users\Vlasnik\AppData\Local\Programs\Python\Python311 game --modpacks aoe2_base

This errored out with:

INFO [py] launching openage v0.5.3-187-g36e39076
INFO [py] compiled by MSVC 19.39.33523.0
INFO [py] running in DEVMODE
INFO [py] openage nyan API modpack is up to date
INFO [py] Found modpack aoe2_base
INFO [py] Found modpack engine
INFO loading configuration files...
INFO [py] loading config file [Union(Directory(C:/Users/Vlasnik/openage\cfg).root @ (b'cfg',), <openage.util.fslike.union.Union object at 0x000002040822F2D0>.root @ (b'assets',))]:cfg/keybinds.oac...
INFO launching engine with root directoryPath(Union(Directory(C:/Users/Vlasnik/openage\cfg).root @ (b'cfg',), <openage.util.fslike.union.Union object at 0x000002040822F2D0>.root @ (b'assets',)):)
INFO loading configuration files...
INFO [py] loading config file [Union(Directory(C:/Users/Vlasnik/openage\cfg).root @ (b'cfg',), <openage.util.fslike.union.Union object at 0x000002040822F2D0>.root @ (b'assets',))]:cfg/keybinds.oac...
INFO Found modpack: aoe2_base
INFO Found modpack: engine
INFO Created game simulation
INFO Activated modpack: engine
INFO Activated modpack: aoe2_base
INFO Using 3 threads (4 available)
INFO [T1] Time loop started
INFO [T2] Presenter: Launching subsystems...
INFO [T2] Presenter: Initializing graphics subsystems...
INFO Loading .nyan file: engine/ability/property/type.nyan
INFO Loading .nyan file: engine/util/lock.nyan
INFO Loading .nyan file: engine/ability/property.nyan
INFO Loading .nyan file: engine/util/animation_override.nyan
INFO Loading .nyan file: engine/util/graphics.nyan
INFO Loading .nyan file: engine/util/diplomatic_stance.nyan
INFO Loading .nyan file: engine/util/sound.nyan
INFO Loading .nyan file: engine/root.nyan
INFO Loading .nyan file: engine/ability.nyan
INFO Loading .nyan file: engine/ability/type.nyan
INFO Loading .nyan file: engine/util/attribute.nyan
INFO Loading .nyan file: engine/util/exchange_rate.nyan
INFO Loading .nyan file: engine/util/game_entity_type.nyan
INFO Loading .nyan file: engine/util/activity.nyan
WARN [T2] Could not find the Qt platform plugin "windows" in ""
INFO Loading .nyan file: engine/util/accuracy.nyan
CRIT [T2] This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Do you know what might be causing this?

Also, unrelated to the above:

The .pyd files don't load properly if their dependencies are missing, even if the files themselves exist. The error message just doesn't communicate this properly.

Should we open an issue about this at Python's repository?

[heinezen]

Do you know what might be causing this?

Could you make a new issue for this? I suspect this is also related to some missing DLL paths.

[heinezen]

Should we open an issue about this at Python's repository?

python/cpython#69841 is the correct issue I think. As far as I understand, Python can't do much about it, since the last part of the message comes from the OS and there is no info returned about paths or names of the missing dependencies :/

[C3pa]

python/cpython#69841 is the correct issue I think. As far as I understand, Python can't do much about it, since the last part of the message comes from the OS and there is no info returned about paths or names of the missing dependencies :/

Thanks for pointing out.

Could you make a new issue for this? I suspect this is also related to some missing DLL paths.

Sure. We can continue that discussion in #1650.

[C3pa]

Thanks for fixing the issue! For anyone wondering, it's now fine to just call this to start the conversion process on Windows 10:

python -m openage convert --jobs 1

[heinezen]

You can even start without --jobs 1 because the multiprocessing was also fixed ;)