Redesigning the build system

[LecrisUT]

@nicholasjng found an interesting issue here with the current build system, basically because all files including those that are not in the git repo are installed, it can fail the scikit-build-core installation because it hits issues when trying to handle a .DS_Store file. For production this is generally not a problem, but it got me looking at the design choices here and I am confused.

---

Design requirements

First let's figure out the design requirements for this project:

• What installation methods are to be supported? Do they all have to go through the python wheel/sdist, or is it desired to be consumed via find_package/FetchContent?
• How is the PyPI package meant to be distributed and consumed? As an sdist only? Compiled wheel with pre-compiled (shared) library files? A noarch wheel (the current option)?
• How is the CMake project desired to be consumed, e.g. as a shared/static library, header-only-like project, etc.?

I don't think any of these choices are a requirement, i.e. you could design the build system to support (almost) any combination of those, but it is a question of what the project maintainers want to support and design.

Redesign suggestion

Focusing on the pip install path, I have a few proposals.

Method 1

scikit-build-core has a wheel.cmake option which can be set to false, meaning the only process done is to install the files under wheel.packages are installed. Here is how it can be used:

• Move all the files to be installed under src/nanobind folder. This has the advantage of scikit-build-core handling which files to be installed which accounts for .gitignore (also un-committed files maybe?).
• Remove the top-level CMakeLists.txt or use it only to generate files like nanobindVersion.cmake or such. If the top-level CMakeLists.txt is removed, set wheel.cmake=false
• Add a cmake.prefix entry-point in pyproject.toml and point it to nanobind python package, which should allow it to pick up cmake/nanobind-config.cmake file which would in turn create the necessary CMake targets and functions

This is the least involved method to implement, and should be equivalent with the current design.

Method 2

This is mostly to show other ways that this can be handled. The design for this method is to align more with a CMake project and allow more ways of distribution, e.g. via FetchContent or find_package without Python package metadata and files.

• The top-level CMakeLists.txt builds and installs a library target, either as a compiled library or INTERFACE one. CMake's FILE_SET and target_sources should help minimize the files installed and consumed to only the relevant files
• Move the python files to a python/nanobind or src/nanobind folder, which could be limited to python packaging distribution, or it could have manual installation instructions to Python_SITEARCH (but I strongly recommend against the manual installation because there is no method of pip remove it)
• If pre-compiled library is provided, wheel.platlib might need to be changed
• Install the target in CMake native paths like ${CMAKE_INSTALL_INCLUDEDIR}, ${CMAKE_INSTALL_DATADIR}/cmake/nanobind which would be expanded as nanobind/include, nanobind/cmake/nanobind, etc. when combined with wheel.install-dir=nanobind
• Add a cmake.prefix entry-point pointing to nanobind python package. CMake internals should handle all the rest from there
• Resolve some alignment issues to make the project usable via FetchContent

---

In either approach the user is unaffected, i.e they would still be able to simply use

find_package(Python 3.8 COMPONENTS Interpreter Development.Module REQUIRED)
find_package(nanobind)
nanobind_add_module(...)

(PS: if the consumer uses scikit-build-core you don't really need the documented execute_process because it populates the CMAKE_PREFIX_PATH with the build's site_packages, but adding cmake.prefix should make it even more robust)

[wjakob]

Let me add @joaander and @henryiii who worked on the current build system.

In general, I will say that the current one serves its purpose, so I am not very inclined to make huge disruptive changes.

[wjakob]

FWIW FetchContent might work, but it is not the recommended way to use this project. For this, please refer to the documentation.

[joaander]

If CMakeLists.txt is installing files it should not be, then you should modify it:

nanobind/CMakeLists.txt

Lines 81 to 96 in b7c4f1a

	install(
	DIRECTORY include/
	DESTINATION "${NB_INSTALL_INC_DIR}"
	)
	install(
	DIRECTORY src/
	DESTINATION "${NB_INSTALL_SRC_DIR}"
	PATTERN "*.py" EXCLUDE
	)
	install(
	DIRECTORY src/
	DESTINATION "${NB_INSTALL_MODULE_DIR}"
	FILES_MATCHING PATTERN "*\.py"
	)

There is no need to redesign the whole build system to solve this problem.

[LecrisUT]

If CMakeLists.txt is installing files it should not be, then you should modify it:

There is no need to redesign the whole build system to solve this problem.

The motivation for the redesign is to avoid having a whack-a-mole situation where one issue is fixed only to uncover another unforeseen edge-case. The presence of the CMakeLists.txt in the top-level would potentially indicate that the system is compiled/generated and/or it can be build+installed outside of the python build-system, but afaics this is not actually the case. Instead the project should be considered to be static installed similar to cython-cmake.

The suggestion of Method1 is in fact very minimal, just re-arranging the project files to better signal the structure of the project, and using more standard CMake conventions such as Use<Pkg>.cmake, Find<Pkg>.cmake.

[hpkfft]

I hope option (2) as documented in https://nanobind.readthedocs.io/en/latest/building.html#finding-nanobind remains supported.

[LecrisUT]

Of course that can be supported, but I would recommend method 2 in that case in order to support more modern workflows like FetchContent. The documented approach is fragile if multiple dependent projects pull nanobind as submodule, but FetchContent would guard against such an issue, among other advantages.

That is the reason for raising the design requirements question.

[wjakob]

This ticket has been open for 3 weeks and discusses relatively disruptive changes to the build system. I prefer to keep issues for things that are truly broken and will convert it to a discussion for now.