Note: this is the proposal as it was initially accepted and does not necessarily reflect the current behavior.
OpenCV is a computer vision library that can optionally be built with CUDA support to massively accelerate certain tasks when using computers with NVidia GPUs. For users without NVidia GPUs, building with CUDA support provides no benefit. CUDA is provided only via a 1.3 GB installer (at the time of this authoring), which requires administrator access to install and modifies the global system state.
Therefore, there is significant value in enabling users to choose whether they find CUDA support valuable for their particular scenario.
The community around OpenCV has built up a library of extensions called OpenCV_contrib. However, these extensions are a source-level patch onto the main OpenCV codebase and therefore must be applied during the core OpenCV build. Further confounding the problem, it is the author's understanding that these community extensions have only been developed with CUDA enabled and cannot be built without that dependency.
Therefore, if CUDA is disabled, OpenCV_contrib must also be disabled. Likewise, when a user requests OpenCV_contrib, CUDA must be enabled. It would be convenient, but not a requirement, to enable CUDA without enabling the community extensions.
Finally, these extensions add additional exports and headers which could be depended upon by other libraries. For maintainers, there must be a way to specify this requirement such that vcpkg install mylib-depends-ocv-contrib
will verify/build/rebuild OpenCV with the community extensions enabled.
The C++ REST SDK is a networking library that provides (among other features) HTTP and Websockets clients. To implement the HTTP client functionality on Windows Desktop, only the core Win32 platform APIs are needed (zlib
is optional).
However, the websockets client is based on Websockets++, which adds mandatory dependencies on boost
, openssl
, and zlib
. Many users of the C++ REST SDK do not use the websockets component, so to minimize their overall dependency footprint it can be disabled at build time. Ideally, these kinds of options would be easily accessible to users in Vcpkg who are concerned about the final size or licensing of their deployment.
SignalR-Client-Cpp depends on the websockets functionality provided by the C++ REST SDK. Therefore, the maintainers of the signalrclient
port would ideally like to express this dependency such that cpprestsdk
will be automatically correctly built for their needs. Note that signalrclient
does not inherently care about boost
, websocketspp
or openssl
-- it depends only on the public websocket client APIs provided by cpprestsdk
. It would be much more maintainable to declare dependencies based on the public APIs rather than the dependencies themselves.
-
General-purpose Open Source projects must be able to easily and succinctly describe their build dependencies inside Vcpkg. This should be no more verbose than a single
vcpkg install
line and, when that command succeeds, there is a strong expectation that all required functionality/headers/imports are available. -
The internal state of the Vcpkg enlistment must be either extremely transparent OR managed by version control (git). This enables larger projects to efficiently transfer the entire state of their customized Vcpkg system between machines (and onto build servers) by having the destination clone and then run a single
vcpkg install
line for the subset of dependencies required. The results of this operation should be as repeatable as reasonably achievable given the current limits of the underlying toolchain.
A key summary of the above motivations is that they are all scenarios surrounding APIs that are not independently buildable from each other. We have an existing solution for APIs that are independently buildable: separate packages. Therefore, we seek to extend the user-facing notion of "packages" to include capabilities and contracts that cannot be made into independent builds.
This document proposes "features" (also called feature packages). These features are intended to model semi-independently toggleable API sets/contracts such that they can be sanely depended upon by other packages. It is not a goal to model exclusive alternatives (such as implementation choices that are not directly user-observable) through this mechanism.
- Individual libraries within
boost
may be reasonably represented as features. - Whether a graphics library is built on DirectX xor OpenGL (where one but not both must be chosen) is not representable as a feature.
From a user experience perspective (i.e. from vcpkg install
) feature packages act as much as possible like completely independent packages. However, internally, any change to a package's features will result in a rebuild of the associated "parent" package. This will invoke a package rebuild experience similar to upgrading.
When using vcpkg install <package>
, some features will be enabled by default. These default features can be avoided by referring to the packages as <package>[core]
and features can be added by supplying them on the same installation line.
Install of a library with default features:
> vcpkg install cpprestsdk
// -- omitted build information -- //
Package cpprestsdk[core]:x86-windows is installed.
Package cpprestsdk[compression]:x86-windows is installed.
Package cpprestsdk[ws-client]:x86-windows is installed.
Removal of that library:
> vcpkg remove cpprestsdk
The following packages will be removed:
cpprestsdk:x86-windows
Removing package cpprestsdk:x86-windows...
Removing package cpprestsdk:x86-windows... done
Purging package cpprestsdk:x86-windows...
Cleaned up D:\src\vcpkg\packages\cpprestsdk_x64-windows
Purging package cpprestsdk:x86-windows... done
Installation of a library with optional features:
> vcpkg install opencv
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
Developer Bob knows he wants OpenCV, so he guesses what the package is called
> vcpkg install opencv
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
Bob attempts to build his application against OpenCV (assuming CUDA), which fails at runtime or compile time indicating that OpenCV wasn't built with CUDA.
Bob comes back to vcpkg, not knowing about the "feature packages" feature. The primary inquiry tools for Vcpkg are search
and list
, so he runs vcpkg search
:
> vcpkg search opencv
opencv 3.2.0 computer vision library
opencv[cuda] support for NVidia CUDA
opencv[contrib] community supported extensions for OpenCV
If your library is not listed, please open an issue at:
https://github.com/Microsoft/vcpkg/issues
He isn't immediately sure what the lack of a version number means, but anything in vcpkg search
can be applied to vcpkg install
, so he runs:
> vcpkg install opencv[cuda]
The following packages will be rebuilt:
opencv:x86-windows
To rebuild with this feature, use:
vcpkg remove opencv:x86-windows
vcpkg install opencv[core,cuda]:x86-windows
Bob follows the instructions...
> vcpkg remove opencv:x86-windows
// -- omitted results as above -- //
> vcpkg install opencv[core,cuda]:x86-windows
// -- omitted build information -- //
Package opencv[core]:x86-windows is installed.
Package opencv[cuda]:x86-windows is installed.
and he can now use OpenCV's CUDA support in his application.
Developer Alice has used cpprestsdk
, built it from source, and she knows about the option to disable websockets. She uses search
to find the complete list of features:
> vcpkg search cpprestsdk
cpprestsdk 2.9.0-2 C++11 JSON, REST, and OAuth library The C++ RES...
cpprestsdk[compression] Gzip compression support in the HTTP client.
cpprestsdk[ws-client] Websocket client support based on websocketspp.
If your library is not listed, please open an issue at:
https://github.com/Microsoft/vcpkg/issues
She decided she only wants cpprestsdk[compression]
, so she installs only that feature:
> vcpkg install cpprestsdk[compression]
// -- omitted build information -- //
Package cpprestsdk[core]:x86-windows is installed.
Package cpprestsdk[compression]:x86-windows is installed.
She receives a quick recursive build that only depends on zlib
.
She's now interested in some additional libraries built on top of cpprestsdk: azure-storage-cpp
and signalrclient
.
> vcpkg install azure-storage-cpp
// -- omitted build information -- //
Package azure-storage-cpp[core]:x86-windows is installed.
> vcpkg install signalrclient
Package signalrclient:x86-windows depends on cpprestsdk[ws-client]:x86-windows.
The following packages will be rebuilt:
* azure-storage-cpp:x86-windows
* cpprestsdk:x86-windows
To rebuild the current package graph with this feature, use:
vcpkg remove cpprestsdk:x86-windows azure-storage-cpp:x86-windows
vcpkg install cpprestsdk[core,compression,ws-client]:x86-windows
vcpkg install azure-storage-cpp[core]:x86-windows
vcpkg install signalrclient[core]:x86-windows
She follows the above script and can use both azure-storage-cpp
and signalrclient
in her code.
Some time has passed, she decided not to use signalrclient
, and she's interested in shipping her application. She wants to minimize her final install size, so she'd like to remove all unneeded packages like boost
and openssl
.
> vcpkg remove boost openssl
The following packages and features will be removed:
* signalrclient[core]:x86-windows
* cpprestsdk[ws-client]:x86-windows
boost[core]:x86-windows
openssl[core]:x86-windows
The following packages will be rebuilt:
* azure-storage-cpp:x86-windows
* cpprestsdk:x86-windows
Removing features requires rebuilding packages.
To rebuild the current package graph without these features, use:
vcpkg remove cpprestsdk:x86-windows azure-storage-cpp:x86-windows signalrclient:x86-windows openssl:x86-windows boost:x86-windows
vcpkg install cpprestsdk[core,compression]:x86-windows
vcpkg install azure-storage-cpp[core]:x86-windows
In the end, her final vcpkg list
outputs:
> vcpkg list
zlib[core]:x86-windows 1.2.11 A compression library
azure-storage-cpp[core]:x86-windows 2.6.0 Microsoft Azure Storage Client SDK for ...
cpprestsdk[core]:x86-windows 2.9.0-2 C++11 JSON, REST, and OAuth library
cpprestsdk[compression]:x86-windows Gzip compression support in the HTTP client.
- Each package can have any number "features".
- Features follow the same naming conventions as packages, but when referenced are always "namespaced" by the parent package.
cpprestsdk[ws-client]
is a completely orthogonal feature frompoco[ws-client]
.
- Features are valid dependencies.
signalrclient
depends oncpprestsdk[ws-client]
- Features can have dependencies (including other features).
cpprestsdk[ws-client]
depends onboost
,openssl
, andwebsocketspp
opencv[cuda]
depends oncuda
opencv[contrib]
depends onopencv[cuda]
boost[python]
depends onlibpython
- Every package has an implicit feature called
core
, which covers the core library with a minimum set of features. All features implicitly depend on thecore
feature of their parent packageazure-storage-cpp
depends oncpprestsdk[core]
cpprestsdk[ws-client]
implicitly depends oncpprestsdk[core]
- Each package declares a list of default features that are enabled when the package is referred to by its raw name, and
core
is always a default feature.cpprestsdk
declaresws-client
andcompression
to be default features. Any unqualified referencecpprestsdk
implicitly meanscpprestsdk[core]
andcpprestsdk[ws-client]
andcpprestsdk[compression]
.opencv
does not declarecuda
norcontrib
to be default features.
As a conclusion of the above, it is expected that all packages will be buildable with all features disabled (just the core
feature) and with all features enabled.
To add the feature CUDA to OpenCV, we will adopt the following syntax in the CONTROL file:
# opencv/CONTROL
Source: opencv
Version: 3.2.0-1
Build-Depends: zlib, libpng, libjpeg-turbo, tiff
Description: computer vision library
Default-Features:
Feature: cuda
Build-Depends: cuda
Description: parallel computing platform
Feature: contrib
Build-Depends: opencv[cuda]
Description: library of OpenCV Extensions
# signalrclient/CONTROL
Source: signalrclient
Version: 1.0.0-beta1
Build-Depends: cpprestsdk[ws-client]
Description: C++ client for SignalR.
# cpprestsdk/CONTROL
Source: cpprestsdk
Version: 2.9.0-2
Build-Depends:
Description: C++11 JSON, REST, and OAuth library ...
Default-Features: compression, ws-client
Feature: compression
Build-Depends: zlib (windows)
Description: Gzip compression support in the HTTP client.
Feature: ws-client
Build-Depends: boost (windows), openssl (windows), websocketpp (windows)
Description: Websocket client support based on websocketspp
- If any feature paragraphs exist, the field
Default-Features
must be present.
Cargo's Features (from Rust): http://doc.crates.io/manifest.html#the-features-section
The proposed feature packages are exceedingly similar to Cargo's Features, with the following changes:
- We avoid any collision problems because features are always namespaced by the owning package
- We do not have a concept of "feature groups", instead we allow dependencies from one feature to another within the same package (Note: This may be how "feature groups" are implemented internally to Cargo -- it was not clear from the documentation).
- Because of the nature of C and C++, it is extremely commonplace that large software packages can have features disabled to remove their dependencies upon other libraries. Changing this configuration requires a rebuild of the package and potentially rippling ABI changes to any downstream dependencies. Therefore, we expect significantly more use of this feature to manage optional API contracts instead of the intended use in Cargo (curation).
- We do not intend feature packages to be used to express the curation relationship, beyond the notion of a "default" set within a package.
Gentoo's USE flags: https://wiki.gentoo.org/wiki/Handbook:X86/Working/USE
Gentoo's USE flags can be shortly summarized as a global set of keywords that is used to make cross-cutting changes to the entire package graph's build configuration. This system standardizes many common settings such that they can be simultaneously toggled for the entire graph.
The most common example of this would be using KDE vs Gnome. A user who knows that, given the choice, they would prefer the KDE/Qt interface can manage the massive space of package configuration efficiently without learning the particular term that each package has decided to call "build using Qt instead of GTK".
USE flags can be customized hierarchically when needed, including at the per-package level. They can be depended upon by other packages, both positively and negatively. USE flags themselves can be used in any boolean expression to determine the complete set of package dependencies, including removing dependencies when flags are enabled.
Problems with USE flags:
- They require coordination from package maintainers to achieve the goal of "portable" flags. This increases the burden of adding a package -- to author a good package, I need to be aware of every uncommon USE flag and evaluate how those could map onto my local configuration space.
- Based on research online, it seems extremely common that users need to tweak flags at a per-package level. This calls into question how valuable the cross-cutting power above is.
- The vast majority of common USE flags are essentially a list of all the common packages and focus on giving the user a view of dependencies (which a package manager is designed to abstract when possible) instead of APIs (which is what users code against).
- Dependency analysis with USE flags becomes a SAT problem with an enormous state space -- P*F bits -- which compounds with any versioning relations. This may work acceptably in practice via heuristics, but it implies that a) there is a looming performance wall which could suddenly create a poor user experience and b) the heuristics may incorrectly model the user's needs, causing a disconnect in desire vs practice, which again leads to a poor user experience.