Skip to content

Latest commit

 

History

History
596 lines (554 loc) · 18.8 KB

literateNix.org

File metadata and controls

596 lines (554 loc) · 18.8 KB
Raw

Literate NixOS Build

Rationale

For utilizing cross system builds, cmake is not good enough. This is especially true for software like supaaYoda which is developed on ArchLinux, which usually has the bleeding edge versions of software.

To this end, for reproducible cross operating system usage, there are only a few viable options:

“Fat” Binary
Basically a binary with all the libraries bundled. This is silly and more than slightly wasteful. It is also slow to compile.
Docker Image
This is actually a great option and will be provided in the future. Caveats include the inherent superuser equivalency for the docker group and other affiliated security issues.
Nix Package
Though not as easy to set up, this is the route considered by this document. Essentially, nix is better than docker in-so-far as non-root users are considered. Though nix is best installed with root, packages can be used without escalating privileges.

Acknowledgements

Conceptually, this document owes its existence to the excellent work of zimbatm and his nix-cpp-demo repository. Of course the opportunity to use noweb is also too good to pass up. Finally, much of the descriptions here are lifted from the nix manual.

The best introduction to nix and a bunch of functional concepts is the nix-pills set of articles. Additionally the utility of --pure along with a simple set of examples is provided in this blog post by Jacek. For an alternate approach to using nix, this blog post by Matthew Bauer is an excellent read.

Implementation

The rest of this document tangles to the appropriate files for nix-shell usage. We will be using the noweb syntax where required. Most of the blocks will be tangled all at once.

Due to the fact that nix-shell respects the system .bashrc, we will not be able to obtain a nix shell directly as the $PATH variable will be overwritten.

We will use the run command to spawn a shell in a non-interactive shell (as opposed to command which spawns an interactive shell) so as to not source the local .bashrc

nix-shell --run bash

Project Root

Every invocation of nix-shell looks for a shell.nix file, failing which a default.nix is required. To this end, we will generate a dummy file which essentially passes control to a yodaStruct.nix file which may also be built directly with nix-build.

We will note that this file simply:

  • defines a variable which is passed to yodaStruct.nix
  • where said variable imports all relevant nix files in a subfolder
# Define
let
# Import
  buildpkgs = import ./nix {};
in
# Pass to
buildpkgs.yodaStruct

Project Source

Although we will be using a file which is meant to build the project, nix-shell will still seek a default.nix in the subfolder, so we shall assert that the form of this is as shown.

<<pR_headerComments>>
# something ? default value ---- Variable declration
# pattern : body ---- Function prototype
{ nixpkgs ? import ./nixpkgs
  , <<pR_arguments>>
   }:
  # Define
  let overlay = self: buildpkgs: with buildpkgs; {
  # All the other nix files
  <<overlayFiles>>
  }; in
  # Ensure reproducibility
  nixpkgs {
  config = {};
  overlays = [overlay];
  }

Essentially we note that we are simply asserting that nixpkgs will be called with the variables defined in the overlayFiles block.

Nix Libraries

Every noweb expansion block under overlayFiles is essentially the expression defining a particular library or helper variable. To clarify matters, they are defined and extended where their constituents are explained.

Helper Functions

To load json src data easily, we will declare and use a simple helper function.

{ fetchFromGitHub }:

# fetches the source described in the <package>.src.json

# path to the .src.json
path:
let
  data = builtins.fromJSON (builtins.readFile path);
in
  fetchFromGitHub { inherit (data) owner repo rev sha256; }

We will need to register the function in the default expression as well.

fetchJSON = import ./fetchJSON.nix { inherit (buildpkgs) fetchFromGitHub; };

Nix Sources

At this stage we will now move towards creating application logic, along with it’s requisite libraries.

Catch2

This is actually handled by conan, and is adapted from Jacek’s blog. It is remarkably trivial to mantain static versions of things with nix though, so it is still useful.

{ clangStdenv, fetchurl }:

clangStdenv.mkDerivation rec {
  name = "catch-${version}";
  version = "2.5.0";

  src = fetchurl {
      url = "https://github.com/catchorg/Catch2/releases/download/v2.5.0/catch.hpp";
      sha256 = "a87d5c0417aaf1c3d16565244a1b643e1999d5838d842823731bc18560268f94";
  };

  # This is a header only library. No unpacking needed. Seems like we need to create
  # _some_ folder, otherwise we get errors.
  unpackCmd = "mkdir fake_dir";

  installPhase = ''
    mkdir -p $out/include/catch
    cp ${src} $out/include/catch/catch.hpp
  '';

  meta = {
    description = "A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later";
    homepage = http://catch-lib.net;
  };
}

Calling the expression

For the actual variable definition which will use callPackage to evaluate the expression defined in the tangled block above, we have:

catch2 = callPackage ./pkgs/catch2.nix { };

As discussed previously, this is now added to the noweb block to be tangled into the output file.

# Package for testing
<<catch2>>

We will also need to add it into our yodaStruct environment.

catch2
catch2

Conan

Unfortunately, conan breaks with the latest (Sun Dec 30 18:14:00 2018) nix expression, so we will override it with our own.

<<conan>>

Where we shall now use the following override.

conan = callPackage ./pkgs/conan/conan.nix { };

The expression itself is not very difficult to understand.

{ lib, python3, fetchpatch, git }:

let newPython = python3.override {
  packageOverrides = self: super: {
    distro = super.distro.overridePythonAttrs (oldAttrs: rec {
      version = "1.2.0";
      src = oldAttrs.src.override {
        inherit version;
        sha256 = "1vn1db2akw98ybnpns92qi11v94hydwp130s8753k6ikby95883j";
      };
    });
    node-semver = super.node-semver.overridePythonAttrs (oldAttrs: rec {
      version = "0.2.0";
      src = oldAttrs.src.override {
        inherit version;
        sha256 = "1080pdxrvnkr8i7b7bk0dfx6cwrkkzzfaranl7207q6rdybzqay3";
      };
    });
    future = super.future.overridePythonAttrs (oldAttrs: rec {
      version = "0.16.0";
      src = oldAttrs.src.override {
        inherit version;
        sha256 = "1nzy1k4m9966sikp0qka7lirh8sqrsyainyf8rk97db7nwdfv773";
      };
    });
    tqdm = super.tqdm.overridePythonAttrs (oldAttrs: rec {
      version = "4.28.1";
      src = oldAttrs.src.override {
        inherit version;
        sha256 = "1fyybgbmlr8ms32j7h76hz5g9xc6nf0644mwhc40a0s5k14makav";
      };
    });
  };
};

in newPython.pkgs.buildPythonApplication rec {
  version = "1.9.1";
  pname = "conan";

  src = newPython.pkgs.fetchPypi {
    inherit pname version;
    sha256 = "0mn69ps84w8kq76zba2gnlqlp855a6ksbl1l6pd1gkjlp9ry0hnf";
  };
  checkInputs = [
    git
  ] ++ (with newPython.pkgs; [
    nose
    parameterized
    mock
    webtest
    codecov
  ]);

  propagatedBuildInputs = with newPython.pkgs; [
    requests fasteners pyyaml pyjwt colorama patch
    bottle pluginbase six distro pylint node-semver
    future pygments mccabe deprecation tqdm
  ];

  checkPhase = ''
    export HOME="$TMP/conan-home"
    mkdir -p "$HOME"
  '';

  meta = with lib; {
    homepage = https://conan.io;
    description = "Decentralized and portable C/C++ package manager";
    license = licenses.mit;
    platforms = platforms.linux;
  };
}

We will also need the data to be defined.

{
  "owner": "conan-io",
  "repo": "conan",
  "branch": "release/1.9.1",
  "rev": "bcb6080d98e7d4e5ed6fafdeb9f3e254c03123e4",
  "sha256": "1bm1c43aswz69rvxp6z61gn310x9k77ixih64kprhdwwzqn1ja4c"
}

FMT

The header only fmt library should also be handled without conan.

{ clangStdenv, fetchJSON, cmake }:
clangStdenv.mkDerivation rec {
  name = "fmtlib-master";
  src = fetchJSON ./fmt.src.json;
  nativeBuildInputs = [ cmake ];
  meta = {
    description = "{fmt} is an open-source formatting library for C++. It can be used as a safe and fast alternative to (s)printf and IOStreams.";
    homepage = http://fmtlib.net;
  };
}

With the standard data definition

{
  "owner": "fmtlib",
  "repo": "fmt",
  "branch": "master",
  "rev": "1b8a216ddf1a3bb612958b912bce5121372dd2e2",
  "sha256": "16h08zdfgbmfslp18y84yd2dwmvq47dnr1chc28srjnpfl3cc7sz"
}

Calling the expression

For the actual variable definition which will use callPackage to evaluate the expression defined in the tangled block above, we have:

fmtlib = callPackage ./pkgs/fmtlib/fmt.nix { };

As discussed previously, this is now added to the noweb block to be tangled into the output file.

# Package for testing
<<fmtlib>>

We will also need to add it into our yodaStruct environment.

fmtlib
fmtlib

YAML Cpp

Since conan will never work with nix, we will simply have to setup nix expressions for each package we need.

# A standard cmake-based build
{ clangStdenv, fetchJSON, cmake }:
clangStdenv.mkDerivation {
  name = "yaml-cpp-master";
  src = fetchJSON ./yaml-cpp.src.json;
  nativeBuildInputs = [ cmake ];
}

With the standard data definition

{
  "owner": "jbeder",
  "repo": "yaml-cpp",
  "branch": "master",
  "rev": "abf941b20d21342cd207df0f8ffe09f41a4d3042",
  "sha256": "01rri88pr8r4lq8vlfbik63kx1fgsq0m5xfg1nfvyvr9fqzpdi86"
}

Calling the expression

For the actual variable definition which will use callPackage to evaluate the expression defined in the tangled block above, we have:

yamlCpp = callPackage ./pkgs/yaml-cpp/yaml-cpp.nix { };

As discussed previously, this is now added to the noweb block to be tangled into the output file.

# Package for testing
<<yamlCpp>>

We will also need to add it into our yodaStruct environment.

yamlCpp
yamlCpp

SharkML

Incredibly, there isn’t much love for the well made shark-ml software.

# A standard cmake-based build
{ clangStdenv, fetchJSON, cmake, boost, openblas, liblapack }:
clangStdenv.mkDerivation {
  name = "sharkML-master";
  src = fetchJSON ./sharkML.src.json;
  nativeBuildInputs = [ cmake boost openblas liblapack ];
}

With the standard data definition

{
  "owner": "Shark-ML",
  "repo": "Shark",
  "branch": "master",
  "rev": "221c1f2e8abfffadbf3c5ef7cf324bc6dc9b4315",
  "sha256": "1h6ggcxpcqdj4x9wjz1njibmvlqmdv9kxm163nk9xivnnx0r6qiz"
}

Calling the expression

For the actual variable definition which will use callPackage to evaluate the expression defined in the tangled block above, we have:

sharkML = callPackage ./pkgs/sharkML/sharkML.nix { };

As discussed previously, this is now added to the noweb block to be tangled into the output file.

# Package for testing
<<sharkML>>

We will also need to add it into our yodaStruct environment.

sharkML
sharkML

yodaStruct Overlay

The main program is also defined and used in the same way as the libraries, so:

yodaStruct = callPackage ./yodaStruct.nix { };

Into the overlay:

# Program expression
  <<yodaStruct>>

Expression

The expression for building the program is conceptually a simple extension of the default.nix process, we declare a function which has a variety of inputs, either defined in the standard packages or locally, and then we simply declare a build script of sorts.

It is only at this stage will we note the concept of runtime dependencies as defined in buildInputs and the build dependencies as defined by nativeBuildInputs.

We are in a position to leverage the project README.md to ascertain the build requirements, and writing out the structure of the project will aid in determining the libraries to be built or overriden.

# Using patterns, and white space negligence
{ clangStdenv
, <<yS_inputs>> }:
  clangStdenv.mkDerivation {
  name = "yodaStruct";
  src = lib.cleanSource ../.;
  nativeBuildInputs = [
  <<yS_buildDeps>>
  ];
  buildInputs = [
  <<yS_runDeps>>
  ];
  }

Where we have leveraged the rather strange design choice of noweb honoring prefix characters for generating sane inputs.

Build Dependencies

Cmake

This is used to actually build things. As such the standard nix package will do.

cmake

Lua

We will require a nix lua setup to work in tandem with the runtime nix packages.

lua

Naturally we will need to pass it in as well.

lua

Conan

To work with windows, conan is needed for handling much of the C++ packages.

conan
conan

Runtime Dependencies

Boost

This is essentially linked against, so it will be used as a buildInput.

boost

Lua Packages

We will not bother building them, since they are already provided.

luaPackages.luafilesystem

To do so, however, we will need to pass the luaPackages function.

luaPackages

Variable Compilation

We will now enable the argument parsing ability of nix-\* commands as enumerated in this outdated gist.

compiler ? "clang"

This will now allow us to pass the compiler argument to our commands:

# Usage Example
# nix-shell --argstr compiler gcc5 --run bash
# nix-shell --argstr compiler clang --run bash

Inputs

Very quickly we shall enumerate the reuired inputs as per the README.

lib
boost
cmake

Nix Package Channel

To pin down the dependencies even further, we will manually determine the branch of NixOS and the package channel in ./nix/nixpkgs. We shall control these parameters by a json file as shown, which is self explanatory.

{
  "owner": "NixOS",
  "repo": "nixpkgs-channels",
  "branch": "nixos-unstable",
  "rev": "ae002fe44e96b868c62581e8066d559ca2179e01",
  "sha256": "1bawyz3ksw2sihv6vsgbvhdm4kn63xrrj5bavg6mz5mxml9rji89"
}

It is pertinent to note that for the json file, comments cannot be added during org-babel-tangle as they cause parsing errors.

Entry

As with other subfolders, we will require a default.nix, for pedagogical purposes, we shall divide the variable into definitions.

# Define
let
  <<nn_pkgVars>>
in
  import src

JSON Parser

We will leverage a json file as the user’s point of entry. That is, we will load data describing our NixOS package channel via this file.

spec = builtins.fromJSON (builtins.readFile ./default.src.json);

Using the JSON

We will fetch the appropriate tar file on the basis of data parsed via the builtins.

fetchTarball = import ./fetchTarball-compat.nix;
src = fetchTarball {
  url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
  sha256 = spec.sha256;
};

In order to marshall the data correctly, we require a compatibility layer on the existing function (fetchTarBall). This is to ensure backwards compatibility with all NixOS versions.

# fetchTarball version that is compatible between all the versions of Nix
{ url, sha256 }@attrs:
let
  inherit (builtins) lessThan nixVersion fetchTarball;
in
if lessThan nixVersion "1.12" then
  fetchTarball { inherit url; }
else
  fetchTarball attrs

Where we note that the @-pattern is used to name the entire set, i.e, both url and sha256 are contained in attrs.