How would I work with a large workspace and only selectively build things?

[TheNeikos]

I am currently working with a larger monorepo with a cargo workspace (+20 crates). However, for my flake I want to ignore most of them and just focus on a select few. How could I do this with crane?
^{I know my way a bit around Nix/NixOs but I did not 'grok' it yet, so any pointers/helping me understand would be amazing!}

The basic hierarchy of the monorepo is:

• cli tool, depends on:
  • [variable amount of crates, but that I can name] that depend on:
    • foo_lib
    • foo_api
  • foo_core, which depends on:
    • foo_api

Now, I'd like that people can do the following:

• Build the crates individually while developing
  • If I work on 'foo_api' I don't want to automatically build everything that depends on it. I think that this is easily covered by just using nix develop somewhere in the tree (or just the root) since the devshell should just expose all the required tools for everything in the graph somehow? (Basically be able to run cargo build/check for any crate in the workspace)
• Build just the cli tool and run it (I can figure out how to expose it as anixos module later on with that I think)

Thank you for the tool!

[ipetkov]

Hi @TheNeikos! The short answer is:

• you can easily build just the crates you want, but changes to unrelated files may result in needing to rebuild the workspace crates
• it may be possible to avoid the rebuilds above but it will take more manual writing of build expressions to achieve (I hope to one day have better automatic support for monorepos but the functionality isn't there yet)

On to the longer/more detailed responses:

Build the crates individually while developing

This is entirely achievable through the dev shell as long as you have the right tools declared. When you run cargo build/check in your terminal you aren't bound by Nix's sandbox, cargo will happily put the artifacts in the target/ directory and its own caching will take care of not rebuilding all dependencies every time!

Build just the cli tool and run it (I can figure out how to expose it as anixos module later on with that I think)

You can definitely use crane to build the CLI tool (so that the binary is available to NixOS modules for example). If the workspace has a ton of other crates that you don't care about being available to other Nix derivations you can always customize the exact cargo command that the build invokes. Here's a rough example:

# This whole example will effectively only build the `foo_cli` crate (and any workspace crates it depends on)
packages.foo_cli = craneLib.buildPackage {
  src = ./.;
  cargoBuildCommand = "cargo build --release -p foo_cli";
};

The downside to this approach is that if you make a change to, say, the bar_api crate, Nix will want to rebuild it all again (that is foo_api, foo_core, foo_lib and foo_cli) since it thinks the entire source path (./.) is required and things will be built in a fresh sandbox. If this isn't an issue, you're done!

But if it is, you could move all the foo crates into their own subdirectory and change the example to:

packages.foo_cli = craneLib.buildPackage {
  src = ./fooCrates; # This line changed!
  cargoBuildCommand = "cargo build --release -p foo_cli";
};

That way Nix will ignore changes to the bar_api crate and only rebuild the derivation when the actual foo_* sources change.

I have some ideas on having crane analyze the workspace layout and automatically construct the right derivations so that each crate can be built with only the sources it relies on so unrelated changes do not cause mass rebuilds, but so far that's all it has been, ideas ;) Hope this helps in the meantime!

[TheNeikos]

Thank you a lot! That does file knowledge gaps I had, so I will try to apply this and then report back!

[dpc]

  cargoBuildCommand = "cargo build --release -p foo-cli";

Would cargoExtraArgs = "--bin foo_cli"; be as good?

The downside to this approach is that if you make a change to, say, the bar_api crate, Nix will want to rebuild it all again (that is foo_api, foo_core, foo_lib and foo_cli) since it thinks the entire source path (./.) is required and things will be built in a fresh sandbox. If this isn't an issue, you're done!

Just to clarify: the external dependencies buildDepsOnly will still reuse the previously build /target, right? I think that's what I'm seeing in my small tests.

  src = ./fooCrates; # This line changed!

@ipetkov :

Are you aware what is the easiest way to filter ./ to effectively achieve ./{fooCrates,commonLib1,common2}. I imagine 99% of the time the workspace is there to share some some common libraries between (otherwise disjoin) packages, and it all comes down to narrowing the set.

Possibly crane could add some support to make it even easier. Eg:

  src = ./;
  srcFilter = "./(fooCrates|commonLib1|common2})" #if done by a regex

or

  src =craneLib.buildSrc(... some argument ...)" #as a crane library utility

[ipetkov]

Would cargoExtraArgs = "--bin foo_cli"; be as good?

Yep that's also a valid alternative

Just to clarify: the external dependencies buildDepsOnly will still reuse the previously build /target, right? I think that's what I'm seeing in my small tests.

Yes, the target artifacts will still be reused, the src = ./fooCrates invocation will use the fooCrates directory as the source root of the derivation (i.e. the builder will not see any other source files)

Are you aware what is the easiest way to filter ./ to effectively achieve ./{fooCrates,commonLib1,common2}. I imagine 99% of the time the workspace is there to share some some common libraries between (otherwise disjoin) packages, and it all comes down to narrowing the set.

nixpkgs has several utilities for filtering out sources, lib.sourceByRegex could be used specifically for your example!

[dpc]

@TheNeikos @ipetkov

I think I've solved this for myself entirely. The only downside is that each application in a workspace will require specifying which directories it is being built from. I guess in theory it would be possible for some more complex automation to figure it out, possibly by calling some cargo subcommand etc. but for now more manual approach seems entirely adequate.

So, in rustshop I will have one big workspace for all the microservices/applications: https://github.com/rustshop/rustshop/tree/main/services and then in the flake.nix I use lib.cleanSourceWith to filter only relevant files/paths: https://github.com/rustshop/rustshop/blob/a264a7fe05e07e44d449998f1ab990a6e2b2f461/services/flake.nix#L64 and build with cargoExtraArgs = "--bin ${name}";.

I possibly could write it more cleanly, but it's a PoC and my Nix-fu is still weak.

[ipetkov]

Yep this is definitely a valid strategy to avoiding rebuilding all binaries in the workspace if only a small subset of the source is changed!

I do want to explore some kind of automated solution for not having to manually keep track of which source files build which binaries (after all, cargo already knows this!), but so far it's just high level ideas :)

[dpc]

Correction: actually working code: https://github.com/rustshop/rustshop/blob/bb73aa85921d86177238e2998017317ab99c1c83/services/flake.nix#L52

I need to include all the Cargo.toml and they all need [lib] and [[bin]] sections filled, so Cargo doesn't have to deduce anything from files that are not there.