Nix packaging, the heretic way (__noChroot = true)

Tags
Nix
Updated at
Jun 24, 2022 8:45 AM
Published at

There are many different ways to package projects with Nix. This is the heretic way 😊

Don’t use this for nixpkgs - PRs will be rejected.

In a repo that is being actively developed, the set of dependencies tends to change quite a bit. And if you decided to package the repository with Nix, you often end up having to keep the Nix code in sync; either by updating a vendorSha256(which can’t be easily calculated), or generating a deps.nix file (which gets out of sync), or by using IFD tricks (which means that build and evaluation aren’t separated anymore), or you get Nix to translate the language lockfile at evaluation time (making the evaluation slower).

The further you go on the right of this list, the more complicated the Nix code gets and deeper the language package manager integration has to be. Which also increases the chances of things breaking apart when updating the language.

In this article, I want to show you another way; the heretic way. It’s one of the fastest, low-code, and reliable ways I found to package things with Nix. And it’s not really talked about, so that’s why I’m writing this article.

Packaging with __noChroot = true

By setting __noChroot = true as an attribute to a derivation, it tells Nix to disable the sandbox during the build on that derivation. This removes some of the purity guarantees of Nix, and

In nix, there are two types of derivations. Let’s call them “input” and “output” derivations.

Input derivations have all their dependencies declared. All the inputs are specified and are combined together to compute the output hash. The hash of /nix/store/<hash>-<name> depends on that computation. The build environment is sandboxed by default and only has access to the input store paths. It cannot reach the Internet or other parts of the filesystem.

Output derivations have inputs, but the hash of /nix/store/<hash>-<name> depends on the outputHash attribute. The build environment is not sandboxed; it’s completely open to the Internet (and other parts of the filesystem). After the build is done, it will compare the hash of the output with the outputHash value and fail if they are not the same. This is what we use to for example download tarballs from the Internet. The first time we compute that hash, and then trust that it will stay the same in the next build. Some people call this Trust On First Use (TOFU) security.

When adding the __noChroot = true attribute to an input derivation, it removes the sandbox (selectively), thus allowing to fetch things on the Internet (and the local filesystem). And we lose the TOFU guarantee that the output will be the same on every run.

It’s the most reliable and simple to implement, based on my experience. And it all relies on one condition: that the nix.conf has sandbox = relaxed set, and then annotate derivations that fetch the language dependencies with __noChroot = true.

Nodejs example

Recently, I was using npmlock2nix to translate the npm package-lock.json to Nix at evaluation time. It’s a great project but unfortunately, it doesn’t support the package-lock.json v2 format. This is not a criticism of the project itself, it’s just a lot of work to keep up with the nodejs ecosystem (and the format itself got even more complicated). But this meant that we were stuck with nodejs 14.x, which is 2 years old and is not actively supported anymore. And because most frontend developers are on macOS, they often don’t have Nix installed and might use a different version of nodejs.

So here is the new heretic solution: fetch the dependencies with __noChroot = true in one derivation, and then symlink them into the main build:

# Fill these arguments how you like
{ nixpkgs # an instance of nixpkgs
, nix-filter # an instance of https://github.com/numtide/nix-filter
}:
let self = 
{
  # Pick the version of nodejs to use
  nodejs = nixpkgs.nodejs_18-x;

  # Build the node_modules separately, from package.json and package-lock.json.
  #
  # Use __noChroot = true trick to avoid having to re-compute the vendorSha256 every time.
  node_modules = nixpkgs.stdenv.mkDerivation {
    name = "node_modules";

    src = nix-filter {
      root = ./.;
      include = [
        ./package.json
        ./package-lock.json
      ];
    };

    # HACK: break the nix sandbox so we can fetch the dependencies. This
    # requires Nix to have `sandbox = relaxed` in its config.
    __noChroot = true;

    configurePhase = ''
      # NPM writes cache directories etc to $HOME.
      export HOME=$TMP
    '';

    buildInputs = [ self.nodejs ];

    # Pull all the dependencies
    buildPhase = ''
      ${self.nodejs}/bin/npm ci
    '';

    # NOTE[z]: The folder *must* be called "node_modules". Don't ask me why.
    #          That's why the content is not directly added to $out.
    installPhase = ''
      mkdir $out
      mv node_modules $out/node_modules
    '';
  };

  # And finally build the frontend in its own derivation
  my-frontend = nixpkgs.stdenv.mkDerivation {
    name = "my-frontend";
    # Use the current folder as the input, without node_modules
    src = nix-filter {
      root = ./.;
      exclude = [
        ./.next
        ./node_modules
      ];
    };

    nativeBuildInputs = [ self.nodejs ];

    buildPhase = "npm run build";

    configurePhase = ''
      # Get the node_modules from its own derivation
      ln -sf ${self.node_modules}/node_modules node_modules
      export HOME=$TMP
    '';

    # TODO: move to different derivation
    doCheck = true;
    checkPhase = ''
      npm run test
    '';

    # This is specific to nextjs. Typically you would copy ./dist to $out or
    # something like that.
    installPhase = ''
      # Use the standalone nextjs version
      mv .next/standalone $out

      # Copy non-generated static files
      cp -R public $out/public

      # Also copy generated static files
      mv .next/static $out/.next/static

      # Re-link the node_modules
      rm $out/node_modules
      mv node_modules $out/node_modules

      # Wrap the script
      cat <<ENTRYPOINT > $out/entrypoint
      #!${nixpkgs.stdenv.shell}
      exec "$(type -p node)" "$out/server.js" "[email protected]"
      ENTRYPOINT
      chmod +x $out/entrypoint
    '';
  };
}; in self

.NET example

For .NET, there is a tool called nuget2nix that is called to generate a deps.nix file, which is then passed to nixpkgs.buildDotnetModule‘s nugetDeps argument. So every time a dependency changes, don’t forget to call that tool again. Because most .NET developers are on Windows, they also don’t have the tool installed, so we wrote a CI step that would check that the file was up to date and push a fixup commit otherwise. Then we re-discovered that dependabot doesn’t have the same permissions and would fail. This dance was starting to get old pretty fast.

So here is the new heretic solution: fetch the dependencies with __noChroot = true in one derivation, and then pass them into the main build:

# Fill these arguments how you like
{ nixpkgs # an instance of nixpkgs
, nix-filter # an instance of https://github.com/numtide/nix-filter
}:
let self = 
{
  # The .NET packages we want to use
  dotnet-sdk = nixpkgs.dotnetCorePackages.sdk_6_0;
  dotnet-runtime = nixpkgs.dotnetCorePackages.aspnetcore_6_0;

  # Fetch all the dependencies in one derivation with __noChroot = true
  nugetDeps = nixpkgs.stdenv.mkDerivation {
    name = "nuget-deps";

    # HACK: break the nix sandbox so we can fetch the dependencies. This
    # requires Nix to have `sandbox = relaxed` in its config.
    __noChroot = true;

    # Only rebuild if the project metadata has changed
    src = nix-filter {
      root = ./.;
      include = [
        (nix-filter.isDirectory)
        (nix-filter.matchExt "csproj")
        (nix-filter.matchExt "slnf")
        (nix-filter.matchExt "sln")
      ];
    };

    nativeBuildInputs = [
      nixpkgs.cacert
      self.dotnet-sdk
    ];

    # Avoid telemetry
    configurePhase = ''
      export DOTNET_NOLOGO=1
      export DOTNET_CLI_TELEMETRY_OPTOUT=1
    '';

    projectFile = "my-api.slnf";

    # Pull all the dependencies for the project
    buildPhase = ''
      for project in $projectFile; do
        dotnet restore "$project" \
          -p:ContinuousIntegrationBuild=true \
          -p:Deterministic=true \
          --packages "$out"
      done
    '';

    installPhase = ":";
  };

  # Build the project itself
  my-api = nixpkgs.buildDotnetModule {
    pname = "my-api";
    version = "0";

    src = nix-filter {
      root = ./.;
      exclude = [
        # Filter out C# build folders
        (nix-filter.matchName "bin")
        (nix-filter.matchName "logs")
        (nix-filter.matchName "obj")
        (nix-filter.matchName "pub")
        (nix-filter.matchName ".vs")
      ];
    };

    projectFile = "my-api.slnf";

    # Replace the `nugetDeps = ./deps.nix` with the derivation.
    # This is only possible for nixpkgs that contains this PR:
    # https://github.com/NixOS/nixpkgs/pull/178446
    nugetDeps = self.nugetDeps;

    dotnet-sdk = self.dotnet-sdk;
    dotnet-runtime = self.dotnet-runtime;

    executables = [
      "MyAPI"
    ];
  };
}

Using it with Flakes

The experimental Nix Flakes feature can have settings be applied on a per-project basis, and that also works with sandbox = relaxed.

diff --git a/flake.nix b/flake.nix
index 1f08bfa..eb95352 100644
--- a/flake.nix
+++ b/flake.nix
@@ -8,6 +8,9 @@
     flake-parts.inputs.nixpkgs.follows = "nixpkgs-lib";
   };
 
+  # Allow __noChroot derivations
+  nixConfig.sandbox = "relaxed";
+
   outputs = { self, flake-parts, ... }:
     flake-parts.lib.mkFlake { inherit self; } {
       systems = [ "x86_64-linux" ];

And the first time developers will nix build the project, they will be prompted if they trust that setting or not:

do you want to allow configuration setting 'sandbox' to be set to 'relaxed' (y/N)? y
do you want to permanently mark this value as trusted (y/N)? y

Explaining __noChroot = true

Understanding the trade-off

When using this solution, we trade some amount of reproducibility for convenience. This is important to understand; we lose the ability to guarantee that the output will be the same. We rely on the language package manager to fetch the content, and always lay it out the same way on the filesystem. Nix itself doesn’t guarantee that anymore.

Also compared to some solutions, the dependencies are chunked as one unit of compilation. If one dependency changes, it will have to re-download the whole set. This means that there will be more disk and binary cache usage. It hasn’t been a practical issue so far for us, but it’s one to be aware of. Getting more granular rebuilds can improve the overall caching.

On the other side, I used to work on a project that required users to set sandbox = false in their nix.conf, which disabled the sandboxing for all derivation. This is the extra heretic way 🙂 With sandbox = relaxed, at least it can be done selectively.

When this is applicable

As I said in the intro, never use this in nixpkgs. nixpkgs is a canonical set of packages where we want to apply the highest level of reproducibility possible.

But if you use Nix as a build system for a project, in a similar manner than Bazel, then this is where this trade-off can start to make sense. Typically you want to be able to trust all the actors that submit PRs to the project as well, so it’s best to use that on a private repo only.

That’s all, hope this was interesting!