I know Garnix quite well. We have the pleasure to count them as one of Numtide's customers. Julian and I launched the Nix Swiss-French meetup together. And I have been using them on my home repository for the past 6 months. So that's the disclaimer and context.

It's also why I know and understand Garnix quite well. Like I said in my post, it's surprising to me that not more people are using it today. To me, Garnix is the perfect Nix CI. Let me share what I see:

Then combine this with blueprint to keep your flake super lean. Mergify + renovate to get automated flake updates. And you have a really nice setup.

The best is when I pull my dotfiles repo and everything is already prebuilt. No amount of customization I do on top of nixpkgs and NixOS is going to penalize me. And Nix is really _the_ language for people that like to fearlessly customize things.

Ok, enough rambling for a day. Does the vision make sense or are there aspects you’re considering that I haven’t touched upon?

The same issue appears in Dockerfile and other developer environments. The developer wants to get something done, something changed, and they are now fixing their environment.

What if the user was in control? What if they could decide when to update their software? And rollback to a previous version if things start breaking?

In order to make that a reality, you would have to describe

This is only

This is something we take for granted when using Nix.

This is what Nix has to offer.

That’s Nix.

However, moving registries without causing disruption takes a long time. There is always a long tail of users that depend on the current location, buried in 20 layers of scripts or recursive Dockerfiles. Projects were being held hostage in some way.

Because Numtide worked with Scarf before, I had migrated https://github.com/nix-community/docker-nixpkgs to it, and had it bound to a custom domain; docker.nix-community.org. So while all the other projects were scrambling away, all I had to do, is redirect the domain to a different registry. The only caveat that I found is that the images must live on the same path prefix, due to a limitation of the docker registry protocol.

In the end, Docker Inc. reverted their stance, but that is a good reminder; if you don’t control the domain, you don’t control the project. It was nice to be able to sit back and relax while all the other projects were scambling away.

NixOS modules that ship with nixpkgs has been slowly adopting it and now we have some experience of how it plays out in practice.

Here I will discuss the pros and cons of settings and its usage: when it's appropriate to use it and when to avoid it. I'll also give you some recommendations.

What is the settings option?

This new approach makes NixOS services configuration more extensible while reducing the number of module options authors have to describe. For instance, instead of writing:

services.foo.extraConfig = ''
  # Can't be set in multiple files because string concatenation doesn't merge such lists
  listen-ports = 456, 457, 458

  # Can't override this setting because the module hardcodes it
  # bootstrap-ips = 172.22.68.74

  enable-ipv6 = 0
  ${optionalString isServer "check-interval = 3600"}
'';

You can now write:

services.foo.settings = {
  listen-ports = [ 456 457 458 ];
  bootstrap-ips = [ "172.22.68.74" ];
  enable-ipv6 = false;
  check-interval = mkIf isServer 3600;
};

So, services now use pure Nix types for different settings instead of configuration file snippets.

Upsides

The most significant benefit of settings is that configuration files now have the same extension properties as the rest of NixOS modules. So it's easier to set and override configuration options for any modules. So, overall, it helps with module composability.

Another benefit is that the module author doesn't have to predict where the extension points should be added; all the configuration is extensible by default.

Another bright side is that the configuration is more likely to be valid, as its evaluation is tied to the Nix evaluation. This moves most typos from runtime issues to nix evaluation time issues (but not semantic issues).

Downsides

The users now have to convert configuration snippets from sources, such as documentation and Stackoverflow pieces of advice, to Nix code, and do it manually! This introduces manual friction and makes it harder to compare to previous or future versions of the configuration to see what changed between them.

This can be especially painful if the Nix data types don't map cleanly to the configuration format. @tazjin showed me a great example of such an issue the other day. A nightmare to work with!

We also had users’ reports that the capitalisation change needs to be clarified. In the nix world, we use camelCase but then setting keys, map 1:1 to the configuration option and might use another rule. Eg: services.biboumi.credentialsFile vs services.biboumi.settings.xmpp_server_ip.

Recommendations to module authors

• Avoid \\settings\\ if the types don't map well. Don't use the settings if there is no precise 1:1 mapping between Nix data types and the target config file.

• Provide a \\configFile\\ escape hatch to the user. Every module that exposes a settings option should also provide a configFile option that contains the generated config file. This gives the user an escape hatch. The option documentation should be explicit that providing an alternative configFile will ignore all the settings options.

config.settings.myservice.configFile = pkgs.writeText "myservice-config.toml" (''
  # Hey, I can now add comments to the config file
'' + ''
  and_compose_snippets = true
'');

• Provide Nix library to load configuration snippets. In order to retain the ability to copy-paste snippets around, it would be nice to provide a pure Nix parsing library that can convert the configuration snippet to Nix data. This is a stretch goal and has some evaluation performance implications. Eg:

services.myservice.settings = lib.fromTOML ''
  user = "hello"
'';

Conclusion

Here, I explored what the settings - the new option in nix modules are. I looked at the upsides and downsides and gave some advice on its usage.

So now you are informed to employ it as intended!

Extra notes

Baseline

The first thing to establish is Nix has a client (nix CLI) / server (nix-daemon) architecture.

They read the same configuration format but look at different places and don’t look at different keys in the file (with some overlapping).

Configuration reference:

Server

The nix-daemon reads from /etc/nix/nix.conf.

The configuration is loaded while the process is starting. If you change the config, don’t forget to restart the nix-daemon (eg: pkill nix-daemon). On NixOS this is handled automatically for you.

The server is mainly interested in configuration keys that change the building environment. Things like where to substitute existing build results from, how the build sandbox is configured, the max-jobs concurrency, …, and who are the trusted-users (more on this later).

Client

The client reads its config from $NIX_USER_CONF_FILES, ~/config/nix/nix.conf and also /etc/nix/nix.conf, merging all the keys from right to left.

The client is mainly concerned about evaluation and CLI settings, like the nix-path, restrict-eval, if flakes are enabled, …

Overriding server configuration from the client (trusted-users)

We’re getting to the meat of the confusion.

When the nix client initiates a build (aka “realizes the derivation”), the nix client reads /etc/nix/nix.conf, and diffs it with its current configuration. If it sees changes in substituters and other build-specific configurations, it will forward them to the server along the build.

Changing build configuration can affect the system’s integrity. This allows replacing /nix/store entries with corrupted payloads (like viruses). And probably escalate permissions to root.

So by default, the server rejects configuration changes. Unless the sending user is part of the trusted-user list. If you ever saw an error message saying you are not a trusted user, that’s why.

On the other hand, this is very handy for setting per-project caches. Typically each project has its binary cache, and changing the server configuration on every project switch takes a while.

TODO: I’m not sure, but I don’t think what is described here works with remote builders. Even if it worked, it doesn’t know what the remote builder config looks like, it’s only comparing it to /etc/nix/nix.conf.

Flakes

Talking about flakes, if the flake.nix has a nixConfig section, the client will also load config from there.

Even if the user is not a trusted-user, this is a standardized place to document your project’s requirements. Eg:

{
  description = "My flake";

  # Points to our cache containing all the build results published by the CI.
  nixConfig.extra-substituters = [ "https://nix-community.cachix.org/" ];
  nixConfig.extra-trusted-public-keys = [ "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=" ];

  # ...
  inputs = {};
  outputs = { self, ... }: {};
}

Before applying the changes, the Nix CLI will ask you if you trust the configuration changes. This makes it safer to have your user be part of the system’s trusted-users list (assuming you trust that user).

Conclusion

As you might notice, I didn’t specify which configuration key applies to the server or the client. This belongs to the reference documentation. Hopefully, somebody will be motivated to fix it. 😇

As usual, ping me if anything here needs some clarification.

This is for people enabling AWS Control Tower on an existing AWS Organization.

AWS provides documentation on how to enroll existing AWS accounts. They mention that the old AWS accounts need an AWSControlTowerExecution role. And then never tells you how to create one.

So here is how:

• Log into the account that needs to be enrolled.
• IAM →Roles → Create Role
• Select trusted entity: AWS Account → Another AWS account. Enter the Management AWS Account ID → Next
• Add permissions: AdministratorAccess ( arn:aws:iam::aws:policy/AdministratorAccess ) → Next
• Name, review, and create:

Simple in retrospect

And sometimes it’s hard to go to your customer/boss and tell them you have to spend the next 3 weeks doing “things right”(tm).

Luckily there is a workaround available, and this is why I’m writing this article. To show a quick but impure alternative that can be used in a pinch.

Don’t use this for nixpkgs - PRs will be rejected. And Hydra doesn’t build those either.

Use __noChroot = true in a pinch

By default, derivations are built in a sandboxed environment, that doesn’t allow them to use the network. This is one of the core features that is used to make builds more reproducible. And also one of the main reasons why an impure build would fail.

By adding __noChroot = true on a derivation, it turns off the sandbox selectively for that derivation. Note that all users also need to have sandbox = relaxed set in their nix.conf or nixConfig.sandbox = "relaxed" in their flake.nix.

So this is not a good solution for open source projects but can work in an enterprise setting, which is the only place I recommend using this.

Example packaging flow-bin

{ nixpkgs ? import <nixpkgs> { } }:
let
  version = "0.105.1";
in
nixpkgs.runCommand "flow-bin-${version}"
{
  # Disable the Nix build sandbox for this specific build.
  # This means the build can freely talk to the Internet.
  __noChroot = true;

  # Add all the build time dependencies
  nativeBuildInputs = [
    # Automatically patchelf all installed binaries
    nixpkgs.autoPatchelfHook
  ];

  # Add all the runtime dependencies
  buildInputs = [
    nixpkgs.nodejs
  ];
}
	# This part is a bit like a Dockerfile, without the apt-get installs.
  ''
    # Nix sets the HOME to something that doesn't exist by default.
	  # npm needs a user HOME.
    export HOME=$(mktemp -d)

    # Install the package directly from the Internet
    npm install flow-bin@${version}

    # Fix all the shebang scripts in the node_modules folder.
    patchShebangs .

    # Copy the node_modules and friends
    mkdir -p $out/share
    cp -r . $out/share/$name

    # Add a symlink to the binary
    mkdir $out/bin
    ln -s $out/share/$name/node_modules/.bin/flow $out/bin/flow
  ''

This kind of approach is quite generally applicable and should work for other languages as well. Of course it’s less reproducible, and if anything changes in the build script, there are no incremental build layers like in Docker.

You could split the build into different phases though, and that’s what we’ll be seeing next.

Example packaging a NextJS project

In this case, the code changes quite often and comes from the monorepo directly. We are not packaging a third-party library, but something that our developers are evolving daily.

We were using npmlock2nix to translate the npm package-lock.json to Nix at evaluation time. It’s a great project but, unfortunately, it doesn’t support the package-lock.json v2 format. This is not a criticism of the project itself, it’s just a lot of work to keep up with the nodejs ecosystem (and the format itself got even more complicated).

This meant that we were stuck with nodejs 14.x, which is 2 years old and is not actively supported anymore. And most frontend developers on the team also didn’t care about Nix and just installed nodejs directly in their macOS, meaning they had the latest version that generates v2 formats.

So __noChroot = true comes to the rescue. Here we split the build and install the node_modules impurely, but keep the core project sandboxed. This allows to minimize the surface and rebuild that happens.

# Fill these arguments how you like
{ nixpkgs ? import <nixpkgs> { }
, nix-filter # an instance of https://github.com/numtide/nix-filter
}:
let self =
{
  # Pick the version of nodejs to use
  nodejs = nixpkgs.nodejs_18-x;

  # Build the node_modules separately, from package.json and package-lock.json.
  #
  # Use __noChroot = true trick to avoid having to re-compute the vendorSha256 every time.
  node_modules = nixpkgs.stdenv.mkDerivation {
    name = "node_modules";

    src = nix-filter {
      root = ./.;
      include = [
        ./package.json
        ./package-lock.json
      ];
    };

    # HACK: break the nix sandbox so we can fetch the dependencies. This
    # requires Nix to have `sandbox = relaxed` in its config.
    __noChroot = true;

    configurePhase = ''
      # NPM writes cache directories etc to $HOME.
      export HOME=$TMP
    '';

    buildInputs = [ self.nodejs ];

    # Pull all the dependencies
    buildPhase = ''
      ${self.nodejs}/bin/npm ci
    '';

    # NOTE[z]: The folder *must* be called "node_modules". Don't ask me why.
    #          That's why the content is not directly added to $out.
    installPhase = ''
      mkdir $out
      mv node_modules $out/node_modules
    '';
  };

  # And finally build the frontend in its own derivation
  my-frontend = nixpkgs.stdenv.mkDerivation {
    name = "my-frontend";
    # Use the current folder as the input, without node_modules
    src = nix-filter {
      root = ./.;
      exclude = [
        ./.next
        ./node_modules
      ];
    };

    nativeBuildInputs = [ self.nodejs ];

    buildPhase = "npm run build";

    configurePhase = ''
      # Get the node_modules from its own derivation
      ln -sf ${self.node_modules}/node_modules node_modules
      export HOME=$TMP
    '';

    # TODO: move to different derivation
    doCheck = true;
    checkPhase = ''
      npm run test
    '';

    # This is specific to nextjs. Typically you would copy ./dist to $out or
    # something like that.
    installPhase = ''
      # Use the standalone nextjs version
      mv .next/standalone $out

      # Copy non-generated static files
      cp -R public $out/public

      # Also copy generated static files
      mv .next/static $out/.next/static

      # Re-link the node_modules
      rm $out/node_modules
      mv node_modules $out/node_modules

      # Wrap the script
      cat <<ENTRYPOINT > $out/entrypoint
      #!${nixpkgs.stdenv.shell}
      exec "$(type -p node)" "$out/server.js" "$$@"
      ENTRYPOINT
      chmod +x $out/entrypoint
    '';
  };
}; in self

Example packaging a .NET project

For .NET, there is a tool called nuget2nix that is called to generate a deps.nix file, which is then passed to nixpkgs.buildDotnetModule‘s nugetDeps argument. So every time a dependency changes, don’t forget to call that tool again. Because most .NET developers are on Windows, they also don’t have the tool installed, so we wrote a CI step that would check that the file was up to date and push a fixup commit otherwise. Then we re-discovered that dependabot doesn’t have the same permissions and would fail. This dance was starting to get old pretty fast.

So here is the new solution: fetch the dependencies with __noChroot = true in one derivation, and then pass them into the main build:

# Fill these arguments how you like
{ nixpkgs ? import <nixpkgs> { } # an instance of nixpkgs
, nix-filter # an instance of https://github.com/numtide/nix-filter
}:
let self =
{
  # The .NET packages we want to use
  dotnet-sdk = nixpkgs.dotnetCorePackages.sdk_6_0;
  dotnet-runtime = nixpkgs.dotnetCorePackages.aspnetcore_6_0;

  # Fetch all the dependencies in one derivation with __noChroot = true
  nugetDeps = nixpkgs.stdenv.mkDerivation {
    name = "nuget-deps";

    # HACK: break the nix sandbox so we can fetch the dependencies. This
    # requires Nix to have `sandbox = relaxed` in its config.
    __noChroot = true;

    # Only rebuild if the project metadata has changed
    src = nix-filter {
      root = ./.;
      include = [
        (nix-filter.isDirectory)
        (nix-filter.matchExt "csproj")
        (nix-filter.matchExt "slnf")
        (nix-filter.matchExt "sln")
      ];
    };

    nativeBuildInputs = [
      nixpkgs.cacert
      self.dotnet-sdk
    ];

    # Avoid telemetry
    configurePhase = ''
      export DOTNET_NOLOGO=1
      export DOTNET_CLI_TELEMETRY_OPTOUT=1
    '';

    projectFile = "my-api.slnf";

    # Pull all the dependencies for the project
    buildPhase = ''
      for project in $projectFile; do
        dotnet restore "$project" \
          -p:ContinuousIntegrationBuild=true \
          -p:Deterministic=true \
          --packages "$out"
      done
    '';

    installPhase = ":";
  };

  # Build the project itself
  my-api = nixpkgs.buildDotnetModule {
    pname = "my-api";
    version = "0";

    src = nix-filter {
      root = ./.;
      exclude = [
        # Filter out C# build folders
        (nix-filter.matchName "bin")
        (nix-filter.matchName "logs")
        (nix-filter.matchName "obj")
        (nix-filter.matchName "pub")
        (nix-filter.matchName ".vs")
      ];
    };

    projectFile = "my-api.slnf";

    # Replace the `nugetDeps = ./deps.nix` with the derivation.
    # This is only possible for nixpkgs that contains this PR:
    # https://github.com/NixOS/nixpkgs/pull/178446
    nugetDeps = self.nugetDeps;

    dotnet-sdk = self.dotnet-sdk;
    dotnet-runtime = self.dotnet-runtime;

    executables = [
      "MyAPI"
    ];
  };
}

Conclusion

So there you have it. I hope that the explanation and examples give you an idea of how to apply this in various contexts, and help unblock some packaging problems that you might have. It’s better to use Nix impurely than not at all, and the sandbox change is really localized and controlled.

Of course, if you want help with pure packaging, you can always reach out to Numtide and we can help you out.

That’s all, hope this was interesting!

There is just one problem with this claim; what if you need to access unfree packages? For example, try running:

$ nix run nixpkgs#slack

error: Package ‘slack-4.22.0’ in /nix/store/fbcgjqs34vllzzppa1y213fbxx01sxn7-source/pkgs/applications/networking/instant-messengers/slack/default.nix:83 has an unfree license (‘unfree’), refusing to evaluate.

       a) To temporarily allow unfree packages, you can use an environment variable
          for a single invocation of the nix tools.

            $ export NIXPKGS_ALLOW_UNFREE=1

       b) For `nixos-rebuild` you can set
         { nixpkgs.config.allowUnfree = true; }
       in configuration.nix to override this.

       Alternatively you can configure a predicate to allow specific packages:
         { nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
             "slack"
           ];
         }

       c) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
         { allowUnfree = true; }
       to ~/.config/nixpkgs/config.nix.
(use '--show-trace' to show detailed location information)

Oops!

This whole error message is misleading. All the solutions proposed don’t work when using Flakes because Flakes evaluation is pure and doesn’t take your environment variables or config into account.

In order to fix that problem, allow me to introduce a new project:

So now with that, you can run:

``plain text $ nix run --no-write-lock-file github:numtide/nixpkgs-unfree#slack

It’s also usable as a flake, so you can point nixpkgs to it:

In the future, I also want to also keep the channels synchronized with nixpkgs so you can run nix run github:numtide/nixpkgs-unfree/`. And potentially provide a binary cache for it.

That’s it for now, have a great weekend!

Discussion

If you want to comment on the article, head over to the NixOS Discourse:

Addendum: how many instances of allowUnfree are there on GitHub?

Thanks to @tazjin for pointing me to a more usable code search: https://sourcegraph.com/search?q=context:global+file:flake.nix+allowUnfree&patternType=literal

At the time of writing, there are 218 results.

Before we jump in, here is a table of all the costs associated with this configuration.

Costs

• Domain: 74 cents per month for me.
• Notion: free, or $4 per month.
• Super: $12 per month.
• Cloudflare: free.

How it works

flowchart LR
    Browser/Client <--> CloudFlare <--> Super <--> Notion
    CloudFlare --> Worker --> CloudFlare

Notion

Notion is a wiki-like SaaS. That is where the content of the site is stored. In principle, I prefer to use open-source software to hack on it, but there is also value in having access to a system that is maintained and improved over time. Since I am already using it daily to take journalling notes, it makes it easy to swap the content around. This very article was first written in my journal. Notion itself allows you to publish pages from their content, but it’s only available at .notion.site/-

Super

That’s where Super comes in. It’s straightforward to set up and bind a published notion page to a domain, with some bells and whistles on top. For example, the CSS can get tweaked, add some privacy-preserving analytics, change the menu a bit.

Cloudflare

One feature that both Super and Notion are missing is page redirects. Since this website had existing content and links pointing from the outside, I wanted to redirect the users to the new URLs. And I knew that with Cloudflare, it’s possible to intercept the traffic.

Cloudflare workers

Now that Cloudflare is in front, it’s possible to use their Worker feature to handle redirects. Once you’ve created a service, here is a little worker script that I wrote:

addEventListener("fetch", (event) => {
  event.respondWith(handleRequest(event.request));
});

const locationMap = {
  // Put your own routes here
  "/NixFriday": "/old-projects/nixfriday",
  "/NixFlakes": "/notes/nixflakes",
};

async function handleRequest(request) {
  const url = new URL(request.url);

  for (const key in locationMap) {
    // if the path as the 'key' prefix
    if (url.pathname.lastIndexOf(key, 0) == 0) {
      // redirect the user to locationURL
      const locationURL = locationMap[key];
      let headers = new Headers();
      headers.set("Location", locationURL);
      return new Response("The page moved to " + locationURL, {
        status: 302,
        statusText: "Found",
        headers: headers,
      });
    }
  }

  return new Response("This path doesn't match anything: " + url.pathname, {
    status: 500,
    statusText: "Server Error",
  });
}

And then attach the routes that need to be redirected to the worker:

What’s missing

RSS feeds. That isn’t very pleasant, and I hope to solve it in the future.

Some syntax colouring. Nix syntax is supported, but not Terraform or TOML.

The MermaidJS block isn’t supported yet. This is a general issue where super has to keep up with notion changes, and the website could be in a broken state in the meantime.

No backup and restore. Their API allows exporting every page to Markdown so somebody can build a service on top.

Conclusion

My previous setup was a more classic one; a Git repository rendered to GitHub Pages. It had all the features that I wanted, except a minor issue; I wasn’t writing articles. Editing, remembering the syntax, previewing, thinking about an appropriate commit message, ... Somehow, the friction was preventing me from writing.

This new setup is far from ideal, but my bet seems to be paying off so far; Notion has less friction as the content gets published directly.

That’s it! Let me know if any details are unclear.

This is a bit of a PSA for the NixOS community (and me), to try and expose something that I see:

💡 dependencies should not create their own instance of nixpkgs

Especially with the advent of Flakes, soon enough, we will end up with 1000 dependencies, each with its own instance of nixpkgs. Given that nixpkgs takes around 100MiB of RAM and a second to evaluate, that can quickly add up.

How we got there

Overlays everywhere. Here is an example of one of my own projects:

pkgs = import inputs.nixpkgs {
  inherit system;
  config = { };
  overlays = [
    (final: prev: {
      fenix = import inputs.fenix {
        pkgs = prev;
      };
    })
  ];
};

nixpkgs overlays are super useful. They are a mechanism that allows taking nixpkgs, and extending it with your own packages and overrides. In most cases, it’s more manageable than forking nixpkgs and managing your own long-running branch. NixOS also doesn’t provide a standard way to have other package sets so it makes sense to have them all in one. Those two reasons are what made them popular.

There is just one problem; overlays are only usable when creating a new instance of nixpkgs. It’s time to stop using overlays (in most cases, see below).

Solution; composition over inheritance

This title doesn’t make 100% sense but you get it, compose instead of extending nixpkgs. Here are a few scenarios that you might encounter with proposed solutions:

Nix classic

Typically, in a Nix classic project, dependencies are pinned using niv, and then you compose the different sources with something like this:

{ system ? builtins.currentSystem }:
let
  sources = ./nix/sources.nix;

  pkgs = import sources.nixpkgs {
    inherit system;
    config = { };
    overlays = [(final: prev: {
      other-dep = import sources.other-dep { pkgs = prev; };
    })];
  };
in
# your code here accessing `pkgs.other-dep`

Instead of creating this one instance with an overlays, split it up like this:

{ system ? builtins.currentSystem
, sources ? import ./nix/sources.nix
, nixpkgs ? import sources.nixpkgs { inherit system; config = { }; overlays = [ ]; }
, other-dep = import sources.other-repo { pkgs = nixpkgs; };
}:
# your code here accessing `nixpkgs` and `other-dep`

Exposing the constructors as a function argument allows a consumer of your project to inject their own instance of nixpkgs in there, and avoid creating a new instance. And also provide their own version of other-dep if they want to.

💡 pkgs has been renamed to nixpkgs to make it clear that it’s just nixpkgs and not a random set of packages.

Nix Flakes

Here is a synthetic example of what a Flake typically looks like:

{
  description = "My flake";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.other-dep.url = "github:other/dep";

  outputs = { self, nixpkgs, other-dep }: {
      packages = nixpkgs.lib.genAttrs [ "x86_64-linux" ] (system:
        let
          pkgs = import nixpkgs {
            inherit system;
            overlays = [(final: prev: {
              other-dep = import sources.other-dep { pkgs = prev; };
            })];
          };
        in
        # your code here accessing `pkgs` and `pkgs.other-dep`
     );
  };
}

Instead of instantiating a new nixpkgs, access nixpkgs.legacyPackages.${system} and then make sure that all dependencies use the same instance of nixpkgs.

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.other-dep.url = "github:other/dep";
  # Use the same version of nixpkgs as us
  inputs.other-dep.inputs.nixpkgs.follows = "nixpkgs";

  outputs = { self, nixpkgs, other-dep }@inputs: {
      packages = nixpkgs.lib.genAttrs [ "x86_64-linux" ] (system:
        let
          p = {
            nixpkgs = inputs.nixpkgs.legacyPackages.${system};
            # other-dep would also access `inputs.nixpkgs.legacyPackages.${system}`
            # thus only using a single instance of it.
            other-dep = inputs.other-dep.packages.${system};
          };
        in
        # your code here accessing `p.nixpkgs` and `p.other-dep`
     );
  };
}

That way, there will only be a single instance of nixpkgs being evaluated, and consumers of your project can again follow the same practice.

NixOS

NixOS is a tough one because there are interactions between the module system and the packages. When using nixos-rebuild, NixOS will create its own instance of nixpkgs, based on the NIX_PATH and channels by default, and configured by the nixpkgs.* options. Or when using flakes, it calls pkgs.nixos that injects its own instance of nixpkgs to the nixpkgs.pkgs option. There isn’t really room to provide more package sets side-by-side.

Luckily NixOS is a bit out of scope for this article because typically NixOS configs are at the root of the dependency tree 😅.

Ideally, we would introduce a new top-level packages attribute that can hold package sets side by side and would be used like this:

{ config, ... }:
{
  systemPackages = [ config.packages.nixpkgs.hello ];
}

Some more arguments against overlays

Did you ever hit hard to debug infinite recursion issues? Without overlays, those are gone.

Given that the instance of pkgs is a global namespace, it can become difficult to reason about it once a few overlays have been added. Are they all using their own prefix inside of that global namespace? Is there any chance they might clash over each other? Are they overriding existing packages? All of this is gone without overlays.

Overlays are opaque before being applied. So tools like nix flake show won’t be able to inspect their content.

When to use overlays

To being said, even with all these arguments against overlays, there are places where they are still useful:

Contrary to what I said, if there are no Nix consumers of your repository, then don’t mind me, go crazy. This article is really aimed at 3rd-party dependencies and hopes to change the status quo.

Another example would be if your project really needs to patch nixpkgs. To get a whole set of nixpkgs out, with some internal dependency replaced with your own version. Imagine needing nixpkgs, but with a different version of OpenSSL, or different build flags. The point is that in these cases, you wouldn’t add new attributes to nixpkgs and only modify existing ones.

Conclusion

In this article we have seen two things; when to use overlays, and how to avoid creating too many instances of nixpkgs. Of course, the reality is always more nuanced than the points and I’m sure you will find corner cases where you still need to reach for those tools. But I hope you got the overall points and that it made sense.

Thanks for reading!

For comments and discussion: https://discourse.nixos.org/t/1000-instances-of-nixpkgs/17347/1

TL;DR: Nix evaluation is 11-17% faster in Nix 2.6 compared to Nix 2.5.1

Methodology

I have both the nix and nixpkgs repo side by side.

nixpkgs is checked out at ac44b27bab615fd49bc94fe22124deae233b5c94 (latest master)

nix is checked out at 2.6.0

Then run nix-build pkgs/top-level/release.nix -A metrics and collect the output.

For nix 2.5.1 I added a space change to pkgs/top-level/metrics.nix to force the rebuild on my machine.

For nix 2.6 I injected the version from the nix repo:

diff --git a/pkgs/top-level/metrics.nix b/pkgs/top-level/metrics.nix
index d413b881eaa..cf065697923 100644
--- a/pkgs/top-level/metrics.nix
+++ b/pkgs/top-level/metrics.nix
@@ -2,11 +2,17 @@

 with pkgs;

+let
+  nix = (import ../../../nix).defaultPackage.${pkgs.system};
+in
+

Then I took all the results and painstakingly inserted them in Notion, for your pleasure and mine:

The problem is that writing is a messy process. Especially when my ideas are unclear, and it takes many small iterations before I get to a point where I am satisfied with the result. And I do not even claim that my writing style is good.

The point is that Git creates too much friction. I do not care about writing commit messages. It's awkward enough to share my ideas with the world. And now I have to worry about sharing my inner workings on top?

Markup languages also add too much friction. Why do I have to remember how to describe a table, properly nest bullet points. Writing needs to be as free as possible.

So for 2022, I am trying something new. I imported all my pages into Notion.so, slapped Super.so in front of it, and voila, I have a dynamically editable website. There are a few drawbacks, like the lack of an RSS feed and no page redirects (DM me, I have a solution for that), but so far, it's working quite well.

At least that is the theory. Let's see what happens in reality.

TL;DR: only use the count attribute to enable resources.

Basic example

The count attribute can be used to instantiate multiple resources with a single resource declaration. Here we instanciate 10 EC2 instances:

Eg:

``plain text resource "aws_instance" "web" { count = 10 ami = "ami-0cdba8e998f076547" instance_type = "t2.micro" }

On the surface it looks useful but it suffers from a number of limitations that make it almost useless.

## Pattern: `enable` attribute

This is pretty much the only viable use-case for the `count` attribute. Use `count = 0` to disable a resource and `count = 1` to enable it.

Even then, evaluation might fail if the resource is disabled and another resource depends on it’s outputs.

Eg:

## Anti-pattern: avoid repetition

Forget DRY with Terraform. Copy-and-paste is your friend :slight_smile:

This is a natural usage of `count`. Don’t do this:

Let’s say I instantiace that module with:

This will work great on the first invocation. The problem is that each `aws_iam_user` is actually a different resource.

Let’s say that later that `bob` leaves the company.

All the users get re-created, invalidating their AWS credentials. With bad luck your current AWS account might be in that list.

So in conclusion, Terraform deals with individual resources. Having them tied to a specific ordering is quite a bad idea as it makes the application of those resource inflexible.

For that use-case it’s better to copy-and-paste the IAM user. Or write a script that generates the terraform code from the list of users.

Flakes was born when Shea Levy, who worked at Target at the time, decided to hire Eelco (who works at Tweag) to solve a set of issues that Target was having.

Flakes is a set of extensions for the Nix language that is currently (Dec 2021) behind an experimental flag.

While adding a set of useful features, Flakes also gathered some controversy in the NixOS community. This article is summarizing to the best of my ability the various viewpoints of flakes.

Flakes features

Flakes are a lot of things at the same time.

1. A single project entry-point

Nix is very flexible and each project tends to have a slightly different shape. Flakes introduces a top-level flake.nix file that serves as the main entry-point to a project. It declares inputs and outputs for the project (more on that later).

2. Dependency management

Flakes introduces a way to declare third-party dependencies. Instead of using a tool like niv, or manually updating versions and sha256, the flake.nix file declares all these inputs. An additional flake.lock file is introduced to pin those dependencies.

3. Pure evaluation

Flakes disables features such as builtings.getenv, builtins.currentSystem, builtins.getCurrentTime. Given that Nix is supposed to help make builds more reproducible, it’s nice that the evaluation is now pure by default.

4. Evaluation caching

With flakes being pure and controlling the inputs, it’s able to cache the evaluation outputs. The cache is currently keyed based on the Git SHA1. This allows speeding up some operations drastically.

5. New CLI

The Nix CLI has historically been confusing. As part of the Flakes effort, the CLI has been re-vamped to live behind a single nix binary, similar to how all git commands are living behind that name.

Note that the nix command was already in progress before flakes.

Arguments against

Here are all the arguments that I have seen that are against Flakes. It’s not because I listed all of them that they are all necessarily valid. Each argument should be weighed independently.

In order of my mind remembering:

a. Keep nixpkgs a monorepo

Some people consider that the biggest value in nix is not the language, but the nixpkgs repo that contains all the package definitions. And that this value is intrinsically tied to having a monorepo because it allows to easily do large refactors. Their fear is that if Flakes becomes stable, that the repository will be split into many different pieces.

Without the linerization of history, it would also become more expensive to fill the binary cache with all the possible combinations of git commits between the repositories.

b. The Flakes RFC was closed

After long conversations, RFC0049 has been withdrawn. Yet Flakes still got implemented, albeit behind an experimental flag. That makes some people grumpy as they perceive it as being a breach of process.

c. recurceIntoAttrs was lost

d. Flake.nix is not exactly a nix file

While flake.nix has the syntax of a Nix file, only the outputs allow function calls. The rest of the structure is more like JSON, where only pure values are accepted.

e. The flake.nix file is not ergonomic

The flake.nix output, in particular, is quite confusing to people. Most projects use the numtide/flake-utils repo to make them more palatable.

f. The strict dependency on Git

A common issue that new users are facing is that flakes will complain that a file doesn’t exist, when in fact it does. The issue is that flakes only consider files that are part of the git index. git add thefile and then things are working again. And Flakes uses the staging area file list, but not the staging area contents. That makes the git experience counterproductive.

The strict dependency on Git means that companies that are using Mercurial or other source control won’t be able to use flakes.

g. Evaluation caching is not useful during development

Because the evaluation is keyed on the git commit, active development on a repository will mostly likely have invalidated the evaluation cache.

h. System tuples are hard-coded with Flakes

Before flakes appeared, nixpkgs was starting to move past the - tuple (eg: x86_64-linux) to allow better expressing system variants such as distinctions between musl or glibc-based systems. As part of flakes’ design, the use is now locked into these tuples as they are mandated by Nix itself.

Flakes also uses these tuples quite a lot, and they are easy to mistype. Especially x86_64-linux that uses both an underscore and a dash.

More?

Let me know if I forgot any major points.

Conclusion

So here we are. With the recent Nix 2.4 release, Flakes is available behind a feature flag in a stable release. It’s being increasingly adopted by more and more people. Even though it hasn’t gone through an RFC, I think we reached a critical mass of users.

It’s unfortunate that Flakes is a bundle of features that all have to be adopted, or not. It puts us in a bit of a Python 3.0 situation, where it splits the community in two, and the transition is painful. To me, there are clearly good arguments on both sides.

With Bernardo, we spent a bit of time looking at voting systems. We were talking about how to improve governance and leadership, and ended up looking at other communities like Debian, Python, Kubernetes, ... In particular, we were interested in their voting systems as a way to make the system more democratic. If the RFCs could be voted on when they are “ready”, it would change the nature of the shepherds to be mostly helping build the best possible RFC.

The rest of my time was spent benchmarking nix and nixpkgs. I think that our community is relying on nixpkgs overlays too much, and I needed those benchmarks to make one of my points. It resulted in a series of articles that I have started writing and will be publishing soon on https://zimbatm.com/. Of course, as with each new article, I will first have to re-do the website entirely with a new static site generator 😊

WIP blog post: https://numtide.notion.site/nixpkgs-instances-are-expensive-b65f68c4a31145aa986fb99496a21e44

But first, here are two triggers that show that I don’t know what I’m doing:

• Randomly upgrade packages in the hope that it will fix a bug.
• Create bigger instance sizes in the hope that it will fix performance issues.

What is bisect debugging?

The idea is simple; apply the Binary search alorithm to the debugging process. If you can map the search space into a flat line, then you are guaranteed to find a solution in O(log n) steps.

The first step is to map the search space. Find two bounds to the problem. On one side it’s working, and on the other it’s failing.

Cut the space in half, and test if the error occurs there. If it is, move the cursor to the half that’s working, otherwise, move it to the half that’s broken.

Repeat until you pin-point the problem.

That’s the overall process.

Of course, reality is messy and doesn’t always cleanly map to linear search space. And finding the exact half is not always measurable. But following that process has been immensely useful to me, over and over again.

Why is it so good?

Mechanical; debugging is an uncertain process. Uncertainty is a big driver for procrastination. Especially as a young engineer, or when under time pressure, the uncertainty can become quite unbearable. Because bisect debugging is mechanical, I find that it helps reduce that uncertainty and focus on finding the root cause of the problem instead. It also helps to communicate to management or other teams what we tried and what the next steps are.

Knowledge; the process of mapping the search space in itself is useful. It helps expose the areas that I don’t understand. In order to build the mental picture of the search space, I have to know what the space is like. Over time, this makes me a better engineer.

Times it doesn’t work

Mapping is too hard.

The search space is not linearizable.

Examples

TODO: Add some worked examples.

A good one is how to debug a networking connection issue.

Bonus: weighted bisect debugging

With experience, it’s possible to speed up the search process even further. If you know where the problem is more likely to be, split to that location instead of the half. Of course, even with experience, you might still be wrong.

Here are some thoughts on this subject, mostly to clarify my own ideas.

On maintaining direnv

I have been maintaining [Direnv] for 10+ years now and have been fortunate. There hasn’t been too many toxic users there. Maybe a shell extension is geeky enough so that mostly knowledgeable people filter through. I have a sense that with experience, also comes respect. We have been there and know what it means to maintain a project.

But still, it’s not easy.

I still remember taking 30min to reply to a single issue, and I still do from time to time. Saying the right thing is difficult and nerve-wrecking. Especially when the user has a good idea. I don’t want to let them down, but also this particular issue is not interesting to me.

The hardest part, to me, is to set the right mood. Fostering collaboration is difficult. I feel like I haven’t succeeded to do that. On the other hand, it’s quite common to have projects with a single main contributor. So is it my fault? Or is the project not suited for a more vibrant collaboration?

To me the whole endeavor feels a lot like some sort of exercise. A communication exercise. It’s never easy, but it gets easier with time. And it has its own type of rewards.

The biggest thing that I had to learn is where I stand. Each new issue is like a test. I found that if I can communicate that clearly, people are generally understanding. Saying yes is easy, it’s mostly about different types of no: out of scope for this project, interesting but needs a PR, …

That’s all for today.

Links to some other opinions

• https://raccoon.onyxbits.de/blog/bugreport-free-support/
• https://lobste.rs/s/yz15mn/about_open_source_community_opennesss

Conceptually this is the biggest change between traditional Source Control Management (SCM) tools and Git. Tutorials should explain that first:

Traditional SCM: code changes -> commit Git: code changes -> staging area -> commit

To understand Git, the important part is to form that mental model. The code changes don’t go to a commit directly. Instead, they are put in that staging area. That area is a virtual place where all the changes get accumulated. When typing git commit, the changes are moved from that area into the commit.

A user who doesn’t have that mental model will find the Git tooling confusing. This feature both provides more flexibility, and makes all the CLI interfaces more complicated to use.

Here is how I translate a number of commands mentally:

• git add somefile: add the changes in “somefile” into the staging area
• git add -p: selectively add changes to the staging area.
• git commit -a: add all the changes of the tracked files into the staging area, and then create a commit from it.

{ ... }:
{
  # Put your NixOS configuration here. Eg:
  services.nginx.enable = true;
}

The first thing to do is to create another NixOS configuration that includes the amazon-image config and your main config. This is what ultimately is going to end-up on the VM:

{ modulesPath, ... }:
{
  imports = [
    "${modulesPath}/virtualisation/amazon-image.nix"
    # path to your config
    ./configuration.nix
  ];
}

TODO: Setup CI here and Eval NixOS to pre-fill the cache.

With that in hand, we can now write a bit of Terraform code:

``plain text variable "name" { description = "Name prefix" }

triggers = { # Force a new deployment if the instance ID has changed. The ID changes if # the instance is re-created for example. machine_id = aws_instance.machine.id } } `

• No auto-scaling: only a single VM gets configured
• No auto-healing: if the VM goes down, it takes another terraform apply` to re-deploy the system configuration.

Upsides

• Simple setup.
• Direct feedback on deployment.
• It's easy to migrate this auto-scaling in the future.

TODO

• Use CI + Cachix to pre-build the NixOS machine.
• Write a terraform aws_image_nixos_custom module for auto-scaling scenarios.
• Secret management → use SSM
• Better SSH key management?

Preparation

Exercise 1 - installation de paquet

Ouvrez la configuration systeme avec sudo nixos-rebuild edit et changez la configuration.

Trouvez un paquet a installer en parcourant https://nixos.org/nixos/packages.html?channel=nixos-20.03 et ajoutez le a environment.systemPackages.

Par exemple:

{
  environment.systemPackages = [
    pkgs.hello
  ];
}

Une fois modifier, enregistrez le fichier et executez sudo nixos-rebuild switch pour activer la nouvelle configuration.

Vous pouvez tester que la nouvelle configuration se soit activee en executant le programme dans la ligne de command:

$ hello
Hello, world!

Exercise 2 - configuration de service

Explorez les options dans https://nixos.org/nixos/packages.html?channel=nixos-20.03 et choisissez un service a installer.

Tout comme dans l'exercise 1 nous allons ouvrir le fichier de configuration avec sudo nixos-rebuild edit , modifier et enregistrer les changements, et finalement executer nixos-rebuild switch pour activer la configuration.

Si vous trouvez la syntaxe nix confuse, n'hesitez pas a consulter ce petit guide (en anglais): https://github.com/tazjin/nix-1p

Prenez notes des etapes et problemes que vous avez rencontrer.

Exercice 3 - retour sur une ancienne configuration

Redemarrer la VM avec reboot. Dans le menu de demarrage vous trouverez toutes les version precedentes de la configuration systeme.

Chosissez une ancienne version et demarrer la machine avec. Constatez que vous etes bien retourner dans un etat precedent en consultant l'etat systeme. (par exemple avec systemctl status ou en executant un des programmes.

$ hello
hello: command not found

Exercise 4 - reclamez de l'espace

Redemarrer le systeme a la derniere configuration ou lancez nixos-rebuild switch.

Lancez df -h pour voir combien d'espace disque est disponible.

Nous allons maintenant supprimer les anciennes configurations. Lances sudo nix-collect-garbage -d pour effacer les vieux profiles et fichiers associes.

Voila, vous pouvez maintenant constanter que vous avec plus d'espace disponible en lancant df -h a nouveau.