Nix is Too Big to Learn All At Once

Epistemic Status: lotus Blooming

I’ve been using Nix and NixOS pretty much non-stop since 2018 and I still find myself learning new things that open up areas that I hadn’t realized were there before. I wanted to take a moment and talk about a few of them because one of the things I’ve realized is that Nix covers so many areas of computing that it’s hard to grasp all at once.

One of the big issues I ran into as I started using NixOS was that all of the documentation that people direct you towards isn’t exactly relevant to what you need to know to use NixOS effectively. Take the Nix Pills series of posts, for example. They start out talking about packaging and making derivations and so forth, and as a new user of NixOS I wasn’t trying to write packages but instead trying to configure my systems to do what I wanted.

Even coming at Nix from the perspective of using it for development environments I probably wouldn’t need to know about how to write a package derivation right at the start.

What did I need to know about in greater detail instead? Modules.

NixOS modules

I needed to know about how the NixOS module system worked.

You might say “Hey, it’s right there in the NixOS manual! RTFM!“.

That would have required me to understand that when I was editing configuration.nixconfiguration.nix, that I was editing a module. Realizing this is hindered even more by the fact that that particular module is using the shorthand version where you don’t have an optionsoptions attribute in the attrset and the configconfig is just the contents of the top level attrset.

Once I learned about them a few years ago, I’ve started modularizing my NixOS configurations, extracting out whole systems so that they can be configured by a single flag. For example, I want an X server on this machine but not that one and the module I wrote handles configuring the server, the display manager, the window manager and so on all behind a single host.xserver.enablehost.xserver.enable flag that I defined.

At a previous job, I had a module for configuring the company VPN on my work laptop. It had to do a lot of changes across the system, but it was all contained and controlled by a single <company-name>.networking.enable<company-name>.networking.enable option. This meant I could turn it all off with a single change if I needed.

Fetchers and build helpers

Another thing I have had to repeatedly remind myself is about all the wonderful things in Nixpkgs under the category of fetchers and build helpers. A lot of these I use regularly.

One that I keep forgetting about is pkgs.writeShellApplicationpkgs.writeShellApplication. This is a “trivial builder” that creates a shell script. So far this is similar to pkgs.writeShellScriptpkgs.writeShellScript or pkgs.writeScriptBinpkgs.writeScriptBin, but it does more than just create an executable script in the store. It also has options for putting packages in the PATH so that you can write your script in a more normal manner.

# Instead of writing something like this...
pkgs.writeScriptBin "foo" ''
  ${pkgs.hello}/bin/hello args args
'';
# Or using something like `getExe`
# You can instead do this...
pkgs.writeShellApplication {
  name = "foo";
  runtimeInputs = with pkgs; [ hello ];
  text = ''
    hello args args
  '';
}

# Instead of writing something like this...
pkgs.writeScriptBin "foo" ''
  ${pkgs.hello}/bin/hello args args
'';
# Or using something like `getExe`
# You can instead do this...
pkgs.writeShellApplication {
  name = "foo";
  runtimeInputs = with pkgs; [ hello ];
  text = ''
    hello args args
  '';
}

In this example, the resulting script contains a line that puts the hellohello package’s binary path into the PATH environment variable. Using writeShellApplicationwriteShellApplication also has the added benefit of checking the script using ShellCheck, and if it fails, the build fails. These things also work if you want to keep your script in a separate file and read it in using builtins.readFilebuiltins.readFile.

Another really useful one that I only recently discovered is requireFilerequireFile. I have a few proprietary fonts that I want to have installed on my systems, but I don’t want to add those files into my NixOS flake. With requireFilerequireFile I can work around that. What requireFilerequireFile does is allow you to tell Nix that there’s a file you expect to have in the Nix store, and if it’s not present, it will fail to build and display a message saying you should add the file manually and retry.

There were some gotchas with this though since both methods it suggests don’t give you the hash you need for the requireFilerequireFile call, so I wrote a simple Bash script to handle it for me in the future:

add-required

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
 
if [[ $#--eq-0| -eq 0 ]] ; then
  echo "Usage:"
  echo "  add-required <absolute-path>"
  exit 1
fi
 
# This command prints some noise to stderr so redirect that to /dev/null
HASH=$(nix-prefetch-url --type sha256 file://"$1" 2> /dev/null)
SRI_HASH=$(nix-hash --to-sri --type sha256 "$HASH")
 
echo "Use the following 'requireFile' invocation:"
echo
echo "requireFile {"
echo "  name = \"$(basename "$1")\";"
echo "  url = <url-to-download-from>;"
echo "  sha256 = \"$SRI_HASH\";"
echo "}"

add-required

#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
 
if [[ $#--eq-0| -eq 0 ]] ; then
  echo "Usage:"
  echo "  add-required <absolute-path>"
  exit 1
fi
 
# This command prints some noise to stderr so redirect that to /dev/null
HASH=$(nix-prefetch-url --type sha256 file://"$1" 2> /dev/null)
SRI_HASH=$(nix-hash --to-sri --type sha256 "$HASH")
 
echo "Use the following 'requireFile' invocation:"
echo
echo "requireFile {"
echo "  name = \"$(basename "$1")\";"
echo "  url = <url-to-download-from>;"
echo "  sha256 = \"$SRI_HASH\";"
echo "}"

Then I can just copy and paste that into a std.mkDerivationstd.mkDerivation for the srcsrc and it just works.

Here’s an example of a derivation I wrote for the Dank Mono font with the relevant part highlighted.

{ stdenv, lib, requireFile, unzip, ... }:
 
stdenv.mkDerivation (finalAttrs: {
  pname = "dank-mono";
  version = "15-Oct-2020";
  src = requireFile {
    name = "Dank-Mono-${finalAttrs.version}.zip";
    url = "https://philpl.gumroad.com/l/dank-mono";
    sha256 = "sha256-zppKaEfi56WPYDS+r/uOkIFDqKdYtTmPm30KxvG3A8Q=";
  };
 
  dontConfigure = true;
  dontBuild = true;
 
  unpackPhase = ''
    ${unzip}/bin/unzip $src -d $out/unpacked
  '';
 
  installPhase = ''
    mkdir -p $out/share/fonts/opentype
    mkdir -p $out/share/fonts/truetype
    mv $out/unpacked/DankMono/OpenType-PS/*.otf $out/share/fonts/opentype
    mv $out/unpacked/DankMono/OpenType-TT/*.ttf $out/share/fonts/truetype
    rm -rf unpacked
  '';
 
  meta = with lib; {
    description = ''
      A typeface designed for coding aesthetes with modern displays in mind.
      Delightful ligatures and an italic variant and bold style.
    '';
    homepage = "https://philpl.gumroad.com/l/dank-mono";
    license = licenses.unfree;
  };
})

{ stdenv, lib, requireFile, unzip, ... }:
 
stdenv.mkDerivation (finalAttrs: {
  pname = "dank-mono";
  version = "15-Oct-2020";
  src = requireFile {
    name = "Dank-Mono-${finalAttrs.version}.zip";
    url = "https://philpl.gumroad.com/l/dank-mono";
    sha256 = "sha256-zppKaEfi56WPYDS+r/uOkIFDqKdYtTmPm30KxvG3A8Q=";
  };
 
  dontConfigure = true;
  dontBuild = true;
 
  unpackPhase = ''
    ${unzip}/bin/unzip $src -d $out/unpacked
  '';
 
  installPhase = ''
    mkdir -p $out/share/fonts/opentype
    mkdir -p $out/share/fonts/truetype
    mv $out/unpacked/DankMono/OpenType-PS/*.otf $out/share/fonts/opentype
    mv $out/unpacked/DankMono/OpenType-TT/*.ttf $out/share/fonts/truetype
    rm -rf unpacked
  '';
 
  meta = with lib; {
    description = ''
      A typeface designed for coding aesthetes with modern displays in mind.
      Delightful ligatures and an italic variant and bold style.
    '';
    homepage = "https://philpl.gumroad.com/l/dank-mono";
    license = licenses.unfree;
  };
})

The difference between regular derivations and fixed-output derivations

This one I only recently learned, so I haven’t had a chance to put it to use yet.

When we use one of the fetchers like pkgs.fetchurlpkgs.fetchurl¹ we’re actually using a “fixed-output derivation” which is why we have to specify a hash.

The hash is the hash of the output of the derivation. By specifying a hash we allow Nix to keep its reproducibility guarantees while allowing us to do things like network access which generally adds unpredictability to our builds.

Regular derivations like the ones you can make with stdenv.mkDerivationstdenv.mkDerivation or the builtin derivationderivation don’t allow network access.

Providing a fixed-output hash (using outputHash and the like) lets us tell Nix that whatever we build, as long as the output hash matches the one we gave it, everything is fine. This is really handy for things that we would like to include in the Nix store, but can’t for one reason or another.

Fixed-output gotchas

There’s a gotcha with fixed-output derivations.

If you change the deriviation so that it generates a different output, e.g. changing a URL you’re fetching from, but don’t update the hash, Nix won’t rebuild it. It finds an output in the store with that hash and skips the build process.

You’ll need to update the hash as well.

Hash calculation tip

An easy way to calculate the hash you will need is to leave the hash as an empty string. Nix will notice and use a dummy value which will cause a mismatch and then you can just copy the correct value from the output and run the build again.

This is a version of “trust on first use” but if you trust your inputs, you should be fine.

An example I can think of is when I was running a modded Minecraft server, I was using packwiz to manage the modpack and fetch the mods and so forth. Because of the way it works, I know the output will always be the same given the same input modpack definition, but I ended up just running that in the server’s pre-start script. Now that I know about fixed-output derivations, I could wrap the fetching of the mods into one and then manage it like any other package, updating the hash as I change the modpack.

Don’t try to learn it all at once

My advice to anyone trying to learn Nix or NixOS is to take it a piece at time. Don’t try to understand it all at once. I think there are too many ways to come at it, as an OS, as a package manager, as a dotfiles manager, all of the above, and trying to understand it all out of the gate will just be overwhelming. The Nix language is pretty simple, all the rest is just extra details.

Take a piece at a time, and ask for help if you get lost. The NixOS forums have been a great place to ask things and get help.

Not to be confused with the builtins.fetchurlbuiltins.fetchurl. These work differently and you generally want to use the Nixpkgs version instead, as the builtin version can have performance implications for evaluating and building your system or packages.

Idle Musings

Nix is Too Big to Learn All At Once

NixOS modules

Fetchers and build helpers

The difference between regular derivations and fixed-output derivations

Don’t try to learn it all at once

Graph View

Backlinks

Nix is Too Big to Learn All At Once

NixOS modules §

Fetchers and build helpers §

The difference between regular derivations and fixed-output derivations §

Don’t try to learn it all at once §

Footnotes §

Graph View

Backlinks

NixOS modules

Fetchers and build helpers

The difference between regular derivations and fixed-output derivations

Don’t try to learn it all at once

Footnotes