Idle Musings

on software, philosophy, and other things.

Nix is Too Big to Learn All At Once

9 min read — Published on 7 February 2024

Epistemic Status: hyacinth Blooming

I’ve been using Nix and NixOS pretty much non-stop since 2018 and I still find myself learning new things that open up areas that I hadn’t realized were there before. I wanted to take a moment and talk about a few of them because one of the things I’ve realized is that Nix covers so many areas of computing that it’s hard to grasp all at once.

One of the big issues I ran into as I started using NixOS was that all of the documentation that people direct you towards isn’t exactly relevant to what you need to know to use NixOS effectively. Take the Nix Pills series of posts, for example. They start out talking about packaging and making derivations and so forth, and as a new user of NixOS I wasn’t trying to write packages but instead trying to configure my systems to do what I wanted.

Even coming at Nix from the perspective of using it for development environments I probably wouldn’t need to know about how to write a package derivation right at the start.

What did I need to know about in greater detail instead? Modules.

NixOS modules

I needed to know about how the NixOS module system worked.

You might say “Hey, it’s right there in the NixOS manual! RTFM!”

That would have required me to understand that when I was editing configuration.nix, that I was editing a module. Realizing this is hindered even more by the fact that that particular module is using the shorthand version where you don’t have an options attribute in the attrset and the config is just the contents of the top level attrset.

Once I learned about them a few years ago, I’ve started modularizing my NixOS configurations, extracting out whole systems so that they can be configured by a single flag. For example, I want an X server on this machine but not that one and the module I wrote handles configuring the server, the display manager, the window manager and so on all behind a single host.xserver.enable flag that I defined.

At a previous job, I had a module for configuring the company VPN on my work laptop. It had to do a lot of changes across the system, but it was all contained and controlled by a single <company-name>.networking.enable option. This meant I could turn it all off with a single change if I needed.

Fetchers and build helpers

Another thing I have had to repeatedly remind myself is about all the wonderful things in Nixpkgs under the category of fetchers and build helpers. A lot of these I use regularly.

One that I keep forgetting about is pkgs.writeShellApplication. This is a “trivial builder” that creates a shell script. So far this is similar to pkgs.writeShellScript or pkgs.writeScriptBin, but it does more than just create an executable script in the store. It also has options for putting packages in the PATH so that you can write your script in a more normal manner.

# Instead of writing something like this...
pkgs.writeScriptBin "foo" ''
  ${pkgs.hello}/bin/hello args args
'';
# Or using something like `getExe`
# You can instead do this...
pkgs.writeShellApplication {
  name = "foo";
  runtimeInputs = with pkgs; [ hello ];
  text = ''
    hello args args
  '';
}

In this example, the resulting script contains a line that puts the hello package’s binary path into the PATH environment variable. Using writeShellApplication also has the added benefit of checking the script using ShellCheck, and if it fails, the build fails. These things also work if you want to keep your script in a separate file and read it in using builtins.readFile.

Using requireFile for private packages

Another really useful one that I only recently discovered is requireFile. I have a few proprietary fonts that I want to have installed on my systems, but I don’t want to add those files into my NixOS flake. With requireFile I can work around that. What requireFile does is allow you to tell Nix that there’s a file you expect to have in the Nix store, and if it’s not present, it will fail to build and display a message saying you should add the file manually and retry.

The url argument tells it what URL to display in the message so you know where to go to get the missing file, e.g. if you forgot about it.

There were some gotchas with this though since both methods it suggests don’t give you the hash you need for the requireFile call, so I wrote a simple Bash script to handle it for me in the future:

add-required
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail
 
if [[ $#--eq-0| -eq 0 ]] ; then
  echo "Usage:"
  echo "  add-required <absolute-path>"
  exit 1
fi
 
# This command prints some noise to stderr so redirect that to /dev/null
HASH=$(nix-prefetch-url --type sha256 file://"$1" 2> /dev/null)
SRI_HASH=$(nix-hash --to-sri --type sha256 "$HASH")
 
echo "Use the following 'requireFile' invocation:"
echo
echo "requireFile {"
echo "  name = \"$(basename "$1")\";"
echo "  url = <url-to-download-from>;"
echo "  sha256 = \"$SRI_HASH\";"
echo "}"

Then I can just copy and paste that into a std.mkDerivation for the src and it just works.

Here’s an example of a derivation I wrote for the Dank Mono font with the relevant part highlighted.

Keeping the source in the Nix Store

Note the highlighted section of the installPhase. Dropping a link to the source path, i.e. the downloaded zip file, into our output path means that it won’t get garbage collected from the store. Otherwise, we’d have to add the file to the store manually again the next time we try to rebuild the system after a GC run.

{ stdenv, lib, requireFile, unzip, ... }:
 
stdenv.mkDerivation (finalAttrs: {
  pname = "dank-mono";
  version = "15-Oct-2020";
  src = requireFile {
    name = "Dank-Mono-${finalAttrs.version}.zip";
    url = "https://philpl.gumroad.com/l/dank-mono";
    sha256 = "sha256-zppKaEfi56WPYDS+r/uOkIFDqKdYtTmPm30KxvG3A8Q=";
  };
 
  dontConfigure = true;
  dontBuild = true;
 
  unpackPhase = ''
    ${unzip}/bin/unzip $src -d $out/unpacked
  '';
 
  installPhase = ''
    mkdir -p $out/share/fonts/opentype
    mkdir -p $out/share/fonts/truetype
    mv $out/unpacked/DankMono/OpenType-PS/*.otf $out/share/fonts/opentype
    mv $out/unpacked/DankMono/OpenType-TT/*.ttf $out/share/fonts/truetype
    rm -rf unpacked
 
    # Drop a link to the src so that it stays in the Nix store
    ln -s ${finalAttrs.src} $out/src
  '';
 
  meta = with lib; {
    description = ''
      A typeface designed for coding aesthetes with modern displays in mind.
      Delightful ligatures and an italic variant and bold style.
    '';
    homepage = "https://philpl.gumroad.com/l/dank-mono";
    license = licenses.unfree;
  };
})

The difference between regular derivations and fixed-output derivations

This one I only recently learned, so I haven’t had a chance to put it to use yet.

When we use one of the fetchers like pkgs.fetchurl1 we’re actually using a “fixed-output derivation” which is why we have to specify a hash.

The hash is the hash of the output of the derivation. By specifying a hash we allow Nix to keep its reproducibility guarantees while allowing us to do things like network access which generally adds unpredictability to our builds.

Regular derivations like the ones you can make with stdenv.mkDerivation or the builtin derivation don’t allow network access.

Providing a fixed-output hash (using outputHash and the like) lets us tell Nix that whatever we build, as long as the output hash matches the one we gave it, everything is fine. This is really handy for things that we would like to include in the Nix store, but can’t for one reason or another.

Fixed-output gotchas

There’s a gotcha with fixed-output derivations.

If you change the deriviation so that it generates a different output, e.g. changing a URL you’re fetching from, but don’t update the hash, Nix won’t rebuild it. It finds an output in the store with that hash and skips the build process.

You’ll need to update the hash as well.

Hash calculation tip

An easy way to calculate the hash you will need is to leave the hash as an empty string. Nix will notice and use a dummy value which will cause a mismatch and then you can just copy the correct value from the output and run the build again.

This is a version of “trust on first use” but if you trust your inputs, you should be fine.

An example I can think of is when I was running a modded Minecraft server, I was using packwiz to manage the modpack and fetch the mods and so forth. Because of the way it works, I know the output will always be the same given the same input modpack definition, but I ended up just running that in the server’s pre-start script. Now that I know about fixed-output derivations, I could wrap the fetching of the mods into one and then manage it like any other package, updating the hash as I change the modpack.

Don’t try to learn it all at once

My advice to anyone trying to learn Nix or NixOS is to take it a piece at time. Don’t try to understand it all at once. I think there are too many ways to come at it, as an OS, as a package manager, as a dotfiles manager, all of the above, and trying to understand it all out of the gate will just be overwhelming. The Nix language is pretty simple, all the rest is just extra details.

Take a piece at a time, and ask for help if you get lost. The NixOS forums have been a great place to ask things and get help.

Footnotes

  1. Not to be confused with the builtins.fetchurl. These work differently and you generally want to use the Nixpkgs version instead, as the builtin version can have performance implications for evaluating and building your system or packages.

Graph View

Backlinks

  • No backlinks found