My Nix Dev Environment Philosophy

Epistemic Status: Blooming

---

I have a certain philosophy about how I build dev environments in Nix, and I don’t think it matches the rest of the community, so I thought I’d take some time to write about it. Maybe it will help others approach this from a different perspective.

What is our goal with dev environments? §

It helps to start by understanding our goal. With Nix dev environments, by which I mean those that use devShells, and any dev environment really, we’re looking for a few properties but the main one I’m interested in is that it should be deterministic¹ — the environment should be the same for everyone using it (as much as possible). In other words, I should be able to blow away my environment, start from scratch and end up with the same environment.

The Nix package manager helps with this, and flakes specifically², in two ways.

• Dependency tracking — Nix keeps track of all the dependencies of the packages it builds such that the entire dependencies of a binary package are kept in sync with the packages that depend on it, and they’re installed together almost as one unit. This is different from something like Flatpack in that the dependencies are all stored as separate artifacts in the Nix store, but you can’t easily install a package without installing its dependencies.
• Flake locking — With Flakes, Nix will create a lock file that tracks the version of all the inputs to the flake. Since Nix tries to guarantee a pure environment for Flakes, the inputs are the only thing that should change the outputs.

Pinning and locking all the things §

So, Nix will let us get a deterministic environment by pinning the inputs, the main one being nixpkgs where the package definitions come from. I mentally split creating a development environment up into two parts:

• System level packages and dependencies — these are all the dependencies for a project like the Python or Ruby interpreter, Rust compiler toolchain, etc.
• Project level dependencies — these are all the dependencies that are usually managed by that ecosystems tooling, e.g. pip, bundle, or cargo/rustc, etc. What I’m talking about here are those ecosystems’ packages that the project depends on, Python or Ruby packages, Rust crates, etc.

The usual recommended approach is to have Nix handle all of that, using a tool like poetry2nix or similar that converts the project’s dependency definitions into a Nix expression that can be used to build the environment. That probably makes sense if you’re trying to package up your project (more on this later), but it makes things complicated when you want to pull in private dependencies, e.g. ones not published to PyPI or similar, ones published to a company’s private package repository hosting like Artifactory or GitHub Packages.

Taking a different approach §

This is where I deviate from the recommended approach in the dev environments I create. I use Nix to set up the project language’s ecosystem. Install the Python or Ruby version I need, etc. and track that version using Nix and the flake lock.

Then for the project level dependencies, I just use the existing ecosystem to handle it. I trust that Ruby’s bundle can keep track of versions correctly with the Gemfile.lock and that each time I run it, I’ll get the same versions as in the lock file.

Examples make things clearer §

I was recently setting up a development environment at work, mostly because that’s what I use, but it’s a good example of the approach. I’m using devshell and flake-parts³, hopefully that doesn’t distract too much from what I’m trying to show.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    devshell = {
      url = "github:numtide/devshell";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
 
  outputs = inputs@{self, nixpkgs, flake-parts, devshell}:
  flake-parts.lib.mkFlake { inherit inputs; } {
    imports = [
      devshell.flakeModule
    ];
 
    systems = [
      "aarch64-darwin"
      "x86_64-darwin"
      "x86_64-linux"
    ];
 
    perSystem = { pkgs, ... }: {
      devshells.default = {
        startup = {
          yarnInstall.text = ''
            if ! [ -d "$PRJ_ROOT/node_modules" ]; then
              yarn install
            fi
          '';
        };
      };
    };
  };
}

Focusing on the highlighted part, what this is doing is ensuring that our Yarn based Node project here has its dependencies installed when we enter the dev environment, either via nix develop or using direnv. Yarn doesn’t seem to have a nice “check” feature like Ruby’s Bundler does that checks to see if any new packages need to be installed, and I don’t like the idea of running yarn install every time we enter the environment (though that could be tenable).

I could create a file to store the hash of node_modules, or package.json to make it able to keep things up to date more, but in my experience I haven’t had to do that yet.

Usually, with something like Ruby Bundler, I also need to specify paths to make sure it doesn’t try to install the packages globally, in essence making a sort of virtual environment without using the usual virtual environment tooling. I find that’s easier in most cases.

What about packaging up the project using Nix? §

This is where things become complicated and I don’t have a good solution yet. Fixed-output derivations might be the way to go, that way I can use the ecosystem packaging tools and allow them to fetch things from the Internet, but that’s generally frowned upon in the Nix world.⁴

This is where the traditional approach is usually more robust, but again it can be difficult to pull in custom dependencies that way. Or you might be dealing with a niche ecosystem where those Nix native tools don’t exist.

Footnotes §

1. Apparently there is some contention around the term “reproducible”, so I’m now somewhat reluctant to use it despite Nix itself using it up until September 2, 2023 (based off the earliest version of the page with the new verbiage). I don’t think restricting it to only mean bit-by-bit identical to be that useful, but my goals are different than the author of that post.

2. You can do this without flakes, but I prefer and am more familiar with flakes, so that’s what I use.

3. The documentation on both of these projects can be difficult to work with. While I think both of these libraries improve the experience, learning to use them is not easy without just reading the Nix code, which requires experience.

4. I came across an issue in the Nix repository where Eelco Dolstra, the creator of Nix, was planning to severely restrict fixed-output derivations. While I understand the concerns I don’t think it’s a good idea because it takes away a tool that can be used in situations where there isn’t a good Nix native alternative. It feels like an instance of perfect being the enemy of good.