The Synergy of LLMs and Expert Knowledge

Ever since I stumbled into using LLMs, I’ve found more and more uses for them in my personal and professional life. Nothing I’m saying here hasn’t been said before. But I want to talk about something that occurred to me this morning.

Writing an XSLT stylesheet §

Last night I spent some time trying to write a Home Manager module for generating the keymap.xml file that Jetbrains IDEs expect. It’s been one of the things bothering me about using Jetbrains IDEs is that they’re not easy to configure declaratively. It’s not low-hanging fruit, so no one has done it yet, but yesterday I realized it would be easier than I expected.

Eventually I want to have it generate the other configuration files, but the keymap is the thing I want to keep in sync most of all across all my systems and Jetbrains IDEs, and the builtin tools for doing that are annoying, and outside the scope of my Home Manager configurations.

The format is pretty straightforward, but I didn’t want to just use string manipulation to generate the XML file. What I wanted was to write a function to take a Nix attrset of a particular form and spit out an XML document.

Looking at some other examples in the Home Manager repository, I realized I could use the builtins.toXML, xsltproc, and an XSLT stylesheet to do it in a way that would be resistant to mistakes and would be guaranteed to generate valid XML or give a useful error.

In the end, I ended up with something that would let me take a Nix configuration that let’s me easily specify key mappings as well as Emacs-style two-stroke sequences using a simple syntax and transform it to what Jetbrains wants. That let’s me take some Nix configuration like this:

programs.jetbrains = {
  pycharm-professional.keymap = {
    parent = "Emacs";
    actions = {
       # Tool windows
       ActivateStructureToolWindow = "alt+2";
       ActivateDatabaseToolWindow = "alt+3";
       ActivateCommitToolWindow = "alt+4";
       ActivateFavoritesToolWindow = null;
       ActivateFindToolWindow = null;
       ActivateRunToolWindow = "alt+5";
       ActivateDebugToolWindow = "alt+6";
 
       # Editor controls
       Unsplit = "ctrl+x 0";
       CloseContent = "ctrl+x k";
 
       # Navigation
       Back = "alt+comma";
       GotoClass = "ctrl+c ctrl+c";
       GotoSymbol = "ctrl+c ctrl+s";
       TextSearchAction = "ctrl+c ctrl+e";
       "Refactorings.QuickListPopupAction" = "ctrl+c ctrl+r";
       "Vcs.QuickListPopupAction" = "ctrl+x ctrl+v";
 
       # I don't remember what this is, but I think it's disabled
       # because of a key mapping conflict
       IpnbCodeCellAction = null;
 
       # Search
       SearchEverywhere = "shift+ctrl+s";
       ShowBookmarks = "ctrl+x ctrl+b";
    };
  };
};

And get this XML as output ¹:

<?xml version="1.0"?>
<!-- Generated by Nix -->
<keymap version="1" name="Nix Managed Keymap" parent="Emacs">
  <action id="ActivateCommitToolWindow">
    <keyboard-shortcut first-keystroke="alt 4"/>
  </action>
  <action id="ActivateDatabaseToolWindow">
    <keyboard-shortcut first-keystroke="alt 3"/>
  </action>
  <action id="ActivateDebugToolWindow">
    <keyboard-shortcut first-keystroke="alt 6"/>
  </action>
  <action id="ActivateFavoritesToolWindow"/>
  <action id="ActivateFindToolWindow"/>
  <action id="ActivateRunToolWindow">
    <keyboard-shortcut first-keystroke="alt 5"/>
  </action>
  <action id="ActivateStructureToolWindow">
    <keyboard-shortcut first-keystroke="alt 2"/>
  </action>
  <action id="Back">
    <keyboard-shortcut first-keystroke="alt comma"/>
  </action>
  <action id="CloseContent">
    <keyboard-shortcut first-keystroke="ctrl x" second-keystroke="k"/>
  </action>
  <action id="GotoClass">
    <keyboard-shortcut first-keystroke="ctrl c" second-keystroke="ctrl c"/>
  </action>
  <action id="GotoSymbol">
    <keyboard-shortcut first-keystroke="ctrl c" second-keystroke="ctrl s"/>
  </action>
  <action id="IpnbCodeCellAction"/>
  <action id="Refactorings.QuickListPopupAction">
    <keyboard-shortcut first-keystroke="ctrl c" second-keystroke="ctrl r"/>
  </action>
  <action id="SearchEverywhere">
    <keyboard-shortcut first-keystroke="shift ctrl s"/>
  </action>
  <action id="ShowBookmarks">
    <keyboard-shortcut first-keystroke="ctrl x" second-keystroke="ctrl b"/>
  </action>
  <action id="TextSearchAction">
    <keyboard-shortcut first-keystroke="ctrl c" second-keystroke="ctrl e"/>
  </action>
  <action id="Unsplit">
    <keyboard-shortcut first-keystroke="ctrl x" second-keystroke="0"/>
  </action>
  <action id="Vcs.QuickListPopupAction">
    <keyboard-shortcut first-keystroke="ctrl x" second-keystroke="ctrl v"/>
  </action>
</keymap>

I’ve never written an XSLT stylesheet before. I may have seen one a few months ago, but that’s the limits of my knowledge. I know they exist and what they do but I couldn’t write one from scratch without a lot of reading first. I’m not against learning XSLT. If my programming path shows anything, it’s that I’m willing to learn almost anything.

But do I need to do that right now? It’s a yak-shave that would keep me from building the thing I’m trying to build and stopping to learn XSLT would sap my energy and I’d probably be less likely to finish. It’s much faster for me to explain what I want to an LLM and iterate (almost like we do in code review) than to try to create something from scratch.

Without an LLM, I probably would have fallen back to the string manipulation approach which would be more error prone and less maintainable than using XSLT, using a hack instead of the right tool for the job.

It almost always gives me code that has a bug of some sort or other², but quickly pointing out the error to it gets it to correct the problem. Simon Willison talks about how what he calls “AI-enhanced development” makes him more ambitious with his projects, and I’ve found that to definitely be true.

The difference between reading and writing §

This reminded me this morning of learning natural languages, and how it’s much easier to read something in a new language than to write it. I think the same thing applies here.

For an expert in a field like programming (I hesitate to call myself that), I can look at code in most languages and know what it’s doing even if I don’t understand all of the details of syntax or what library functions exist. For small problems, it’s actually faster to just ask an LLM to write something and then iterate towards a solution using my own expertise than it would take to go learn that thing enough to write it from scratch.

This is similar to natural language learning in that production of new sentences is much harder than listening or reading sentences made by others who know the language already. You don’t get tripped up by the rules of grammar when you’re listening as much as you do when trying to speak.

Now that I have an example of an XSLT stylesheet, I can probably build the others I will need for other configuration files, so in that way, using an LLM has sped up my learning process considerably.

An LLMs usefulness still requires human judgement §

We are still quite far from being able to tell an LLM to create an entire software product and it being able to do it on the first try. In my experience, it still requires a good deal of hand-holding, especially when you get off the beaten path, e.g. asking it to write Nix expression language.

I’ve been using it at work to get up to speed with Terraform, because even though I don’t know everything about Terraform, I can look at it and see if what the LLM is suggesting is reasonable or not. I still rely on others on the team to point out any problems I missed.

It’s also sometimes easier to ask an LLM about something and then go verify it, than to find that information using Google search. But that “go verify it” step is so important.

There’s something tangentially related here, about how I use spellcheck usually. I don’t know where I picked up this advice but it’s been invaluable.

Instead of just using spellcheck to correct my mistakes, I go back and try to correctly spell the misspelled word. In this way I learn from my mistakes rather than become dependent on the tool. In the same way, I usually manually type out what an LLM has given me instead of copy-pasting it. That way I get a feel for the language I’m using/learning faster than if I just copy-paste.

Footnotes §

1. There’s some in-between transformation that happens to turn this configuration into something easier to handle in XSTL, since builtins.toXML is very generic. I’m also not using the typical enabled = true; pattern here, because the way it’s coded up right now, an IDEs presence in the attrset determines whether it’s on or not.

2. I didn’t realize until later that I was using GPT-3.5 and not GPT-4, so I’m certain that it’s probably better with GPT-4. At some point I’ll start using a local LLM, but I need to set that up.