Epistemic Status: 
Sprouting — I appreciate the irony of having an epistemic status on a page that mentions “epistemic uncertainty”. I don’t know what I don’t know, and unlike an LLM, I know better than to say otherwise.
This past weekend, I found myself surprised to finally be reading up and learning about Large Language Models (LLMs). I’ve been very skeptical of all the AI hype. Anytime there’s large amounts of money flowing towards something, I’m quite suspicious.
But I was reading about MUDs, a type of text-based multiplayer game usually based on tabletop role playing games, and how one particular codebase had added initial support for using LLMs to drive NPC interactions. That sent me towards looking up how LLMs could be made to interact with their world. My rudimentary understanding was that these could only output natural language, which seemed to indicate that connecting them to external actions would be difficult.
I’ve been having a running dialog with a friend of mine about these and how they could be applied to programming tooling, and one of the ideas that has come up in our discussions would involve using LLMs for AI-assisted development at a higher level than just programming, e.g. keeping track of what work you’ve done so you can ask for a summary the next day, with the LLM interacting with its “surroundings”.
Now after some reading, I have to admit, learning and thinking about LLMs has taken over part of my brain. I’ve had trouble thinking about anything else this week after the reading I have done, and the reading list I’ve assembled. I feel like it’s time to sit down and try to write something about what I’ve learned, and what I think about all of this.
I may sound critical in this post but...
I think these things can have real, useful applications. And there are neighboring technologies, like embedding vectors, that are really powerful tools. Having started to look at all these tools, I’m really excited to explore and see what they can be used for.
But like every powerful tool, we need to be careful with how we use them. The story of the man in Belgium who committed suicide because of an LLM-backed chatbot should be kept in mind when working with these. The human tendency to anthropomorphize and form parasocial relationships with celebrities and inanimate objects makes the use of LLMs very tricky. The tech world’s collective rush to slap “AI” on everything is going to cause serious harm if we aren’t careful.
There’s both more and less here than just the hype would suggest
There’s definitely something interesting going on with these LLMs. They can do some pretty amazing things, especially when you start to understand what they’re actually doing. Somehow we’ve managed to create things that we don’t quite understand how they work or how to make them do the things we want them to do, but I don’t know that I would want to call them intelligence.
There are definitely things that make them seem intelligent. One of the initial papers I found was one called Generative Agents: Interactive Simulacra of Human Behavior. In it, the researchers were able to create a simulated village with interacting agents that exhibited some interesting emergent behaviour over the course of two simulated days. For example, they gave one of the characters the idea of throwing a Valentine’s Day party and without any further intervention, that character invited others, and the others “remembered” the party and came, and in one case another character invited their “crush” to the party.
The way they constructed these agents was interesting, given an LLM’s lack of anything like memory, which I go into later. I think the approach is applicable to other things outside these sorts of simulacra.
Simon Willison’s blog in particular has been helpful in giving me a crash course in these things (beyond what little I already knew), and I’d recommend going and reading what he has to say about these things1. His view on the technology is that the more we all know about it, the safer we might be. I hope he’s right.
I’ve also been reading several papers about the subject, and building up a reading list of others to go read. Answering my original question of how to make LLMs interact with the world is a paper called ReAct: Synergizing Reasoning and Acting in Language Models. Some of these things have started to be integrated into systems like GPT-4’s API.
It actually feels like LLMs would be great tools for realizing what I call the other AI, augmented intelligence, *I prefer “augmented intelligence”, but I can see the benefit of calling it “intelligence augmentation” or “amplification”, i.e. IA instead of AI. which is what I believe we should be working towards. Tools to help us do more2, not tools to replace us, or to be used against us3, or to become an artificial slave class for us.
Prompt injection is a serious problem
One thing in particular Simon is very adamant about is that people aren’t taking prompt injection seriously enough. There is currently no reliable solution to what is essentially a huge gaping security hole in these things, and that makes a lot of the really cool and useful applications instead very dangerous. The solutions that are being offered are only partial solutions, and because we’re talking about security this means they’re aren’t solutions at all.
To put this in terms that most software engineers should understand, it’s like if someone asked you:
Would you feel safe deploying an application to the public Internet if the SQL injection mitigation had only a 95% success rate of preventing SQL injection attacks?
Any responsible software engineer would have to answer in the negative.
Without a 100% solution to prompt injection, anyone putting these into production where they’re open to user input and where they can trigger potentially dangerous actions is asking for trouble.
I spent way too much ”hammock time” thinking about this problem this week (my ADHD brain didn’t want to think about anything else, especially late at night), and I’ve come to a similar conclusion that Simon has, of separating the system that interacts with untrusted inputs from the system that needs to take potentially destructive actions. It’s not an easy solution, and it still has problems, but it’s honestly better than most of what seems to be out there.
The essential problem is one of trust. Because the instructions to the LLM use the same “channel” as the user provided input, the LLM can’t separate them out and treats both with the same weight and authority. No amount of coercion is going to convince it not to be confused, because the malicious instructions don’t look any different from the legitimate instructions. Delimiters don’t work, asking nicely doesn’t work, using another system to determine if an injection happened doesn’t work. All these approaches can and have been circumvented.
It’s a difficult problem, and one I’ll probably keep thinking about off and on, whether I want to or not 
.
Oversimplification isn’t helping matters
The common refrain from detractors (myself included) has been something along the lines of:
These things are just overblown Markov chains. All they do is predict the most likely next word, and then do that again, and again.
While that description isn’t technically wrong (except they’re not Markov chains), it’s glossing over a huge important piece of information. The context that these LLMs use is huge. *The “context” of an LLM is the input provided. That’s the prompt and any other instructions provided as natural language to the model. GPT-4 can use up to 32,000 tokens (the OpenAI marketing website says “over 25,000 words”) in its context, so when it’s predicting the most likely next token, it’s using all of that context to do it. And then adding that new token to the context and doing it again, and again, and again.
Chatbots don’t chat
Another related point of confusion is how chatbots built on LLMs actually work. The naïve understanding that most people probably have is that you send a message to it, it “thinks” about that and sends a response, you send another message to it, that same thing “thinks” about that and sends a response. This implies that the LLM has memory, but it doesn’t.
What’s actually happening is that each time a message is “sent”, either way, it’s just being concatenated onto all of the previous text in the context. Then when you send it a new message, the LLM is given all of the previous context with your message appended as a long string of input. It’s a stateless operation, with nothing kept between each message except the context.
This becomes a problem when you have a long “conversation” with a chatbot. Eventually the context exceeds the size that the LLM can take, and so it has to be truncated. It’s like the LLM has a short memory span, and if you talk to it long enough, it won’t “remember” where the conversation started and very likely starts to become incoherent.
This is much less of a problem as the context size on these models has increased in the past few months, and I can think of clever ways off the top of my head for truncating the context that might work better than a naïve approach, but it’s still something to be aware of.
They can be great for rubberducking and similar creative tasks
Rubberducking, the idea of talking out a problem you’re working on with another person, or in some cases an actual rubber duck, can lead to new ways of looking at a problem. Tools like ChatGPT (or a locally running LLM) can be used for this sort of thing, as long as we keep in mind that ChatGPT will lie to you.
It has no motivations to lie or tell the truth. It’s just trying to complete a sentence in a satisfactory way.
~ Simon Willison, Don’t Trust AI to Talk About Itself
For technical problems, an experienced engineer can easily see and correct mistakes that an LLM makes, and the fact that they make mistakes doesn’t make them useless for this task. Microsoft released a paper about how they got GPT-4 to write a 3D game running in a browser with a “zero-shot” prompt. *A “zero-shot” prompt is where you ask an LLM to complete a task without providing any examples. A “one-shot” or “few-shot” prompt is where you provide one or more examples of the task you want the LLM to complete. Geoffrey Litt has a great quote about it in his post about opening up programming to more people via LLMs.
In general, GPT-4 feels like a junior developer who is very fast at typing and knows about a lot of libraries, but is careless and easily confused.
~ Geoffrey Litt, Malleable software in the age of LLMs
Simon’s article about AI-enhanced development talks about how he uses it to take on projects that he wouldn’t have without it.
I think it’s probably great for that sort of thing, especially for someone like me who sometimes struggles with getting started on a project. If I can start with GPT-4 or another LLM and get something up and running quickly that gets most of the way to a starting solution, I could easily take it from there. I could see it being really great for getting the boilerplate of setting up a project out of the way.
Because programming is a task where the results are “provable”, i.e. it works or it doesn’t, it’s less susceptible to the problem of hallucination. And an experienced programmer can guide it and catch its simple mistakes.
It also helps to remember the cutoff for training data. For most of OpenAI’s models that’s September 2021, *{As of November 2023} which means they can’t know about newer libraries and will resort to making stuff up.
I don’t think this will directly lead to AGI
The “holy grail” of AI research is an Artificial General Intelligence, a system that could do anything that a human can do. LLMs aren’t even thinking yet, so I think that goal is still a long way off.
There was a talk I watched this week by Thomas Dietterich where he lays out some of the problems that LLMs aren’t suited to solve, and then provides some suggestions for where we could go. *He mentions the distinction of aleatoric vs epistemic uncertainty which I find very helpful in thinking about the problems here.4
I wouldn’t say that LLMs are a dead end, they’re very powerful tools at what they do best, but I don’t think they’ll lead to the solution by themselves. We’ll need to combine them with other things. Thomas references a paper in his talk that might be relevant, which I haven’t read yet, titled Deep Learning Needs a Prefrontal Cortex. If we can augment these things with other systems, we might get where we want to go.
AI ethics in a capitalist society
I’m just going to mention this briefly, but I think it’s very important. There are lots of ethical questions that LLMs bring up that we don’t have good answers to.
Is it ethical to train LLMs on a scraping of the Internet, or books under copyright, without the permission of the authors of that content? Is it ethical to train LLMs on art created by living artists? Is it ethical to release software that can lead to real harm like it did with the Belgian man I mentioned earlier? Is it ethical to deploy these tools widely when they are prone to spouting wildly inappropriate things?
I’m concerned, given that the vast majority of these things are being developed by for-profit corporations who are driven to maximize short-term shareholder value5 racing to try to get to AGI first, that those ethical concerns won’t be adequately addressed, and that the very destructive values of our for-profit world will get baked into these systems, never mind the displacement that could occur as these systems and things like them eliminate jobs.
I think Naomi Klein made a good argument in an article in the Guardian, in response to tech leaders who think this stuff is going to lead to a better world all by themselves.
If earning money is to no longer be life’s driving imperative, then there must be other ways to meet our creaturely needs for shelter and sustenance. A world without crappy jobs means that rent has to be free, and healthcare has to be free, and every person has to have inalienable economic rights. And then suddenly we aren’t talking about AI at all – we’re talking about socialism.
Because we do not live in the Star Trek-inspired rational, humanist world that [Sam] Altman [CEO of OpenAI] seems to be hallucinating. We live under capitalism, and under that system, the effects of flooding the market with technologies that can plausibly perform the economic tasks of countless working people is not that those people are suddenly free to become philosophers and artists. It means that those people will find themselves staring into the abyss – with actual artists among the first to fall.
~ Naomi Klein, AI machines aren’t ‘hallucinating’. But their makers are.
Right now we’re nowhere near Star Trek’s Federation. Instead we’re living in the Ferengi Alliance, with the Rules of Acquisition.
Rule 95: Expand or Die
Rule 111: Treat people in your debt like family… exploit them.
Ending on a more positive note
I don’t want to sound like I’m against these things, because I’m not. Like I said many times, I think they have the potential to be very useful, especially in augmenting creativity.
I think we should definitely learn all we can about these things, how to make them useful tools, while being cognizant of the dangers that exist.
To quote Simon one last time:
In AI, multiple things are true at the same time. The potential for harm is enormous, and the current systems have many flaws—but they are also incredibly empowering on an individual level if you can learn how to effectively use them.
~ Simon Willison, Thoughts on AI safety…
Footnotes
-
I’m going to cite Simon’s blog a lot below. I usually prefer to use multiple sources, but his posts have plenty of reliable citations that I think he’s a reliable source. “But you don’t have to take my word for it.”
-
I’m very much against the “augmented intelligence” that involves cybernetic implants. In our hyper-captialist society, I could never trust such a thing to do my bidding and not the bidding of the manufacturer (a point somehow missing from most cyberpunk novels).
-
The real sci-fi future I worry about isn’t The Matrix or The Terminator, but Dune. I grew up watching and rewatching the Alan Smithee edit of the 1984 Dune (Alan Smithee is a pseudonym used when someone doesn’t want credit for the work, in this case David Lynch), which includes a prologue that explains the Butlerian Jihad as starting when thinking machines were created and then used by men to enslave others who had become dependent on those machines. A future of widespread use of things like self-driving cars concern me, because then where I can go is determined not so much by me but by the manufacturers. I believe any surrender of this sort of autonomy is inevitably going to lead to abuse. As for the 1984 Dune movie, the best cut is the fan edit Dune Redux, if you can find it.
-
The short version is that epistemic uncertainty is “uncertainty caused by a lack of knowledge”, whereas aleatoric uncertainty is “uncertainty due to inherently random effects”. LLMs lack the ability to distinguish between the two, which is why they hallucinate answers when they don’t know the answer instead of returning something like “I don’t know”. This paper seems like a good introduction to the concept, but I haven’t read it yet.
-
There was an argument I read once that suggested thinking of corporations as “slow moving AI” not really under the control of any of the individuals who think they are in control of it. I wish I could remember where I read it.