AI Hallucination is just Man-Guessing

One time I was out drinking with some Swedish folks and they told me about the word killgissa. The word means something like ‘man-guessing’, or ‘guy-guessing’, and it refers to when you sound like you know what you’re talking about but you’re actually just guessing.

I reckon AI hallucination is just man-guessing, but on your behalf. To explain, I first have to convince you that human reason isn’t actually that reasonable. With any luck it’ll make you better at managing your own processes of reason and your AIs. Let’s see.

We’ll begin with some scholarship. In the opening paragraphs of The Enigma of Reason, Mercier and Sperber point out:

Most of us think of ourselves as rational. Moreover, we expect others to be rational too. We are annoyed, sometimes even angry, when we see others defending opinions we think are deeply flawed.

Their ‘enigma of reason’ is precisely this:

if we humans are all endowed with this power of distinguishing truth from falsity, how is it that we disagree so much on what is true?

Their eventual point is, as I say, that reason isn’t actually that reasonable.¹

I think their perspective is worth considering. But then, I want to talk about how the same exact thing kind-of explains AI hallucinations. Not simply the fact of them, but also, why it could be said that:

We are annoyed, sometimes even angry, when we see [AI] defending opinions we think are deeply flawed.

Then we’ll talk about what to do about it. Let’s begin.

¶Moral dumbfounding as a microcosm of the failure of reason

When I teach ethics, I like to start with a curious little phenomenon I’ve described a few times in other places. It’s called moral dumbfounding, and it’s like a little microcosm of Mercier and Sperber’s enigma. Imagine this:

Julie and Mark are brother and sister … traveling together in France on summer vacation … One night they are staying alone in a cabin near the beach … they tried making love … a new experience for each of them. Julie was already taking birth control pills, but Mark uses a condom too, just to be safe. They both enjoy making love, but they decide not to do it again. They keep that night as a special secret, which makes them feel even closer to each other. What do you think about that? Was it OK for them to make love?

Most of the people in any given classroom won’t be very enthusiastic about the siblings’ experiment. This isn’t really that surprising. What is surprising is what happens when I ask why they think it’s wrong for Julie and Mark to make love. Assuming the students aren’t trying to be too clever, we’ll usually end up having an exchange that looks something like this:

1. Them: “The baby… it’ll come out messed up”
2. Me: “But they used two forms of contraception”
3. Them: “Fine, well, it’s illegal”
4. Me: “Not in France”
5. Them: “Ok, so it’ll ruin their relationship”
6. Me: “It clearly says it brought them closer”
7. Them: “Uh, well, it’ll make everyone else feel weird”
8. Me: “It just as clearly says they keep it a secret”

Sometimes people throw in off-piste stuff—“it’s against religion” for example, or “even with two forms of contraception, there’s still a chance of getting pregnant”. But everyone recognises that we’re beginning to grasp at straws.

It’s called ‘moral dumbfounding’ because lots of researchers like to keep the dialogue going—“lots of religious taboos aren’t things you think are wrong”, or “fine, Mark is sterile, what now?” Eventually people² will end up backed into a corner where they have no more reasons—they still think incest isn’t really that sweet, but they’re dumbfounded when it comes to explaining why.

We don’t have to be quite so ruthless.² In our hypothetical exchange, we’ve already demonstrated all we need. People aren’t ‘reasoning’ here. Their reasons are shit. All the evidence against their reasons is there right in front of them.

¶Thinking isn’t fast or slow, it’s lazy

This kind of shoddy ‘reasoning’ is such a prevalent feature of the human experience that you can find examples of it everywhere. Plenty of people have developed a programme illustrating similar failures of human reason. In philosophy, it dates back as far as you can go. It’s no particular surprise, then, that when psychology pivoted from philosophy toward science, it recognised the problem as early as William James, the grand-daddy of the modern discipline.

What’s a bit different about psychology is the misconception that we’ve solved it. A domain called dual process theories suggests that true reasoning is a different kind of thinking than shoddy reasoning, and we so often engage in shoddy reasoning because this more true, special kind of reasoning is very costly. For example, as I describe elsewhere, the most popular account names these kinds of thinking:

1. System one is fast, unconscious, automatic, and intuitive.
2. System two is slow, conscious, effortful, and deliberative.

So when you drive a familiar route and find yourself at the end with no memory of the drive, that’s System one—fast, automatic, unconscious. Or when you tell me what 1+1 is. Or read these words.³

When you’re driving to a new place, and you need to work out how long it’ll take and which route will have less traffic, that’s System two—deliberative, effortful, and slower. Or calculating 26*49. Or working out what these words mean, rather than just reading them.

Fast vs slow thinking, Kahneman called them, and it’s a pretty neat dichotomy if you’re going to put all human thinking in two buckets.

So neat, in fact, that he’s hardly the first to have had the idea.

And indeed, so neat that, as Mercier and Sperber put it:

Talk of “system 1” and “system 2” is becoming almost as common and, we fear, often as vacuous as talk of “right brain” and “left brain” has been for a while.

The basic idea is that ‘system 1’ processes are shortcuts that usually lead to the right outcomes, but in unusual circumstances lead to errors. So, if I asked you whether brother-sister incest was normally a good idea, your instinct to respond that it might be bad for the baby is normally going to be sensible. But, if you were in the class where I ask you about Julie and Mark, who used two forms of contraception, then it’s not going to be so sensible. In this case, it’s only because the example is so carefully designed that it doesn’t work.

For situations like this, some would have you believe, we require ‘system 2’ thinking—the slower, more effortful kind of thinking that can double-check the work of ‘system 1’. To error-correct in those circumstances where ‘system 1’ isn’t going to work.

This is, sadly, not quite right for a great number of reasons. I spend a lot of time talking about it here, and here. Even dual-process theorists don’t actually think this way. The only people who endorse this strawman are half-assed leadership consultants, TikTokers and… my department at Sandhurst. Because the idea is sticky. But I don’t need to make you read any of those articles to convince you to abandon the notion yourself. We can just consider our typical exchange above about Julie and Mark.

If there really were two systems of thinking—the usually correct, but sometime error-prone ‘system 1’, and the more reliable, costly, but error-checking ‘system 2’—then wouldn’t the error-checking ‘system 2’ come online after the first mistake in reasoning about why Julie and Mark fucking is “wrong”? If so, how do we explain all the shitty reasoning that follows that first mistake, when we’re corrected? And if not, precisely what is required to bring ‘system 2’ online?

This is, more-or-less, the problem with dual-process theory. The core concept is cute, but it’s so fuzzy around the edges that it doesn’t really help us get at the complicated features of the human experience. As I say, It’s something that dual-process theorists themselves grapple with, but the very fact that it’s still called dual-process theory hints at just how sticky it is.

¶Reasons are just one lazy tool

One of the reasons the idea of fast (system 1) and slow (system 2) thinking is so popular is because it feels true. It feels like our shoddy reasoning comes about when we fail the think hard enough, or something like this.

We can explain this without getting ourselves trapped, like our dual-process theorists above, and assuming you’ll permit me to skim over some of the detail. I like to refer, sometimes, to the idea of the brain as a ‘lazy controller’. I do it on this site sometimes, but you’ll get a better treatment at my other project, Neurotypica. Very briefly, my ‘lazy controller’ is the idea that one conclusion we might draw from how the brain treats difficult tasks is that it responds quite lazily. As dual-process theorists point out, deliberative thinking is expensive. But the brain doesn’t need to respond to difficulty by switching on some distinct, slower and expensive ‘system 2’. It has a bunch of tools available to it, from cheap to costly. Why would it go from the lowest price to the highest price, if it thought it could just use the next most expensive thing than the one that just failed?

Essentially, putting all expensive thinking into one bucket is fun, but it’s not really that helpful. This is true, in particular, for the concept of “reason”, and it’s most obvious when we start pulling apart the concept of inference.

Inferring cause and effect is a very basic capacity for living creatures. Pamela Lyon has an entire article about the inferential capabilities of the humble bacteria, as in fact do I. You don’t need an article to believe me though. You know this. You infer where a ball will be, in order to catch it. That same basic principle describes a great deal of what almost all living organisms do, and you don’t need someone to explain that to you.

Importantly though, inference is not the same as reasoning. David Hume famously pointed out that:

Animals … are not guided in these inferences by reasoning: Neither are children: Neither are the generality of mankind, in their ordinary actions and conclusions

You’re not appealing to reasons to catch a ball that someone has thrown at you. You are doing something quite different. Something more automatic. Something cognitively cheap. Some kind of internal calculation of trajectory and speed that happens when you see the ball leave their hand.

Similarly, you don’t need reasons to infer that your loved one is in a bad mood. You hear it in their tone, or the slump of their shoulders. Some kind of pattern recognition that comes from knowing both them, but also knowing the way people express unhappiness in your culture.

These kind of inferences live in the same kind of space as reasoning, but they have nothing to do with reasons as we normally understand them. As Mercier and Sperber put it:

automatic inference in perception and deliberate inference in reasoning are at the two ends of a continuum. Between them, there is a great variety of inferential processes doing all kinds of jobs. Some are faster, some slower; they involve greater or lesser degrees of awareness of the fact that some thinking is taking place …

And reasoning? Reasoning is only one of these many mechanisms.

Their book spends some time exploring these. For example, automatic inferences we’re unaware of, like those that characterise visual illusions or when we create false memories, and those automatic inferences we are aware of, like intuitions, are different things. In both cases, we don’t really have an awareness of how the knowledge came into our minds—intuition, false memories, and visual illusions are all unconscious processes. But in the case of intuition, we are at least aware of the content. You might not know you’re viewing an illusion, or that a memory isn’t quite correct, but almost by definition, you know that you have an intuition about something.

Their book even teases apart different kinds of intuition. You might have an intuition that you’ve seen someone before, and you might have an intuition that your colleague is about to make a mistake on the job, for example. One—facial recognition—seems to be a far more innate predisposition of human cognition. The other—knowing your colleague is about to do something career-limiting—is something you develop: expertise of some kind.

Equally, you have innate reflexes that make you jerk away from spiders, and the intuitive knowledge that spiders can hurt you, but the latter is developed in a different way from the former. Perhaps it’s even developed from the former, but it’s certainly not the same thing as the former. If I ask you ‘are spiders dangerous’, you’re not simulating a spider attack to respond intuitively. You just know.

And even these two less innate kinds of intuition—spider danger knowledge and colleague fuck ups—seem like different kinds of gut feeling. Expertise isn’t really the same as internalised information.

Reasons are distinct again. You know spiders can hurt you. This is based on reasons. Someone has told you spiders are venomous at some point, or your parent once told you that you should stay away from them, or you’ve had the misfortune of being bitten, or whatever. But you don’t require the reasons to access the knowledge.

Indeed, reasons can be completely distinct from the intuitions those reasons are about, because reasons can be wrong. You might have the intuition that the spider walking along the table in front of you is dangerous, and—if asked—you might give the reason that it’s a big black spider. But this reason is just guesswork about your own mind. You’re assuming that the reason you’re scared of the spider has to do with how you think scary spiders look—big and black—when the real reason might just be some kind of reflex related to fast-moving bugs.

In fact, reasons can worsen your decision-making when asked to justify your intuitions. In the most famous study on this, people asked to give reasons for how tasty they thought a jam was would rate the jam much less tasty than expert jam tasters, especially when compared with people who just ate the jam, judged the jam, and didn’t give reasons for it. Reasons made them worse at rating jam than people who didn’t reason.

As the authors say, “[a]nalyzing reasons can focus people’s attention on nonoptimal criteria”.

¶Reasons are a tool for social interactions

So, reasons are a very distinct form of inference. Sometimes we use them to generate inferences, and sometimes we use them to explain inferences, but they are their own, special thing.

Mercier and Sperber reckon reasons are specifically a social technology. Reasoning evolved so that we could persuade, coordinate, and police each others’ claims. They point out:

much experimental evidence suggests that people quite often arrive at their beliefs and decisions with little or no attention to reasons. Reasons are used primarily not to guide oneself but to justify oneself in the eyes of others, and to evaluate the justifications of others (often critically). When we do produce reasons for guidance, most of the time it is to guide others rather than ourselves.

Once again, we don’t really have to explore the experimental evidence to illustrate what they’re talking about. We can just return to Julie and Mark. The core of the moral dumbfounding paradigm is the ongoing, shitty attempts at justifying some kind of moral intuition. Are students really motivated by the potential for a messed up baby? If so, why did they quickly move on to worrying about the legality, and then to their relationship, and then to the societal impact?

When combined with the laziness of human cognition—always reaching for the cheapest tool to solve a problem—Mercier and Sperber’s ‘Enigma of Reason’ looks a little different. Reason is so flawed because it isn’t to guide the formation of belief or the making of decisions. We have the rest of inference to help us with that. We’ve had the rest of inference sharpening itself against the world around us for huge evolutionary timescales. Reason is newer. Reason requires language.⁴ And so, reason seems more like a tool designed specifically for social consumption. On this view, the flaws of reason become features.

For example, we really do seem to mostly produce reasons after the fact. This is, at a minimum, because we only really need to reason when our intuitions are in question. Remember the dual-process theorists—our ‘fast’ thinking already handles most things well. We almost never need to double-check our work. It’s only if someone mistrusts us, then we’re prompted for reasons to justify our intuitions. Importantly, in this case, we’re going to have to come up with reasons that are good enough to convince others to put aside their own intuitions.

Relatedly, in a social environment, the kinds of shitty reasons we see produced in moral dumbfounding are entirely sensible. Cognition is lazy. Why come up with good reasons at the outset, when bad reasons will be caught, evaluated, and tossed out by others, who are themselves only likely to accept good reasons? Thinking is expensive, so we outsource it to others.

In both cases, we can see why reason sometimes looks so incredibly flawed. When we reason alone we get neither of these advantages. We might have a sense that an intuition is wrong, but we only have to match our own intuition to another intuition of ours to resolve the problem—not, usually, a particularly high standard. In fact, it’s often very easy to simply shave off the uncomfortable edges of a poorly fitting intuition than match it to anything else at all. This is especially the case because, we’re much more likely to accept our own, cheap reasons when we offer them to ourselves.

Reason is for dialogue—with ourselves, since that’s possible, but much more beneficially with other people. As Mercier and Sperber put it:

reason is biased; reason is lazy; reason makes us believe crazy ideas and do stupid things. If we put reason in [a social environment], these traits make sense: a myside bias is useful to convince others;⁵ laziness is cost-effective in a back-and-forth; reason may lead to crazy ideas when it is used outside of a proper argumentative context. We have also repeatedly stressed that all of this is for the best—in the right context, these features of reason should turn into efficient ways to divide cognitive labor.

We are biased towards our own perspective, and only seriously quality check other sides. Outside of a proper back-and-forth dialogue, reason is likely to produce something other than rationality.⁶ We just pile up superficial reasons for whatever intuition we started off with, rightly or wrongly.

¶AI is only (simulated) reasons

So, that is how Mercier and Sperber solve the enigma of reason for humans. I don’t think one must entirely agree with them to be enthusiastic about just how much of our flawed reasoning the argument explains.

More interesting is that their argument happens to neatly explain AI reasoning too. Mercier and Sperber point out that:

Most of us think of ourselves as rational. Moreover, we expect others to be rational too. We are annoyed, sometimes even angry, when we see others defending opinions we think are deeply flawed.

The same thing—an expectation of rationality—makes us furious when AI does messes it up. Furious and confused. As this NYT article put it a couple of months ago:

A new wave of “reasoning” systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.

But, if we treat reason as a social tool, then this statement actually explains itself. We’re expecting AI to have perfect reasons, because—at least in theory—it has all the knowledge of the internet at hand. But reason isn’t really that interested in knowledge, per se. It’s related to intuitions in concert with dialogue. And AI is at a disadvantage when it comes to both of these things. I reckon that AI isn’t “hallucinating” when it gets stuff wrong so much as it’s reasoning like someone who is crippled at reasoning.

To start, the generative large language models we’re talking about when we use the word ‘AI’ are entirely language. They are trained to take in language and produce relevant tokens in response. Tokens are meaningful sub-parts of language. Humans intervene occasionally to reinforce good responses, and stop bad ones, but the entire process is fundamentally based around language alone.

This makes large-language model AIs pure reasons—there are no independent intuitions here. LLMs don’t have beliefs, or goals, or incentives as we would understand them. I speak about this more in my AI isn’t that scary and AI consciousness articles, but the core idea is that this language production is much more like walking is to us than thinking. They are simply responding to the information put in, they aren’t producing it for themselves. They are simulating a process of reasoning from intuition, which is rather different from reasoning.

You can tell that this is true because the errors they make are often quite different from the errors we make. A reasoning human might get a citation wrong, but they aren’t going to generate completely fictitious citations—they’re motivated to actually convince you to understand their intuition, not glide over something that feels right. In contrast, the AI isn’t reasoning over an intuition they’re trying to convince you of, it’s reasoning over a response to your input. They’re simply generating the most plausible-seeming story from the patterns in their training data. Confident, fluent, and sometimes nonsense because they’re completely indifferent to the underlying process. They don’t care whether you believe them or not.

So AI is at a disadvantage because they have no intuitions to reason from in the first place. They’re also at a disadvantage on the other side—they get less help from dialogue.

Mercier and Sperber speak about the ‘myside bias’ in reasoning humans—we reason for our intuitions and against opposing intuitions. We do this because it’s (at least) a division of labour thing—we just need to develop our own side, and we can rely on others to develop the opposing sides, and eventually the intuition that has the best and least superficial reasons will come to dominate.

If humans suffer ‘myside bias’, AI suffers ‘yourside bias’. We prompt these models to be helpful and agreeable. Even if you ask them to challenge you, they’re only doing it to be of use. Unlike reason between two humans, you don’t need to produce reasons to satisfy it that your intuition is the best intuition. It has no intuitions. It will simply produce reasons in whatever direction makes you respond positively.

In the same way that human reason isn’t flawed, when viewed in the correct context, AI sycophancy isn’t flawed either. It’s just simulated reason—reason with no intuition, and no social friction. It’s like arguing with a really well-informed version of yourself.

This actually makes the sycophancy more insidious, because sometimes you don’t even have a ‘side’ for the AI to ‘yourside’. Since you’re often asking the AI about stuff you don’t know about, you get whatever side it selects in the absence of your scrutiny. You get some weird ‘third side’ that’s informed by whatever latent conversational space you’ve pushed it into.

Now you’re both hallucinating, and this seems almost entirely sufficient to explain all the AI psychosis going around, even in the absence of mental illness.

¶What to do about it?

As I was writing this article, I came across a couple paragraphs in Mercier and Sperber’s book that nicely explain the core issues for both humans and AI:

The myside bias, coupled with lax evaluation criteria, make us pile up superficial reasons for our initial intuition, whether it is right or wrong. Often enough, however, we don’t start with a strong intuition. On some topics, we have only weak intuitions or no intuitions at all—a common feeling at the supermarket, when faced with an aisle full of detergents or toilet papers. Or sometimes we have strong but conflicting intuitions—economics or biology? Allan or Peter? Staying at home with the kids or going back to work?

These should be good conditions … Reason has a perfect opportunity to act as an impartial arbiter. When the reasoner has no clear preconception, the myside bias is held at bay and reason can then guide the reasoner’s choice, presumably for the better … …
Still, [experiments demonstrate that in these cases] reason doesn’t do the job classically assigned to it. It does not objectively assess the situation in order to guide the reasoner toward sounder decisions. Instead, it just finds reasons for whatever intuition happens to be a little bit stronger than the others. Humans are rationalization machines.

This is particularly problematic in groups:

When you agree with someone, you don’t scrutinize her arguments very carefully—after all, you already accept her conclusion, so why bother? When like-minded people argue, all they do is provide each other with new reasons supporting already held beliefs. Just like solitary reasoners, groups of like-minded people can be victims of belief polarization, overconfidence, and belief perseverance

AI is this but worse. AI is just optimising for plausible continuations, not rational truth. It’s only going to improve when you add some kind of grounding—some kind of AI version of intuition, like tools or retrieval—as well as a truly adversarial role.

And this is it. The core insight is this. Solving the enigma of reason for humans is the same as solving it for AI. Put another way, The solution to AI hallucination is the same as the solution to human hallucination. You need useful social friction and some kind of innate grounding.

In humans this looks like red-teaming or debate, and the development of data literacy and expertise that improves the intuitions you’re reasoning from.

In AI this looks like adversarial role adoption and self-critique, combined with tool-use that helps verify the information it’s producing against something more objective than its own reason simulations.

In both cases, ban man-guessing. Simple.⁷