Memory and imagination both use the same architecture

God gave us memory so that we might have roses in December.

J.M. Barrie

Memory, like many things in the brain, is a bit of a mysterious function, but it’s also one of the first cognitive functions people think might be worth improving. However, the way we typically think about memory makes that quite difficult.

It has been tempting, since computers, to think of our memory like some kind of database storage, but this is probably not how most, if any, memories work. Rather, memory seems like it can be broken into some number of different kinds. But that also doesn’t really seem to quite get at the thing. Let me explain why, then let me show you how the architecture of the brain does a better job of understanding how memory works, and how we can get it to work for us.

¶Memory ‘stores’

Trivially, we could split up memory into long-term memory, short-term memory, and working memory. Short term memory refers to when we pick information up, but don’t manage to store it for the long term. Long term memory refers to information that we have managed to store for the long term. Working memory refers to information that we’re paying attention to right now, though this could be information we just picked up or information we had stored for the long term. This is a very common way of thinking about things.

But this kind of multistore model of the brain probably doesn’t quite capture all the places memory is held. For example, there also appears to be some kind of sensory store—for a time, senses hold on to the information they’re receiving until we figure out whether we need that information or not. You’d notice this when you’re asked “are you listening to me”? You weren’t, but that doesn’t stop you from perfectly recalling the last 30-odd seconds of them talking. This kind of memory of a thing is probably not the same as something you were paying attention to, but which nevertheless didn’t make it into your long term memory.

Similarly, not all long-term memory is the same. Another trivial sort of approximation we can make is that there are implicit and explicit forms of long-term memory. Implicit memories are those that we aren’t conscious of. Think of muscle memory (usually called procedural memory), where your body remembers, perhaps even more accurately, when you don’t think about doing a thing than when you do. Explicit memories, on the other hand, are those that we are conscious of, and we bring into mind effortfully, like semantic information (dates, names, places, etc) or episodic memories (memories of events or ‘episodes’). And we shouldn’t forget emotional memory—nostalgia for example, or the tension you feel when you go somewhere that something bad once happened.

And those—long and short term memory—are the less controversial kinds. Working memory is the messiest of all. Alan Baddeley was the first modern architect of our concept of working memory, and he split it into four kinds: a phonological loop, which processes speech; a visio-spatial sketchpad, where visual and spatial information is processed; an episodic buffer, which ties the two together; and the central executive, which sits in the middle and coordinates the process. But attention researchers often wonder whether these are simply forms of straightforward paying attention to things. Are ‘working’ memories so much stores as they are active processes?

Of course, many people aren’t very happy with any of these models, and slice these things up even further, but I’m not sure how much value there is in that. Even in the slices we have, it’s not clear what we should be doing to improve our memory. How does short term become long term? How do we improve our working memory? Does our skill with our visio-spatial sketchpad influence the quality of the episodic buffer? Should we practice each kind of memory individually, or do they have to be operated together to be trained? These questions are the questions that make those little apps that improve your ‘memory’ or ‘cognitive ability’ such failures. Treating memory like a bunch of modules means you never really get at the whole thing. A more useful exercise is thinking about what kind of architecture could support these kinds of memories.

¶Memory as a map

Looking at the brain returns us to the idea of memory as a process. One very common feature of the neocortex—the wrinkly sheet wrapped around the outside of the brain—are these very intricate maps of perceptual spaces. At the back of the brain, we have the visual cortex, which has detailed maps of orientations and contrasts and colours. At the sides, we have the auditory, motor, and sensory cortices, which map frequencies, movements, and tactile sensations respectively. Each of these regions appear to code for very specific aspects of perception in a way that could be drawn out in a map.

So, for instance:

the auditory cortex has neurons that code for specific frequencies of sound, but as you move away from the auditory cortex and towards the motor cortex, you find parts of the brain that are active for language. This is sensible indeed, because language is both a motor task (producing sounds) and an auditory task (words necessarily sound like things).

Or:

as you move away from the region of the brain that codes for body parts we move and toward the vision part and you start to see clusters of cells that enthusiastically respond to things like motion perception—something that is both about moving and seeing movement. Move away from the vision part and towards the hearing part and we see regions of the brain that frenetically respond to things in the world that are both audio and visual, like a barking dog. Move away from the hearing part and towards the body part and you find the language centres of the brain—something that is both about hearing speech and producing speech with our body.

The neuroscientist Michael Graziano looks at the motor cortex as an ‘ethological action map’—this part of the neocortex seems to have distinguishable functional zones which each map to different categories of behaviour. Reaching and grasping, for example, is right next to chewing and swallowing, two kinds of behaviour that often go together. It seems like, perhaps, all of the neocortex does something like this.

Even more abstract parts of the neocortex also seem to map things. The anterior temporal lobe (at the sides of the brain) seems to code for semantic objects (concepts of things). Different patches of brain respond to specific kinds of object—the fusiform gyrus codes faces, and another nearby region houses for example. As you move away from these regions, it codes for more abstract ideas like the semantic concept of home. That might activate the ‘house’ patch, but also other regions because your home is not simply your house.

In essence, certain regions of the brain appear to represent aspects of the world in detail, and as one moves away from these regions, it represents more general features of the world.

Probably, then, a great deal of memory relies on the pathways we create and develop in the processing space of the cortex. Those beautiful neural maps already describe the world and your interactions within it. Why would you use a seperate structure to remember all that information? Wouldn’t be very efficient. So, as we move through the world, we are actively mapping it, and much of our memory is likely the product of this map we are inscribing into our neurons.

¶Other kinds of memory

We could see this mapping process that happens in the neocortex as a kind of perceptual or bodily memory. Really, if you think about it, this could capture the majority of our memory processing—everything you perceive and how it relates to the actions you carry out. But we still have to account for more emotional and declarative (i.e. recalling information and events) kinds.

Emotional memory is harder to explain. We know, for example, that the amygdala plays a role in charging events with emotional significance. This is associated with better recall. It’s a sensible mechanism that helps us to learn the significance of events in the world. But where emotion comes from in the first place is a more difficult question. Some think, me included, that emotions are just another kind of perception. A sort of perception of the body. A visceral sense of goodness or badness shaped by primitive mechanisms of the body that happen long before they’re handed over to the mind for refinement.

Declarative memory is harder still. I have noted before that:

The hippocampus appears to play an extremely important role in the storage of memories. Damage to the hippocampus almost always results in amnesia of some kind, but amnesia of declarative memory (i.e. the recall of information and events). Often, hippocampal damage still leaves people with the ability to use skills they have learned, and possibly learn new skills (even while not remembering that they learned them). The hippocampus also appears to hold information related to the space around you, and that memory of the space with specific ‘place’ cells.

So the hippocampus could be a place where we get some real specificity—not merely the map of us in the world, but an actual map of the details of the world. And indeed, the hippocampus has:

a reciprocal relationship with our verbalisations. Language helps us to ‘chunk’ memory items together giving us access to more detail, more easily. For example, I don’t need to describe a chair to you. By communicating the word chair, I have already communicated a large chunk of information: an object for sitting, a seat and a back, some kind of legs or foundation, and so on.

So, by mapping our perceptions of the world to language objects, we might get a pretty good chunk of detail about the world packed together in quite a tiny space, using the very same infrastructure we use to declare that stuff. Maybe that’s all we need for declarative memory?

¶Remembering, forgetting, and imagination

Even if we don’t understand them completely, thinking of our memory as some kind of neural map—maps of our patterns of acting in the world, maps of the feelings of goodness and badness that went along with those patterns, and maps of the details of the world at the time in the form of place cells, or via the medium of language—is much more satisfying to me than trying to imagine a series of computer databases optimised for different kinds of storage.

It also helps explain the highly context dependent nature of memory. It’s much easier to remember things in the same context as you learned them. For example, when you move from one room to another to do something, you might forget why you went into the room in the first place. This is because the context of the room is part of the memory. It’s not just the memory of the thought, but the memory of the thought in the context of the room you were in when you thought it.

If memory is largely a map of your experiences in the world, then where and how and under what circumstance something happened will have an enormous impact on how you remember it. The denser the map around a memory (the more context there is), the more easily one can find it to remember, particularly if you’re in the neighbourhood where you learned it in the first place.

This idea of a map of memory also helps us understand a bit more about how we forget things, as well as how we imagine things. If you consider:

Our brain is comprised of two kinds of cells: grey matter and white matter. Grey matter refers to our neurons: the basic signalling unit of the brain. Each neuron is connected to nearby neurons and passes electrical and chemical signals to these neighbours. These chains of interconnected signalling devices make up a complex series of’pathways’ for information about the world to travel, and in this way neurons pass information about our perceptions to our muscles so that we can make the appropriate responses.
…
White matter refers to the other kinds of brain cells—glial cells. White matter fits all around the neurons (the ‘grey matter’) and takes care of them—protecting them, repairing them, and improving their performance. Importantly, the more you practice something—a particular pathway of input to output—the more white matter will grow around those pathways—all the better to support them in doing this thing that we do all the time. This is wonderful for speed and performance. But it means that these neurons, now surrounded by all this construction, can’t change their connections very easily anymore. It means these neurons are quicker at doing some things, but much less good at doing others. The patterns in our brains become increasingly rigid.

These pathways, then, are the physical form of our memory maps. In these wired pathways of interconnected neurons is real information about our senses, about the actions those senses prompt, and about the states of our body—our emotions—that are commonly associated with those relationships, all working in tandem with the world we have been acting in.

When we forget, it’s not that we are forgetting something entirely. All that needs to be lost are the pathways that connect the thing we should be remembering to the thing we’re doing now. You’ve almost certainly had an experience like this. You’re wandering around on a Wednesday. You have the vague sense you should be doing something, but you’re not sure what. Then your dentist’s SMS system reminds you that you have an appointment and all of a sudden where there was nothing but a vague sense of something, you have a flood of information. The time of the appointment; what it was for; how much it’ll cost; the name of your dentist; where you were when you decided to book; how much you hate the dentist; and so on. The full memory was there, it’s just that the link hadn’t been established.

This is, in fact, why many researchers wonder if there’s any kind of limit to our long-term memory. Maybe there are hoards of lost information in the depths of our brain, linkless and forgotten. It also helps explain some of the weirder features of forgetting and remembering, like repressed memories, but I’ll let you read that article for more rather than rehashing it here.

The primary thing, though, that we can learn by turning to the brain is that memory, for the most part, almost certainly uses the same architecture as our processing of the present. We’re just adding to the map.

Maybe the most interesting corollary to this is that our imagination probably does too. Imagination seems very much like you’re just ‘remembering’ the future by putting together existing pathways in likely configurations that haven’t happened yet. This is poignently illustrated by the work of Adrian Owen. In some of his most famous experiments, he’ll ask completely comatose patients to imagine playing, say, tennis. An astounding number of them, despite having no indication of having heard anything at all, will show very similar neural activity as we might expect to see in someone actually playing tennis.

¶Outro

If memories are maps rather than databases or stores, it implies a couple of obvious things.

Firstly, if you’re trying to remember something, or remember to do something, then place the thing squarely in a map that already exists. A memory that isn’t connected to any of this existing infrastructure will be lost—linkless and forgotten. So, for example, the classic memory palace was used by Cicero to remember his orations in the Roman senate. By attributing a speaking point with a decoration in a well-known home, or a building on a well-known street, he was able to ‘walk’ in his mind through his speaking points, collecting them as he spoke. Similarly, many social scientists talk about ‘chaining’ or ‘stacking’ habits—planning to do new habits after a well-established, but similar habit—so that you remember to do the new one better. A new skincare routine built on top of an existing hygiene routine will be much easier to remember than trying to do it after dinner or something like this.

On the other hand, our memory maps grow rigid as we strengthen them and develop them. Remember:

the more you practice something—a particular pathway of input to output—the more white matter will grow around those pathways … wonderful for speed and performance. But it means that these neurons, now surrounded by all this construction, can’t change their connections very easily anymore

Our memory maps, then, also act as a kind of filter. They’re not really a map of the world, but a map of our world. They’re a map of our biases, our prejudices, our expectations, our hopes, our fears. They’re a map of our identity. And so, as we get older, our memory maps become less flexible, and our ability to see things in different ways becomes more limited. We might better remember what we’ve done, but because our imagination uses the very same architecture, we become increasingly unable to imagine a different future. To stop our maps from holding us prisoner to our memories, we have to work to combine the parts of the maps which don’t usually touch, as I write more about here and here.

It’s not really that ground-breaking, is it? But I find it kind of sweet to think that the same part of the brain that manages my memories also manages my future, and that by acting on one, I act on the other. This, actually isn’t always a good thing, but there’s something rather pretty about this closed loop of time running through our minds, I reckon.