Knowledge is just easy measurement

I complain a lot about brain science. It, like many scientific disciplines, leaves you feeling like you know less coming out than you did going in.

But this isn’t entirely fair. It has taught me a lot. Just not strictly about the brain.

Maybe the most important thing it has taught me is that much of what we ‘know’ about any given thing is simply whatever is easiest to measure.

This wouldn’t be a problem, except that when we have an easy measurement we tend to stop there. The most shallow data point becomes the only one.

Let me give you two examples. First up, more than you ever wanted to know about the brain (and yet also less), and then we’ll apply the same logic to my old bug-bear, therapy.

¶Almost everything we know about the brain is at the surface

There are two ways of measuring brain activity in humans.

One way is to measure the blood volume and blood flow. This method is called fMRI, and is predicated on the fact that the brain uses blood as fuel.¹ So the basic idea is, the more blood moving about, the more active the brain nearby probably is—it’s demanding more fuel. The nice thing about this is that we can get pretty good information about where activity is happening in the brain with this method. fMRI can be measured in units called voxels. These are cubes of brain typically about 3mm across.² So we can say, you know in these 20-odd 3mm cubes of brain or whatever, we had more activity when the participant was doing x than when they were doing y. So that brain region is involved in x, and not y. This is essentially what you’re looking at anytime you’ve seen a picture of a brain with a coloured patch; it’s the visual equivalent of saying this brain region eats more blood during the task.

The problem with fMRI is that it’s shit at telling us when something happened. Blood flow happens much slower than thoughts. It takes about 7 seconds for the specific physiological thing that fMRI measures to happen³. Many thoughts can happen in those 7 seconds. You can, in theory, get better resolution on that by making the experiment longer, but you’re reliant on participants who get fatigued or fall asleep. If you do an fMRI experiment longer than an hour, other researchers will say things like ‘oh wow’ or ‘poor participants’ or just emit a deep belly laugh at your naivete. So most people don’t get much better resolution, because they don’t (or can’t) scan for long enough. As such, most of our findings in the brain reflect two the kinds of neural activity fMRI is good at—really strong activity and really slow activity.

The other way of measuring brain activity looks at the electical or magnetic activity that happens on the scalp as a result of all the neurons firing underneath. You’ll see this called MEG or more commonly EEG.⁴ M/EEG can get us millisecond resolution on neural activity, but it’s shit at telling us where that activity is happening. We can say something happened at the front, or at the back, or maybe even at the sides, but not much more than that.

There are some solutions for this, the most obvious is by combining the result of fMRI with our M/EEG findings. This is good, but it’s intrinsically biased to a specific structure of the brain. M/EEG measures scalp activity. The cortex is the brain structure that’s all wrinkled up and pressed up against the skull. By combining fMRI and M/EEG we can get nice resolution and where and when things are happening in the cortex, but thats only one brain structure. Underneath that are a bunch more, hugely important structures that M/EEG have a hard time measuring because they’re so far away.

So, we have a literature that, by necessity, concentrates on:

1. Big and/or slow neural changes, wherever they might occur in the brain;⁵ and
2. Fast changes that take place across a large swath of the structure closest to the skull: the cortex.

¶The (only?) major discovery of human brain science

This means that we’re really quite stuck when we’re trying to figure out what the brain is doing, outside of one major discovery. We know that the cortex maps perceptions to actions, because this is the easiest part of the brain to measure. We can extrapolate from that quite a lot, philosophically, which is why embodied cognition and predictive cognition are so popular as frameworks in research and AI and with anyone who says things like ‘my priors’ instead of ‘my expectations’. These frameworks rely pretty much solely on the idea that the brain maps predictable perceptions to predictable actions.

But it would be a mistake to think that this is all the brain does. It’s just what’s easiest to measure. And yet, this is basically the major extent of neural research. Really, read the function section of the wikipedia page. Lots of fancy words, written by brain science PhD students and postdocs like me while procrastinating. It’s not a bad representation of what’s out there in the journals.

And just like in the journals, you come away feeling like nothing was really said except what regions are ‘involved’⁶ in tasks (i.e. big, slow neural changes), and lots about how perceptions get mapped to actions.

Before we talk about why that’s a problem, let’s be very clear. Don’t be tricked by the addition of studies at the microscopic level. So, in the section on ‘learning and memory’, they reference long term potentiation of neurons. It sounds like this is saying something important, and maybe different. But this is just basic neuroplasticity or “neurons that fire together, wire together”:⁷ mapping things to things.

This kind of microscopic research always starts out in slugs and bees and rabbits and whatnot because we can get ethics approval to cut into their heads and plug electrodes in there while they do our tasks. Non-human animal research is not quite so limited as human research, and so we find really interesting things in the deeper sub-cortical structures. But the only ones we can translate to human research well are the ones that translate well to the cortex, because you can’t do the same kind of surgical research on humans. As such, while the initial research into memory happened in the hippocampus of the rabbit, we’ve really only concentrated on long-term potentiation as a mechanism because that happens a lot on the cortex. As such, we still can’t explain non-straightforward patterns of learning and memory—like why do people suffer from dissociative amnesia where the brain dramatically ‘forgets’ some period of time, but then ‘remembers’ it later on. Or the, possibly-related (but who knows) phenomenon of repressed memories. These things require something more complex than these easy to measure features of neural activity, and seem to be highly-dependent on sub-cortical brain structures. We’re left stymied.

You don’t really need to understand this to realise that microscopic-level research that talks about the properties of neurons and neurotransmitters and such have been practically lack-lustre in humans. Just consider that two of our most impressive neurobiologically inspired drugs are Parkinsons medication and antidepressants/anxiety medication and both of these just drown the brain in a neurotransmitter⁸—it’s like pouring petrol in the engine bay of a car and hoping some gets into the petrol tank.

Or, take another example that I hear alot. If we go back to our wikipedia brain function section, we see ‘motivation’. This sounds a little different from mapping predictable things to predictable things. Here, we know that dopamine is involved in things we find obviously rewarding (like taking a drug you’re addicted to, or having sex). We discovered this in rats in a place that’s hard to measure in humans. We got excited and managed to find it in humans too. But again, on investigation, it’s just mapping input to output, perception to action—do x thing and it will feel good. It’s just the same easy thing to measure, and this time we managed to trace the same thing we knew existed from cortical studies or animal surgery studies all the way into the basal ganglia. We call this the ‘reward pathway’ and pat ourselves on the back, but is it that impressive? Do you come away knowing more than ‘rewarding things are rewarding, and so we do them again’?

I want you to pay attention to people who talk about the brain now. Notice what they say. You’ll hear ‘<insert brain region> does x thing’. Here’ they’re not actually giving you any information about the brain at all, outside of what brain region is eating more blood when x thing is happening. A variation on this is ‘<insert greek letter> frequency does y thing’. This is just them noticing the same thing in M/EEG—what frequency is more prominant when y thing is happening. Or something like ’<insert neural pathway/neurotransmitter> is the <pathway/neurotransmitter> responsible for z thing, which just tells you that we see more neurotransmitters when z happens than not.

What they won’t be able to tell you, more likely than not, is what the mechanism is.⁹ How does the brain region or frequency do the thing you’re saying it does? It’s just us doing the same thing we see the brain doing! Finding things that predictably happen together, and making a fuss. We do predictable things in response to predictable things and so does the brain… Amazing.

¶Does it matter?

If you’ve managed to make it this far, you might be wondering why this is a problem. Nervous systems map inputs to outputs. Even bacterial cells have signalling pathways that are reminiscent of this. Maybe that’s all we need?

Maybe. But it certainly hasn’t shed much light on the stuff we actually care about. For example, what in the brain is required to have a self-concept? It’s very difficult to imagine that this arises solely from a mapping from perception to actions. The self reflects on those mappings. What is doing that? If it’s just input-output mappings, then obviously other animals have some kind of preliminary self too. To what extent? When does it change into something that is such a dominant feature of conscious life as it is for us?

Or, what about something less poignant. You can’t text and drive because you can’t switch your attention between the two tasks fast enough to do either well. We can explain driving with our perception-action mappings. We can even explain texting, to some extent. But what the hell is the attention-switcher? What brain thing is deciding what perception-action mapping to pay attention to and how? How can we harness that so people stop killing themselves on the roads?

These are questions no one has very good answers to. The computational approach to mind has predominated for decades now and wiped out much of the once rich discussion about what other ways of processing information the brain might be able to do. And we are not much further down the road to understanding cognition and intelligent behaviour as a result. This is perhaps most evident in AI research. ChatGPT is surprisingly good at writing things, but it’s also surprisingly dumb. Because the brain is probably doing more than the easiest thing to measure, and an AI which does only that easy thing is working at a handicap.

¶Now therapy and I’ll keep it short

If you haven’t heard of CBT, or its trendy sibling ‘mindfulness’,¹⁰ then you haven’t known someone who’s done therapy. It is the most widespread method of psychological counselling deployed in the western world. Why? Because it’s easy to measure.

CBT has long been taught to psychology students as the most empirically robust method of treating psychological wellbeing for the most common disorders (anxiety and depression).

This is because of CBTs fairly unique method of delivery. It’s designed around a discrete problem, with a time-limited approach. You come in and say ‘I’m depressed’ and the counsellor gives you a questionnaire which confirms that fact. Then you come in for around 10 more sessions to do some psychoeducation (‘this is how depression works’) followed by some related homework (‘given depression works thus, try out this tool and see how it improves your depression this week’). The therapist gives you the same questionnaire again at the end, and you are told how much less depressed you are.

This method is so effective because very little relies on the therapist. Your therapist needs to know how the process works, and how to teach you the tools. But the process is almost completely standardised, as are the homework assignments. The therapist’s main job is to convince you that the tools work (psychoeducation) and that you should try them (homework assignments), and then titrate the difficulty to you. The bulk of the rest is done by you. So we see CBT delivered via workbook, online portal, telephone, and so on, and these interventions are as good, or better than in-person therapy.

Because of this reliability, ease of delivery, and because the time-limited nature is built in, we can easily measure how well we’re doing. After session 10, have your scores on whatever questionnaire improved? Unless you have something very complicated going on, it would be hard for them not to. You’ve come away with more understanding of the problem, and 10 weeks practicing addressing it with various tools. Perhaps needless to say, governments love this. It’s easy to fund a 10 week program of therapy, with evidence in support no less. No-brainer. And the funding makes more people do the therapy, which gives us more measurements, which makes the therapy more popular to deploy. And so on. CBT wins.

But is it actually better than other kinds of therapies?

¶CBT is no triumph

CBT is unequivocally better than no treatment, or ‘treatment as usual’ which is some kind of unspecific supportive counselling from a doctor or other treating clinician. This is true for CBT with basically any psychological problem,¹¹ despite the remarkable range of symptoms and causes. The effect-sizes are always trending towards medium—we’re talking treatment efficacies of about 60-70% (people score better at the end than 60-70% of people at the start).

But an improvement for 1 in 4 people isn’t, you know, amazing. It’s certainly no triumph of more than 100 years of modern psychological research. And on close inspection, we find that in comparison to other methods of therapy, it might not be the best. There are plenty of meta-analyses around that call into question CBTs dominance, you don’t need to google long. And the list of alternative therapies is long.

There’s another question here too. CBT is all about the discrete problem. We improve you, you know, five points on a sixty point inventory of depression or something like this. Sure, it’s better than it was. The numbers tell us that. But does it feel better? The numbers don’t tell us anything about that.

And this is all made sharper in the face of a recent study that over the last 40-odd years, CBT has been declining ever more rapidly in effectiveness. I’m not entirely sure we need to dwell too much on the reasons for this—a simple call to the law of large numbers tells us that the larger the sample size (the more people doing CBT), the more likely we are to see an effect regress to the mean of the general population (the more likely the effect of CBT will reduce). But what we can draw a link to is the fact that people drop out of CBT at far higher numbers than any other mainstream therapeutic method. Despite the growing numbers of the treated, CBT obviously isn’t improving how it makes it’s clients feel.

¶CBT is just easiest to measure

So why are psychology students taught to sneer at other forms of treatment? Because CBT is so easy to measure in comparison. Let’s take the most common alternative to CBT, psychodynamic or psychoanalytic therapy. Unlike CBT which very much concentrates on the here and now, psychoanalytic approaches care much more about our history of being in the world. CBT tells you what to do when you have an anxiety attack or a bout of depression. Psychoanalysts will work hard to get to the root of why you have an anxiety attack, or a bout of depression. Usually, these are thought to be a result of early childhood experiences and the subsequent unconscious defences that we have built to protect us.

This kind of subjectivity is an absolute nightmare to make empirical. First of all, when do you do your questionnaires? Psychoanalytic therapy takes as long as it takes to uncover the experiences and the unconscious processes. Might be ten for you and forty for me. How do we treat those as equivalent treatment packages? Or, one of the tools of the psychodynamicist is transferance, where I act out the negative feelings I have for some person from my past on the therapist in the room. This relies heavily on the therapist reminding me somehow of the person from the past. How do we measure how effective transferance is? Every therapist and every client is going to have different interactions, and these reminders are going to be more or less strong, or even more or less present.

The issues go on and on, and you get very similar subjectivity problems in approaches like existential and humanist therapies. We can measure these things, but it’s a very different proposition to measuring CBTs nice 10 session structure. Less subjective therapeutic methods also continue to spring up, and though they tend to cut closer to the more cognitive-behavioural discrete problem solution, they are all messier than CBT to empiricise and are often thought to have too little data to draw definitive conclusions.

But, the studies that we do have seem to indicate that almost all these approaches have very similar effect sizes to CBT. It’s hard to characterise how substantial that improvement is (it’s rarely x points on a whatever point scale after y number of sessions), but this is often superficially non-trivial. The most random and impressive¹² is that of the Alcholics Anonymous programme. For long-term efficacy, AA seems to be at least as good at, if not better than CBT at treating alcoholism. Somewhere around 30-40% of people seem to remain sober a year later, where CBT tends to hover around 30% over the same timeframe. Treating alcoholism is no mean feat—it’s not a few points on a whatever point scale, it’s a huge change in a completely systemic problem that effects almost every aspect of life in dramatic ways. And AA’s method was developed outside the psychological discipline, almost 100 years ago, and to boot it relies heavily on the development of a relationship with a higher power. Not quite CBT.

The fact that CBT is easy to measure wipes all of these facts right out of the young, impressionable psychology student’s mind. It took me almost a decade of study and practicing as a counsellor myself to become suspicious. We are blinded by the scientific ritual, and are encouraged to value only one kind of knowledge production, ignoring all the methods that are at least equally valuable, and certainly worth noting. This is most evident in the placebo effect. Something that is as effective as anti-depressants, and yet we laugh it out of the room more often than not. It all obscures the most valuable feature of any psychotherapy—the therapeutic alliance. The relatonship between therapist and client accounts for the most variance in almost any treatment modality.¹³ Spending more time on this, and less on the easy to measure solution might encourage us to find a therapy approach that fits the client rather than fits our empirical impulses.

The fact that CBT is so easy to measure means that we as clinicians are at odds with the needs of the client. We’re here with our questionnaires and our tools and our 10 sessions (no more, no less), scaring clients off in huge numbers for the coldness of the approach. Off they go into the arms of those methods more comforting but less well-regulated, and we have the audacity to sneer as they go.

It’s alarming really. Not just the generalised paternalistic attitude, but the places the clients end up. Take Jonah Hill’s movie Stutz. A popular film demonstrating an essentially existential approach, where the core feature is the trusting relationship between Jonah and Phil. Its beautiful. But what if that therapist wasn’t quite so professonal? If they had something more predatory, or even just more careless in mind? Forming bonds with people in moments of vulnerability is no thing to take lightly.

¶Outro

Ok, so I lied about being short on that last one. But it all seemed worth saying. The point is that the easy measurement seems like the place we often stop when it comes to carving out our world at it’s meaningful joints.

This wouldn’t really be a problem, by itself. It’s not that everyone always must go deeper. But it does become a problem when we let the easy measurement wipe away all the other avenues for exploration. When the easy measurement becomes a point of stagnation.

The easy measurement is knowledge, but it’s too often the end of knowledge too.