What happens in your brain and what happened to Albert's brain

So we’ve finally arrived. The end of our 19 quadrillion mile journey, and here we are inside your brain. In your visual cortex to be precise, under that bony lump you just felt at the back of your head. The pattern of light has been carried from your eyes to your brain and is now a pattern of flickering activity in billions of brain cells. This pattern still has to be decoded and put together before we can really claim to have been seen. This is probably one of the most difficult things your brain does.

"What's difficult about seeing?"

Seeing looks easy but is a lot more complicated than it seems. So complicated that it’s one of the things computers can’t do well. If computers could be smug, they would boast about being able to calculate millions of times faster than humans or being able to beat anyone on the planet in a game of chess. They couldn't boast about being able to see.

"Seeing doesn’t feel hard. I can certainly see better than I can play chess.”

That because human brains are built for seeing not playing chess. A four year old human will see better than the best supercomputer on the planet. A computer can certainly be programmed to recognise particular shapes but ‘seeing’ properly needs understanding of what an image means, and that requires an ability to think. Understanding and thinking are things that can’t easily be broken into little steps. So computers are great at some things but they can’t yet think for themselves or see for themselves.

“How much of our brains is involved in seeing?”

Almost half of your brain is working on vision in one way or another. If half of your brain was dedicated solely to playing chess then chess would be pretty effortless too. When you are sitting still, like now reading this blog, your brain is using just under a third of all the oxygen you breathe and burning about a third of the calories. So with half your brain working on vision, that means that a sixth of what you eat is used by your brain just to see or roughly half your lunch every day. Things like speech are done by tiny areas on just one side of the brain. Hearing is done by parts of the brain that are about one tenth of the size of the visual areas. The vision areas are just as big if not bigger than the areas using for thinking and what are called ‘higher mental functions’ - the things that make you humans clever and different to other animals. Even the brain areas involved in moving arms and legs (or anything else) are also surprisingly small. So compared to anything else you do, the fact that so much brain power is needed for seeing proves the point that seeing must be very complicated indeed.

“Does the brain just work like a really fast computer?”

Your brain works in a very different way to a computer which helps in vision and lets you do lots of things at the same time. In a computer there usually just one central chip, reasonably enough called a central processor unit or CPU, that does one thing at a time and does all the actual computing. Even though you might feel that you can only think about one thing at a time, your brain is actually made of trillions of little CPU’s called brain cells. Each of them can be working on one small part of whatever you are doing. So you can walk, see, talk and listen all at the same time. It’s what computer scientists call parallel processing, with lots of little computers sharing out one big problem.

“Do we understand anything about how the brain helps us to see?”

We know quite a lot but much more we still have to work out. The main challenges of seeing can be broken down into a few stages. First the pattern of light has to be separated to pick out different objects. This is the ‘What’ part of vision. For this the brain looks at edges, sudden changes in the pattern of light. As soon as an image reaches the visual part of the brain, the position and angle of any edges are picked out first. Colour and how things are moving can also help to ‘glue’ patterns of light together into an object. This gives the brain a basic sketch of what’s there.

“It sounds like creating a cartoon in reverse. You start with line drawings and make a simple story board with just a few of the details.”

Is seeing like drawing a cartoon in reverse? I suppose it is a bit. Well the next step is for this outline sketch or story board to be sent to next level of brain cells that start adding understanding by recognising different features.

“That would be where the cartoonist adds facial expression.”

Well your brain does this in a special area of the brain, the inferior temporal cortex that helps to recognise faces. You can find the inferior temporal cortex easily enough. Find the bony lump behind your ear and it’s just above that on the inside of your head. In this area there are brain cells that are good at finding bits of faces noses, eyes and mouths. These brain cells then lead onto the next level where the pattern of recognised features, like a police identikit picture, leads you to realise that the pink thing over there is your grandmother. You can still recognise this shape as your granny even if her legs are hidden from view by the suitcase, she is wearing new clothes you have never seen before and is perhaps even looking a little older since she last visited. That’s the real trick of recognition to be able to pick out something familiar in an unfamiliar setting, something brain scientists call invariance.

“Hmm. Do we have a brain cell for everyone we know?”

There was an idea, first suggested by Jerry Lettvin in the 1970’s that people might have a single grandmother cell (and one for everything else they recognise); a brain cell that fires when your granny appears. This master brain cell has billions of cells working for it sifting visual information. The granny cell sits on the top of the pile like a queen and makes the final decision if that person really is your granny. It is a nice idea in some ways but it seems the brain works a little differently. Rather than a single brain cell for each memory, scientists now think that recognition and memory are coded as patterns in large groups of brain cells. Otherwise one small knock on the head and your granny could vanish before your eyes.

“So is that the secret of seeing, being able to recognise what you see?”

Recognising things is important, but knowing where they are is just as important. The image your brain receives from the eye is, like the image made by any camera, flat or two dimensional. In this flat image light from all the objects in view is superimposed. Distance disappears and the only difference between objects is the pattern of light. Looking at a photograph none of this is very obvious to you of course. Your brain works this out for you so that you can tell effortlessly where one object stops and the next begins. Even in a two dimensional photograph your brain seems to instantly recreate some sense of depth, a notion of where things are in relation to one another and puts a label on what all these objects are. So it looks simple enough to work out to how far things are away, but it’s not as simple as it looks.

“How does our brain do that?”

The brain uses lots of tricks for working out how far away things really are. Cover one of your own eyes and then the other. The world will look slightly different in the two eyes, things close up seem to jump from side to side compared to the background as you change from one eye to the other. The brain has learnt to use these differences between the two eyes to work out where things are in depth. Your brain takes the two slightly different images from your two eyes and combines them into the one image that you see. The differences that seem to be lost in the process are converted by your brain into a sense of depth that is called stereopsis. You can also judge depth with just one eye. Move your head from side to side with one eye shut. Nearby objects move from side to side, but further away things seem to move less.

"That's why pigeons keep nodding their heads as they walk in the park! You told me about that earlier."

Well remembered. A pigeon’s eyes are on opposite sides of the head so they can’t use stereopsis because the two eyes are always looking in opposite directions. The final part of vision to understand what you are looking at and how all the different objects at different distances relate to each other. To do that you often need to understand how the world is put together and what sorts of things are likely and unlikely. That’s how some visual illusions work. If you have a drawing where there are two possibilities that are both equally likely, your brain can't make up its mind and you see things first one way and then the other. Take a look at these drawing of a cubes.

“What about them?”

The middle one, number 2, is a famous cube called a Necker cube. Look at it and decide whether it is sitting in the same orientation as cube 1 or cube 3.


“It’s the same as 1, no number 3. It’s impossible it keeps changing.”

Exactly, because it is ambiguous the brain can see it both ways. In this case the image in your head flips between the two possible real world shapes because the brain can’t decide which is correct. Your brain tries to take what your eyes see and work out what object is out there making that pattern of light. Sometimes it will create lines that don’t exist.

“What do you mean?”

Look at this picture, a Kanizsa triangle, what do you see?

“A black triangle sitting on three dots on top of another triangle.”

There is no black triangle there.

“Yes there is.”

No it just looks like a real triangle even though it doesn’t have sides because to your brain it looks more likely to have a triangle sitting on three sound dots than to have three dots with wedges out of them that line up so perfectly. The white triangle behind makes the effect stronger but it works even without that. The sides of the imaginary black triangle are called an illusory contours. The other thing that helps us to make sense of what we see is our memory. If there is an hidden image in a visual illusion it can takes ages to see it. But once you’ve see and remembered you can look at the same picture 10 years later and see the hidden image straight away. Take a look at this picture. Can you see the dog in the picture?

“No, where is it?”

Put the mouse over the picture and you’ll see. Now whenever you see that picture you brain will see the dog straight away. Without memory seeing the dog is very hard. Once the memory is in your head it is impossible not to see the dog.

So seeing is not as simple as it looks. The whole process of seeing and perceiving merges into almost every other aspect of what your brain is doing. Remembering, thinking, learning and interpreting are all part of seeing. How humans actually manage this part of vision is still mysterious. Crack this and you’re close to unravelling what it is to be human.

“We still don’t know how our brains see?”

Three hundred years ago William Molyneux said it’s not the eyes that see ‘it is the soul’. And three hundred years later he’s still not far from the truth. A lot has been learnt about seeing but there is a lot more we don’t know. The secret of seeing is there somewhere in your brain which is getting a stream of images through your eyes, from the moment you open your eyes first thing in the morning until they close in sleep at night. Even though you don’t think you are working that hard, parts of your brain are working flat out. For every second of your waking hours your brain is untangling the patterns of light reaching your eyes to make sense of them. Continually trying to work out the shape of the world and what is happening ‘out there’. You see things, wonder about them, and learn about them, seeing merges into thinking. What you are doing right now, reading, is from a vision point of view quite simple, but the patterns of light and dark on the page get transferred almost directly to ideas, thoughts and memories.

“Albert, this may be a bit of sore point, but do you know what happened to your brain when you died?”

At the time no, because I wasn’t using it any more. But since I’ve come back I read that story. It is not a happy one for me nor, it seems, for Dr Harvey who stole my brain in the first place.

“I read about that. He lost his job because he wouldn’t give your brain back to the university of Princeton.”


It seems that my brain was chopped into pieces and left in a two jugs for decades. They even published pictures of my brain in a medical journal. When scientists eventually got around to studying the brain they found my brain had a few unusual features but was smaller than most brains. Sperm whales have brains five times the size of the average human. Does that make them five times smarter than us?

"No, they just have bigger bodies so they have bigger brains too."

So the size of the brain as a percentage of our weight would be a better way to work out how smart we are.

“I would think so.”

So on that basis mice are 50% smarter than humans.

“So I guess brain size is not that important.”

It is much more important what you do with your brain than what shape or size it is. One of the largest preserved brains is in a pot at Cornell University. It used to belong to a man called Edward Rolloff who was a self taught expert in languages but also a murderer. When he was being hanged for his crimes his last words were; ‘Hurry up I want to be in Hell in time for dinner.’

What were your last words?

“I told them to the nurse who was with me as I passed away. Sadly she didn’t speak German so she just look blankly back at me. That was my last memory, a sadness not at dying but that those last words would be forgotten forever. Do you speak German?”

“Sorry, apart from ‘auf Wiedersehen’, barely a word.”

That’s a shame, so there is no point in telling you either. If I say so myself, as last words go they were rather good.

“Tell me in English then.”

It wouldn’t be the same. So much would be lost in translation.