Seeing sound, tasting color
Imagine Thursday. Does Thursday have a color? What about the sound of rain — does that sound taste like chocolate? Or does the sound of a saxophone feel triangular to you?
For about 3% of the population, the sharp lines between our senses blend together. Textures may have tastes, sounds, shapes, numbers may have colors. This sensory crosstalk is called synesthesia, and it's not a disorder, just a different way of experiencing the world.
To learn about the neuroscience behind this fascinating phenomenon and what it tells us about how our brains perceive the world, we were fortunate enough to speak with David Eagleman, a neuroscientist, author, and entrepreneur here at Stanford. Eagleman has long been fascinated by synesthesia and what it means about how our perceptions shape our reality.
We also discuss Eagleman's work with Neosensory, a company that develops technology to help individuals with hearing loss by translating sound into vibrations on the skin. The episode highlights the adaptability and plasticity of the brain, offering a deeper understanding of how our perceptions shape our reality.
In addition to his research, Eagleman is a prolific communicator of science — the author of several books including Livewired and Incognito and host of the PBS series "The Brain with David Eagleman" and the new podcast series "Inner Cosmos".
Enjoy!
SUBSCRIBE on Apple Podcasts, Spotify, Google Podcasts, Amazon Music or Stitcher. (More options)
Learn more
- Livewired (book)
- Incognito (book)
- Wednesday Is Indigo Blue (book)
- Neosensory (website)
- Synesthete.org (website)
- Inner Cosmos with David Eagleman (podcast)
Episode Credits
This episode was produced by Michael Osborne, with production assistance by Morgan Honaker, and hosted by Nicholas Weiler. Cover art by Aimee Garza.
Episode Transcript
Nicholas Weiler:
Welcome back. This is From Our Neurons to Yours, the show where we crisscross scientific disciplines to bring you to the frontiers of brain science. I'm your host, Nicholas Weiler.
Imagine Thursday. Does Thursday have a color? What about the sound of rain? Does that sound taste like chocolate? Does the sound of a saxophone feel triangular to you? For about 3% of the population, the sharp lines between our senses blend together. Textures may have tastes, sounds, shapes, numbers may have colors. This sensory crosstalk is called synesthesia, and it's not a disorder, just a different way of experiencing the world.
To learn about the neuroscience behind this fascinating phenomenon and what it tells us about how our brains perceive the world, we were fortunate enough to speak with David Eagleman, a neuroscientist, author, and entrepreneur here at Stanford. Eagleman has long been fascinated by synesthesia and what it means about how our perceptions shape our reality. I started by asking Eagleman about why some people experience the world in this dramatically different way.
David Eagleman:
Yeah. In general, Mother Nature is always trying as much variation as she can in everything, height, weight, shoulder width, whatever, people look different. There's all kinds of things that Mother Nature is always trying out to see if it works well, and synesthesia is that kind of in the cognitive domain in terms of our consciousness of the world. You can have different ways of being conscious. And actually, I just want to say, one of the things that I've been doing for the last several years now is really looking at all the different ways that people can experience the world in ways that are now measurable to us but you wouldn't have thought. So for example, synesthesia, if the person next to you is a synesthete and you're not, you don't realize that they are having a different experience of reality. But it's the same with, let's say, how well you imagine something in your mind's eye.
So if I say, imagine an ant crawling on a red and white tablecloth towards a jar of jelly, some people see that like a movie in their heads, and some people have no particular visual experience at all. You can use this with anything, like how much people hear their internal voice. Some people hear their voice all the time like a radio, and other people just have silence, there's nothing, there's a spectrum. So traditionally in neuroscience, we've looked at the big differences between people as in, oh, that person has psychopathy and the rest of us don't, or that person has schizophrenia and the rest of us. But the truth is that everything now is being appreciated as being on a spectrum, first of all. So you don't just have psychopathy or schizophrenia, there's a whole spectrum there. But more importantly, there's all these different other ways that people can experience the world differently.
Nicholas Weiler:
Well, that's so fascinating. I think that's one of the reasons why people... I'll speak for myself, that I certainly found neuroscience so interesting is to understand what makes our experience unique. So you mentioned that we now have better techniques for answering some of these questions. Can we now measure things in a way that would let us see the differences between the brain of someone with synesthesia and someone who doesn't have such strong synesthesia or between different types of synesthetic experience?
David Eagleman:
Yeah. For many years, my colleagues and I have done neuroimaging with synesthesia, and what you see generally is just more crosstalk between areas. So it's like two countries with porous borders between them. That's what it's like with different brain areas for example, between let's say letters and color. Now, it's slightly more complicated than that, but essentially that's it. The network is just running a little bit differently and so as a result, you have these blended senses. Also, my lab has been trying to chase down the genetics underlying this to understand how does a tiny genetic change between you and the person next to you, how does that lead to a totally different perception of the world. And I'm actually calling this perceptual genomics, which is... I mean, for many decades people have been pulling the genes for aortic stenosis or predisposition to diabetes or whatever, but this is, hey, how does a difference between my ACTG and yours lead to us experiencing the world in a different way?
Nicholas Weiler:
Well, how many different ways does this experience manifest itself?
David Eagleman:
My colleagues and I have found at least 80 different forms of synesthesia, essentially any cross-blending of the senses that you can imagine, we found somewhere. Now, some are more common than others, but you imagine temperature, taste, smell, touch, all kinds of aspects of vision, all of these things fall under the umbrella of synesthesia. And I've actually started to hypothesize that there are several other things that we probably should put under the umbrella of synesthesia. Just as an example, in autism, a lot of kids have what we call sensory processing disorder, where particular sounds will be really aversive to them. They can't stand the sound of a vacuum cleaner or the texture of a sweater or something like that. That actually is exactly like synesthesia, where you've got this cross-blending of the senses, it is just that in sensory processing disorder, instead of it triggering a color or a sound or something, it triggers something aversive.
Nicholas Weiler:
Fear or anxiety or something like that.
David Eagleman:
I think it's actually just an aversive feeling like, oh, that's disgusting, or that's-
Nicholas Weiler:
Oh, interesting.
David Eagleman:
... ugh in some way.
Nicholas Weiler:
Okay. So yeah, more direct experience of the thing itself being. Because I read you writing somewhere that for people who have certain forms of synesthesia, the color Monday is just obviously blue or they're identical. The Monday just is blue. It's not that Monday looks blue, it's not that Monday reminds you of blue, it's that in some senses they are the same thing. Is that accurate, or am I misremembering this?
David Eagleman:
No, that's right. I mean, it's self-evidently true that Monday is blue and that's it, and Tuesday is orange, and Wednesday is purple, and so on. And of course, it's different for each synesthete. And interestingly, we had a paper some years ago with two of my colleagues at Stanford who found that some kids born between, let's say 1969 to let's say 1984, a lot of children in that range had a very particular set of colors with their letters. I can't remember what. I think A was red and B was orange, and C was yellow, and D was green, or something like that. And then it cycled those colors. And that was very unusual because typically when we find synesthetes, it's different for each person.
I had collected a database with about 60,000 synesthetes in it at this point. So we went through there and found hundreds and hundreds of people who had this particular pattern of colors, and it turns out that particular pattern is the Fisher-Price magnet set. So there's some amount of imprinting that can happen. So all these kids born in this very particular window of time when that set was popular imprinted on those colors. Otherwise, it's very difficult to chase down the origin for most people.
Nicholas Weiler:
Very interesting. Well, I want to come back to that idea of where do these sensory substitutions, where does this imprinting happen and when can it happen. Would you be able to describe for us where this happens in the brain?
David Eagleman:
Sure. So there are particular regions of your brain that care about letters and numbers and other regions of your brain that care about color, and these happen to be next-door neighbors. And so what happens in synesthesia appears to be that there's more crosstalk between those areas. So you see a letter and that triggers a color experience.
Nicholas Weiler:
It is generally for areas that are spatially adjacent to each other in the brain?
David Eagleman:
That is generally the case.
Nicholas Weiler:
Fascinating. Okay. So we were talking a little bit about how people develop synesthesia. You're looking at the genetics of why some people have this trait very strongly and other people less so, but this is not something that only occurs in childhood or as a long-term personality trait because you're also very interested in the idea of sensory substitution more broadly. You have a company called Neosensory that's developed a wristband to help people with hearing difficulties by translating sound into vibrations on the skin, is that right?
David Eagleman:
That's correct, yeah.
Nicholas Weiler:
So what's the connection between synesthesia and this sensory substitution?
David Eagleman:
The issue that I've always been interested in is how our brains construct our reality. And the weird part about brains, of course, is that they're locked in silence and darkness and all that you have going on in brains are spikes, these little electrical spikes running around and the chemical releases that are caused by those. So you've got this giant forest of 86 billion neurons all spiking off between ten and a hundred times per second each neuron. But somehow this constructs your reality, your colors, when you hear a sound, whether you have a feeling on your fingertips when you get a taste in your mouth, all this stuff is just an internal construction. So what I got interested in many years ago was this question of sensory substitution, which is, could we just feed information into the brain via unusual channels and could the brain figure out what to do with it?
So the thing that we did in my lab is to take people who are deaf, who have severe or profound hearing loss, and capture sound and translate it into patterns on their skin. So we're doing what the inner ear normally does, which is take sound and break it up from high to low frequency and push it over to the brain, and we just transfer that to a different part of the body. And that works, and people who are deaf can come to understand better and better what is happening in the auditory world around them. So we originally built this as a vest, and then as you said, I spun off a company called Neosensory, and we ended up shrinking that down to a wristband with vibratory motors all along the band. And it works really well.
And what we've done since that time, by the way, so not only do we do that for deafness, but we also do that for high-frequency hearing loss. And that's what happens to people as they get older, they lose just the high frequencies. So what we're doing is a very sophisticated machine learning that's just listening in real time for high-frequency parts of speech called phonemes, things like S or V or K or TH or things like this. It just listens for those in real-time and buzzes you in different ways to say, oh, I just heard an S, oh, I heard a V, I heard a TH, and so on. And so what happens is your brain can learn how to combine two sources. It's doing fine at the medium and low frequencies in your ear, and it's clarifying the high frequencies from your wrist.
Nicholas Weiler:
How interesting. So it's sort of providing a little extra context in a way. I'm amazed that it works so well with speech. Is that a level of detail that you can get with people who have more profound hearing loss, or is there less detail that you can provide there?
David Eagleman:
Good question. The resolution is low. So somebody who is deaf with the wristband can understand, oh, there's somebody talking, there's a dog barking, there's a knock at the door, there's a car passing, things like that really easily. Understanding speech, speech is a little too fast and detailed to be able to just listen as well as you could with an ear using the wristband. But with the Clarify, which is the same hardware just running a totally different algorithm, all it's doing is just telling you about a few different high-frequency phonemes. So it's just saying, oh, I just heard an S, I just heard a V, and so on. And what that does is you're doing most of the work and it just says, hey, this is a clarification of what's going on out there.
Nicholas Weiler:
And is the experience that people report with this that they are hearing better?
David Eagleman:
Exactly. What happens is... I'll tell you what typically happens, actually. People say, "I don't know whether this thing is working or not." You have to wear it for about a week or so for your brain to figure this out. So they say, "I don't really know if it's working or not," and they take it off, and then their spouse yells at them and says, "You have to put that thing back on because you're not hearing me without it, but with it, you just don't have any problem hearing me." So they usually discover through external sources that they're hearing much better.
Nicholas Weiler:
Yeah. So you don't even perceive the difference necessarily, but the results speak for themselves.
David Eagleman:
Yeah, exactly.
Nicholas Weiler:
Well, this brings up this sort of larger question about our brain's capacity for learning and change, which I know is a big interest of yours as the topic of your most recent book, Livewired. With synesthesia, I've often wondered... You talked about it being on a spectrum and most things in the brain being on a spectrum. And I've often wondered if synesthesia is sort of an extreme version of the human brain's capacity for metaphor and analogy, and how we say that music sounds sharp or a-
David Eagleman:
Or the tie is loud.
Nicholas Weiler:
Yeah, tie is loud. Exactly.
David Eagleman:
Or a person is sweet.
Nicholas Weiler:
Yeah, exactly. So one, I wonder if those are things that we all experience, and two, whether it's something that we can enrich our lives with by adding more of those experiences.
David Eagleman:
Well, you're exactly right in your intuition that the brain is all about metaphor, everything is always getting cross-connected. What's interesting is that we don't call something synesthesia unless it's rare in the population. For example, if you ask people, let's say you play the high note on a piano and you play the low note and you say to a person, okay, which one is brighter? Almost everyone answers the high note. And if you say, which one is larger, almost everyone answers the low note. And so we have these cross-sensory correspondences that we all hold, but if we all hold them, we don't even find it interesting, we don't call that synesthesia, but what that illustrates is that... Yeah, I mean, metaphors are such a beautiful and important part of language. It allows you to just quickly understand something by mapping it over to the physical domain.
As far as can we experience it more, either you're a synesthete or you're not, there is drug-induced synesthesia where people on various sorts of psychedelics will experience synesthesia. Also, people often experience it when they're really tired. For example, if I'm really tired and falling asleep and somebody slams the door, I'll see colors. But in general, we're mostly stuck with whatever reality we're experiencing. And by the way, I'll just mention that for a synesthete, it sounds so amazing to experience their world that way, but for the person, it's no big deal, it's just that's what reality is. Let's assume you have normal color vision, and if someone else were colorblind, they'd say to you, "Oh my gosh, it must be the most amazing thing." And you'd say, "Oh, yeah. Well, normally I don't even think about it, but I guess it's okay."
Nicholas Weiler:
Right. One of the reasons I ask this is, one of my favorite visual portrayals of something like synesthesia is from the movie Ratatouille. I'm sure that people mention this to you all the time. And there's a beautiful sequence in which one of the main characters is just eating a strawberry and eating a piece of cheese, and it's translated into this gorgeous musical and animation of what those flavors are like and what they're like in combination. And then his brother tries it, and it's very, very faint. He doesn't really get it. So this is why I'm wondering... You have this company Neosensory to help people who have sensory loss, but I wonder if there are places that you see things like this incorporating some of this sensory crosstalk as something that anyone can do as sort of a practice to enrich our sense of the world, even if it's not like synesthesia really.
David Eagleman:
Interesting. I think you only find this in the artistic community. So people have done lots of things. For example, Scriabin, the composer, put together an organ that shines different colors of light when you hit different notes. So there's a multisensory experience. But again, if you're not synesthetic, I don't know that there's a way to experience it yourself. In fact, many, for example, music composers or visual painters who are synesthetic have written a piece of music or painted something thinking, oh, this is going to mean so much to other people, but of course, it doesn't mean anything to you. In fact, it doesn't even mean anything to another synesthete because it's different for them.
So there's a sense in which we're all trapped inside our own reality, and there are so many beautiful things that we all experience, but we can't easily experience each other's reality. I'll just give you one more example of that. You and I maybe have normal color vision. We have three types of color photoreceptors in our eye, but what's now known is that a very tiny fraction of the female population has not just three, but four types of color photoreceptors, which means they see colors that we can't even imagine. They're having a different experience of color. And so would we love that? Yes, we'd love that. But to them, that's just reality.
Nicholas Weiler:
So thinking again about the way that people experience the technology you've developed with Neosensory and being able to translate these vibrations into clarifying context for speech or just general representation of the auditory world. I want to come back to your book, to Livewired, and to sort of the bigger concept of how adaptable our brain is, how changeable our brain is. Is there a bigger message we can take away from the fact that someone who has lost hearing can restore some of that functionality through another sense?
David Eagleman:
Yeah. I mean, the big lesson, of course, is just that the brain is so fluid and flexible. And it's not normally thought of this way. So one of the main classes that I teach is brain plasticity, and my hope is to give students a completely different kind of understanding of what the brain is actually up to. Because in a typical textbook, you find a diagram that says, "Okay, look, this part of the brain is for vision. This is for hearing, this is for touch, and so on." And it seems like it's a map with set borders, but in fact, it is tremendously fluid and it's always changing. So of course we know that when somebody is born blind or goes blind, the visual cortex gets taken over, and if they go deaf, their auditory cortex gets taken over. No part of the brain lies fallow, and that's why sensory substitution is possible because it just doesn't matter how you put the information in there as long as it gets there.
And so the model I developed on this over many years is what I call the potato head model, which is just, you remember as a kid plugging in the eyes, the ears, the nose, whatever, into a potato. That's what the brain is like. It just figures out, okay, I've got these two spheres on the front of my skull that convert photons into spikes, oh, I've got these things on the side of my head that convert air compression waves into spikes, oh, I've got this nose and mouth that convert mixtures of molecules into spikes, but the language is all the same on the inside, it's just zeros and ones. And one of the things that I've just always been obsessed with and wrote a lot about this in Livewired is that when you look across the animal kingdom, you find all kinds of different sensors like snakes pick up on infrared, and the black ghost knifefish picks up on perturbations and electrical fields, and lots of birds and cows and insects pick up on the magnetic field of the earth and so on.
But their brains are really similar to ours. You don't need to reinvent the principles of the brain to just plug in new sensors. And same with output, you got animals all around us, including other mammals who have fins and prehensile tails and wings and whatever. And the brain doesn't need to be reprogrammed, it just says, oh, okay, I got it. This is what I can control. I'll just figure out how to do it by babbling, by trying things out. Oh, yeah. Ooh wow, that locomotes me. Cool. And so that's what you get with a really plastic brain. I think that was Mother Nature's big discovery is that she doesn't have to reinvent the brain every time, instead, all she has to do is tinker around with the peripheral devices and the brain will figure it out.
Nicholas Weiler:
Amazing. Yeah, just a general-purpose device for surviving in this crazy world.
David Eagleman:
Exactly.
Nicholas Weiler:
Well, thank you David Eagleman so much. It's been a fascinating conversation and a pleasure to have you on.
David Eagleman:
Great. So nice to be here.
Nicholas Weiler:
Thanks again to our guest, David Eagleman. We'll include links to his work and more information about synesthesia and sensory substitution in the show notes. If you're enjoying the show, please subscribe, share with your friends, and write us a review. It helps us grow as a show and bring more listeners to the frontiers of neuroscience. We'd also love to hear from you, leave us comments or give us a shout-out on social media at Stanford Brain. From Our Neurons to Yours is produced by Michael Osborne with production assistance from Morgan Honaker. Our logo is by Aimee Garza. I'm Nicholas Weiler. See you next time.
