A new precision neuroscience of language

We talk with neuro-linguist Cory Shain about our brains' ability to process language – a new Big Ideas in Neuroscience project to understand this quintessential human superpower
Nicholas Weiler
From Our Neurons to Yours Wu Tsai Neuro Podcast

Right now, as you're reading this sentence, something remarkable is happening in your brain. Light waves from your screen hit your eyes, transform into electrical signals, and take on meaning. You understand what you're reading. This is language — our human superpower.

But despite 150 years of intensive research, we still do not have a complete picture of how the brain actually accomplishes all of this. We don't even have a good answer to a seemingly simple question: Where in the brain does language happen? It turns out, the answer may be different in different people.

Today we'll hear from neuro-linguist Cory Shain, one of the leaders of a new Big Ideas in Neuroscience project here at Wu Tsai Neuro that is combining multiple brain recording techniques to build individualized maps of the language network—and use these insights to improve brain implants for people who've lost the ability to speak or write due to brain injury or illness.

Shain is an assistant professor of linguistics at Stanford Humanities and Sciences, and leads the “Precision Neuroscience of Language” project with Wu Tsai Neuro / HAI Faculty Scholar Laura Gwilliams, an assistant professor of psychology at Stanford Humanities and Sciences; Frank Willet, an assistant professor of neurosurgery at Stanford Medicine; and Jaimie Henderson, the John and Jene Blume - Robert and Ruth Halperin Professor and a professor of neurosurgery at Stanford Medicine.
 

View all episodes

SUBSCRIBE on Apple Podcasts, Spotify, Amazon Music and more.

Cory Shain smiles at the camera in front of a blurred backdrop of Stanford architecture
Cory Shain is an assistant professor of linguistics at Stanford Humanities and Sciences and a Wu Tsai Neuro affiliate. 

Learn More

Episode credits

This episode was produced by Michael Osborne at 14th Street Studios, with sound design by Mark Bell. Social media strategy is by Julia Diaz, and additional editing by Nathan Collins. Our logo is by Aimee Garza. The show is hosted by Nicholas Weiler at Stanford's Wu Tsai Neurosciences Institute and supported in part by the Knight Initiative for Brain Resilience. 

Share the show:

If you're enjoying our show, please take a moment to give us a review on your podcast app of choice and share this episode with your friends. That's how we grow as a show and bring the stories of the frontiers of neuroscience to a wider audience. 

Get in touch:

We want to hear from YOUR neurons! Email us if you'd be willing to help out with some listener research, and we'll be in touch with some follow-up questions.

Transcript

Nicholas Weiler: (00:11)

This is From Our Neurons to Yours, a podcast from the Wu Tsai Neurosciences Institute at Stanford University, bringing you to the frontiers of brain science. I'm your host, Nicholas Weiler.

(00:26)

Right now, as you're listening to me talk, something remarkable is happening in your brain. Sound waves from your speaker or your headphones are hitting your eardrums, getting transformed into electrical signals, and somehow almost instantly those signals take on meaning in your brain. You understand what I'm saying. You might even be formulating a response in your head. And if you are, you can reach us at neuronspodcast@stanford.edu. We'd love to hear from you.

(00:54)

Anyway, language is our human superpower. But despite 150 years of intensive research, we still do not have a complete picture of how the brain actually accomplishes all of this. We don't even have a good answer to a seemingly simple question. Where in the brain does language happen? This isn't just a theoretical puzzle, by the way. Imagine you're a physician caring for someone who's lost the ability to speak due to a brain injury or disease.

(01:24)

Today, we are developing the technology to record brain activity and decode it to restore speech. But to do that effectively, you need to know exactly where to place those recording devices in each individual's unique brain. A new Big Ideas in Neuroscience Project here at Wu Tsai Neuro aims to make progress on these questions by combining multiple brain recording techniques in the same people to understand how each of their brains processes language, going from whole brain language networks all the way down to the computations of individual neurons and circuits. The project is led by Wu Tsai neuro Faculty Scholar, Laura Gwilliams, brain computer interface developers, Frank Willett and Jaimie Henderson, and today's guest, neurolinguist Cory Shain. Let's get to my conversation with Cory.

(02:17)

I'd love to start with this question of what are we doing right now? As we're having this conversation, as you are hearing what I'm saying, and maybe formulating a response or waiting for me to stop talking so you can say your piece, what is our best understanding of what is actually going on in your brain right now?

Cory Shain: (02:42)

I love this question. And I think it's easiest to answer it first by thinking about the problem that the brain needs to solve in order to be able to converse, and then digging into how the brain might be kind of breaking those problems up. So if you think about what you're doing in a conversation, you can be in one of two roles. You can either be comprehending or you can be producing. So roughly listening or speaking, but using comprehension and production as a way to cover also sign languages, so we also sign.

(03:12)

And the two processes are kind of mirror images of each other. So as you Nick are talking to me, and I'm trying to comprehend what you're saying, I need to receive physical inputs from the world somehow. So we're using a spoken language. And so these are pressure waves that are hitting my eardrum. And then I need to kind of decompose those into their constituent parts that are meaningful for language as opposed to not meaningful. So for example, any background noise or particular idiosyncrasies about the way that you said a given word. So I need to kind of map what you're saying into more stable and abstract representations of your intended meaning.

(03:57)

And then that needs to kind of be conceptualized somehow. So I need to link up the message that you sent me, like the form of the message, to its contents. So what it is that you are expressing to me about the world or what type of information it is that you're asking from me. And as part of that, I need to make use of my knowledge of you and what types of knowledge we have in common versus what types of knowledge are kind of privileged to me. And I need to use that information in order to make judgments about what you would likely want to hear in response, like what types of responses would be useful. So for selecting a message to convey, and then choosing the form that that message ought to take in language.

(04:40)

And now I want to make my utterance, and so I have to go backwards. I have to figure out how to map this high level kind of conceptual abstraction that is the meaning, the thought that I want to recreate in your head via these linguistic structures that will allow you to decode that thought. So I have to do a little bit of mind reading to figure out how accurately you'll be able to recover what I want you to recover based on different ways I could package that information, and then I need to act. I need to coordinate my vocal motor tract in order to be able to produce the pressure waves that are going to hit your ear.

(05:19)

And our picture of how it works out in the brain, I'll just preview, is pretty incomplete. And that's in part why we want to study this more through the Big Ideas Project that we'll be talking about later. So on the comprehension side, we know that these pressure waves get kind of routed to your cortex through a series of transformations in the brainstem and the midbrain and then eventually arrive at a particular spot in the cortex, which is like this outer shell of neurons in your brain that do a lot of the thinking that's kind of like, I like to think, I don't know, for your listeners who have seen brains, have seen images of brains before, I tend to think of the brain via this silly image of a frog that's crouched ready to jump. And so the primary auditory cortex is like the inner thigh of the frog, Just tucked in there.

Nicholas Weiler: (06:12)

That is a very vivid image. I have never thought about the brain as a frog before, but I like that. Okay. We're an inner thigh of the frog.

Cory Shain: (06:19)

Right. So that's where the sensory signals are coming in. And that part of the cortex doesn't really care so much about the contents of those signals. It's mostly interested in their acoustic properties, how high pitched they are, and other aspects of the kind of auditory signal itself, irrespective of whether it's speech or car honking or any other sound that I might counter in the environment. But then nearby those, moving further out, I guess on the thigh of the frog, there are some regions that are still pretty auditory but become more and more selective for speech specific as opposed to other types of sound properties. So we have-

Nicholas Weiler: (06:57)

And those are maybe like filtering the sounds that are coming in to detect specific speech sounds might be one way to think of it.

Cory Shain: (07:05)

Yes. Yeah, exactly. It's a detector of sorts. Of all the different types of sounds I could be hearing, I'm hearing one that sounds a lot like a P, right? And so I'm going to encode my response to that hypothesis about the incoming speech signal. And so we start getting areas that start caring more about whether or not the sound that we're hearing is speech, and if so, what it's kind of like meaning relevant properties might be. And then something happens.

Nicholas Weiler: (07:36)

Then we wave our hands.

Cory Shain: (07:37)

Yeah. Something happens, and then we get a mental picture or some sort of mental format for the meaning. And then vice versa, once we have that conceptualization, and we want to utter it in language, then that needs to be routed through motor cortices that are going to control my larynx, so my vocal chords and my lips and my tongue, and the other kind of parts of my body that are needed in order to create this delicate dance of speaking. And so what I just described is what we can say on the basis of consensus. And then everything that I just skipped over, all the interesting stuff, is where we enter into a territory that it is not uncharted, it's heavily charted, but there's still quite a lot of disagreement as to what the contents are.

Nicholas Weiler: (08:24)

Yeah. I mean, this is obviously a very deeply important aspect of neuroscience to me. I mean, we named our show From Our Neurons to Yours, which is really a reference to this fact that somehow we are able to transmit meaning from some representation of meaning in my neural networks through this medium of speech or sign to through your ears and into your brain where it forms a version of that meaning in your brain.

(08:55)

I like to say like, human beings do have telepathy. We just have to make these incantations with our mouths in order to transmit thoughts to each other, which is still pretty good. And the thing that's so, so remarkable and that I'd love to interrogate more as we go through this conversation is that it doesn't feel like a sequence of steps. It doesn't feel like I am decoding what you are saying, creating meaning, thinking about what I'm going to say, and then translating that into speech.

(09:25)

It is so fast, it is so instinctive, and it seems like it happens all at once. When you get into a conversation, it becomes more of a mind meld where you're like, you both are having these models of some kind of meaning, whatever the thing is you're talking about, and you're pushing each other's models back and forth in real time through your speech.

(09:48)

And so what you're describing, we have all these regions in our brain, and we're going to talk about some work that you have coming out soon that is trying to get at like, what is this network? What is this network in the brain that's doing all this? So you and Laura Gwilliams, who we've had on the show before, Jaimie Henderson, also a friend of the show and Frank Willett are involved in this study of the neuroscience of language. And together you now have this Big Ideas in Neuroscience Project called the Precision Neuroscience of Language Project. So we'll get back to this more, I think towards the end of our conversation, but just very briefly, what is the big idea of this project?

Cory Shain: (10:32)

Yeah. So I should say that the big idea is informed by our different entry points into language neuroscience across the three labs and four professors that you just mentioned. So Laura and I are each running labs touching on different aspects of language neuroscience. I primarily come at language neuro from a functional neuroimaging perspective. So like fMRI, which is a methodology that gives us a lot of resolution on where things are happening in the brain. But it's very slow. And language, as you just said, is this really fast domain where things are changing rapidly in time at a scale that we can't easily detect in MRI.

(11:12)

And so a lot of language neuroscientists, including Laura and her lab, are focused instead on the time domain. So what is the sequence of transformations and computations, millisecond by millisecond, that's happening that enable us to go seemingly at warp speed to translate between two completely superficially incommensurable formats of the sounds hitting my ear, and then the subjective experience of having a thought transferred to me by someone else.

(11:41)

And then Frank and Jaimie are interested in translating these insights into helping people with different types of speech impairments, currently focused primarily on paralysis of people who have lost partially or completely the ability to articulate language, but nonetheless have language and thought otherwise preserved.

(12:02)

And so the big idea comes in unifying all of these different entry points into a common goal of developing mechanistic theory that would be precise enough to allow us to make predictions about which types of representations and computations are happening in language where, with enough precision that we could know what would be damaged. So how the brain will respond to particular linguistic stimuli, and given a patient's clinical needs, which parts of the brain we ought to try to read out particular types of information from.

(12:45)

And so in this respect, the goal of the project is not different from the goal of language neuroscience broadly for 150 years. I mean, there's nothing unique about that goal. It's pretty aspirational. But I think that our contribution, our big idea, is to try to do this in detail at the level of individual brains across different modalities that all have various strengths and weaknesses in terms of the types of insights that they can offer us about brain function. So to an extent that hasn't been done before, we hope to connect up within an individual kind of large scale whole brain mapping of functions all the way down to tiny even cell level mapping of kind of like the microcircuitry and its function in order to help us understand the spatial temporal code for language in the level of detail that hasn't been possible before.

Nicholas Weiler: (13:39)

Fantastic. Well, you said each of you has a different entry point into the neuroscience of language. So let's take those one at a time and sort of see what we have learned so far at these different levels, and then at the end we can bring this back together and say, okay, now what will we learn when we are able to, as you say, bring these different levels together in an individual brain to try to understand the connections between them?

(14:34)

You have a preprint that's up now on bioRxiv, understanding this sort of brain-wide language network and how it differs between people. Could you describe at a high level what was new, what was different about what you did here that is giving us a new understanding of this network?

Cory Shain: (14:54)

Sure. So I'll back up pretty far actually, to 2009 when a-

Nicholas Weiler: (14:59)

Time machine.

Cory Shain: (15:00)

... postdoc at MIT named Ev Fedorenko wrote a paper with her advisor arguing that the reason why our picture, as I mentioned before, of how the brain organizes language is so blurry is that we haven't been kind of looking at the right units of analysis. So one, the brain's a very high dimensional piece of machinery.

Nicholas Weiler: (15:24)

By which we mean complex and lots of different moving parts.

Cory Shain: (15:27)

Yeah, exactly. There's so many different ways in which we might kind of, for example, align different parts of two different brains in order to understand functioning. So we need to choose some level of analysis at which we are going to try to make claims. And that's often an assumption of any empirical investigation of brain function.

(15:48)

And so this all goes all the way back to the 1800s when people were first starting to discover some important asymmetries in how important different parts of the brain were for say expressive language. So famously like with Paul Broca discovering people who could comprehend language but had a really hard time articulating it. And then reliably finding that those regions kind of recurred in systematic parts of the brain.

(16:18)

And so neuroimaging of language has followed a similar trajectory where you'll try to study some functions. So for example, having a task that will require more or less effort and some hypothesized dimension of interest. So for example, more versus less syntactically complicated language. If you're interested in studying how the brain kind of puts together the different words in order to form structured representation of their meaning.

Nicholas Weiler: (16:43)

So can you see differences in brain activity depending on how complicated the sentences are that you're trying to understand?

Cory Shain: (16:48)

Right. Yeah. And so you'll do this at the level of an individual. So different people come in, they'll do your task, you'll have a hard versus an easy condition or like a complex versus a simple condition, or something. And then you'll be interested in the difference. So which brain areas are more responsive during the complex condition. And so that gives you a map of each individual. But you don't want to make claims about Cory's brain, you want to make claims about the brain with a capital T, like the human brain. And so then how do we make those claims at the level of the population? Well, the standard way to do this is to just assume that the same regions, however region is defined, do the same things in everybody's brains, and then you can just take an average.

Nicholas Weiler: (17:29)

So you take a whole bunch of brains, average the activity during the same task together, and say the average is the thing that's in common.

Cory Shain: (17:35)

Yeah. At each spot, at each location in the brain. And then you're like, okay, so this spot does this and this spot does this. And what Ev and Nancy were pointing out was that you can do that mathematically and you can get an answer, and the answer may reveal things about our species broadly, but it's a possibility that the average that you're getting may not look anything like the individuals that went into it. And then they made a rather explicit argument that in fact it doesn't. And that when you take these averages, you can get distortions, both like conflating functions that are distinct, as well as failing to find functions that are really there but just variably located between people.

(18:19)

And so this kind of triggered a move that hasn't yet fully, I think, taken root within language neuroscience to try to take some of the individual variation into account, which is a little different than what you typically think of with individual variability in science where you're trying to understand what causes people to differ in some traits.

(18:38)

Here it's more how do we take into account the kind of like unique functional neuroanatomy of each person in our study in a way that will allow us to make apples to apples comparisons across people so that we can kind of figure out the right level of description for how the brain broadly is organized, allowing for everyone's instantiation of that organization to differ at least to some extent.

Nicholas Weiler: (19:03)

So right. So if my brain and your brain are, we both have places that are probably parsing the syntax, the word order, the grammatical components of sentences, but if they're a few millimeters off from one another, you need to be able to see what the network is in my brain and in your brain. And if you average them, you're not going to get the thing that's in common, which is the structure, not the location. Maybe that's one way to think about it. So in your study, you basically tried to create individualized language maps within each of, what was it, 1,000, 2,000 subjects?

Cory Shain: (19:43)

Yeah, it was almost 1,200 subjects. And some of them were scanned multiple times. So it was almost 2,000 different scanning sessions, which we treated separately.

Nicholas Weiler: (19:51)

And so how different did you find the networks? Firstly, how did you define the network? We don't have to get into too much detail about that, but when we say a network, what are we talking about, and two, how different were they from each other?

Cory Shain: (20:05)

Yeah. So the question of what the definition of a network is a good one, and it varies between people. So this paper is kind of like one of a long line of papers from my group and similar groups arguing for the existence of a network in the brain that is reliably and selectively engaged in and critical for language processing, so both production and comprehension in people. And so this is a controversial claim already, the degree to which there is kind of like language selective tissue in the brain versus tissue that happens to have a different kind of selectivity profile that simply gets recruited in service of language.

Nicholas Weiler: (20:41)

I see. So other people would argue that we have meaning, we have listening, we have various general purpose brain networks that we recruit for language, but they're not unique to language.

Cory Shain: (20:55)

Right. So just to give one simple example, there's a kind of hidden structure behind most of the sentences that we hear and produce that we're probably not aware of. So to us, it feels like words are just come out one after the other, like beads on a string. But for example, if I say that Cory talked to Nick this morning, then I can form a question whose answer is Nick from that statement. So who did Cory talk to this morning? But I can't form a question whose answer is morning as, what did Cory talk to Nick this? And so that difference has like long engaged linguists and has led to a consensus view among linguists that there is more going on beneath the surface than just these like sequencing one after-

Nicholas Weiler: (21:45)

My favorite example of this is that it sounds normal for me to say the old gray house, but somewhat deranged for me to say the gray old house.

Cory Shain: (21:53)

That kind of stuff too. And then ambiguities are another fun one. So a old newspaper headline was, sisters reunited after 17 years at checkout counter.

Nicholas Weiler: (22:04)

Right.

Cory Shain: (22:05)

And so the listener may have noticed that there's kind of two possible meanings there, and the wrong one is typically the first one that you get, that they've been at this checkout counter for 17 years, when it's actually reunited after 17 years. And so that's an attachment ambiguity. You have this deeper structure, and the after 17 years can attach in two different places and that leads to two different meanings. So there's many different reasons to think that there's kind of like this deeper structure behind our utterances.

(22:30)

And there are many other domains of cognition that have similar substructures. So for example, if I'm trying to figure out how to make a peanut butter sandwich, I kind of have to decompose it into a sequence of actions, but that are organized into different tasks and subtasks and further subtasks and so on. So you might think that, rather than having a region of the brain that does parsing of abstract linguistic structure, you might have a region of the brain that does parsing of just temporal structure in general and it happens to get applied to language as well as task comprehension or even like hierarchical structure in music, what have you. So it would be a domain general system that gets reliably implicated in language simply because language has properties needed to engage that system.

(23:15)

So this is all a foil. I think that there actually is a rather language selective brain network, and this study is part of the evidence for that. And so historically this has been defined using tasks at the individual level. So you would have some tasks that you think is like, I'm going to call this language. So one famous such task is to compare sentences versus matched length lists of non-words. So people read them one word at a time, only in the sentences condition it's meaningful and in the non-words list condition it's not, but they're otherwise very perceptually similar conditions. And so the regions that respond more to the sentences are the ones that care about those meaning differences.

(23:54)

And it turns out that you get a distributed kind of primarily left lateralized network that looks a lot like what people would typically associate with language and the temporal lobe and in the inferior frontal lobe, but also other places when you have people do that task. So right now I've only defined it in terms of one task. So it's not really a network. This is just simply the set of regions that responds during a task. But it turns out historically that it's been now shown that there's a bunch of different tasks that are all qualitatively different from each other that will give me basically the same set of regions.

(24:22)

So I'll get those regions if I instead have people listen to intact speech versus the same speech that's been low passed filtered. So it just sounds like... So it has similar perceptual structure, but it lacks meaning. And so that's when you start getting into the notion of a network. The hypothesis is that there's some sort of intrinsic structure in the brain that works together in support of some set of functions, and that we can kind of convergently identify that structure through many different pathways. And so our paper is another one of those pathways that kind of takes the reliance on tasks out altogether, and just looks at how the regions of the brain co-vary with each other over time in arbitrary tasks.

Nicholas Weiler: (25:06)

That's really interesting. Yeah, that was what I was going to ask you about. So one of the things that was so cool in this study is that you are seeing that this is a network that is active, as you were saying, this isn't a network that's active not only when doing languagey things like comprehending or speaking or reading or whatever, but these brain networks are talking to each other. You can see their activity going up and down together or following each other in activity to create a unified network that is both related to language and also active when you're doing other things. So I'd love to get, what is the implication of that? That you're identifying this network, which some people have said there isn't really a language network, but you're saying, no, there is. This is a network. It's always involved in language, and it's also active when we're doing other things. What does that mean to you?

Cory Shain: (25:57)

It means that what we've called a language network on the basis of task data is really there in some kind of deep sense. That this is an intrinsic property of how the adult brain at least is organized in most people, and then that creates a kind of theory external object of study in the brain. So we no longer have to, like we can characterize it on the basis of what we found previously about its function, say for example, that it seems to care a lot about language and not much other stuff. But as future experiments might kind of like revise our interpretation of its function or nuance it, that doesn't fundamentally alter the finding that this is an important thing that we need to explain about and understand about adult brain function.

Nicholas Weiler: (26:42)

One of the things that you showed is that this network is distinct in different people. That, as we were saying before, it wouldn't be effective to average this across your 1,200 subjects because it's different in space and maybe it's different in strength between different regions.

(26:58)

I guess this comes back to this question of how do we understand the precision neuroscience of language? How is my brain doing language differently from your brain, or at least where is my brain doing things? Where is your brain doing things? What is your interpretation of this? I could think of a few possible interpretations of these differences in the language network, which is, one, it's the same network, it just happens to form in slightly different places in different brains. Or two, maybe these differences in the network reflect the differences we all know in the way that we each use language. Some people are great talkers, some people are great readers. I listen to a lot of podcasts with famous writers and they're not always as good at speaking as they are at writing. So those are sort of the two immediate things that came to mind. I don't know if there's anything in this paper that can answer that question, but what's your take on it?

Cory Shain: (27:53)

I love this question. And no, there's nothing in that paper, but I think that we are starting to get some traction on the answer, which is I think a bit of both. I think there's a deep sense in which this network, despite its slightly different topography in different brains, is the same. So in kind of unpublished stuff that we are currently working on, we're finding that you can actually get it by cross correlating between brains.

(28:20)

So if both of us listen to a story, and I find the language network in myself, for example, based on our functional connectivity approach in the preprint, and then I grab a region of that network and look at its correlation and activity across the different regions of your brain, then it's going to pick out roughly the same network that I would have picked out if I localized the language network in you.

(28:44)

In other words, I can actually use the response signatures of regions of these networks to figure out the correspondences between people in ways that winds up aligning closely with what we're finding at the individual level. So that's an important piece of evidence for sameness. And beyond this, I like the work of Saima Malik-Moraleda, who has shown that, across something like three dozen languages, when you give people a roughly comparable language localizer task, you get qualitatively the same network in the sense that the variability between languages is not greater than the variability between individuals within the same language.

Nicholas Weiler: (29:18)

So that's like same network, but different implementation in people.

Cory Shain: (29:22)

Yeah. So that's evidence that we're justified in calling these the language network in a deep sense. But there are differences. So again, different work by Saima Malik-Moraleda looked at what they called hyperpolyglots. So these are people who self-reported at least speaking very many languages. And they found that the language networks of these people were systematically smaller than the language networks of monolinguals, so people who only speak one language.

(29:49)

This is a puzzling finding. You might expect that a brain network that has more responsibilities would have to be bigger. It may be evidence of kind of like efficiency pressures. But within the languages that they speak, the responses to those different languages are highly overlapping. There's some influence of our experience it seems on the structure of the network, but there's still, I think, a ton to be discovered about that question.

Nicholas Weiler: (30:15)

And that brings me to another thing I wanted to ask about is we can see that there are individual differences in these brain-wide language networks from your data, but this is sort of where you come up against some of the limitations of using fMRI, which can set some slow variations in metabolism. It's measuring blood flow. So how much energy are different parts of the brain using over seconds, I think if I have that right. But the actual processes are clearly happening much faster than that within these regions.

(30:50)

And that brings us to this distinction between the networks, the sort of brain-wide communication between regions that we've been talking about, and the circuits, which I think of as like these are how the neurons in those regions are actually talking to each other and representing information that is getting passed around in the network. And this is where work like Laura Gwilliams' work comes in where she's looking much more at like, what are these neurons within these circuits doing and how might that do some of the work that we know is going on with language?

(31:23)

And that could inform our understanding of what are all these different parts doing, and what does it mean if like this part is stronger in me and that part is stronger in you? Because we don't yet understand exactly what those parts are doing. Maybe you can tell us a little bit about what we currently understand from Laura's work and others about the kinds of things that are going on in these actual brain circuits within the network that you're talking about. Because what I'm interested in is what is the gap. You can show the network at this level. They can look at the cells at this other level with different techniques. What are they finding out and what is the gap between that and what you're seeing?

Cory Shain: (32:07)

Yeah. So I think there's two really nice pieces from Laura's group recently that illustrate I guess the other entry point into this Big Ideas grant, looking at two different spatial scales. So I think the one you're talking about with respect to circuits, which I think you mean like it's like a handful of neurons, for example, or neurons are ranged within a single cortical column. So what we would consider from an fMRI perspective to be like one place.

Nicholas Weiler: (32:33)

A pixel.

Cory Shain: (32:34)

Right. But that actually have a very rich structure on their own. So I'll start there. So Laura and colleagues were some of the first to use these really high resolution neuropixels probes, which I believe you've talked about previously on the show that allow you to record from the same cortical column at many different sites simultaneously.

Nicholas Weiler: (32:55)

It's basically a long electrode with lots of different sites so that it can pick up signals from a bunch of different neurons.

Cory Shain: (33:02)

Right. And so there you're getting spiking. So you're getting as close as possible to the actual temporal signal generated by neural activity. And so that takes away the temporal blurring that you just mentioned, which is a serious problem in MRI, and allows you to kind of time lock the neural responses you're seeing to particular properties of the stimulus.

(33:24)

And they found really many interesting things, but just to give like a sampling of them, one is that even over the course of the depth of a single cortical column, there are rich differences in functional tuning with some cells that seem to prefer different types of speech properties than others. And so this is I think very interesting to us and one of the major motivators for our project because it suggests that what we are looking at when we look, for example, an an fMRI image is something like a Monet painting where like we've stepped far enough back to see this kind of a big picture structure, but when we zoom in, things look quite different.

(34:05)

A voxel in MRI that appears to have a kind of uniform functional tuning may in fact be composed of pieces that have very different tuning. And so from a science perspective, we're really interested in what the space of possibilities are for that relationship. How can circuit function generate potentially quite different looking network function, and how does knowledge of network function constrain what's possible at the circuit level?

Nicholas Weiler: (34:31)

Right. And some of the things, we've talked with Laura before on the show about things like neurons being able to hold on to particular speech sounds for several seconds, much longer than the speech sound is actually heard. And I think the same for words, which could allow you to think about things like, okay, neurons are holding onto the information so they can combine it with what comes next and you can build like, this is what the word is, this is what the sentence is. But still that's a far cry from understanding how do we get syntax, and certainly from understanding where does meaning come from? Where are these representations that we are trying to communicate with one another?

(35:10)

One of the things I thought that was very interesting in your preprint was you said that this kind of suggests that the network is many different parts of the network are doing lots of different things, that is that there's not one brain region that is doing just this computation and one brain region that is doing just that computation. But that it's this big network that's processing a lot of things in parallel, which feels like a big, that's a shift that we've seen a lot in neuroscience over the past five, 10 years, in part due to these techniques like the neuropixels probes, because you can actually record in many places at a time and see, wow, all these places are involved at the same time. It's not quite as sequential and logical as we might imagine.

(35:52)

So I think the place I want to take this, and we could go much deeper on any one of these pieces, is to sort of the translational side, which is we've talked a lot about the big picture science, the understanding the language network, understanding what makes different people's language networks distinct, what allows us to convey meaning to each other.

(36:24)

But also there's this aspect that folks like Jaimie Henderson and Frank Willett are deeply involved in actually putting this to use and doing brain recordings in people who have lost the ability to speak due to paralysis of one kind or another, where they can use computer algorithms to essentially decode what the brain is trying to say. But for now, a lot of that has been restricted to the motor areas of the brain, the parts of the brain that normally actually control the speech muscles themselves.

(37:01)

So it's very similar to other work like building a robotic arm that's decoding what the brain is, allows the brain to move this robotic arm. So I think the question is, how do we get to a place where we can do a similar type of decoding in these areas of the brain that are responsible for the higher levels of speech, for encoding the meaning that you're trying to express? Is that one of the directions that this work could take us?

Cory Shain: (37:32)

Yes. That's one of our main goals is to answer that question. And I would say that the answer is currently unknown. There is an element of interest, I think, within our project on the motor cortex itself. So even before we step out to higher level language, you need to know where in motor cortex you ought to place your probe.

(37:51)

So we find that, or I should say Jaimie and Frank find across their multi-site BrainGate collaboration, that you get the best performance for these decoders for attempted speech in regions of motor cortex, but that the performance varies a lot between people, presumably in part as a function of how speech selective the implanted sites are.

(38:16)

So there appears to be a distinction in the motor cortex between regions that control just the muscles, like the orofacial movements, and regions that appear to be kind of relays between higher level language areas. And those orofacial motor areas that are kind of like speech motor areas. They seem to contain some sort of higher-level representation of the speech sounds that need to be produced. And it's really those regions that we think are the ones that you want to hit if you're going to try to decode attempted speech from motor cortex.

Nicholas Weiler: (38:47)

If there are individual differences between people, where do you stick your electrodes?

Cory Shain: (38:52)

Right. Yeah. So it goes all the way down. But certainly the problem becomes very pronounced when we start wanting to work with populations for whom that itself, so the speech motor planning part, may no longer be intact. So they may still be able to think, they may still be able to formulate the notion of a planned message without even being able to program the motor movements that we would otherwise want to read out.

(39:15)

And in those cases, we need to know where else to look. And the language network is a very good candidate, I think, or at least an interesting candidate. And that's one of the goals is to explore that in more depth. And so this requires a synthesis between the neuroimaging side and the translation side. So before surgery, we are hoping to have people do lots of different both fMRIs studies to kind of map them in space as well as faster time resolution methods that Laura is an expert in, like magnetoencephalography, which can give us a notion of the computations and representations.

(39:54)

And so both of those pieces of information together we think will be highly informative about optimal targeting for the sensors that will lead to the best results for patients who need it across a range of different types of disorders rather than the currently fairly narrow types of paralysis that Frank and Jaimie have seen the most success in, like ALS.

Nicholas Weiler: (40:13)

Yeah. And we're really excited. We have this new magnetoencephalography device that just arrived and got installed at Wu Tsai Neuro, which is going to enable some of these studies. And it's complementary to your fMRI studies. We now have these magnetic detectors of brain activity that's non-invasive. Plus the EEG, the electrical sensors of brain activity.

(40:41)

And then there are also people like the folks who Frank and Jaimie are working with who actually have electrodes inside their skulls that can help get an even more refined idea of what are exactly the cells that are firing, how can we connect the actual firing of neurons to the signals, the magnetic and electrical signals that Laura and others can record, and then these bigger picture metabolic fMRI brain imaging signals that you're an expert in.

(41:14)

So just to sort of take us out, I mean, this is a very new project. This just got launched a few months ago, and you're all coming together to do this work. Can you give us a flavor before we close of how are you actually going to combine these different modalities of studying the brain, and what are just one or two things you hope will come out of that?

Cory Shain: (41:39)

Yeah. So the procedure that we might envision, let's say for example, for a patient with paralysis who's eventually going to be implanted with a BCI, so a brain computer interface sensor, that will help decode speech. The goal would be to enrich the data collection protocol before their surgery with lots of scanning from fMRI and MEG simultaneously. And so fMRI will give us across a range of different tasks as well as rest. We'll be able to do a lot of different analyses that kind of map their brain in detail spatially. And then from MEG, which gives us some traction on space too, better than many other high temporal resolution methods. And we'll be able to get a sense of what types of representations appear to be involved in processing language for that individual.

(42:27)

So already there we have the ability to link up two different types of analyses with complimentary strengths and weaknesses. And then likewise, we can use the fMRI data to give us a sense of which functional networks in the brain might be the sources, like the neural origins of the high temporal resolution computational measures that we're getting from MEG. So if we see a signature of some representation or some process happening at a given moment in time, which brain system did that likely originate from.

(42:59)

So already we can kind of bring these two modalities into conversation. And then both of those together will, hopefully, we hope to allow us to triangulate on ideal sensor placements in surgery. Which then we can, because these participants are awesome and consent to do silly sounding tasks in their homes after their surgery, we can then directly test different hypotheses about the function at these broader scales against a function that we can directly measure at the circuit level from the implanted arrays.

Nicholas Weiler: (43:33)

That's fascinating. So you have, because there are these patients who we’re working on finding ways to help, you have the ability to say like, "Okay, now that you've got these electrodes in your brain, we're really trying to understand what is the computation? Is this doing syntax? Is this doing meaning? What is this?"

(43:49)

Well, this is so exciting. I know that we'll have a whole lot more to discuss about this project as it gets underway and as you start getting experimental data from this. But Cory, thank you so much for coming on the show and giving us this whirlwind tour of what we know currently about the neuroscience of language and where we're heading with this Precision Neuroscience of Language Big Ideas Project.

Cory Shain: (44:14)

Thanks so much, Nick. This was really fun to talk to you.

Nicholas Weiler: (44:17)

Thanks again so much to our guest, Corey Shain. He's an assistant professor of linguistics at Stanford and one of the leaders of the new Precision Neuroscience of Language Initiative here at Wu Tsai Neuro. To read more about his work and the initiative, check out the links in the show notes. If you enjoyed this episode, please be sure to subscribe for more conversations from the frontiers of brain science.

(44:38)

We also love hearing from listeners. If you have thoughts about the show or questions about the brain you'd like to hear us discuss in a future episode, send us an email at neuronspodcast@stanford.edu, or leave us a comment on your favorite podcast platform. While you're there, please give us a rating and share the show with your friends. Might seem like a small thing, but it is tremendously valuable for us to be able to bring more listeners to the frontiers of neuroscience.

(45:05)

Coming up on From Our Neurons to Yours.

(45:31)

From Our Neurons to Yours is produced by Michael Osborne at 14th Street Studios, with sound designed by Mark Bell. Our social media strategy is by Julia Diaz, additional editing by Nathan Collins. Our logo was designed by Aimee Garza. I'm Nicholas Weiler. Until next time.