Predicting the brain: the Stanford Digital Brain Project

Two of the projects' leaders, Dan Yamins and Russ Poldrack, explain the ideas behind the effort and the experimental campaign that will bring them to fruition
Nicholas Weiler
From Our Neurons to Yours Wu Tsai Neuro Podcast

This week, Wu Tsai Neuro launched the Stanford Digital Brain Project, an ambitious effort to build predictive models of the human brain.

So today on the show, we’re bringing you two conversations to get you up to speed on what this new initiative is and why it matters.

First we’re going to revisit a conversation from last year with Wu Tsai Neuro Faculty Scholar Dan Yamins about the vision for building digital models of the brain. We talked about how such models would let neuroscientists test hypotheses about brain function or develop personalized interventions for brain diseases in a virtual environment far faster than can be done in the lab.

Then, in the second half of the episode, we'll bring you a conversation with cognitive neuroscientist Russ Poldrack, one of the leaders of the new initiative, about what it's actually going to take to build this thing. We talk about the experiments now getting underway to capture the brain data that'll be needed to train these models, and why collecting this kind of data is already a remarkable scientific undertaking on its own, well before anyone actually builds a model out of it.

View all episodes

SUBSCRIBE on Apple Podcasts, Spotify, Amazon Music and more.

Dan Yamins and Russ Poldrack
Wu Tsai Neuro Faculty Scholar Dan Yamins, an associate professor of psychology and computer science, and Russ Poldrack, the Albert Ray Lang Professor of Psychology.

Learn More

Episode credits

This episode was produced by Michael Osborne at 14th Street Studios, with sound design by Mark Bell. Social media strategy is by Julia Diaz, and additional editing by Nathan Collins. Our logo is by Aimee Garza. The show is hosted by Nicholas Weiler at Stanford's Wu Tsai Neurosciences Institute and supported in part by the Knight Initiative for Brain Resilience. 

Share the show:

If you're enjoying our show, please take a moment to give us a review on your podcast app of choice and share this episode with your friends. That's how we grow as a show and bring the stories of the frontiers of neuroscience to a wider audience. 

Get in touch:

We want to hear from YOUR neurons! Email us if you'd be willing to help out with some listener research, and we'll be in touch with some follow-up questions.

Transcript

Nicholas Weiler (00:11):

Welcome back to From Our Neurons to Yours, the podcast from the Wu Tsai Neurosciences Institute at Stanford University, where we bring you to the frontiers of brain science. I'm your host, Nicholas Weiler.

(00:29):

Recently on the show, we've been bringing you a lot of stories about the frontiers of psychiatry. We've heard how conditions like depression have historically been diagnosed and treated mostly based on outward symptoms, not actually based on a deep understanding of what's happening inside our brains. But neuroscientists are now starting to understand more specifics about the actual brain networks that are involved in these disorders. And we have new tools, things like transcranial magnetic stimulation that might be able to shift those networks back towards health.

(00:59):

The problem is that, as we may have mentioned once or twice, the brain is complicated and it's still hard to know exactly which circuits in this complex system to target or how to target them well in a particular person. One way researchers are trying to close that gap is by building a digital model of the whole system, something detailed enough to represent how different brain networks interact so we can understand what's going on in a disorder and predict what a given treatment would actually do.

(01:28):

About a year ago, we talked with Stanford's Dan Yamins about this idea, about how you might build a foundation model of a human brain trained on brain data, the way large language models are trained on text. Since then, Dan and a group of colleagues at Stanford have moved from vision to action. They've launched a new initiative called the Stanford Digital Brain Project here at the Wu Tsai Neurosciences Institute, which is going to build just such a model.

(01:53):

So today we decided to bring you two conversations. First is our interview with Dan from last year where he laid out the big idea for us, what a brain foundation model is and why it matters. Then in the second half of the episode, we'll bring you a conversation with cognitive neuroscientist Russ Poldrack, one of the leaders of the new initiative. In the second conversation, we get into what it's actually going to take to build this thing, the experiments now getting underway to capture the kind of brain data that will be needed to train these models and why collecting this data is already a remarkable scientific undertaking on its own, well before anyone actually builds a model out of it.

(02:32):

To set up last year's conversation with Dan, we had just had a conversation on the show with Laura Gwilliams about large language models like ChatGPT and comparing them to our brains. So I started by asking Dan whether it is fair to compare what he envisions building to these large language models we've all gotten used to.

Dan Yamins (02:55):

Yeah, I think that the large language model analogy is pretty good. The idea is to capture brain data, not the data like the memories of any one individual or anything like that, but more like the processes of information processing, decision-making and brain circuits more generally from the brain and capture those in a model, kind of like the models that are underlying large language models, but now trained to make predictions about and to mimic some of the processes of the human brain.

Nicholas Weiler (03:30):

And so is the idea there that sort of like large language models are trained on a whole lot of text and then they can generalize from basically any input to a reasonable output, this would be trained on a whole lot of brain data and it could similarly generalize, it can give you an insight and predict what the brain would do in any given situation?

Dan Yamins (03:53):

I think that's the ideal. Whether or not it will actually be able to do that remains to be seen. Large language models are definitely not perfect in their predictions, and so I think we're likely to be imperfect here as well. But the goal is to take a first step at trying to capture patterns of brain processing in a neural network model. Again, without it being like any individual person or something like that, which would be quite beyond our capacities at this point, but just more basically trying to capture some aspects of how the brain processes vision input, how it processes audio input, how it sequences the way that you move your limbs to express your decisions about what to do in response to different things you might see or hear, or achieve certain goals.

(04:41):

Those algorithms are kind of written in the brain right now. The brain implements processes, algorithms of information processing and motor decision-making and memory and detention and all sorts of things like that. The brain implements that. We want to make an active copy of those algorithms. And it won't be a perfect copy, but it will be, as best as we can, a kind of record of and a runnable predictive model for the algorithms that are implemented in the brain effectively. That's the goal.

Nicholas Weiler (05:11):

I love that idea of, "Is this a way that we can capture what is the algorithm or the algorithms that the brain is running?" And I'd love to delve into the details a little bit more with you. But first actually, so why at this moment in neuroscience, what would people do with a model like this?

Dan Yamins (05:26):

Well, so the key reason why we want to do this is because unlike a kind of direct AI goal, what we are interested in doing is helping work with needs of humans. Humans have real brains and those brains are often functioning really well, but sometimes there are things we'd like to fix about them effectively. So in the form of some neuropsychiatric conditions, in the form of just understanding how the brain functions so we can help keep it healthy, the core goal is to take the model of this kind and help us basically do biomedical outcomes. We want to be able to use the model, the digital brain as a neuro biomedical discovery and diagnostic design platform.

Nicholas Weiler (06:10):

There's this intersection in neuroscience, it seems to me from all the people we've been talking with, where we're getting a better sense of how some of the circuits in the brain work or at least which circuits seem to be connected to certain disorders or functions of the brain. We're also developing better and better tools for stimulating or quieting brain circuits, essentially sort of shifting those circuits into healthier function. And so now it seems like this AI component that you're talking about, these foundational models, maybe those give us the link here where we can say, "Well, what would happen if we did this particular type of stimulation? How do we expect the brain will respond to that? How is that going to affect how the brain is processing sensory input or affecting behavior?"

Dan Yamins (06:57):

Yeah, exactly. So if you have a digital brain, it won't have a personality or anything like that, but what it will be able to do is you can show it images or you can make it listen to sounds, you can have it issue motor commands, do the brain equivalent of move an arm. You can have it analyze how it would remember something or what it would pay attention to. And by virtue of that, you can sort of be like, "Okay, well, let's say I would like to ask, 'What are the predicted responses of the brain to a set of diagnostic stimuli for say ADHD or for epilepsy or for depression?'" You could use the model to make those predictions without having to actually do anything with an individual real person, so that basically you can get lots of tests quickly at low cost and be able to analyze real data from individuals in a way that is much more effective.

(07:50):

And similarly, if you wanted to model the effect of a psychiatric drug, you could ask, "Okay, look, we can measure how psychiatric drugs operate," and we can ask, "What does that mean in terms of the way that the brain processes of information and make predictions about, okay, ex post facto, if we had used this drug instead of that drug, or if we had used no drug at all?" It could allow us to basically do simulations of brain outcomes without having to actually take up people's time and effort and energy to do those things in the lab and also quite disruptively. And so the brain model can help us sort of automate those processes in a low cost and flexible way.

Nicholas Weiler (08:28):

Right. It reminds me of something Laura Gwilliams was telling us about the idea that with the language models, you can actually go into the computational neural network architecture that's underlying the model and deactivate some neurons, digital neurons and see the effect, do these digital lesioning experiments. And so I imagine that you could have something similar here where you're like, "Okay, well, we've now basically downloaded the algorithm," as you put it, "or reverse engineered maybe the algorithm into this digital model. And now we have a system that we can use as a model," again, like Laura was putting it, a model organism. It's a digital model organism that you can go in and have so much more control, so much more ability to modify whatever you want, see how that affects things, whether that's understanding what the circuits are normally doing or testing out potential therapies.

Dan Yamins (09:21):

Exactly. Yeah. I mean, the brain's like a network, right? So when you put information through, it sets off a cascade of responses through different parts of the system. And also at the output, if you sort of follow that back up the chain, you can see where things come from. Basically, the digital brain model gives you the ability to simulate those pathways and learn a lot from being able to do so. For example, I don't know if people are familiar with TMS technology, transcranial magnetic stimulation. That's a technology that people use right now to treat depression in drug-resistant cases. That's a device that basically puts a very gentle magnetic pulse into a part of the brain, and that seems to really help reset some of the neural circuitry to help people recover from depression. But those tools are, while very novel and very exciting, people don't know how to point them exactly right now.

Nicholas Weiler (10:13):

Right.

Dan Yamins (10:14):

So what the optimal ways to configure the TMS device so as to be able to achieve the best and effect with the least intervention, that's something that people would love to be able to know how to do. A simulator model, basically digital brain model could be used to support exactly that kind of optimal treatment design strategy. Basically you want treatments that are the least amount of time and that are the least disruptive to the brain, but also really effective at helping reset the circuits so that people can get out of a depressive loop, that TMS has been effective at that to some extent, but there's so much more to be done to be able to focus those things to really help people in the least invasive, most effective way we can.

Nicholas Weiler (10:56):

I love the terminology you're using of a brain simulation. I think that's a really intuitive way of thinking about this. You're simulating how a brain might act. I want to be clear, this is something that does not exist yet as a whole brain-wide simulation. It's something that I know you've been talking about that folks like Andreas Tolias at Stanford have been talking about and others. And I want to get into a little bit the question of, well, how would you build something like this?

(11:21):

But first, maybe to make sure that listeners understand what is the thing we're talking about? What is a foundation model? What is this kind of generative model? How would it work? I wonder whether it makes sense to go back and talk a little bit about the brain modeling that you've been doing for many years and that neuroscientists have been doing for many years in computational neuroscience. So you mentioned something to me last time we talked, which is you've observed that when you train computational neural network models to perform a task, the better they perform, the more human-like or the more brain-like their performance is, the more the actual activity of those models starts to look like the equivalent activity in the brain. And I wonder what is the jump that you're making between those kinds of models, the kinds of models that neuroscience has been using for decades, and what we're now talking about with the foundation models, these kinds of architectures that have come out in the past few years?

Dan Yamins (12:23):

Well, okay, so one of the things that's been really exciting over the past 10 years is that models from artificial intelligence originally created to do tasks, visual recognition, auditory recognition, even language tasks, have been shown to be pretty good quantitative models of brain areas that are known to support those tasks. That's kind of been the nexus, the core driving result in this burgeoning neuro AI field, neuroscience AI. Models that come out of artificial intelligence end up being surprisingly good quantitative predictors of brain data with their internals. Basically you have a neural network, it's trained to do some task like take an image in and report out something about the image, like the category of some objects in it.

Nicholas Weiler (13:10):

"Is this a cat or a dog? Is this a car or a building?"

Dan Yamins (13:13):

Yeah, "Is it a cat or a dog?" Or, "Where is the cat?" Or, "What pose is the dog at?" Or, "What pose is the hand at?" Or other things. There's many different tasks that people try to solve in computer vision. Anyway, it turns out that computer vision models, they have internal layers and states and stuff like that. And you can ask how good are those internal layers at predicting neural responses in the middle, the kind of internal parts of the brain that do that task? And it turns out, surprisingly perhaps, that AI models are quite good models of the brain in a kind of fine-grained quantitative fashion.

Nicholas Weiler (13:47):

So the artificial neurons seem to be acting kind of like the biological neurons?

Dan Yamins (13:51):

Yeah, that's right.

Nicholas Weiler (13:52):

Or maybe the computations are somehow similar?

Dan Yamins (13:54):

There's a lot of debate as to what it means for computations to be similar. So I shy away from that statement, but what I would say is just that you can measure brain data the way we normally do, by measuring the responses of neurons in response to stimuli or in different conditions, and you can measure neural networks as if they were candidate brains. And then you can ask quantitatively how correct are the AI models at describing the brain data? And actually it turns out that good AI models are by far today's best state-of-the-art quantitative models of the actual brain. Even though they weren't trained on any kind of brain data in a direct way, they end up learning things that make their internals look like the different parts of the brain in a pretty strong way, pretty good mappable way.

Nicholas Weiler (14:39):

And does that suggest to you maybe there's an optimal way of doing this that the brain and the AI models are converging on the best way of doing this particular task?

Dan Yamins (14:50):

Yeah. It suggests that there's not too many different solutions to the tasks that the brain needs to do. Another way to say that is kind of in an evolutionary sense, the brain is highly constrained by behavior. So we as humans do lots of challenging things as a job, you have to do those jobs well. And there's not that many different possible solution types. So if you figured out how to solve it with AI, then you end up having to basically stumble on the brain-like solution. And so it's a kind of principle of convergent evolution in a way. And I think that that's turned out to be pretty powerful driver of connection between artificial neural networks and real brains. That's the core foundational principle of neuro AI field. That is like the preconditions under which the digital brain work can really thrive.

Nicholas Weiler (15:39):

So tell me about that transition. So we're saying that if you train sort of a general AI model, it's got neural networks, sort of in the way that's sort of the foundational architecture of a lot of these networks. If you train it to do tasks, it starts to look more like how the brain functions. So tell me about the leap to now, let's actually train that model on actual brain data. Let's train it specifically to act like the brain. How are these models that you're talking about different and what's the additional value that that gets us?

Dan Yamins (16:08):

Well, so earlier you said that when you were talking to Laura, you heard about the ways large language models are kind of like potential model organisms, but that the models weren't exactly like the brain. And I was going to interrupt you and I decided not to, but I was going to ask you, how do you know they're not like the brain?

Nicholas Weiler (16:24):

Right.

Dan Yamins (16:24):

I mean, probably they're not perfectly like the brain. That's totally true. But actually, one way to interpret what I just told you about neuro AI is actually that you get surprisingly like the brain by doing an AI task. And actually people have shown that LLMs are by far our best model of language areas in the brain today. So to some extent, the AI models are kind of like the brain, but they're certainly not perfectly like it. And of course they're not, aside from the fact that they're in silico and all that, they were optimized for probably quite different things. The brain was built by evolution, by social interactions and all these different real world things over the course of eons, effectively.

(17:01):

The AI models have some of those same constraints. They're optimized, they're learned to solve human-like tasks in some way. Predicting text on the web has kind of got some social component to it and it's got some descriptions of the worlds in it and all these things. And so it's not totally dissimilar to the way that the brain is trained probably, but it's not the same by any means. And so the leap from training a network to solve a task and then asking do its internals of the neural network look like the true brain to just directly training the neural network on brain data is like the leap from going from a theoretical science to an empirical science.

(17:41):

When we find that neural networks that solve a task look like the brain, that gives us a theory of the brain. It says the brain is kind of like it was optimized for the task that we found that the neural network is good at. And that's really cool and it's very exciting because it tells us a underlying principle for why the brain is as it is, why the neurons are as they are. But what it doesn't do is optimize the connection directly between what we really find in the brain and the neural network. And that means that there's some limitations on those models as correct brain models. Naturally there would be. And so that leap that you're talking about is trying to close that gap by instead of focusing on an AI task and then seeing afterwards how good is it as a model of the brain, instead directly go from establishing that link and making the models as brain-like as we can.

Nicholas Weiler (18:28):

That makes a lot of sense. So I now want to get back to that question I raised earlier. This is not something that exists as a whole as you're describing it. I think pieces of this are in process. People are working on things like generative models or foundation models for the visual system or other systems in the brain. What would it take to actually make this a reality? What would need to go into a model like this to make it useful?

Dan Yamins (18:53):

Well, as you say, pieces of this have been things that people have been working on. The neuro AI field as a whole for the last 15 years has been building models of different parts of the brain. What's really needed to move that to this much more biologically direct link is to have a lot more brain data. The reasons why we weren't doing the direct match to brain data from the beginning was exactly that we didn't have anywhere like enough data to do it. So people may be familiar with the idea that training neural networks takes a lot of data, right? Large language models are actually trained on much more text data than any one person sees in their lifetime by a large fraction, like 10,000 times that or something. And so their models are quite data-hungry. I don't think we're proposing to go to the scale of the internet essentially with our brain data, that would be hard.

(19:38):

But we now are in the situation where the ability to scale up brain data collection tremendously is something that's burgeoning right now and is happening in a number of really exciting ways. And so the idea is to collect that data and that larger scaling up of existing models that have been sort of at medium scale so far is a really important piece of closing that gap, of getting something that's enough data from people that the model has a chance of being, as I say, not like in any individual, but capturing enough about the common features of people that it can be used as a simulator for the kind of biomedical purposes that I mentioned earlier.

Nicholas Weiler (20:13):

And we're talking about having a large cohort of folks with fMRI data, other kinds of data, and it would probably have to include what are the people looking at, what kinds of videos are they watching, what kinds of behaviors are they doing? So I guess my question is how comprehensive does this need to be to capture the essence of the algorithm that you're interested in?

Dan Yamins (20:37):

That's a good question. We'll try to make it as comprehensive as we can. We want to basically collect data from high resolution MRI recordings as well as another technique that's called MEG, which is kind of putting a magnetic cap on the head and it's a non-invasive technique for recording high frequency temporal signals from the brain. So between the MRI data and the MEG data, you could triangulate what the responses are pretty well. And as you say, the subjects who would come in and volunteer to have their brain recorded, as many people do now, would come into this experiment and be recorded just looking at stimuli, doing basic visual, auditory, memory, attention tasks, motor tasks, things that are just everyday life types of activities.

(21:26):

And of course we'd like it to be as comprehensive as we can. So we're targeting the things that seem easy about being human, like understanding what you're seeing when you're looking at it, understanding the words that you hear, being able to move your arms to grasp things and open bottles and stuff like that, or remember some basic things that you've just heard, not really the stuff that's hard about being a human. So I don't think that our models will rival real humans in terms of their complex decision-making capacities for large life events or anything like that. So we're not targeting that level of complexity, but at a somewhat lower level complexity, pretty comprehensive.

Nicholas Weiler (22:07):

But you could probably see something about how watching a sad movie makes you feel sad, right?

Dan Yamins (22:12):

Exactly.

Nicholas Weiler (22:13):

And what are the connections between those things?

Dan Yamins (22:15):

Yeah, exactly. So things that are a little bit more automatic for humanness, essentially. I don't think we're going to be able to use this model to be a therapist or something like that, but what we would be able to do is potentially use it to detect some of the core things that you would use to think about your everyday life.

Nicholas Weiler (22:36):

We're almost out of time. I would love to hear you just paint a picture for us in just a minute or so of what it would be like to run a neuroscience lab in 10 or 20 years when we have these foundation models of the brain, these large brain models.

Dan Yamins (22:52):

Right now, neuroscience is sort of transitioning from being a single lab driven pre-theoretical science to being a more mature science where results from one observation strongly constrain others, and also where the scale of projects is beginning to go beyond what any one individual lab can do and where there's a tight coupling between your theories, in this case neural network based theories, and the actual data that you would both collect and also the experiments you would design to figure out what to do next. So what I'm hoping will happen is that maybe, let's say five to 10 years in the future, more neuroscience labs will be using shared models and will be contributing data to the creation of those shared models in a more self-organized but collaborative fashion.

(23:47):

I also think that the kinds of questions that people will ask will be more formulated in terms of using the neural networks as hypotheses for the brain. And so the theories can be turned to ask interesting questions about how you think the brain works and test them in a kind of theoretical and predictive way first, as opposed to first get the data and then make the hypothesis. I think it'll start to go the other way a bit. And then maybe most excitingly for me is I think that neuroscience has been pretty disconnected from its ultimate applications for a long time. The whole purpose of neuroscience in the long run is to help people with their brains effectively.

Nicholas Weiler (24:24):

Right. We all have brains. We'd like them to work well.

Dan Yamins (24:26):

We want to keep them healthy and we want them to work well. And so I think that the actual use of the neuroscience experiments and models to help do treatments and diagnoses and things like that, I think that's going to start to become much more real. And I hope that will be the biggest outcome because that's what justify the whole endeavor.

Nicholas Weiler (24:44):

Well, fantastic. Dan, this is such a fascinating conversation. I'd love to keep talking about this. I think we're out of time for today, but I'd love to have you back and keep talking about these issues. Thanks so much for joining us.

Dan Yamins (24:53):

Yeah. Thank you so much for having me and for asking such great questions.

Nicholas Weiler (25:01):

That was my conversation with Dan Yamins from last year. And as I mentioned at the top, we have now launched the Stanford Digital Brain Project here at Wu Tsai Neuro to make this vision a reality. It is a monumental effort bringing together scientists from across campus to collect the most detailed naturalistic portrait of the human brain in action ever attempted.

(25:21):

To talk about where things stand today, I asked Russ Poldrack, one of the project's leaders, to join us on the show. Russ is an expert in neuroimaging. He's the chair of the Department of Psychology and leader of the Stanford Center for Open and Reproducible Science, and he's been deeply involved in building out the framework and organizing these data collection efforts. I started by asking him to give us a sense of the scope of the imaging experiments the team is planning.

Russ Poldrack (25:48):

It's ambitious in a couple of ways. One in just the amount of data that we plan to record. So we're setting up now for the first sort of data collection campaign where we will aim to image each person about 100 times, both with high field functional MRI imaging and using magnetoencephalography using this system called the OPM-MEG system. There have been studies that have imaged individuals more than 100 times on, say, MRI. I did one of those on myself a decade ago and other people have done similar things. But even for this one campaign, no one that I know of has collected these kinds of data from an individual at such a scale. And that's just the first of what we hope will be on the order of 10 campaigns looking at different aspects of mental function. And so if we're successful in that, then we will have a dataset that's orders of magnitude bigger than anything that exists now.

Nicholas Weiler (26:51):

I'd like to get back to these sort of different technologies that you're combining and what they're going to teach us. But it seems like the individual experiments are also going to be kind of unusual, not what you might expect from sort of a run-of-the-mill cognitive neuroscience experiment. Could you describe for us what these experiments are actually going to look like and what makes them different from what we often see in brain imaging experiments?

Russ Poldrack (27:16):

Sure. Maybe I'll first say something about how we usually do brain imaging experiments with people, which is we put them in a scanner and we give them often very simplified kinds of tasks. For example, we might be showing them individual pictures and having them say, "Is this a picture you've seen before?" Something like that. In my lab, we might be having people choose actions and occasionally telling them to stop. But we build experiments that are very tightly controlled with the aim of being able to ask very specific questions about what a particular brain system might be doing at any point during that experiment.

(27:49):

But what we need in order to build the kind of models that we want to build coming out of this study is a different type of experiment that really tries to look as broadly as possible across the nature of human experience. So instead of wanting to hold things very tight in terms of controlling the stimulation, what we really want to do is give people stimuli that are as varied as possible so we can really map out what's the range of human experience so that our models can come to encompass that entire range. Because if the models only see a small slice of human experience and human brain activity, that's all they're going to be able to model. So we need to be able to capture a broad range.

(28:28):

The first experiment that we're doing is going to be showing people videos, having them do different tasks, sometimes tasks that involve attending to different parts of the screen, other times tasks that involve some kind of memory for previous images, and also having them potentially describe the images that they're seeing. There's a really broad set of kinds of videos you might watch, and we're trying now to select the videos that can really span that range. And then you might be watching different videos for different reasons, and we want to kind of capture the different things people might be doing with a video rather than just having them do a very specific isolated task on a very tightly controlled set of videos.

Nicholas Weiler (29:07):

Yeah, that seems very clever. I was thinking about this and I mean, you can take the videos as maybe a stand-in for seeing things in real life, but even if you just think of it as we all spend time watching videos, I'm going to know, "Have I seen this one before?" So there's an aspect of memory there, and then, "I need to remember the video so I can describe it later." So there's another aspect of learning and memory. And then there's how it makes you feel, there's the emotional aspect to it, and maybe what you're selectively remembering about it. Then there's the language part, where you're actually describing the thing, so taking your experience and your memory and turning it back into language and a description.

(29:44):

So it seems like it is going to activate all of these different networks. And that's something that Dan was telling us about last year when he was describing the idea of these models, which is part of the challenge in building sort of a holistic understanding of what's going on in the brain is that all of these different brain networks and systems that we've been studying are all active all the time. They're all influencing one another. And so what this is trying to get at is, can we see how those networks are engaging with each other and interacting so that we know if this goes up, that's going to have an effect downstream on this other system in the brain? Am I getting that right? Is that part of the idea here?

Russ Poldrack (30:22):

Yeah, we want to be looking at brain activity as a whole and understanding how that relates to the kinds of things that are happening in the person's experience, the kinds of actions they're taking, ultimately with the goal of being able to take any kind of experience a person has and predict what their brain activity patterns would be over time as they have that experience. That's the ultimate goal of this model.

Nicholas Weiler (30:48):

And so I know that you're probably very focused on getting this first set of experiments underway. Can you give us a sense of how this sort of longer set of campaigns you mentioned will build on that? I know your group is interested in things like cognitive control, which I don't imagine would be deeply involved in this first set of experiments. So how do additional experiments that you plan to run add on different aspects to this same underlying model?

Russ Poldrack (31:12):

Yeah, no, we're already thinking about what the next campaigns will be, and we're hopeful that maybe the next campaign will be one that has people making a much broader set of decisions while they're being scanned. An obvious platform that people have been using for a little while and that we are now thinking a lot about how we might use is the various types of video games, simulations, things that people do often for fun. If you're doing a game that involves navigating through some space or you're doing a driving simulator, there's various ways that we could have people engage in a really broad set of actions and then see how do their brain systems activate and respond in the context of all of those different types of decisions.

Nicholas Weiler (31:59):

Plus who doesn't want to be paid to lie in a scanner and play video games? That sounds great.

Russ Poldrack (32:04):

Exactly.

Nicholas Weiler (32:04):

Well, you are going to be asking quite a lot from these research participants. I mean, one of the things I'm interested in here is that the plan is to do this in a fairly small number of people. And I wonder, is that because you're going to be asking a lot of each of them and you can't necessarily do that with 100 people? Or is there something else here about maybe we don't need a really large group of people? Usually we do because we need statistical power, but I'm curious if this is a constraint because of the experimental design or maybe it's a feature of how you've designed these experiments.

Russ Poldrack (32:38):

It's both a constraint and a feature. It's mostly a constraint just in terms of throughput. So if we had 10 MRI scanners and an army of people who could run subjects every day, then we could in theory get 100 sessions in a year from 100 people. But even though this is a huge project, it's still constrained by things like the availability of the MRI scanner. And so the goal right now is to get roughly 10 people scanned roughly 100 times on each of those modalities, MRI and MEG, over the course of a year. So that means if each of those sessions is about an hour, they're going to need to come in for roughly four hours a week for two MRI sessions, two MEG sessions. So that's a pretty substantial commitment on the part of the people who are going to participate. We'll pay them for their time, but it's going to really require a commitment.

(33:34):

One of the other reasons that we're focusing on individuals is that even though humans in general have the same brain organization in broad strokes, there's detailed organization within every person. And what we know is if you try to take a bunch of people and put them all together, you'll see the big picture, but you won't see the fine details that we really need in order to be able to make decisions about people. For example, like we would envision one will one day in the context of brain disorders, you really need to be able to build person-specific models. And so that's the rationale for getting enough data per person to be able to build really robust models for those individuals. Now, the idea is once we have those models on some number of people, we can start to use those existing models to then adapt to new people without needing necessarily as much data. That's the hope in the end.

Nicholas Weiler (34:28):

Before we move on, Russ, you used the phrase person-specific model, and I just wanted to get a sense of what you meant by that. I don't want to give the impression that we're talking about uploading someone's brain into a digital model or something like that.

Russ Poldrack (34:41):

No, that's a good point. The goal is really to develop models that can predict brain activity. And we could build one based on data from a lot of people, and those models would be okay at predicting data from any new person. But if we want models that can really precisely predict brain activity for a specific individual in the way that we will want to, for example, for clinical purposes, then we need to use a bunch of data from that person to generate that model. So when I say a person-specific model, I don't necessarily mean a model of what the person is thinking. I mean a model that is attempting to do the best possible job of predicting brain activity based on that particular person.

Nicholas Weiler (35:25):

Right. And part of the whole idea is we don't want to be giving generic brain stimulation. We want to be giving stimulation ultimately that is tuned to your particular brain and the place where we want it to be performing better. It reminds me of the conversation we had with your colleague Cory Shain a few months ago on the show where we were talking about trying to understand the brain networks responsible for language and that it turns out if you average over lots of people, you're going to miss the fact that there are actually individual differences in where those different components of the language network exist that you're going to lose in the mix, in the average. There is variability between people and how these networks work and in the specific location where some type of linguistic processing might occur. And so you actually have a risk of losing that individual information if you're averaging too much over too many people.

Russ Poldrack (36:20):

Yeah, that's exactly the rationale here. We really need this amount of data, we think, in order to be able to build those person specific models.

Nicholas Weiler (36:28):

So you mentioned these two different imaging technologies that are going to be used in these experiments, and each participant's going to do all the tasks in both of these things. There's the fMRI and then there's this OPM-MEG. And if I understand correctly, I mean each of these instruments are among the most advanced of their kind in the world, and I don't think they've ever been combined in this way before. So I'd love to hear two things. One, if you could give listeners a quick refresher on what these two different things are measuring, and then help us understand why combining them is going to be so valuable for this dataset and the models that you'll be building.

Russ Poldrack (37:04):

Sure. Yeah. So both the MRI system and the MEG system are meant to be measuring activity in the brain, but they measure it in very different ways. And so the MRI system, we call it an ultra high field MRI system. The magnet has a strength of 7 Tesla. If you go to a hospital and just have to get an MRI scan for your knee or something like that, most of those scanners will be either 1.5 Tesla or 3 Tesla. So this is a much stronger magnet. And what that does is allows us to get much finer resolution in terms of seeing where the activity is happening in the brain down to the one millimeter level or potentially even smaller. And MRI is going to help us see the entire brain at a pretty high spatial resolution. Because MRI is measuring blood flow. That blood flow response is relatively slow, it emerges over several seconds, whereas the actual activity of the neurons is happening over say 50 milliseconds, much faster timescale.

(38:02):

So what MEG does is allows us to measure that. It actually measures it by the system, this optically pumped magnetometer system. It's basically a bunch of little sensors that sit right outside the head. It's like a little helmet that one puts on. And each of them is measuring tiny changes in magnetic fields. And basically what happens is when neurons in the brain fire, they fire in a coordinated way and they cause these electrical fields, and any electrical field has an associated magnetic field that goes along with it. These are tiny. They're many orders of magnitude smaller than the magnetic field of the earth. But using this technology and putting a person in a room that's very magnetically shielded, we can actually sense brain activity at a very fast timescale. So now we can see activity at the millisecond timescale. It's not quite as good at localizing it in space. And so the two tools are kind of very well aligned in that they fall on different parts of the trade-off of high spatial resolution versus high temporal resolution.

Nicholas Weiler (39:07):

Yeah. And just to give people a sense, and we're talking about millimeter resolution in space and millisecond resolution in time, that millisecond resolution in time is pretty close to the speed at which neurons are actually firing. Just to make sure that it's sort of clear, there are probably thousands of cells in those cubic millimeters that you're measuring because neurons are even smaller than that.

Russ Poldrack (39:29):

That's right. There's probably 100,000 or so.

Nicholas Weiler (39:32):

Right, 100,000 or so. So you're not necessarily getting down to the level of individual neurons, but probably that's not what you need. What we're trying to understand is the networks that are engaged, the different parts, rather than trying to understand this neuron connects to that neuron and so on. So what is it about that that's important for building the actual models that will come out of this dataset?

Russ Poldrack (39:55):

Yeah. So because the two techniques give us basically different lenses on the brain, we think that you can't get a really solid picture of how the brain works by just taking that one lens. For example, if you're an astronomer, you might want to look at the visible light that's coming from the sky or you might want to look at infrared light that's coming from the sky and they're going to tell you different things. Similarly here, these two different measures are going to tell us different things about how the brain works. And we're pretty sure that either one of them alone, because every tool has biases and artifacts that are going to kind of flavor the images, to the degree that you see something with one tool, you're never quite sure, is this really there or is it due to some kind of artifact in the images or some sort of thing that's a known bias of the method?

Nicholas Weiler (40:45):

Right. Is this just an MRI thing, an artifact of MRI that comes out sometimes?

Russ Poldrack (40:50):

Exactly. If you can put them both together... So for example, there's a thing that people are really interested in measuring, which has to do with the difference in basically how much excitation is going on the brain, like how much turning on of neurons and how much inhibition. We call that E-I balance, excitatory-inhibitory balance. You can try to measure that either with functional MRI or with MEG, but each side there are questions about whether it's getting the right answer. If you can put them together, then to the degree that they tell you the same thing, you can be much more certain that you've canceled out the problems on each side.

Nicholas Weiler (41:23):

Right. It's interesting that these are two magnet-based technologies, but they're measuring totally different things, where the MRI is measuring this slow blood flow signal that we understand to be related to the brain's response. When something is active, it needs to send more blood there to get more oxygen to that part of the brain. And that happens over a couple of seconds, but it's a reliable indicator of where the brain is active. And then in the case of OPM-MEG, as you said, it's measuring these tiny, tiny little magnetic fluctuations that are caused by the brain's electrical signaling.

(41:56):

I mean, they're so tiny that, we just did a story on this on our website, we installed this new system at one of our community laboratories at Wu Tsai Neuro and had to build this whole, I think it was a 13,000 pound magnetically shielded room within a room in order to shield this device that's so sensitive from the Earth's magnetic field, from a bus passing by outside the room, which would otherwise make it impossible to measure these incredibly tiny signals.

(42:24):

It's a really cool system. And I don't know if anyone's seen an MEG before, typically, they're this giant helmet thing that has super cooled sensors and basically fills up half the room. And this is just a little headset that goes on your head and eventually it should be able to let people even stand up and walk around at some point, I believe. So yeah, really interesting to be able to combine these two frontier systems.

(42:48):

In combining these systems, the datasets that you're collecting are going to be unique, pretty one of a kind in scope and depth, as we said. And you're going to be sharing all of this data openly with the research community. Can you give us a sense of how that's going to work and how you envision other members of the community using this data?

Russ Poldrack (43:09):

Yeah. So my lab and I have been huge advocates for data sharing in neuroimaging for a long time and have built a lot of infrastructure, both for how you format the data and actually a platform called OpenNeuro where people can go and obtain lots of shared neuroimaging data. And so we have a lot of experience already and sort of know how to do this. Now, one special aspect of this particular dataset is that there's a lot of data on one particular person for each person who we've imaged. And so we need to be confident that we can protect their privacy where we're sure that there's no way that it could be traced back to the individual person. Other aspects of the data that might have more information that could potentially be used to identify the person will share in a more restricted way, where we'll make sure that the other people who we're giving the data to are going to protect it. But all the data will be available to other researchers.

(44:01):

The goal is ultimately going to be to be able to give the data to other researchers so that they can go try to train their own models, because it's going to be such a unique dataset and because the field is still very data limited. People have been trying to build these types of brain models and have seen that they're nowhere near the limits of where the models could get if they had more data. They're very far from that. And so the idea is we are going to share these data with other researchers. Other researchers are also going to share their data. Let's say we have 100 people where we have varying amounts of data. We can start to imagine, how do we bring all those together? We could then start using that to then test how well we can adapt it to one individual person. And so that's the goal.

Nicholas Weiler (44:45):

Yeah. I mean, even aside from the models, obviously this is the Stanford Digital Brain Project, the ultimate goals is building this detailed model of the human brain that'll be useful for researchers and for clinicians, as we talked extensively with Dan about. But just having this dataset is going to be, I mean hopefully, pretty transformative for neuroscience. It's just a way of seeing in detail the interactions in the brain that hasn't been seen before.

(45:12):

Let's talk about the models. So how do you take all this data, let's say you're gathering each of these experimental campaigns, how does that then sort of funnel into building a foundation model that represents at a sort of statistical level that allows you to predict what brain activity we would expect to see if someone was doing some other task that wasn't included in this exact experiment?

Russ Poldrack (45:35):

So the first thing we have to do is a lot of processing on each of the different data types just to get them ready so that they're in a format and they've been processed in a way that they're kind of ready to be worked with. We've gotten rid of effects of things like when people move their head. We've basically made the data sort of ready to be modeled, and that's a whole enterprise on its own. Fortunately, we have a lot of experience doing that stuff too. Once you have that, then the idea is we're not going to have necessarily everything we'd want to know about what a person is doing at any point in time. And so one of the ways that we've learned you can train models is through what we call self-supervision. So just like the large language models that people use are trained just by trying to predict the next thing in the text that they're looking at.

(46:22):

One of the things that we can do with brain data is try to just predict the next pattern of activity at the next millisecond or the next second if it's MRI. And that's a starting point just for getting the models to understand how brain activity changes over time. Then we can, to the degree that we know what the person's doing, we know what video they're watching, we know what game they're playing, we've recorded their behavior, we know where their eyes are looking, all of that sort of stuff, we can also start feeding that into the model. Today's models can bring together all these different types of information across different timescales, across different data types, but ultimately their goal is just, "Can you predict the next thing that's going to happen?"

(47:02):

Once you have models that can do that, then what you can start doing is say, "Hey, let's pretend we don't know what the brain's doing at all. All we have is the video and the behavior and where their eyes are looking. Can we predict what the entirety of the brain activity would look like then?" Because that's ultimately what we want to do. We want to know if I, for example, change something about what you're thinking, how will that change your pattern of brain activity? Or ultimately, if we stimulate your brain in a particular way, how will that change the pattern of brain activity? So that's how we get to those ultimate models is by starting by just training them to predict the next thing and then starting to do that across a broader set of different types of inputs.

Nicholas Weiler (47:42):

Yeah, I love the way that you're describing that. The goal of the project is to create this tool for neuroscience that would let you have sort of a virtual model, a thing where you could ask like, "Okay, what would I expect the brain to do in this situation?" Or if someone's experiencing depression, for example, and their brain networks look this way, we've talked with Corey Keller about brain stimulation and we've talked with Leanne Williams about detecting patterns of brain activity associated with disease. Once we can do that in a more precise way, then you could use this model to say, "Well, I expect that given this person's particular pattern of brain activity that seems to be associated with their depression, we can give this pattern of brain stimulation and we think that's going to be the most effective way of helping them get over that depression with the minimal side effects."

(48:31):

But I also recognize that given how all of us are feeling about AI these days, I can imagine people might be feeling a little fearful about the idea of building foundation models that incorporate so much information about how our brains function, whether there are privacy concerns or whether it's about just what kind of power does this give to an AI system. Can you help set our minds at ease about that? What makes this different from those sort of fears that we might have coming from images that we've seen in science fiction and concerns we have about some of the large language model systems that are out there today? Is that something that you are worried about? And if not, why not?

Russ Poldrack (49:08):

I'm definitely worried more generally about the power of AI systems and what they can do. And every day we're seeing new examples of how that can potentially go wrong. I think one of the guardrails here is that this system is simply going to be a predictive system. The systems that I think most people are worried about right now out in the world are these systems that can actually go out and engage in actions, what we call agentic AI systems. This is simply going to be a predictive system. And so there's no chance for it to go out in the world and do things, and then we as scientists will take those predictions and use them. We have to be careful about how we might use them. But I'm not worried in the same way that I'm worried about what agentic AI systems out in the world can do in terms of breaking into computers or things like that.

Nicholas Weiler (49:57):

So I have two more questions I want to ask. We've been talking about everything that's going to go into these experiments. All the data is then going to be brought into a model. This is a team of almost a dozen researchers from across campus that are involved in the different aspects of this. And I wonder if you could just give us a sense, we talked with Dan last year, now this is actually coming to pass. What has it taken to get all of these different people working together to make this project possible?

Russ Poldrack (50:25):

I think what's bringing us together is the excitement about being able to be part of something that's this potentially transformative. It's the kind of thing that each of us has probably wanted to do some version of or some part of for a long time, but we've never had the resources to do it. It's a lot of work. We're all spending many hours a week now working on different parts of it, be it picking out the videos to show the people in the scanner or creating the ways to manage the data once they come off the scanners. All of that requires a ton of effort because this is new in various ways. Now, many of us are people who have each individually maybe collaborated in pairs or maybe two or three at a time, but never has a group of this size and kind of breadth across campus come together to do one project in this sort of way.

Nicholas Weiler (51:15):

Great. And so I know this last question's going to be sort of a crystal ball question. Science progresses at the speed that it progresses, but what kind of timeline are you envisioning for the next steps in these experiments and when we might start to see some of these models coming out and being used in research or even in the clinic?

Russ Poldrack (51:33):

I think we're hopeful that we'll start collecting data on the first campaign early in the coming year. That's an ambitious goal. We're hopeful that we can get there. Beyond that, the hope is that we go on for probably five more years and continue these data collection campaigns and hopefully ramp up the velocity of the data collection over time once we've kind of figured out how to do that. In terms of where we need to be to be able to have useful modeling outputs, by the time we have that full first dataset, we should be able to be getting some interesting things. Now I think some of the real power is going to come once we have multiple, what we call tranches, multiple different types of data. So it's probably going to be several years before we can fully flesh out the promise of this kind of very rich and highly diverse sort of dataset.

Nicholas Weiler (52:24):

Excellent. Well, as it moves forward, we'll keep having you and Dan and other members of the project come back on the show and tell us how it's going and what's next. So Russ Poldrack, thanks so much for coming on From Our Neurons to Yours. This has been fascinating.

Russ Poldrack (52:37):

Yeah, thanks. It was a pleasure.

Nicholas Weiler (52:41):

Thanks again so much to our guests, Dan Yamins and Russ Poldrack. Dan is a faculty scholar here at Wu Tsai Neuro and an associate professor of Computer Science and of Psychology at Stanford. Russ is the Albert Ray Lang Professor of Psychology and Director of the Center for Open and Reproducible Science at Stanford. To read more about their work and the Stanford Digital Brain Project, check out the links in the show notes. And I should say, the project is not yet recruiting research participants, but stay tuned. They will be soon.

(53:10):

If you enjoyed this episode, be sure to subscribe for more conversations from the frontiers of brain science. We also love hearing from you. If you have thoughts about the show or questions about the brain you'd like to hear us discuss in a future episode, send us an email at neuronspodcast@stanford.edu or leave us a comment on your favorite podcast platform. And if you're someone who's emailed us and I haven't gotten back to you yet, I am so sorry. I have a list of people I'm about to get back to. While you're at it, please give us a rating and share the show with your friends. It may seem like a small thing, but it is really tremendously valuable for us to be able to bring more listeners to the frontiers of neuroscience.

(53:47):

Coming up on From Our Neurons to Yours...

Ehsan Adeli (53:50):

We use body language every day. And we've all experienced this, someone may tell you just that, "I'm fine," but their posture, facial expression, movement, eye contact, or level of engagement may tell you something completely different. So behavior is an external manifestation of the internal state of the mind.

Nicholas Weiler (54:13):

From Our Neurons to Yours is produced by Michael Osborne at 14th Street Studios, with sound design by Morgan Honaker. Our social media strategy is by Julia Diaz. Additional editing by Nathan Collins. Our logo was designed by Aimee Garza. I'm Nicholas Weiler. Until next time.