Q&A: Using software engineering to bring back speech in ALS

Erin Kunz took a circuitous route to  studying the brain. The third year PhD student in Electrical Engineering started her career developing autonomous vehicles at General Motors (GM) — but now she uses her software engineering and machine learning skills in the Neural Prosthetics Translational Laboratory at Stanford, led by faculty members Jamie Henderson and the late Krishna Shenoy.

Neural prostheses, also called brain-computer interfaces (BCIs), are an emerging technology intended to restore neural functions that have been lost through illness or injury. The newest generation of BCIs are implanted directly into the brain and can both deliver electrical signals—to stimulate specific nerves—and record the brain’s electrical activity. In patients who have lost the ability to move their upper or lower bodies due to stroke or spinal cord injury, BCIs are being developed to restore limb movement, and allow use of phones and iPads. In other settings, BCIs are being used to facilitate communication in the form of artificial ears and handwriting decoder devices.

Erin Kunz studies speech production with a clinical trial participant with ALS using brain-computer interface (BCI) technology. Photo by Steve Fisch.

Here at Stanford, Kunz is developing brain-implanted BCIs to help restore speech to individuals with Amyotrophic Lateral Sclerosis (ALS), a progressive neurodegenerative disease that leads to loss of muscular function. Her innovative work is supported by the Ketterer-Vorwald Neurosciences Interdisciplinary Graduate Fellowship.

Our beloved colleague Krishna Shenoy passed away January 21, 2023. Read more about his life and impact on the field here. ​​

We spoke with Kunz to learn about her path to neuroscience research, her science fiction-sounding PhD project, and how the interdisciplinary Wu Tsai Neuro community has influenced her work.

Can you tell us a bit about your transition from working in the automotive industry to a neuroscience-focused PhD?

I did my undergrad at Berkeley in mechanical engineering and I grew up in a big car family, so automotive work was very appealing. During my undergrad I drifted from the more traditional mechanical side of engineering to controls and computer science and started getting interested in machine learning. My first job at GM was working on software for autonomous vehicles.

Closeup of a brain-computer interface (BCI) in action. Photo by Steve Fisch.

On the personal side, I became interested in neuroscience because my father was diagnosed with ALS while I was in college. That had a huge impact on my life, and he ultimately passed away about a month after I graduated. The Shenoy lab came on my radar at that time because they work on assistive technologies for patients with ALS. Having been a “human speech-decoder” for my father, I appreciate how much assistive technologies, particularly communication BCI, can help patients and their families.

I spent a few years at GM starting off my career, and then started wondering what's out there in the BCI  field. I thought, what the heck, I'll apply to grad school and see what happens. I applied to Master's programs, and ended up doing my Master's at Stanford in Electrical Engineering. I really made an effort to get involved with neuroscience research and get to know professors in that field, then applied to stay on for the PhD program.

You're developing brain implants to help people with paralysis be able to speak again. What’s the high-level goal for this project?

Currently, I'm working with a participant who has ALS. Her speech is very limited—she can produce sounds but they aren't intelligible. Because she can't communicate conversationally the way that you or I can, we are developing a technology that can take in her neural signals of attempted speech, pass them through a computer, and output the words that she wants to say. Specifically, our program decodes phonemes, which are individual sound units of language, and then that sequence of phonemes goes into a language model that produces the actual text output.

A big part of the project is to actually understand the neuroscience behind speech production. Because a lot of neuroscience uses animal models, and animals don’t talk, our understanding of speech has been mechanistically limited. Now that we have a participant with a BCI in this area of the brain, we can do a lot of experimentation and understand how our speech cortex actually works, which is really exciting.

The Neural Prosthetics Translational Lab team exchange smiles with their ALS trial participant. Photo by Steve Fisch.

That sounds super cool—can you tell us more about how the BCI you’re developing functions?

Our lab is one of only a small number of labs that works with human participants with implanted BCIs. Each device is a square of just under four millimeters containing electrodes. Every individual we work with undergoes voluntary neurosurgery to have it implanted into the part of their neocortex that controls movement. We use the device to record their neural signals as they do different tasks, both for developing applications technologically and also to add to our understanding of how the brain controls movement more broadly. The patient populations we work with are individuals either with quadriplegia [loss of voluntary movement in all four limbs] or ALS, so they have to meet a pretty substantial threshold to be able to have this sort of voluntary intensive surgery to be in the study.

How are you using the software skills you developed at GM and in your electrical engineering training?

I would say a lot of what we do is software engineering or machine learning. We have what we call “the rig,” which is a big stack of multiple computers that the neural signals go through. We write software programs that interpret neural signals in real time, so there's a lot of machine learning in developing those algorithms.

The work we do is super interdisciplinary. I mean, it requires understanding neuroscience, understanding machine learning, and all of the analysis that goes into that as well as having a good grasp on your intended behavior — in this case speech and language that are actually incredibly complex and have entire fields of study in their own right. Right now I'm taking a linguistics course called Articulatory Phonetics, about how we actually produce sound. It’s a pretty broad range of fields that I have had to come up to speed with pretty quickly.

How has Wu Tsai Neuro’s support and the Institute’s community helped to advance your work?

I'm part of the NeuroTech Training program at Wu Tsai Neuro, in addition to being a Ketterer-Vorwald Neurosciences Interdisciplinary Graduate Fellow. We have weekly seminars with different speakers from industry or from other institutions, which has been really cool. A lot of times I'll walk away with some idea that may be totally tangential to my own research, but has surprising applications in how we could analyze our own data, for example.

What do you see yourself doing next?

At the moment I'm thinking I want to do a postdoc either in this lab or a cousin lab. We're part of a consortium called BrainGate, and there’s a handful of labs around the country that do similar research. I definitely want to continue doing research in the prosthetic electrophysiology brain-computer interface field.

Kunz’s mentor, Krishna Shenoy, passed away following our interview — we circled back with her to ask about how his mentorship has influenced her work.

How did you first meet Krishna, and what made him stand out as a mentor?

I met Krishna during pre-Covid times when I joined Stanford for my Master's and I would attend his office hours to discuss the state of our field. Coming from industry, I was still feeling out the idea of doing a PhD, and Krishna was really inspirational to me. He was not only a great scientist, but a great mentor and a very approachable person.

You could tell he was doing this research and in this field because he enjoyed science and mentorship. His achievements came as a side effect of doing what he enjoyed, rather than chasing prestige. He used to say that he would stay in academia as long as it was fun, and he stayed in it his whole career. This really resonated with me — that’s the way I want to do science as well.

Can you share some examples of Krishna's mentorship and how he supported his lab?

Krishna always encouraged work-life balance. We’d often spend much of our regular one-on-one research meetings talking about how we were doing outside of lab and sometimes didn’t get to the science. We weren’t just his students — we were friends and colleagues.

I don’t know if he always had that balance for himself — I remember getting comments from him on my application for my Ketterer-Vorwald Neurosciences Interdisciplinary Graduate Fellowship at midnight on a Friday. But I got the sense that he genuinely enjoyed reviewing and commenting on our work, even late at night.

But he also reminded us that science would always be there. There was a time when I was overwhelmed juggling my research and outside responsibilities, and he helped me find perspective. He reminded me that science is only one part of life and encouraged me to take care of my personal responsibilities first because the science would always be there when I was ready for it.

Our lab gatherings at the Society for Neuroscience (SfN) annual meeting, where we would share embarrassing stories and strengthen friendships, demonstrated his emphasis on fostering strong relationships. After Krishna’s death, alumni from the lab have reached out to us to offer support both academically and personally. It speaks to how good of a mentor Krishna was. There’s a whole Shenoy Alumni Network that stays in contact and supports one another.

How do you feel about continuing the work without Krishna?

It’s bittersweet. ​​I’m working on a new project that’s awesome and cutting edge, but he doesn’t know about it anymore. I’d love to chat with him about the research and brainstorm through the highs and lows.

On the other hand I feel a sense of purpose in completing the projects in his honor, knowing his passion for the research we were doing.

A lot of us have been sharing the punchline Krishna always used in lab meetings during the darkest days of the pandemic. Whatever the bleak news of the day was, Krishna would say, ‘Now save us with science!’ That was how he saw science — even when times are tough, we can come together and take comfort in the work we are doing together.

Krishna Shenoy, 1968-2023. Photo by Rod Searcey.