New brain implant turns visualized letters into text
Brian Stetka
When we move, sense or speak—or do just about anything—our brain generates a specific pattern of electrical activity. And for decades, scientists have been connecting those impulses to machines, not only to understand and treat brain diseases but also to help people with disabilities. Brain-computer interfaces, or BCIs, can restore movement in people with paralysis and may help treat neurological and psychiatric diseases.
The next frontier in BCIs may be things like the lowly text message; typing still poses maddeningly difficult challenges to bioengineers. A study published today in Nature reports on a brain implant that will allow people with impaired limb movement to communicate with text formulated in their mind—no hands needed.
Developed by a team at Stanford University, the artificial intelligence software, coupled with electrodes implanted in the brain, was able to “read” the thoughts of a man with full-body paralysis as he was asked to convert them to handwriting. The BCI transformed his imagined letters and words into text on a computer screen—a form of “mental handwriting.” The technology could benefit the millions of people worldwide who are unable to type or speak because of impaired limbs or vocal muscles.
As the paralyzed person in the study imagined writing a letter or symbol, sensors implanted in his brain detected patterns of electrical activity. An algorithm then interpreted these signals and traced the path of his imaginary pen. Credit: Frank Willett et al., Nature; 2021
Previous work by co-senior study author Krishna Shenoy of Stanford had helped analyze the neural patterns associated with speech. It also decoded imagined arm movements so that people with paralysis could move a cursor ploddingly on a keyboard screen to type out letters. But this technique only allowed them to type around 40 characters per minute, far lower than the average keyboard typing speed of around 190 characters per minute.
Shenoy’s team’s new work focused on imagined handwriting as a way to improve the speed of communication for the first time. And the researchers hope it will reach, at very least, smartphone texting rates. Their technique allowed the study subject, who was 65 years old at the time of the research, to mentally type 90 characters per minute. That rate is not far from average for most senior texters, who can typically type around 115 characters per minute on a phone.
“This line of work could help restore communication in people who are severely paralyzed, or ‘locked-in, ’” says Frank Willett, lead author of the paper and a research scientist at Stanford’s Neural Prosthetics Translational Laboratory. “It should help people express themselves and share their thoughts. It’s very exciting.”
The study participant suffered a spinal cord injury in 2007 and had lost most movement below his neck. In 2016 Stanford neurosurgeon Jaimie Henderson, co-senior author of the paper, implanted two small BCI chips into the patient’s brain. Each of the chips had 100 electrodes capable of sensing neuronal activity. They were implanted in a region of the motor cortex that controls movement of the arms and hands, allowing the researchers to profile brain-activity patterns associated with written language.
“This study is an important and clear advance for intracortical brain-computer interfaces,” says Amy L. Orsborn, a member of the department of bioengineering at the University of Washington. “One obvious reason why is because they achieved a huge leap in performance on a challenging but important task like typing. It’s also the most significant demonstration to date of leveraging established tools in machine learning like predictive language models to improve BCIs.”
“I saw this research initially presented at a poster in 2019 and think it’s great!”, says Mijail D. Serruya, an assistant professor of neurology at Thomas Jefferson University, who studies BCIs in stroke recovery but was not involved in the research. “I think it clearly shows that fine motor trajectories can be decoded from neocortical activity.”
Serruya adds that his research could align with Willett’s in helping to treat people who have suffered brain trauma or a stroke. “We have shown that motor control signals can be decoded [following a stroke], implying that some of the decoding approaches developed by Willett might have applications beyond people with spinal cord injury,” he says.
Yet Serruya also has one quibble with the new research—a hesitation he posed to Willett a few years ago: he believes that while focusing on restoring communication via written letters is intuitive, it may not be the most efficient means of doing so.
“Why not teach the person a new language based on simpler elementary gestures, similar to stenography chords or sign language?” Serruya asks. “This could both boost the speed of communication and, crucially, decrease the mental effort and attention needed.”
But for now, Willett is focused on mentally decoding our more familiar forms of communication—and he wants to repeat the typing experiment with other paralyzed people. He explains that while translating the brain’s control over handwriting is a significant first step in reclaiming someone’s ability to communicate, decoding actual speech—by analyzing what someone intends to say—is still a major challenge facing researchers, given that we generate speech much more quickly than we write or type.
“It’s been a hard problem to decode speech with enough accuracy and vocabulary size to allow people to have a general conversation. There’s a much higher signal-to-noise ratio, so it’s harder to translate to the computer,” Willett says. “But we’re now excited that we can decode handwriting very accurately. Each letter evokes a very different pattern of neural activity.”
As for when text-and-speech-decoding technology might be available to the public, Willett is cautiously optimistic. “It’s hard to predict when our method will be translated into a real device that anyone can buy,” he admits. “Of course, we hope it will be soon, and there are companies working on implantable BCI devices now. But you never know when someone will succeed in translating it. We hope it’s within years and not decades!”