An NIH-funded research team at the University of California, San Francisco has developed a brain-computer interface that allows severely paralyzed people to communicate using both speech and upper-body gestures at the same timeโa first for the technology.
Using machine learning to interpret brain signals, the system translated thoughts from two participants into commands controlling a full-body virtual avatar. While brain-computer interfaces have previously enabled either speech or gesture individually, this is the first to combine both modes of expression simultaneously. "Conversation is about much more than the words being spoken. It's a multilayered, dynamic process involving the whole motor cortex," said Edward Chang, a professor of neurological surgery at UCSF and corresponding author of the research. "This proof-of-concept shows us it's possible for a BCI to restore some of this freedom and flexibility."
People with amyotrophic lateral sclerosis (ALS) or brainstem strokes often become severely paralyzed, losing verbal and non-verbal communication abilities. Current eye-tracking technology, the standard of care, enables text-to-speech but is slow, offers limited forms of expression, and can be exhausting to use.
The researchers implanted a thin strip of sensors called an electrocorticography array onto the motor cortex of patients. Computer models called decoders translated brain signals into commands controlling a digital avatar. For this study, participants attempted to speak specific phrases or perform common gestures such as hand waves or thumbs-up signs, both separately and concurrently. The team discovered that brain signals associated with simultaneous speech and gestures were fundamentally different from signals produced by either activity alone, and decoders trained on simultaneous data performed better than those trained on isolated speech or gestures.
Debara Tucci, director of NIH's National Institute on Deafness and Other Communication Disorders, said the results "give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication." The current system uses a wired connection to external processing units, but Chang said his team will soon test a fully implantable, wireless version with better long-term prospects.