Episode 105 - Bionic Babble, From Silence to Speech

Image from Willett, FR, et. al. A high-performance speech neuroprosthesis. Nature (2023). Diagram of the decoding algorithm. First, neural activity (multiunit threshold crossings and spike band power) is temporally binned and smoothed on each electrode. Second, a recurrent neural network (RNN) converts a time series of this neural activity into a time series of probabilities of each phoneme (plus the probability of an interword ‘silence’ token and a ‘blank’ token associated with the connectionist temporal classification training procedure). The RNN is a five-layer, gated recurrent unit architecture trained using TensorFlow 2. Finally, phoneme probabilities are combined with a large-vocabulary language model (a custom, 125,000-word trigram model implemented in Kaldi) to decode the most probable sentence. Phonemes in this diagram are denoted using the International Phonetic Alphabet.

Neuroscientists have developed a bionic device powered by AI that helps aphasic patients regain their speech, turning futuristic concepts into reality.¹ This innovative approach is revolutionizing treatment and offering hope to many.

1. Willett, F.R., Kunz, E.M., Fan, C. et al. A high-performance speech neuroprosthesis. Nature (2023). https://doi.org/10.1038/s41586-023-06377-x