High-Resolution, Non-Invasive Imaging of Upper Vocal Tract Articulators Compatible with Human Brain Recordings

PLoS One. 2016 Mar 28;11(3):e0151327. doi: 10.1371/journal.pone.0151327. eCollection 2016.

Abstract

A complete neurobiological understanding of speech motor control requires determination of the relationship between simultaneously recorded neural activity and the kinematics of the lips, jaw, tongue, and larynx. Many speech articulators are internal to the vocal tract, and therefore simultaneously tracking the kinematics of all articulators is nontrivial--especially in the context of human electrophysiology recordings. Here, we describe a noninvasive, multi-modal imaging system to monitor vocal tract kinematics, demonstrate this system in six speakers during production of nine American English vowels, and provide new analysis of such data. Classification and regression analysis revealed considerable variability in the articulator-to-acoustic relationship across speakers. Non-negative matrix factorization extracted basis sets capturing vocal tract shapes allowing for higher vowel classification accuracy than traditional methods. Statistical speech synthesis generated speech from vocal tract measurements, and we demonstrate perceptual identification. We demonstrate the capacity to predict lip kinematics from ventral sensorimotor cortical activity. These results demonstrate a multi-modal system to non-invasively monitor articulator kinematics during speech production, describe novel analytic methods for relating kinematic data to speech acoustics, and provide the first decoding of speech kinematics from electrocorticography. These advances will be critical for understanding the cortical basis of speech production and the creation of vocal prosthetics.

Publication types

  • Research Support, N.I.H., Extramural
  • Research Support, Non-U.S. Gov't
  • Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

  • Algorithms
  • Biomechanical Phenomena
  • Brain / anatomy & histology
  • Brain / physiology*
  • Brain Mapping
  • Diagnostic Imaging / methods*
  • Electrocorticography
  • Female
  • Humans
  • Jaw / anatomy & histology
  • Jaw / innervation
  • Jaw / physiology
  • Larynx / physiology
  • Lip / anatomy & histology
  • Lip / innervation
  • Lip / physiology
  • Male
  • Models, Neurological
  • Phonetics
  • Somatosensory Cortex / anatomy & histology
  • Somatosensory Cortex / physiology
  • Speech Acoustics*
  • Speech Production Measurement / methods
  • Tongue / anatomy & histology
  • Tongue / innervation
  • Tongue / physiology
  • Vocal Cords / anatomy & histology
  • Vocal Cords / innervation
  • Vocal Cords / physiology*