Meta Spirit LM is trained with a word-level interleaving method on speech and text datasets to enable cross-modality generation. We developed two versions of Spirit LM to display both the generative semantic abilities of text models and the expressive abilities of speech models. Spirit LM Base uses phonetic tokens to model speech, while Spirit LM Expressive uses pitch and style tokens to capture information about tone, such as whether it’s excitement, anger, or surprise, and then generates speech that reflects that tone

Spirit LM lets people generate more natural sounding speech, and it has the ability to learn new tasks across modalities such as automatic speech recognition, text-to-speech, and speech classification. We hope our work will inspire the larger research community to continue to develop speech and text integration.