Tortoise TTS

Tortoise TTS is a multi-voice text-to-speech system that prioritizes high-quality audio generation with realistic prosody and intonation. The model's architecture innovatively combines techniques from image generation, specifically using both autoregressive transformers and diffusion models, adapting these for speech synthesis. As detailed in the architectural design document, the system consists of five distinct models working in concert.

The core architecture comprises four main components:

1. An autoregressive decoder that predicts speech token probability distributions based on text and reference audio
2. A contrastive model (CLVP) for output re-ranking
3. A denoising diffusion probabilistic model (DDPM) for converting speech tokens to high-quality spectrograms
4. A UnivNet vocoder for converting spectrograms to waveforms

A key innovation, dubbed the "Tortoise Trick," involves fine-tuning the DDPM on the autoregressive latent space to enhance both efficiency and output quality, as described in the research paper.

Training and Performance

Tortoise TTS was trained on an extensive dataset of approximately 50,000 hours of speech data. This included:

• LibriTTS and HiFiTTS datasets (896 hours combined)
• 49,000 hours of audiobooks and podcasts, transcribed using a fine-tuned wav2vec2-large model
• Training was conducted using the DLAS trainer on 8 NVIDIA RTX-3090 GPUs over a year

The model's generation speed was initially slow, taking about 2 minutes per medium-sized sentence on a K80 GPU. However, significant performance improvements have been achieved, reaching a real-time factor (RTF) of 0.25-0.3 on 4GB VRAM and sub-500ms latency with streaming capabilities.

Features and Usage

Version 2.1, released on May 2, 2022, introduced several key features:

• Random voice generation
• User-provided conditioning latents
• Custom pretrained model support
• Performance improvements

The model offers multiple usage methods:

• do_tts.py for single phrase generation
• read.py and read_fast.py for processing larger text files
• socket_server.py for socket streaming
• A programmatic API supporting deepspeed, kv cache, and float16 precision

Voice customization is achieved through reference audio clips, and the model includes prompt engineering capabilities for controlling emotional tone. Users can manipulate the voice latent space, enabling techniques like voice averaging.

Ethical Considerations and Impact

The model's ability to generate highly realistic voices has raised ethical concerns regarding potential misuse. To address this, the developers provide a classifier model, tortoise-detect, specifically designed to identify Tortoise-generated audio. The project encourages community feedback and collaboration to address these ethical considerations.

References

1. Tortoise Architectural Design Document - Detailed technical architecture
2. Tortoise TTS GitHub Repository - Source code and implementation
3. Research Paper - Academic publication describing the model
4. DLAS Trainer - Training framework used
5. Ocotillo Tool - Transcription tool for training data
6. Example Outputs - Demonstration of model capabilities

The model is licensed under the Apache 2.0 license, making it freely available for both research and commercial applications.