README.md
Orpheus TTS - Text-to-Speech Fine-tuning with Unsloth
This project demonstrates how to fine-tune the Orpheus 3B text-to-speech model using Unsloth for efficient training and inference.
Overview
Orpheus is a text-to-speech (TTS) model that converts text into natural-sounding speech. This implementation uses:
- Unsloth for efficient LoRA fine-tuning (30% less VRAM, 2x larger batch sizes)
- SNAC (Stochastic Neural Audio Codec) for audio tokenization at 24kHz
- Hugging Face Transformers for model training and inference
Features
- ✅ Fine-tune Orpheus 3B model with minimal GPU memory
- ✅ Support for single-speaker and multi-speaker TTS
- ✅ Expressive speech with emotion tags (laugh, sigh, gasp, etc.)
- ✅ Audio generation with customizable parameters
- ✅ Automatic WAV file export of generated speech
- ✅ LoRA adapter training for efficient fine-tuning
Installation
Prerequisites
- Python 3.8+
- CUDA-compatible GPU (recommended: 16GB+ VRAM)
- CUDA toolkit installed
Install Dependencies
IMPORTANT: The datasets package version must be between 3.4.1 and 4.0.0 for compatibility.
pip install sentencepiece protobuf "datasets>=3.4.1,<4.0.0" "huggingface_hub>=0.34.0" hf_transfer
pip install -r requirements.txt
For Colab or specific environments, you may need:
pip install --no-deps bitsandbytes accelerate xformers peft trl triton cut_cross_entropy unsloth_zoo
pip install --no-deps unsloth
Project Structure
orpheus/
├── orpheus_sft_unsloth.py # Full training + inference script
├── inference.py # Standalone inference script
├── requirements.txt # Python dependencies
├── README.md # This file
├── lora_model/ # Saved LoRA adapters (after training)
└── generated_audio/ # Generated audio outputs
Usage
Training
Fine-tune the model on your dataset:
python orpheus_sft_unsloth.py
The training script will:
- Load the pre-trained Orpheus 3B model
- Apply LoRA adapters for efficient fine-tuning
- Train on the MrDragonFox/Elise dataset (or your custom dataset)
- Save the fine-tuned LoRA adapters to
lora_model/
Training Parameters:
- Batch size: 1 per device
- Gradient accumulation: 4 steps
- Learning rate: 2e-4
- Training steps: 60 (configurable)
- LoRA rank: 64
Inference
Generate speech from text using the fine-tuned model:
python inference.py
Or use it as a module:
from inference import OrpheusInference
# Initialize the model
tts = OrpheusInference(
model_path="unsloth/orpheus-3b-0.1-ft",
lora_path="lora_model" # Optional: load fine-tuned adapters
)
# Generate speech with emotion tags
prompts = [
"Hey there my name is Elise, <giggles> and I'm a speech generation model.",
"I missed you <laugh> so much! It's been way too long.",
"This is absolutely amazing <gasp> I can't believe it worked!"
]
audio_files = tts.generate(
prompts=prompts,
output_dir="generated_audio",
temperature=0.6,
top_p=0.95,
max_new_tokens=1200
)
print(f"Generated {len(audio_files)} audio files")
Emotion Tags (Expressive Speech)
Orpheus supports special emotion/expression tags to create more expressive and natural-sounding speech:
Supported Tags:
- Laughter:
<laugh>,<giggles>,<chuckle> - Emotions:
<sigh>,<gasp> - Physical sounds:
<yawn>,<cough>,<sniffle>,<groan>
Usage:
prompts = [
"Hey there <giggles> welcome to my channel!",
"I missed you <laugh> so much!",
"That's so beautiful <sigh> it brings back memories.",
"I'm so tired <yawn> after working all day.",
"This is incredible <gasp> I can't believe my eyes!"
]
audio_files = tts.generate(prompts=prompts)
How it works:
- Tags are enclosed in angle brackets:
<tag> - During training, the model learns to associate these tags with audio patterns
- The Elise dataset contains 336 occurrences of "laughs", 156 of "sighs", etc.
- If your custom dataset lacks these tags, you can manually annotate transcripts where the audio contains those expressions
Multi-Speaker Support
For multi-speaker models, specify the voice name:
tts.generate(
prompts=["This is a test <laugh> with emotion."],
voice="speaker_name" # Specify the speaker
)
Dataset Format
The training script expects datasets with the following structure:
Single-speaker:
text: The text to be spokenaudio: Audio file witharrayandsampling_ratefields
Multi-speaker:
source: Speaker identifiertext: The text to be spokenaudio: Audio file witharrayandsampling_ratefields
Example dataset: MrDragonFox/Elise
Model Architecture
Audio Tokenization (SNAC)
- Sample rate: 24kHz
- Multi-layer hierarchical codec (3 layers)
- 7 tokens per frame (1 + 2 + 4 from the three layers)
- Duplicate frame removal for efficiency
Special Tokens
- Start of human: 128259
- End of human: 128260
- Start of AI: 128261
- End of AI: 128262
- Start of speech: 128257
- End of speech: 128258
- Pad token: 128263
Output
Generated audio files are saved as WAV files in the generated_audio/ directory:
- Format: WAV (PCM)
- Sample rate: 24kHz
- Naming:
output_0.wav,output_1.wav, etc.
Memory Usage
Typical memory requirements:
- Training: ~12-16GB VRAM (with LoRA and 4-bit quantization)
- Inference: ~8-10GB VRAM
- CPU RAM: ~16GB recommended
Troubleshooting
Common Issues
1. Dataset version error:
Ensure datasets>=3.4.1,<4.0.0 is installed
pip install "datasets>=3.4.1,<4.0.0" --force-reinstall
2. CUDA out of memory:
- Reduce
max_new_tokensduring inference - Use
load_in_4bit=Truewhen loading the model - Reduce batch size or enable gradient checkpointing
3. Multi-GPU issues:
- Set
CUDA_VISIBLE_DEVICES=0to use only one GPU per_device_train_batch_size >1may cause errors on multi-GPU setups
Performance Tips
- Faster Inference: Use
FastLanguageModel.for_inference(model)before generation - Memory Optimization: Enable 4-bit quantization with
load_in_4bit=True - Better Quality: Adjust
temperature(0.4-0.8) andtop_p(0.9-0.95) parameters - Longer Audio: Increase
max_new_tokens(each ~7 tokens = 1 audio frame)
Resources
License
This project uses models and libraries with their respective licenses:
- Unsloth: Apache 2.0
- Transformers: Apache 2.0
- Orpheus model: Check model card on Hugging Face
Citation
If you use this code in your research, please cite:
@misc{orpheus-tts-unsloth,
title={Orpheus TTS Fine-tuning with Unsloth},
author={Unsloth AI Team},
year={2024},
url={https://github.com/unslothai/unsloth}
}
Contributing
Contributions are welcome! Please feel free to submit issues or pull requests.
Acknowledgments
- Thanks to Etherl for creating the original notebook
- Unsloth AI team for the efficient training framework
- Hugging Face for hosting models and datasets