Update model card with demo audio + eval results
Browse files
README.md
CHANGED
|
@@ -14,27 +14,17 @@ base_model: openbmb/VoxCPM2
|
|
| 14 |
|
| 15 |
LoRA adapter fine-tuned on VoxCPM2 for bedtime story narration in English.
|
| 16 |
|
| 17 |
-
## Training
|
| 18 |
-
- **Base
|
| 19 |
- **Dataset**: LJSpeech (6,550 clips, ~12h — half-data experiment)
|
| 20 |
- **Method**: LoRA (r=32, alpha=64, targeting DiT attention layers)
|
| 21 |
-
- **Best
|
| 22 |
-
- **Validation loss**: 0.872 (best across 6 experiments)
|
| 23 |
- **GPU**: A100-80GB via Modal
|
| 24 |
|
| 25 |
-
##
|
| 26 |
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
model = VoxCPM.from_pretrained("openbmb/VoxCPM2", device="cuda", load_denoiser=True, optimize=True)
|
| 31 |
-
# Apply LoRA weights manually or load via adapter
|
| 32 |
-
```
|
| 33 |
-
|
| 34 |
-
## Evaluation results
|
| 35 |
-
|
| 36 |
-
| Experiment | Config | Train Loss | Val Loss |
|
| 37 |
-
|-----------|--------|-----------|----------|
|
| 38 |
| exp1 | r32, lr=1e-4 | 0.823 | 0.918 |
|
| 39 |
| exp2 | r16, lr=5e-5 | 0.765 | 0.899 |
|
| 40 |
| exp3 | r64, lr=1e-4 | 0.838 | 0.908 |
|
|
@@ -42,5 +32,31 @@ model = VoxCPM.from_pretrained("openbmb/VoxCPM2", device="cuda", load_denoiser=T
|
|
| 42 |
| exp5 | r32, lr=2e-4 | 0.832 | 0.896 |
|
| 43 |
| **exp6** | **r32, lr=5e-5, half data** | **0.935** | **0.872** |
|
| 44 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
## Part of DreamVoice
|
| 46 |
-
|
|
|
|
| 14 |
|
| 15 |
LoRA adapter fine-tuned on VoxCPM2 for bedtime story narration in English.
|
| 16 |
|
| 17 |
+
## Training
|
| 18 |
+
- **Base**: `openbmb/VoxCPM2`
|
| 19 |
- **Dataset**: LJSpeech (6,550 clips, ~12h — half-data experiment)
|
| 20 |
- **Method**: LoRA (r=32, alpha=64, targeting DiT attention layers)
|
| 21 |
+
- **Best**: exp6 (r=32, lr=5e-5, half data) — val loss 0.872
|
|
|
|
| 22 |
- **GPU**: A100-80GB via Modal
|
| 23 |
|
| 24 |
+
## Evaluation (6 experiments)
|
| 25 |
|
| 26 |
+
| Exp | Config | Train Loss | Val Loss |
|
| 27 |
+
|-----|--------|-----------|----------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
| exp1 | r32, lr=1e-4 | 0.823 | 0.918 |
|
| 29 |
| exp2 | r16, lr=5e-5 | 0.765 | 0.899 |
|
| 30 |
| exp3 | r64, lr=1e-4 | 0.838 | 0.908 |
|
|
|
|
| 32 |
| exp5 | r32, lr=2e-4 | 0.832 | 0.896 |
|
| 33 |
| **exp6** | **r32, lr=5e-5, half data** | **0.935** | **0.872** |
|
| 34 |
|
| 35 |
+
## Demo Audio
|
| 36 |
+
|
| 37 |
+
### English bedtime stories (VoxCPM2)
|
| 38 |
+
|
| 39 |
+
| Mood | Audio |
|
| 40 |
+
|------|-------|
|
| 41 |
+
| Magical | [english_magical_mid.wav](samples/english_magical_mid.wav) |
|
| 42 |
+
| Funny | [english_funny_high.wav](samples/english_funny_high.wav) |
|
| 43 |
+
| Calming | [english_calming_low.wav](samples/english_calming_low.wav) |
|
| 44 |
+
| Dreamy | [english_dreamy_low.wav](samples/english_dreamy_low.wav) |
|
| 45 |
+
|
| 46 |
+
### LoRA vs Stock comparison
|
| 47 |
+
|
| 48 |
+
| Type | Audio |
|
| 49 |
+
|------|-------|
|
| 50 |
+
| Stock VoxCPM2 | [comparison_stock_0.wav](samples/comparison_stock_0.wav) |
|
| 51 |
+
| LoRA exp6 | [comparison_lora_exp6_0.wav](samples/comparison_lora_exp6_0.wav) |
|
| 52 |
+
|
| 53 |
+
### Reference voice
|
| 54 |
+
- [reference_voice.wav](samples/reference_voice.wav) — voice used for cloning
|
| 55 |
+
|
| 56 |
+
## Files
|
| 57 |
+
- `lora_weights.safetensors` — LoRA adapter weights (12.8 MB)
|
| 58 |
+
- `lora_config.json` — LoRA configuration
|
| 59 |
+
- `training_state.json` — Training state
|
| 60 |
+
|
| 61 |
## Part of DreamVoice
|
| 62 |
+
[DreamVoice](https://huggingface.co/spaces/build-small-hackathon/dreamvoice) — bedtime stories in a parent's cloned voice.
|