Starstruk Logo MarkStarstruk

v0.4

modelscope/FunASRv0.4Aug 27, 2024by zkoch

AI Summary

Ultravox v0.4 upgrades the Whisper encoder from small to medium, expands multilingual support to nine languages, and significantly improves BLEU scores for speech translation.

Key Highlights

  • Whisper encoder upgraded from Whisper-small to Whisper-medium
  • Expanded multilingual support for ar, de, en, es, fr, it, ja, pt, and ru
  • Significant performance improvements in zero-shot speech translation metrics
  • Increased training data size to approximately 5k hours

New Features

  • Offline batch inference mode
  • Gradio demo updates to support text/voice conversation
  • AutoProcessor.from_pretrained support
  • bfloat16 default on MPS

Full Release Notes

Hey everyone,

We're releasing **Ultravox 0.4** today. The [weights](https://huggingface.co/fixie-ai/ultravox-v0_4) have been pushed to Hugging Face (along with updated datasets for training). If you're using the Ultravox APIs, v0.4 is the new default.

There are two key differences between 0.3 and 0.4:
* We've upgraded the Whisper encoder from Whisper-small to Whisper-medium
* We've trained on a larger set of multi-lingual data. Previous versions of Ultravox were only trained on English. Supported languages are now `ar`, `de`, `en`, `es`, `fr`, `it`, `ja`, `pt`, `ru`.

v0.4 builds upon the work in 0.3 and continues to show improved speech understanding. Our primary method of evaluation is zero-shot speech translation, measured by [BLEU](https://en.wikipedia.org/wiki/BLEU), as a proxy or general instruction-following capability (the higher the number the better). `ca` and `zh` are examples of model performance for languages not included in training.

|  | Ultravox 0.3 | Ultravox 0.4 |
| --- | ---: | ---: |
| **en_ar** | 9.07 | 28.07 |
| **en_de** | 22.67 | 25.60 |
| **es_en** | 24.10| 31.03 |
| **ru_en** | 22.52 | 38.96 |
| **en_ca** | 24.87 | 27.49 |
| **zh_en** | 4.26 | 10.08 |

This version of Ultravox continues to use a frozen Llama 3.1 8B pre-trained core, but we've roughly doubled the size of the data and the overall training time. The speech adapter was trained on ~5k hours of speech from LibriSpeech, Common Voice, Peoples Speech, and AnyInstruct. The training time on 8xH100s is roughly 170 minutes. We expect to increase the size of our training sets by 1-2 orders of magnitude over the next few months. For comparison, 0.3 was trained on ~2.5k hours of audio.

We'd love to hear feedback on your experience with Ultravox, along with feature suggestions. Roadmap coming soon.

## What's Changed
* Update gradio demo to support text/voice conversation by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/75
* Offline batch inference mode by @liPatrick in https://github.com/fixie-ai/ultravox/pull/82
* Live reload for Gradio demo by @juberti in https://github.com/fixie-ai/ultravox/pull/89
* Working AutoProcessor.from_pretrained by @farzadab in https://github.com/fixie-ai/ultravox/pull/92
* Use bfloat16 by default on MPS by @juberti in https://github.com/fixie-ai/ultravox/pull/95
* Add retry and filter in ds tool by @liPatrick in https://github.com/fixie-ai/ultravox/pull/81
* Change tokenizer padding_side to left for eval by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/96
* Make v0.4 release by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/99


**Full Changelog**: https://github.com/fixie-ai/ultravox/compare/v0.3...v0.4