v0.3

Hey everyone,

We're officially making **Ultravox 0.3** available today. The [weights](https://huggingface.co/fixie-ai/ultravox-v0_3) have been pushed to Hugging Face (along with updated datasets for training), and the model training code has been updated as well. We’re also opening up early preview access to our Ultravox APIs through our managed service. For more information on that, please go here: [https://fixie-ai.github.io/ultradox/](https://fixie-ai.github.io/ultradox/)

v0.3 demonstrates substantially improved speech understanding. Our primary method of evaluation is zero-shot speech translation, measured by [BLEU](https://en.wikipedia.org/wiki/BLEU), as a proxy or general instruction-following capability (the higher the number the better):

|  | Ultravox 0.2 | Ultravox 0.3 |
| --- | --- | --- |
| **en_de** | 12.07 | 22.68 |
| **es_en** | 15.17 | 24.10 |

This version of Ultravox uses a frozen Llama 3.1 8B pre-trained core. The speech adapter was trained on 2.5k hours of speech from both LibriSpeech and CommonVoice. The training time on 8xH100s is roughly 80 minutes. We expect to increase the size of our training sets by 1-2 orders of magnitude over the next few months. For comparison, 0.2 was trained on ~1.5k hours of audio.

In addition to increasing the overall size of the training set, v0.3 also introduces two other important changes. The first is that we’re augmenting the ASR data sets with synthetic data in the form of generated continuations. The second change is that we’ve migrated to a Knowledge Distillation approach for calculating loss. Combined, both of these approaches result in much higher speech to text alignment in the adapter. You can learn more in their [respective](https://arxiv.org/abs/2309.00916) [papers](https://arxiv.org/abs/2405.19041).

The key benefit of better adapter alignment is that it makes it easier to customize Ultravox to particular needs and use cases by allowing it to extend any pre-trained LLM (including fine-tuned versions) with speech capabilities while retaining core capabilities across modalities. If this is something that interests you, please [get in touch](mailto:hello@fixie.ai).

We’d love feedback on the model, so please let us know what works well and what doesn’t. To make testing easier, we built a new Gradio demo. To run it, simply run `just gradio` inside of the Ultravox folder.

## What's Changed
* Remove legacy directory by @farzadab in https://github.com/fixie-ai/ultravox/pull/1
* Improved Evaluations by @farzadab in https://github.com/fixie-ai/ultravox/pull/2
* Audio Encoder to bfloat16 by @farzadab in https://github.com/fixie-ai/ultravox/pull/4
* Whisper encoder + No 30 second padding by @farzadab in https://github.com/fixie-ai/ultravox/pull/5
* Optionally include "passage" in BoolQ samples by @juberti in https://github.com/fixie-ai/ultravox/pull/6
* Add tts_tool, for converting a HF dataset to audio by @juberti in https://github.com/fixie-ai/ultravox/pull/12
* Add logging code by @juberti in https://github.com/fixie-ai/ultravox/pull/19
* Fixes the default HF model name by @cezarc1 in https://github.com/fixie-ai/ultravox/pull/13
* Update Hugging Face link by @simonw in https://github.com/fixie-ai/ultravox/pull/17
* Don't run tests on docs changes by @juberti in https://github.com/fixie-ai/ultravox/pull/21
* Local tokenizer and processor for more consistent CI by @farzadab in https://github.com/fixie-ai/ultravox/pull/16
* Tool for uploading to HF Hub by @farzadab in https://github.com/fixie-ai/ultravox/pull/15
* Remove mlflow dependency by @juberti in https://github.com/fixie-ai/ultravox/pull/23
* Switch from Pip to Poetry by @juberti in https://github.com/fixie-ai/ultravox/pull/24
* Tool for adding new synthetic columns by @farzadab in https://github.com/fixie-ai/ultravox/pull/14
* entails -> provides a rationale for by @farzadab in https://github.com/fixie-ai/ultravox/pull/27
* Add @file syntax to ds_tool by @juberti in https://github.com/fixie-ai/ultravox/pull/28
* datasets: Handle converting `int16` audio data in `VoiceSample`. by @shaper in https://github.com/fixie-ai/ultravox/pull/26
* Allow for toggling training and eval on/off by @farzadab in https://github.com/fixie-ai/ultravox/pull/29
* Add Eleven and Fireworks support to ds_tool by @juberti in https://github.com/fixie-ai/ultravox/pull/31
* Don't fail basic inference due to missing OAI key by @juberti in https://github.com/fixie-ai/ultravox/pull/34
* BoolQ for Training and Eval by @farzadab in https://github.com/fixie-ai/ultravox/pull/30
* Extending `ds_tool` for SODA conversational dataset by @farzadab in https://github.com/fixie-ai/ultravox/pull/32
* Add streaming support, using HF TextStreamer by @juberti in https://github.com/fixie-ai/ultravox/pull/46
* Minor fixes to ds_tool and infer_tool by @juberti in https://github.com/fixie-ai/ultravox/pull/36
* SODA Dataset for Training by @farzadab in https://github.com/fixie-ai/ultravox/pull/35
* HF pipeline to run Ultravox independent of Ultravox repo by @farzadab in https://github.com/fixie-ai/ultravox/pull/49
* Runs Tags for filtering by @farzadab in https://github.com/fixie-ai/ultravox/pull/51
* More validations by @farzadab in https://github.com/fixie-ai/ultravox/pull/48
* CoVoST 2 dataset by @farzadab in https://github.com/fixie-ai/ultravox/pull/53
* Speech Translation Evals by @farzadab in https://github.com/fixie-ai/ultravox/pull/54
* Update ds_tool.py by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/52
* Llama3.1 by @farzadab in https://github.com/fixie-ai/ultravox/pull/56
* Make so infer_tools works with a single arg for filename by @cdiddy77 in https://github.com/fixie-ai/ultravox/pull/55
* HF Model loading fixes by @farzadab in https://github.com/fixie-ai/ultravox/pull/59
* Separate files for eval logs by @farzadab in https://github.com/fixie-ai/ultravox/pull/61
* Add "without any explanation" to ST prompt by @farzadab in https://github.com/fixie-ai/ultravox/pull/60
* Support KL loss by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/63
* [ds_tool] Tools with Audio by @farzadab in https://github.com/fixie-ai/ultravox/pull/62
* Add basic data_processing test by @juberti in https://github.com/fixie-ai/ultravox/pull/64
* Add generic dataset by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/67
* Fix TypeError: non-default argument 'template' follows default argument, and filter out audio by @liPatrick in https://github.com/fixie-ai/ultravox/pull/69
* Filter out audio in map sample by @liPatrick in https://github.com/fixie-ai/ultravox/pull/72
* Update caching to use prefix by @liPatrick in https://github.com/fixie-ai/ultravox/pull/76
* Add weighted sampling in InterleaveDataset by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/70
* Update default config to ultravox_v0.3 by @zqhuang211 in https://github.com/fixie-ai/ultravox/pull/84

## New Contributors
* @cezarc1 made their first contribution in https://github.com/fixie-ai/ultravox/pull/13
* @simonw made their first contribution in https://github.com/fixie-ai/ultravox/pull/17
* @shaper made their first contribution in https://github.com/fixie-ai/ultravox/pull/26
* @cdiddy77 made their first contribution in https://github.com/fixie-ai/ultravox/pull/55

**Full Changelog**: https://github.com/fixie-ai/ultravox/commits/v0.3