Starstruk Logo MarkStarstruk

r1.3.0

the-dev-tools/dev-toolsr1.3.0Dec 14, 2022by SmileGoat

AI Summary

This release introduces large-scale pre-trained models like Whisper and Wav2vec2, enhances TTS with prosody prediction and adversarial loss, and moves audio processing to the dedicated paddleaudio library.

Key Highlights

  • Added Whisper CLI/Demos for multi-language recognition and translation
  • Added Wav2vec2-ASR (English and Chinese) CLI and demos
  • Added end-to-end Prosody Prediction pipeline for TTS
  • Added Adversarial Loss for Chinese-English mixed TTS
  • Moved audio processing from paddlespeech to paddleaudio

Breaking Changes

  • Moved paddlespeech/audio to paddleaudio

New Features

  • U2/U2++ Conformer dygraph to static conversion
  • Whisper CLI and Demos
  • Wav2vec2-ASR CLI and Demos
  • Streaming TTS fastdeploy serving
  • SSML for Chinese Text Frontend
  • Prosody Prediction pipeline
  • Adversarial Loss for mixed TTS
  • TTS Paddle-Lite x86 inference
  • TTS Android Demo

Full Release Notes

## HighLIght

### S2T
- Support U2/U2++ Conformer dy2static, and [U2/U2++ C++ High Performance Streaming ASR Deployment](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/speechx/examples/u2pp_ol/wenetspeech). @zh794390558 
- Add [Wav2vec2ASR-en](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/examples/librispeech/asr3), wav2vec2.0 fine-tuning for ASR on LibriSpeech.  @Zth9730 
- Add [Whisper CLI and Demos](https://github.com/PaddlePaddle/PaddleSpeech/pull/2640), support multi language recognition and translation.  @zxcd 
- Add [Wav2vec2 CLI and Demos](https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/demos/speech_ssl), support ASR and Feature Extraction.  @Zth9730 
- Add whisper. #2640 #2704 by @zxcd
- Fix gpu training hang. #2478 by @Zth9730 
- Support u2++ based cli and server. #2489 #2510 by @Zth9730 
- Add wav2vec2-en. #2518 #2527 #2637 by @Zth9730 
- Add wav2vec2-zh cli. #2697 by @Zth9730 

### T2S
- Add seek for BytesIO. https://github.com/PaddlePaddle/PaddleSpeech/pull/2484 by @ZapBird 
- Add mix finetune. https://github.com/PaddlePaddle/PaddleSpeech/pull/2525 https://github.com/PaddlePaddle/PaddleSpeech/pull/2647 by @lym0302 
- Add streaming TTS fastdeploy serving. https://github.com/PaddlePaddle/PaddleSpeech/pull/2528 by @HexToString 
- Add SSML for Chinese Text Frontend. https://github.com/PaddlePaddle/PaddleSpeech/pull/2531 by @david-95 
- Add [end-to-end Prosody Prediction pipeline](https://github.com/PaddlePaddle/PaddleSpeech/tree/develop/examples/csmsc/tts3_rhy) (including using prosody labels in Acoustic Model). https://github.com/PaddlePaddle/PaddleSpeech/pull/2548 https://github.com/PaddlePaddle/PaddleSpeech/pull/2615 https://github.com/PaddlePaddle/PaddleSpeech/pull/2693 by @WongLaw 
- Add [Adversarial Loss](https://arxiv.org/pdf/1907.04448.pdf) for [Chinese English mixed TTS](https://github.com/lym0302/PaddleSpeech/blob/develop/examples/zh_en_tts/tts3). https://github.com/PaddlePaddle/PaddleSpeech/pull/2588 by @lym0302
- Fix frontend bugs. https://github.com/PaddlePaddle/PaddleSpeech/pull/2539 https://github.com/PaddlePaddle/PaddleSpeech/pull/2606 by @yt605155624 
- Add TN for English unit. https://github.com/PaddlePaddle/PaddleSpeech/pull/2629 by @WongLaw 
- Add male voice for TTS. https://github.com/PaddlePaddle/PaddleSpeech/pull/2660 by @lym0302 
- Add double byte char for zh normalization. https://github.com/PaddlePaddle/PaddleSpeech/pull/2661 by @david-95 
- Add TTS Paddle-Lite x86 inference. https://github.com/PaddlePaddle/PaddleSpeech/pull/2636 https://github.com/PaddlePaddle/PaddleSpeech/pull/2667 by @yt605155624 
- Add greek char and fix #2571. https://github.com/PaddlePaddle/PaddleSpeech/pull/2683 by @david-95 
- Add Slim for TTS. https://github.com/PaddlePaddle/PaddleSpeech/pull/2729 by @yt605155624 

### Audio
- Move paddlespeech/audio to paddleaudio. https://github.com/PaddlePaddle/PaddleSpeech/pull/2706 by @SmileGoat 

### Demo
- Add TTSAndroid demo. https://github.com/PaddlePaddle/PaddleSpeech/pull/2703 by @yt605155624 

## New Contributors
* @ZapBird made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2484
* @HexToString made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2528
* @dahu1 made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2554
* @kFoodie made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2664
* @zxcd made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2640
* @michael-skynorth made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2666
* @heyudage made their first contribution in https://github.com/PaddlePaddle/PaddleSpeech/pull/2688

**Full Changelog**: https://github.com/PaddlePaddle/PaddleSpeech/compare/r1.2.0...r1.3.0