v0.9.0

tracel-ai/burnv0.9.0Sep 6, 2023by nathanielsimard

AI Summary

Launch of the Burn Book and Model repository, introduction of three new optimizers, and numerous new tensor operations and training metrics.

Key Highlights

Launch of Burn Book and Model repository.
Three new optimizers: AdamW, AdaGrad, RMSProp.
New tensor operations including conv_transpose and adaptive pooling.
New training metrics (CPU/GPU temperature, memory use).

New Features

Burn Book documentation.
Model repository (SqueezeNet, Llama 2, Whisper, Stable Diffusion).
Optimizers: AdamW, AdaGrad, RMSProp.
Tensor ops: cast, clamp, abs, max_pool, adaptive_avg_pool, conv_transpose, not, dim iterator.
Training metrics (temperature, memory).
WGPU Autotune and Matmul optimization.
ONNX support (reshape, transpose, binary, concat, dropout, avg pool, softmax, conv1d/2d, tanh, clip).

Full Release Notes

Burn v0.9.0 sees the addition of the Burn Book, a new model repository, and many new operations and optimizations.

# Burn Book

The Burn Book is available at https://burn-rs.github.io/book/

- Burn Book setup and plan @nathanielsimard @wdoppenberg @antimora
- Motivation & Getting started @louisfd @nathanielsimard
- Basic Workflow: from training to inference @nathanielsimard @louisfd
- Building blocks @nathanielsimard
- ONNX models @antimora
- Advanced sections @nathanielsimard

# Model repository

The Model repository is available at https://github.com/burn-rs/models

- Setup @nathanielsimard
- Add SqueezeNet @antimora
- Multiple models made with Burn @gadersd
  - Llama 2
  - Whisper
  - Stable Diffusion v1.4

# Changes to Burn

## Neural networks

- Three new optimizers
  - AdamW @wdoppenberg
  - AdaGrad @CohenAriel
  - RMSProp @AuruTus
- Custom initializer for transformer-related modules @wbrickner
- Cross Entropy with label smoothing and weights @ArvidHammarlund

## Tensors

- Many new operators
  - cast @trfdeer @nathanielsimard
  - clamp, clamp_min, clamp_max @antimora
  - abs @mmalczak
  - max_pool1d, max_pool with dilation @caiopiccirillo
  - adaptive_avg_pool 1d and 2d @nathanielsimard
  - conv_transpose 1d and 2d, with backward @nathanielsimard
  - Not operator @louisfd
  - Dim iterator @ArvidHammarlund
- More tests for basic tensor ops @louisfd


## Training

- New training metrics @Elazrod56
  - CPU temperature and use
  - GPU temperature
  - Memory use
- Custom training and validation metric loggers @nathanielsimard
- Migration from log4rs to tracing, better integration in a GUI app @dae
- Training interruption @dae
- New custom optimize method @nathanielsimard

## Backends

- WGPU backend
  - Autotune @louisfd @nathanielsimard
    - Cache optimization @agelas
  - Pseudo-random number generator @louisfd
  - Fix configs @nathanielsimard
  - Matmul optimization @louisfd
- ndarray backend
  - Optimization of argmin/argmax @DrChat
  - Optimization of conv2d @DrChat
- Candle backend @louisfd
  - Support for all basic operations
  - Work in progress

## Dataset

- Option for with or without replacement in dataset sampler @nathanielsimard

## Import & ONNX

- Refactor, performance, tests and fixes @antimora @Luni-4 @nathanielsimard, @gadersd
- New operators @Luni-4 @antimora @AuruTus
  - Reshape
  - Transpose
  - Binary operators
  - Concat
  - Dropout
  - Avg pool
  - Softmax
  - Conv1d, Conv2d
  - Scalar and constants
  - tanh
  - clip

## Fix

- Hugging Face downloader Windows support @Macil
- Fix grad replace and autodiff backward broadcast @nathanielsimard
- Fix processed count at learning completion @dae
- Adjust some flaky tests @dae
- Ability to disable experiment logging @dae

## Configuration

- Rewrite publish and checks scripts in Rust, with cargo-xtask @luni-4 @DrChat
- Add Typos verification to checks @caiopiccirillo @antimora
- Checks for Python and venv environment @mashirooooo
- Feature flags for crates in different scenarios @dae

## Documentation

- Configuration doc for vscode environment setup @caiopiccirillo
- Jupyter notebook examples @antimora
- Readme updated @louisfd

# Thanks

Thanks to all aforemetioned contributors and to our sponsors @smallstepman and @premAI-io.