v1.6.0

rust-lang/rustlingsv1.6.0May 27, 2026by amirivojdan

AI Summary

This release introduces transliteration support for Farsi and Tajik, new text transforms for data augmentation and Persianization, and improved model download reliability through latency-based mirror selection. It also features a refactored conjugation system and localized UI enhancements.

Key Highlights

  • Farsi ↔ Tajik Transliteration using a quantized ByT5 model
  • Persianizer transform for loanword replacement with 1,700+ mappings
  • Text Noise transforms (Keyboard, OCR, Whitespace) for data augmentation
  • Rule-Based Informal Classifier for colloquial text detection
  • Iran Mirror with Latency-Based Selection for faster downloads

Breaking Changes

  • Dropped Python 3.10 support; package now requires Python 3.11+

New Features

  • FarsiToTajik and TajikToFarsi transliteration functions
  • Persianizer transform for auto-replacing foreign loanwords
  • NumberToWords transform for Persian number conversion
  • KeyboardNoise, OCRNoise, and WhitespaceNoise transforms
  • RuleBasedInformalClassifier for dependency-free detection

Full Release Notes

# What's New in v1.6.0

This release introduces transliteration support, a suite of new text transforms for data augmentation and Persianization, faster and more reliable model downloads, and a refactored conjugation system. It also includes several enhancements to the Shekar Studio Web UI.

## Highlights

- **Farsi ↔ Tajik Transliteration** – added a new `transliteration` module powered by a quantized ByT5 model. Use `FarsiToTajik` and `TajikToFarsi` (both exposed at the package top level) to convert between Persian (Arabic script) and Tajik (Cyrillic script).
- **Persianizer for Loanword Replacement** – the new `Persianizer` transform suggests and substitutes native Persian alternatives for foreign loanwords, backed by a curated mapping of 1,700+ entries sourced from beparsi.com. Use `Persianizer()` to auto-replace, or `.suggest()` to get ranked alternatives with positions.
- **NumberToWords Transform** – the new `NumberToWords` transform converts numeric digits (both Persian and Arabic-Indic) into their Persian word form (e.g. `۱۲۳` → `صد و بیست و سه`).
- **Text Noise Transforms for Data Augmentation** – three new transforms for synthetic noise generation, useful for training robust models and benchmarking:
  * `KeyboardNoise` – simulates typos based on Persian keyboard adjacency (substitution, insertion, deletion, repeat, shift).
  * `OCRNoise` – simulates OCR errors based on visual character confusions.
  * `WhitespaceNoise` – corrupts whitespace and ZWNJ structure (deletion or swapping between space ↔ ZWNJ).
  All three accept independent per-operation probabilities and a `seed` for reproducibility.
- **Rule-Based Informal Classifier** – added `RuleBasedInformalClassifier` for fast, dependency-free detection of colloquial/informal Persian text using keyword matching over informal vocabulary and conjugated verb forms. Inspired by [Persian-Informal-Text-Detector](https://github.com/MahtaFetrat/Persian-Informal-Text-Detector).
- **Iran Mirror with Latency-Based Selection** – the model hub now selects the fastest available mirror automatically (currently `shekar.ai` and `ir.shekar.ai`), significantly improving download speed for users in Iran and reducing failures when one mirror is unreachable.

- **Shekar Studio Enhancements** – the built-in web interface (`shekar serve`) now includes:
  * A new **Transliteration** panel with Persian (Tajik) support
  * Localized UI with **English** and **Persian (Tajik)** translations alongside Persian (Farsi)
- **Refactored Conjugation Engine** – conjugation logic has been split into separate formal and informal pipelines for cleaner code, better test coverage, and more accurate informal verb generation.

## Other Changes

- Dropped Python 3.10 support; the package now requires **Python 3.11+**.
- Bumped `onnxruntime` minimum to `>=1.26.0`.
- Changed `YaNormalizer` default to 'standard'.