Starstruk Logo MarkStarstruk

hailo-encoder-v0.1.0-iter156b

ruvnet/RuVectorhailo-encoder-v0.1.0-iter156bMay 3, 2026by ruvnet

AI Summary

Releases the first compiled Hailo-8 encoder HEF for the all-MiniLM-L6-v2 model on Pi 5 + AI HAT+ hardware.

Key Highlights

  • First known compiled `.hef` for `sentence-transformers/all-MiniLM-L6-v2`.
  • Achieves 67.3 embeds/sec throughput, 9.6x faster than CPU fallback.
  • Documents mask-aware vs CPU fallback behavior and accuracy differences.
  • Provides build provenance for the HEF file.

New Features

  • all-MiniLM-L6-v2 HEF binary
  • 9.6x throughput over CPU fallback
  • CPU fallback support
  • Build provenance documentation
  • Mask-aware vs CPU fallback notes

Full Release Notes

First-known compiled `.hef` for `sentence-transformers/all-MiniLM-L6-v2`
on the Hailo-8 NPU (Pi 5 + AI HAT+ target).

## Artifact

- **File**: `encoder.hef`
- **Size**: 15,758,361 bytes (~15 MB)
- **sha256**: `cdbc892765d3099f74723ee6c28ab3f0daade2358827823ba08d2969b07ebd40`
- **DFC**: v3.33.0
- **Hardware**: hailo8

## Architecture

Single-input encoder block: takes pre-computed FP32 hidden states
`[1, 128, 384]`, returns post-encoder FP32 `[1, 128, 384]`. Input
embedding lookup (`word + position + token_type + LayerNorm`) is
handled host-side via candle's `BertEmbeddings` reimplementation
in `crates/ruvector-hailo/src/host_embeddings.rs`. Post-NPU
mean-pool + L2-normalize handled in `inference.rs`.

## How to use

```bash
bash crates/ruvector-hailo-cluster/deploy/download-encoder-hef.sh \
    /var/lib/ruvector-hailo/models/all-minilm-l6-v2

# Then build worker with:
cargo build --release --features hailo,cpu-fallback \
    --bin ruvector-hailo-worker \
    --manifest-path crates/ruvector-hailo-cluster/Cargo.toml
```

## Performance

Measured on cognitum-v0 (Pi 5 + AI HAT+) via cluster-bench at
concurrency=4:

| Metric | Value | vs cpu-fallback |
|---|---:|---:|
| throughput | 67.3 / sec | 9.6× |
| p50 latency | 57 ms | 10× |
| p99 latency | 152 ms | 5.4× |
| cache hit (in-process) | 15.86 M / sec | 226,000× |

## Build provenance

Compiled iter 156b (commit `ffa3e90a6` on branch `hailo-backend`).
The four Hailo Dataflow Compiler v3.33 SDK bugs that blocked
transformer-encoder compilation are documented in ADR-167 + ADR-175,
all worked around from user-space:

1. KeyError on internal layer name → calibration dict keying
2. AccelerasValueError shape mismatch → NCHW reshape
3. ElementwiseAddDirectOp Keras deserialize → `acceleras` Layer
   `register_keras_serializable` monkey-patch (the breakthrough)
4. tf_rgb_to_hailo_rgb align → single-input encoder form

See `crates/ruvector-hailo-cluster/deploy/compile-encoder-hef.py`
to rebuild from source against your own corpus.

## Note on accuracy vs cpu-fallback

The single-input HEF runs full encoder attention with no padding
mask. cpu-fallback's `BertModel.forward` applies the real mask, so
the two embedders produce vectors in different spaces (cosine ~0.44
between matching texts). Both internally preserve semantic ordering
(`sim(close) > sim(far)` Δ=+0.23). The cluster's iter-143 fingerprint
separates HEF and cpu-fallback workers automatically so they never
mix in dispatch. A mask-aware HEF compile is documented as future
work in ADR-175.