v3.10.20

ruvnet/ruflov3.10.20May 30, 2026by ruvnet

AI Summary

Release v3.10.20 introduces an optional cross-encoder reranker that significantly improves search relevance, pushing top-1 hit rate from 80% to 90% and top-3 hit rate from 80% to 100%. The feature uses a lazy-loaded MS-MARCO model combined with the existing hybrid approach via a weighted combination (default 0.5 hybrid + 0.5 cross-encoder), though it comes with ~25× latency cost (984ms vs 39ms).

Key Highlights

Cross-encoder reranker pushes top-1 hit rate from 80% → 90% and top-3 from 80% → 100% on evaluation harness
Combined hybrid + cross-encoder (0.5:0.5) outperforms either approach alone in ablation studies
Lazy-loaded model via @xenova/transformers with graceful degradation when unavailable
MRR@3 improved from 0.800 to 0.933
New MCP tool parameters: rerank, hybridWeight, ceWeight for fine-tuning relevance vs latency tradeoff

New Features

Cross-encoder reranker using Xenova/ms-marco-MiniLM-L-6-v2 model (int8, ~30MB)
Lazy-loaded singleton via AutoTokenizer + AutoModelForSequenceClassification
One-shot load policy - returns null immediately after failed load, no retry loops
New neural_patterns MCP tool params: rerank (boolean), hybridWeight (number), ceWeight (number)
Response includes crossEncoderScore when rerank is enabled
5 new tests covering graceful-degradation contract
Support for both single-logit (sigmoid) and binary-logit (softmax) model heads

Full Release Notes

## What ships

**Cross-encoder reranker** (opt-in) — Xenova/ms-marco-MiniLM-L-6-v2 (int8,
~30MB) lazy-loaded via @xenova/transformers, gracefully degrading when
unavailable. Pushes top-1 from 80% → 90% and top-3 from 80% → **100%** on
the same A/B harness.

## Cumulative SOTA push (3.10.17 → 3.10.20)

| Metric | 3.10.17 cosine | 3.10.18 hybrid | 3.10.19 multi-field | **3.10.20 +rerank** |
|---|---:|---:|---:|---:|
| **Top-1 hit rate** | 0% | 50% | 80% | **90%** |
| **Top-3 hit rate** | 0% | 70% | 80% | **100%** |
| **MRR@3** | 0.000 | 0.583 | 0.800 | **0.933** |
| Top-1 diversity | 100% | 80% | 100% | **100%** |
| Avg query latency | 29 ms | 41 ms | 39 ms | 984 ms (opt-in) |

## The ablation that drove the architecture

| Configuration | Top-1 | Top-3 | MRR@3 |
|---|:---:|:---:|:---:|
| Hybrid only (3.10.19) | 8/10 | 8/10 | 0.800 |
| Cross-encoder alone (over top-30 pool) | 6/10 | **10/10** | 0.733 |
| **Combined 0.5·hybrid + 0.5·CE (3.10.20 default)** | **9/10** | **10/10** | **0.933** |

Cross-encoder alone finds *all* relevant docs in top-3 but loses top-1 —
MS MARCO's calibration on short commit subjects is noisy. Hybrid is the
opposite: strong top-1, weaker top-3. Linear combination captures both.

Weight grid-search confirms a broad plateau:

| hybrid : ce | top-1 | top-3 | MRR@3 |
|---|:---:|:---:|:---:|
| 0.5 : 0.5 (default) | 9/10 | 10/10 | 0.933 |
| 0.4 : 0.6 | 9/10 | 10/10 | 0.933 |
| 0.3 : 0.7 | 9/10 | 10/10 | 0.933 |

## Why opt-in

Latency cost is ~25× hybrid (1.0 s vs 39 ms per query at N=385). The default
hybrid path stays for hot paths and batch retrieval. Callers needing SOTA
relevance flip `{rerank: true}` per call.

## What changed in code

1. **`src/memory/cross-encoder-rerank.ts`** — lazy-loaded singleton via direct
   `AutoTokenizer` + `AutoModelForSequenceClassification`. The xenova v2
   `pipeline('text-classification')` API can't handle `{text, text_pair}` pairs
   reliably; the lower-level API does. Handles single-logit (sigmoid) AND
   binary-logit (softmax) heads.

2. **One-shot load policy** — after a failed load, subsequent calls return
   null immediately. No retry loops in hot paths.

3. **`neural_patterns` MCP tool** — three new params:
   - `rerank: boolean` (default false)
   - `hybridWeight: number` (default 0.5)
   - `ceWeight: number` (default 0.5)
   - Response includes `crossEncoderScore` when rerank is on.

4. **5 new tests** in `__tests__/cross-encoder-rerank.test.ts` covering the
   graceful-degradation contract (no network needed — forces failure with a
   guaranteed-bad model name).

## Reproduce

```bash
git clone https://github.com/ruvnet/ruflo && cd ruflo
npm install && ( cd v3/@claude-flow/cli && npx tsc )

# Unit tests (no network) — 44 total
( cd v3/@claude-flow/cli && npx vitest run __tests__/cross-encoder-rerank.test.ts __tests__/hybrid-retrieval.test.ts __tests__/pretrain-from-github.test.ts )

# Live A/B (cross-encoder downloads ~30MB on first run)
cd v3/@claude-flow/cli
node scripts/pretrain-from-github.mjs
node scripts/benchmark-pretrained-retrieval.mjs              # 3.10.19 default → 80% top-1
RERANK=1 node scripts/benchmark-pretrained-retrieval.mjs     # 3.10.20 + rerank → 90%/100%
HYBRID=0 node scripts/benchmark-pretrained-retrieval.mjs     # cosine baseline → 0%
```

## Honest limits

- N=385, 10 queries, regex-relevance proxy. Direction (0% → 90% top-1) is
  robust to noise; absolute numbers could shift on a different corpus. A
  labelled held-out evaluation is the right next gauge.
- 30 MB cross-encoder model downloads on first run. Subsequent runs hit
  local cache.
- The remaining 10% top-1 gap is one query that the regex can't see clearly
  — may be genuinely ambiguous or a regex-proxy artefact.

## What's next

- **Labelled held-out corpus** for tighter relevance confidence intervals
- **Larger cross-encoder** (ms-marco-MiniLM-L-12-v2) if quality matters more
  than latency
- **Learned distiller** (#2241 round-D) — still tracked

## Install

```bash
npx ruflo@3.10.20    # latest / alpha / v3alpha all aligned
```

Full ADR: [v3/docs/adr/ADR-080-cross-encoder-reranker.md](v3/docs/adr/ADR-080-cross-encoder-reranker.md)