Starstruk Logo MarkStarstruk

v0.0.1

every-app/open-seov0.0.1Jun 27, 2026by chonknick

AI Summary

Initial public release of pulpie, a fast, CPU-friendly web content extraction tool using encoder models (Orange family) that achieves high fidelity to MinerU-HTML.

Key Highlights

  • Initial public release of pulpie
  • Fast, CPU-friendly extraction using small encoder models
  • MinerU-HTML parity verified by offline byte-parity and ROUGE-5 benchmarks
  • Three model sizes available: Small (210M), Base (610M), and Large (2.1B)
  • Support for clean Markdown and HTML output

New Features

  • Core extraction functionality
  • Markdown output support
  • HTML output support
  • Orange encoder models (Small, Base, Large variants)
  • MinerU-HTML integration and preprocessing
  • WebMainBench benchmarking capabilities

Full Release Notes

First public release of **pulpie** — fast, CPU-friendly web content extraction from HTML using encoder models.

## Install

```bash
pip install pulpie            # core
pip install pulpie[markdown]  # + markdown output
```

## What it does

pulpie extracts the main content from raw HTML and returns clean HTML or markdown. It classifies HTML blocks with small encoder models (the **Orange** family), reaching autoregressive-extractor quality at a fraction of the cost.

```python
from pulpie import Extractor

extractor = Extractor()             # downloads pulpie-orange-small (210M) on first use
result = extractor.extract(html)
print(result.markdown)              # clean markdown
print(result.html)                  # clean HTML
print(result.n_main, result.n_other)
```

## MinerU-HTML parity

pulpie's `simplify`/`reconstruct` faithfully reproduce MinerU-HTML's `simplify_html` / `map_to_main` (Apache-2.0), the segmentation format the Orange models were distilled on. Verified two ways:

- **Offline byte-parity:** 65/65 tests reproduce MinerU's exact output across 13 real-page fixtures.
- **End-to-end ROUGE-5** on full WebMainBench English (6,647 pages, orange-small):

  | Simplification | ROUGE-5 F1 |
  |----------------|-----------|
  | MinerU-HTML (reference) | 0.8625 |
  | **pulpie (this release)** | **0.8625** |

  Byte-faithful end-to-end (Δ vs MinerU = +0.0000), a +0.14 improvement over the prior simplification.

## Model zoo

| Name | HF repo | Size | Notes |
|------|---------|------|-------|
| Orange Small | `chonkie-ai/pulpie-orange-small` | 210M | Best value; default |
| Orange Base | `chonkie-ai/pulpie-orange-base` | 610M | Distilled |
| Orange Large | `chonkie-ai/pulpie-orange-large` | 2.1B | Highest quality |

## Credits

Built on MinerU-HTML / Dripper (Ma et al., 2025) — their `simplify_html` preprocessing, block-annotation scheme, and the WebMainBench benchmark are foundational. Thank you for releasing your tools and data.