vr0.6.0

unclecode/crawl4aivr0.6.0Apr 22, 2025by unclecode

AI Summary

v0.6.0 introduces world-aware crawling with geolocation and locale support, table extraction to CSV/pandas, browser pooling for performance, and network/console logging capabilities.

Key Highlights

World-aware crawling (geolocation, timezone, locale support)
Table-to-DataFrame extraction for CSV export
Crawler pool with pre-warming for lower latency
Network and console log capture with MHTML snapshots
MCP socket and SSE endpoints with playground UI

Breaking Changes

Browser strategy consolidation; legacy docker modules removed
`ProxyConfig` moved to `async_configs`
Server migrated to pool-based crawler management
FastAPI validators replace custom query validation
Docker build now uses Chromium base image

New Features

Geolocation configuration with accuracy settings
Browser pooling manager in SDK and Docker API
Table extractor exporting to CSV or pandas
Stress-test framework for 1k+ URL runs
Docs v2 updates with TOC and code buttons

Full Release Notes

## 🚀 0.6.0 — 22 Apr 2025

### Highlights
1. **World‑aware crawlers**:
```python
crun_cfg = CrawlerRunConfig(
        url="https://browserleaks.com/geo",          # test page that shows your location
        locale="en-US",                              # Accept-Language & UI locale
        timezone_id="America/Los_Angeles",           # JS Date()/Intl timezone
        geolocation=GeolocationConfig(                 # override GPS coords
            latitude=34.0522,
            longitude=-118.2437,
            accuracy=10.0,
        )
    )
```

2. **Table‑to‑DataFrame** extraction, flip `df = pd.DataFrame(result.media["tables"][0]["rows"], columns=result.media["tables"][0]["headers"])
` and get CSV or pandas without extra parsing.  
3. **Crawler pool with pre‑warm**, pages launch hot, lower P90 latency, lower memory.  
4. **Network and console capture**, full traffic log plus MHTML snapshot for audits and debugging.  

### Added
- Geolocation, locale, and timezone flags for every crawl.
- Browser pooling with page pre‑warming.
- Table extractor that exports to CSV or pandas.
- Crawler pool manager in SDK and Docker API.
- Network & console log capture, plus MHTML snapshot.
- MCP socket and SSE endpoints with playground UI.
- Stress‑test framework (`tests/memory`) for 1 k+ URL runs.
- Docs v2: TOC, GitHub badge, copy‑code buttons, Docker API demo.
- “Ask AI” helper button, work in progress, shipping soon.
- New examples: geo location, network/console capture, Docker API, markdown source selection, crypto analysis.

### Changed
- Browser strategy consolidation, legacy docker modules removed.
- `ProxyConfig` moved to `async_configs`.
- Server migrated to pool‑based crawler management.
- FastAPI validators replace custom query validation.
- Docker build now uses a Chromium base image.
- Repo cleanup, ≈36 k insertions, ≈5 k deletions across 121 files.

### Fixed
- Session leaks, duplicate visits, URL normalisation.
- Target‑element regressions in scraping strategies.
- Logged URL readability, encoded URL decoding, middle truncation.
- Closed issues: #701 #733 #756 #774 #804 #822 #839 #841 #842 #843 #867 #902 #911.

### Removed
- Obsolete modules in `crawl4ai/browser/*`.

### Deprecated
- Old markdown generator names now alias `DefaultMarkdownGenerator` and warn.

### Upgrade notes
1. Update any imports from `crawl4ai/browser/*` to the new pooled browser modules.  
2. If you override `AsyncPlaywrightCrawlerStrategy.get_page` adopt the new signature.  
3. Rebuild Docker images to pick up the Chromium layer.  
4. Switch to `DefaultMarkdownGenerator` to silence deprecation warnings.

`121 files changed, ≈36 223 insertions, ≈4 975 deletions`