v0.5.0.post1

unclecode/crawl4aiv0.5.0.post1Mar 4, 2025by unclecode

AI Summary

v0.5.0.post1 is a major release focusing on scalability and flexibility, featuring deep crawling strategies, memory-adaptive dispatching, browser profiling, LXML scraping mode, and Docker CLI support.

Key Highlights

  • Deep Crawling System with BFS, DFS, and BestFirst strategies
  • Memory-Adaptive Dispatcher for scaling to thousands of URLs
  • Multiple crawling strategies (Browser-based vs. HTTP-only)
  • Docker CLI (`crwl`) for convenient access
  • Browser Profiler for persistent authentication states

Breaking Changes

  • Several breaking changes (migration guidance required)

New Features

  • LLM Content Filter for intelligent markdown generation
  • URL redirection tracking
  • LLM-powered schema generation utility
  • Robots.txt compliance support
  • Proxy rotation with authentication and session persistence
  • PDF processing for local and remote files

Full Release Notes

# Crawl4AI v0.5.0.post1 Release

**Release Theme: Power, Flexibility, and Scalability**

Crawl4AI v0.5.0 is a major release focused on significantly enhancing the library's power, flexibility, and scalability. 

## Key Features

1. **Deep Crawling System** - Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies, with page limiting and scoring capabilities
2. **Memory-Adaptive Dispatcher** - Scale to thousands of URLs with intelligent memory monitoring and concurrency control
3. **Multiple Crawling Strategies** - Choose between browser-based (Playwright) or lightweight HTTP-only crawling
4. **Docker Deployment** - Easy deployment with FastAPI server, JWT authentication, and streaming/non-streaming endpoints
5. **Command-Line Interface** - New `crwl` CLI provides convenient access to all features with intuitive commands
6. **Browser Profiler** - Create and manage persistent browser profiles to save authentication states for protected content
7. **Crawl4AI Coding Assistant** - Interactive chat interface for asking questions about Crawl4AI and generating Python code examples
8. **LXML Scraping Mode** - Fast HTML parsing using the `lxml` library for 10-20x speedup with complex pages
9. **Proxy Rotation** - Built-in support for dynamic proxy switching with authentication and session persistence
10. **PDF Processing** - Extract and process data from PDF files (both local and remote)

## Additional Improvements

- LLM Content Filter for intelligent markdown generation
- URL redirection tracking
- LLM-powered schema generation utility for extraction templates
- robots.txt compliance support
- Enhanced browser context management
- Improved serialization and config handling

## Breaking Changes

This release contains several breaking changes. Please review the full release notes for migration guidance.

For complete details, visit: https://docs.crawl4ai.com/blog/releases/0.5.0/