v0.5.0.post1
unclecode/crawl4aiv0.5.0.post1Mar 4, 2025by unclecode
AI Summary
v0.5.0.post1 is a major release focusing on scalability and flexibility, featuring deep crawling strategies, memory-adaptive dispatching, browser profiling, LXML scraping mode, and Docker CLI support.
Key Highlights
- Deep Crawling System with BFS, DFS, and BestFirst strategies
- Memory-Adaptive Dispatcher for scaling to thousands of URLs
- Multiple crawling strategies (Browser-based vs. HTTP-only)
- Docker CLI (`crwl`) for convenient access
- Browser Profiler for persistent authentication states
Breaking Changes
- Several breaking changes (migration guidance required)
New Features
- LLM Content Filter for intelligent markdown generation
- URL redirection tracking
- LLM-powered schema generation utility
- Robots.txt compliance support
- Proxy rotation with authentication and session persistence
- PDF processing for local and remote files
Full Release Notes
# Crawl4AI v0.5.0.post1 Release **Release Theme: Power, Flexibility, and Scalability** Crawl4AI v0.5.0 is a major release focused on significantly enhancing the library's power, flexibility, and scalability. ## Key Features 1. **Deep Crawling System** - Explore websites beyond initial URLs with BFS, DFS, and BestFirst strategies, with page limiting and scoring capabilities 2. **Memory-Adaptive Dispatcher** - Scale to thousands of URLs with intelligent memory monitoring and concurrency control 3. **Multiple Crawling Strategies** - Choose between browser-based (Playwright) or lightweight HTTP-only crawling 4. **Docker Deployment** - Easy deployment with FastAPI server, JWT authentication, and streaming/non-streaming endpoints 5. **Command-Line Interface** - New `crwl` CLI provides convenient access to all features with intuitive commands 6. **Browser Profiler** - Create and manage persistent browser profiles to save authentication states for protected content 7. **Crawl4AI Coding Assistant** - Interactive chat interface for asking questions about Crawl4AI and generating Python code examples 8. **LXML Scraping Mode** - Fast HTML parsing using the `lxml` library for 10-20x speedup with complex pages 9. **Proxy Rotation** - Built-in support for dynamic proxy switching with authentication and session persistence 10. **PDF Processing** - Extract and process data from PDF files (both local and remote) ## Additional Improvements - LLM Content Filter for intelligent markdown generation - URL redirection tracking - LLM-powered schema generation utility for extraction templates - robots.txt compliance support - Enhanced browser context management - Improved serialization and config handling ## Breaking Changes This release contains several breaking changes. Please review the full release notes for migration guidance. For complete details, visit: https://docs.crawl4ai.com/blog/releases/0.5.0/