feat: Add CrewAI integration for autonomous web crawling

[saturnabitsick]

Summary

This PR adds comprehensive CrewAI integration to Crawl4AI, enabling autonomous AI agents to use Crawl4AI for web scraping, data extraction, and research tasks. The integration provides CrewAI-compatible tools that seamlessly connect crawling capabilities with multi-agent workflows.

List of files changed and why

• crawl4ai/crewai_tool.py (NEW) - Core integration module providing 4 CrewAI tools: crawl_website(), extract_data_from_url(), crawl_multiple_urls(), and search_and_crawl(). Includes global crawler instance management and async-to-sync wrapper for compatibility.
• crawl4ai/init.py - Updated to export CrewAI tools with optional import (graceful fallback if CrewAI not installed).
• docs/crewai_integration.md (NEW) - Comprehensive user guide including installation, quick start, detailed tool documentation, 4+ use cases, best practices, performance tips, troubleshooting, and advanced examples.
• CREWAI_INTEGRATION.md (NEW) - Implementation architecture guide covering design decisions, feature overview, and integration points.
• INTEGRATION_SUMMARY.md (NEW) - Quick reference summary of the integration with key features, file structure, and next steps.
• examples/ (NEW directory) - 4 practical examples:
  • crewai_example_1_basic.py - Simple website crawling
  • crewai_example_2_multi_agent.py - Multi-agent research workflow
  • crewai_example_3_competitive_analysis.py - Competitive intelligence gathering
  • crewai_example_4_data_extraction.py - Structured data extraction
• examples/README.md (NEW) - Guide to all examples with setup instructions and tips.

How Has This Been Tested?

• Verified module syntax correctness
• Tested imports with and without CrewAI installed (optional dependency)
• All 4 tools properly decorated with @tool decorator for CrewAI compatibility
• Async-to-sync wrapper tested for function call conversion
• Error handling verified for edge cases
• Documentation examples reviewed for completeness
• Git commits verified and created successfully

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• New functionality is well-documented with examples
• Backward compatibility maintained - no breaking changes
• Optional dependency handling - CrewAI not required for core functionality