[Bug]: Docker requests can no longer specify deep crawl params

[markwainwright]

crawl4ai version

0.9.4

Expected Behavior

When calling the Docker /crawl endpoint I should be able to specify:

• Deep or single-page crawl
• filter_chain with params e.g. URL pattern filters
• url_scorer with params e.g. keywords
• max_pages and max_depth (clamped to a configurable limit to prevent DoS)

Current Behavior

Since 0.9.0 any requests with deep_crawl_strategy are rejected with 400.

This removes a ton of functionality that I'm currently relying on - for example:

• If I want to conditionally run single-page vs deep crawls, I would now need to run two docker containers in parallel, each statically configured to do each type.
• I currently dynamically generate URL path filters based on varying input URLs. This is now impossible because they cannot be defined statically.

According to the changelog, this was removed from the API to improve security. However, I believe this could be re-added with no increased security risk, beyond clamping a few values (e.g. max pages) to a server-configurable limit to prevent DoS - which seems to already be implemented in deploy/docker/governor.py.

An alternative would be to allow users to opt-out of such security limitations, for example in environments where they are already validating inputs before calling the API.

OS

Linux

Python version

3.12

Browser

No response

Browser version

No response

Error logs & Screenshots (if applicable)

No response

[SohamKukreti]

Thanks for raising this issue @markwainwright.
I've raised PR #2340 which adds an opt-in parameter CRAWL4AI_ALLOW_DEEP_CRAWL (off by default) and also patches some security holes in deep crawl.
Feel free to try it out and let us know if it works for you!