Add compose v5 compatibility, legacy-field warnings, UI error fixes

[SohamKukreti]

Summary

Companion to the self-hosting docs update; all changes verified against a built 0.9.2 image.

docker-compose.yml:

• Express the PID cap as deploy.resources.limits.pids instead of pids_limit: Compose v5 normalizes the two together and rejected the file ("can't set distinct values on 'pids_limit' and 'deploy.resources.limits.pids'"), which also broke compose down. Identical behavior on Compose v2.x.

entrypoint.sh:

• When no CRAWL4AI_API_TOKEN is set, explain that the loopback bind is the container's own (published ports reset connections) and how to fix it. The old one-line notice read as "available at 127.0.0.1:11235" and sent users debugging the wrong thing.

server.py / schemas.py:

• /screenshot and /pdf: output_path (removed in 0.9.0) was silently dropped; requests still succeed but now return a warning that no file was written and results are fetched via GET /artifacts/{artifact_id}.
• Legacy hooks.code (removed in 0.9.0) parsed as an empty hook list and reported {"status": "success", "attached": []} with hooks enabled. The field is now captured (never executed) and the response reports status "ignored" plus a warning; /crawl/stream sends X-Hooks-Warning. With hooks disabled, any hooks payload still returns 403 (unchanged).
• Add "/" to the auth gate's exact-match public paths so the existing root -> /playground redirect is reachable; previously the middleware returned a bare 401 before the route ran. /monitor and all data routes remain gated.

static/playground/index.html:

• The streaming branch never checked response.ok and showed a green "Success" banner for 401 bodies; it now errors before reading the stream. All branches surface the server's detail message instead of a generic "Request failed", and 401s hint at the token bar (distinguishing "no token set" from "token rejected").

tests:

• Add deploy/docker/tests/test_legacy_compat.py covering the root redirect, output_path warning, legacy-hooks handling, and the compose PID-cap placement (13 behavioral tests, TestClient-based).

Fixes #2091

List of files changed and why

• deploy/docker/entrypoint.sh
• deploy/docker/schemas.py
• docker/server.py
• docker/static/playground/index.html
• deploy/docker/tests/test_legacy_compat.py
• docker-compose.yml

How Has This Been Tested?

Ran old tests and also created a new test file to test the changes. Manually verified as well.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[SetagGnaw]

Nice work on the legacy-compat warnings. One interaction seems worth a look.

The pre-existing gate in /crawl and /crawl/stream fires on the mere presence of a hooks object, without looking inside it:

if crawl_request.hooks and not HOOKS_ENABLED:
    raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")

crawl_request.hooks parses into a truthy HookConfig whether the payload carries declarative hooks, legacy code, or both, so all three shapes get the same 403 with the same advice. Before this PR the gate could not do better: pydantic dropped the unknown code field, so a code-only payload was indistinguishable from an empty one by the time it reached the gate. Your schema change is what makes the distinction expressible at all (hooks.code and hooks.hooks both survive parsing now), but the PR only uses it after the crawl, in the warning block.

Since CRAWL4AI_HOOKS_ENABLED defaults to false, that ordering means a legacy 0.8.x caller on a stock deployment never reaches the new hooks.status: "ignored" + warning path. They hit the 403 first, and its remedy is misleading for them twice over: enabling the flag won't make inline code run (it was removed entirely in 0.9.0), and it nudges operators into enabling the declarative-hooks surface just to appease code that can never execute. I see test_any_hooks_payload_403_when_disabled pins the 403-when-disabled behavior, so I take the gate itself as deliberate. The ask is only that the gate use the distinction your schema change just made possible.

Smallest version, no status codes change: keep firing on presence, but branch the error detail on what is actually inside, e.g.:

if crawl_request.hooks and not HOOKS_ENABLED:
    if crawl_request.hooks.code:
        raise HTTPException(
            403,
            "Inline hook code (hooks.code) was removed in 0.9.0 and cannot be "
            "enabled; it was not executed. Use declarative hook actions instead "
            "(GET /hooks/info), which are additionally disabled on this server "
            "(CRAWL4AI_HOOKS_ENABLED).",
        )
    raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")

Probably worth hoisting into a small helper, since the gate appears in both /crawl and /crawl/stream.

Related question: did you consider rejecting hooks.code payloads with a 400 in both cases (hooks enabled and hooks disabled), with a clear "removed in 0.9.0" error, instead of 200 + warning? A caller that never inspects the warning field reads the 200 as "my hook ran", which is the same silent-failure shape this PR is eliminating, just moved from the field level to the status level. Curious whether backward compatibility drove the accept-and-warn choice.

[SohamKukreti]

@SetagGnaw Good catch on the ordering. On a stock deployment (CRAWL4AI_HOOKS_ENABLED defaults to false) a legacy hooks.code caller hits the presence-based 403 and never reaches the "ignored" + warning path, and the 403's remedy is actively misleading for them: enabling the flag won't ever make inline code run.

I've adopted your suggested shape: keep firing on presence (the 403-when-disabled contract stays pinned by test_any_hooks_payload_403_when_disabled), branch only the detail on hooks.code, and hoist it into a small helper since the gate appears in both /crawl and /crawl/stream plus a test pinning the code-only 403 message so the two error texts don't drift back together.

On your 400 vs. 200 + warning question: I considered rejecting hooks.code payload with a 400, and backward compatibility did drive it, but with a specific shape in mind:

1. A 400 now would be a second breaking change for the same clients. Since 0.9.0 (two releases), a hooks.code payload with hooks enabled has returned 200 with complete, usable crawl results, the inline code never influenced the crawl in 0.9.x, so the results aren't degraded by its absence, only the hook side-effects are missing. Pipelines that adapted by simply not caring about hook effects (or that only ever used hooks for nice-to-haves like scroll/settle) currently work; a 400 would break them to protect callers who depend on hook effects, and those callers already see broken output, so they have a detection signal today.
2. Rejecting discards work the server can fully perform. The request minus code is a completely valid 0.9 crawl. That's also why mixed payloads argue against 400: declarative hooks + legacy code would be rejected wholesale even though everything except the dead field executes correctly.
3. It's consistent with the output_path decision in the same PR: serve the thing we can serve (crawl results / artifact), attach an explicit machine-readable signal about the part we can't (warning, status: "ignored", X-Hooks-Warning on the stream). One philosophy across both legacy fields.

Thanks for the review!