Feat/issue 42 agent first web proxy

[rizzoMartin]

Description

Adds data_engineering/semantic_web_proxy, a deterministic proxy that reduces a web page to its semantic core before it reaches a context window.

Raw HTML is mostly not content. Scripts, styling, navigation, consent banners and footer link farms dominate a typical page, and every one of those bytes costs tokens and dilutes model attention. The skill fetches a public http(s) URL behind an SSRF guard (or accepts HTML the host already holds), strips everything non-semantic with trafilatura, and returns concentrated Markdown, plain text, or JSON plus an estimated token saving.

Measured on a live page: https://en.wikipedia.org/wiki/Markdown goes from roughly 77.8k to 5.0k estimated tokens, a 93.5 percent reduction. The bundled fixture corpus, which uses small synthetic pages with far less script weight, lands at 65 to 80 percent. Both figures are documented on the catalog page rather than only the flattering one.

Acceptance criteria mapping

Criterion (issue and review thread)	Where it lands
Takes a URL, strips non-semantic elements, returns token-efficient Markdown or JSON	skills/data_engineering/semantic_web_proxy/proxy.py (extract_semantic); test_skill.py::TestExtractSemantic
Reports tokens saved	skill.py::_token_savings; test_skill.py::TestTokenSavings
Fetch-first for v1, no headless browser	proxy.py::fetch_html; no Playwright dependency added
Warn clearly when a page looks JavaScript-only instead of returning nothing	proxy.py::looks_like_js_shell; test_skill.py::TestLooksLikeJsShell, TestEmptyJavascriptShell; fixture tests/fixtures/semantic_web_proxy/js_shell.html
Replace the focus_element string with flags mapped to trafilatura	manifest.yaml parameters include_comments, include_tables, include_links; test_skill.py::TestExtractSemantic::test_comments_*
Savings expressed as a percentage of context	token_savings.reduction_pct always, context_saved_pct when the caller passes context_window; test_skill.py::TestTokenSavings
SSRF guard before any fetch, same idea as the other URL skills	proxy.py::is_safe_public_url; test_skill.py::TestIsSafePublicUrl, TestFetchHtml
Split fetch from extract so tests run offline against fixture HTML	fetch_html is the only network-touching function; the whole suite is offline (tests/fixtures/semantic_web_proxy/*.html)

Deviations from the issue and thread, for review

Four deliberate departures. Each is easy to reverse if you would rather I follow the original text.

1. requests instead of httpx. requests is already a core dependency, is filtered out of the generated extras by extras.py::CORE_DEPENDENCIES, and matches security/deceptive_ui_guard and compliance/tos_evaluator. Using httpx would have added a new package to [all], which CI installs.

2. Token counting defaults to a heuristic, not cl100k_base. tiktoken downloads its vocabulary over the network on first use, which collides with the repository rule that bundle tests mock model downloads and CI does not download models, since [all] is installed in CI. The default is the same four-characters-per-token basis optimization/prompt_rewriter uses. cl100k_base is available through a hand-maintained opt-in extra, data_engineering_semantic_web_proxy_tokenizer, guarded with importlib.util.find_spec the way deceptive_ui_guard guards Playwright. When tiktoken is absent the skill falls back and emits tokenizer_unavailable rather than failing.

3. Output shape differs from the issue example. The issue specifies a flat {"status", "token_count_saved", "semantic_payload"}. status and semantic_payload match, but token_count_saved became token_savings.tokens_saved inside an object that also carries original_tokens, semantic_tokens, reduction_pct, context_saved_pct, tokenizer and estimate. Reporting the saving as a percentage, as requested in the thread, needed more than one scalar. Flagging it because the issue's Ideal Inputs and Outputs is part of the written spec.

4. Redirects are followed manually and revalidated on every hop. fetch_url_html in guard.py pairs a pre-flight host check with allow_redirects=True, so a 302 into 169.254.169.254 is not caught. This skill uses allow_redirects=False with a bounded hop loop that reruns the guard each time. I did not touch guard.py, since that is outside this issue, but security/deceptive_ui_guard probably has the same gap and may deserve its own issue.

Note on scope

Beyond the skill bundle and its catalog and index entries, this PR adds one row to the curated list in tests/test_examples_smoke.py so the new offline demo is covered by CI, in the spirit of #237. Happy to drop it if you would rather keep the diff to the bundle.

Verification

Run locally against a clean tree:

• python -m black --check . — 198 files unchanged
• python -m flake8 . — 0, both the critical selection and the full run
• pytest skills/ — 365 passed, 1 skipped
• pytest tests/ — 384 passed, 4 skipped
• python scripts/sync_extras.py --check — in sync
• pytest tests/test_skill_issuer.py tests/test_registry_identity.py tests/test_registry_docs.py tests/test_card_ui_schema.py tests/test_extras_sync.py — 41 passed
• skillware test data_engineering/semantic_web_proxy — 55 passed
• Wheel smoke: built the wheel, installed it into a clean venv, scripts/wheel_smoke_test.py exits 0 and reports the correct pip install "skillware[data_engineering_semantic_web_proxy]" hint

Local interpreter is Python 3.14. I did not run the suite on the 3.10, 3.11 and 3.12 matrix, but all new modules parse against the 3.10 grammar and trafilatura>=2.0.0 declares requires-python >=3.10 with wheels resolving cleanly for 3.10. CI is authoritative here.

news.ycombinator.com is unreachable from my environment, so the Hacker News thread case from the issue is covered by the thread_with_comments.html fixture rather than verified live.

Type of Change

• New Skill — new registry bundle under skills/
• Skill Upgrade — changes to an existing skill under skills/
• Bug Fix — incorrect runtime or framework behavior
• Documentation — docs, README, CONTRIBUTING only
• Framework Feature — skillware/core/ loader, env, adapters
• CLI — skillware/cli.py, docs/usage/cli.md
• Examples — examples/*.py, agent loops, examples/README.md
• Packaging — PyPI wheel, pyproject.toml, MANIFEST.in
• RFC / meta — templates, labels, CI, or large design doc

Checklist (all PRs)

• Linked GitHub issue (Fixes #… or Refs #…)
• Scope matches the issue — no unrelated refactors
• python -m black --check . and flake8 pass locally (or CI-equivalent subset)
• pytest skills/ and pytest tests/ pass locally when relevant
• CHANGELOG.md updated under [Unreleased] when user-visible behavior changes
• examples/README.md updated if this PR adds, renames, or removes a runnable script
• Ran pytest tests/test_registry_docs.py when skills, examples index, or agent-loops matrix changed

New or updated skill

Bundle and metadata

• Skill at skills/<category>/<skill_name>/ (from templates/python_skill/ or equivalent)
• manifest.yaml: name (full ID), version, description, parameters, constitution, real issuer
• Optional: short_description, issuer.github, issuer.org, requirements, env_vars

env_vars is intentionally absent: the skill needs no API key and no configuration.

Effect, Directive, Assurance

• Deterministic skill.py (Effect; no ad-hoc LLM-generated execution paths)
• instructions.md (Directive) explains when and how to use the skill
• card.json (Presentation) issuer matches manifest when present
• test_skill.py (Assurance) covers execution and schema expectations
• SkillLoader.load_skill("<category>/<skill_name>") succeeds (or deps documented)

55 bundle tests plus 9 maintainer-layer tests in tests/skills/data_engineering/test_semantic_web_proxy.py, driven over four fixture pages. Every test is offline: fetch_html is patched on the effect module, or the pure extraction path is exercised directly. An output fixture lives at tests/fixtures/card_ui_schema/data_engineering__semantic_web_proxy.json, generated from real execute() output so every ui_schema.fields[].key resolves.

Documentation and catalog

• docs/skills/<skill_name>.md and row in docs/skills/README.md
• Usage Examples for Gemini, Claude, OpenAI, DeepSeek, Ollama per skill usage template

Also added: a row in the docs/usage/agent_loops.md matrix, a row in examples/README.md, and a chaining example with security/prompt_injection_firewall that I ran end to end before documenting it.

Constitution and safety (skills only)

Read-only by construction. The skill issues HTTP GET requests and parses the response; it never submits forms, authenticates, or follows non-content actions. No LLM is called inside the skill, so identical HTML and options always produce an identical payload.

Network safety: only public http and https URLs are accepted. Non-http schemes such as file:// are rejected, along with loopback, .local, private, link-local, reserved and multicast addresses, resolved through socket.getaddrinfo before any request is issued. Redirects are followed manually, capped at five hops, and the guard reruns on every hop so a redirect cannot walk into a metadata endpoint. Responses are capped at 2 MB and must carry an HTML-like content type.

Output safety: semantic_payload is attacker-influenced third-party text and may carry prompt injection aimed at the calling agent. The constitution and instructions.md both direct hosts to treat it as data and to pass it through security/prompt_injection_firewall before it enters a context window, which is the text-channel half of the defense chain in docs/security/skill-trust-model.md.

Honest limits: token_savings.estimate is always true, and the catalog page states the counts are indicative for budgeting rather than billing figures. JavaScript is never executed, and a page that appears client-rendered is reported with page_likely_requires_javascript instead of a silently empty payload.

Permission is deliberately out of scope: this skill does not read robots.txt. The catalog page points at compliance/tos_evaluator as the pre-fetch partner when a site's terms are in question.

Related Issues

Fixes #42

[rosspeili]

Thanks @rizzoMartin, excellent work on #42. Fetch/extract split, SSRF guard with redirect re-validation, JS-shell honesty, fixture corpus, and firewall chaining section are top tier reasoning man. Tests and catalog quality are strong (59 bundle tests + integration, and registry/card/smoke pass on my side as well.

One blocker before merge:

• Update docs/usage/install_extras.md manually:
  • Category data_engineering row, add semantic_web_proxy and trafilatura>=2.0.0
  • Skill extras table, add data_engineering_semantic_web_proxy and data_engineering_semantic_web_proxy_tokenizer

pyproject.toml is synced i think, the install guide tables are the gap (CONTRIBUTING step after sync_extras.py), and skill history can point at merge SHA once landed, you can do a follow up, or I also do ocassional sweeps after skills change or land in hotfixes or chores so nw.

Otherwise good to merge from here. <3