diff --git a/AGENTS.md b/AGENTS.md index 5038863e..5eaea458 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,14 +1,14 @@ # AGENTS.md -> Primary entry point for AI agents integrating the resolver as a skill. -> Deep reference is in **[agents-docs/](agents-docs/README.md)**. +> AI agent integration reference for the web doc resolver. +> Deep reference is in **[agents-docs/README.md](agents-docs/README.md)**. ## Named Constants ```bash readonly MAX_LINES_PER_SOURCE_FILE=500 readonly MAX_LINES_PER_SKILL_MD=250 -readonly MAX_LINES_AGENTS_MD=200 +readonly MAX_LINES_AGENTS_MD=150 readonly DEFAULT_MAX_RETRIES=3 readonly DEFAULT_RETRY_DELAY_SECONDS=5 readonly DEFAULT_POLL_INTERVAL_SECONDS=5 @@ -63,16 +63,10 @@ After every completed task, append to `.agents/metrics.jsonl`: } ``` -Append-only. Never truncate. dora-report reads this file. - ## YAML Workflow Style Rule All `.github/workflows/*.yml` must include `# yamllint disable-line rule:truthy` on the `on:` line. -## Session Bootstrap - -`docflow.json` drives context injection at agent startup. `hooks/session-start.sh` can be run manually to verify environment readiness. - ## Repository Structure ```text @@ -106,82 +100,16 @@ Detailed reference material in `agents-docs/`: ## Coding Workflow -### Branching & Commits - -- Branch naming: `feat/`, `fix/`, `chore/`, `docs/` -- Commit format: Conventional Commits (`type(scope): description`) - -### PR Checklist - -- `./scripts/quality_gate.sh` passes -- Linting clean (`ruff`, `black`, `cargo fmt`, `cargo clippy`, `npm run lint`) -- No new secrets (verified via Gitleaks) -- `AGENTS.md` updated if structure changed - -### CI & Codacy Rules (NEVER SKIP) - -**ALL GitHub Actions checks MUST pass before merge.** No exceptions. - -**Codacy MUST be up to standards before merge.** If Codacy shows `ACTION_REQUIRED`: - -#### Codacy Issue Resolution Protocol - -**NEVER skip, suppress, or ignore Codacy issues without following this protocol:** - -1. **Analyze**: Run `codacy pull-request gh --output json` to get all issues -2. **Research**: For each issue, web-research the pattern against official docs and best practices: - - Check the rule's official documentation (ESLint, Biome, Semgrep, etc.) - - Determine if it's a genuine code quality concern or a false positive - - Document findings in the PR description or comments -3. **Fix**: If the issue is genuine, fix the code. Commit, push, re-verify. -4. **Verify**: Confirm the fix resolves the issue without introducing regressions -5. **Ignore (last resort only)**: If and only if the issue is a verified false positive: - - Document WHY it's a false positive (with links to docs/best practices) - - Use `codacy pull-request gh --ignore-issue --ignore-reason FalsePositive` - - Add an inline comment explaining the rationale - -**Key principles:** - -- Fix first, ignore never (unless verified false positive) -- Always document the reasoning behind any decision -- Never assume an issue is a false positive without verification -- Web research against official docs is mandatory before dismissing any issue - -**Never merge with:** - -- Any failing GitHub Action (even if "pre-existing on main") -- Codacy `ACTION_REQUIRED` status -- Merge conflicts -- Required reviews missing - -### Test Commands - -- **Python**: `pytest -m "not live"` -- **Rust**: `cd cli && cargo test` -- **Web**: `cd web && npx playwright test --project=desktop --project=mobile --project=tablet` - -## Release Workflow - -> **Do NOT use `gh release create` manually.** The CI/CD pipeline handles releases automatically. - -### Correct Release Steps - -```bash -# 1. Bump versions -python scripts/sync_versions.py --set $VERSION - -# 2. Commit -git add -A && git commit -m "chore(release): v$VERSION" - -# 3. Tag and push (triggers CI/CD) -git tag -a v$VERSION -m "Release v$VERSION" -git push origin main --tags -``` - -### What CI/CD Does Automatically - -- Runs Python + Rust test suites -- Builds binaries: Linux x86_64, macOS aarch64, Windows x86_64 -- Generates build attestations -- Extracts changelog from `CHANGELOG.md` -- Creates GitHub release with binaries + install instructions +- **Branch Naming**: Prefix branches with `feat/`, `fix/`, `chore/`, or `docs/`. +- **Commit Format**: Conventional Commits style: `type(scope): description`. +- **PR Checklist**: + 1. All tests must pass successfully. + 2. Linting must be completely clean (`ruff`, `ruff-format`, `cargo fmt`, `cargo clippy`, `npm run lint`). + 3. Verify that no new secrets are introduced (via Gitleaks). + 4. Ensure `AGENTS.md` is updated if there are structural or tree modifications. +- **Quality Gate Command**: `./scripts/quality_gate.sh` +- **Test Commands per Layer**: + - Python: `pytest -m "not live"` + - Rust: `cd cli && cargo test` + - Web: `cd web && npx playwright test --project=desktop` +- **File Size Limit**: Source files must not exceed 500 lines max per source file. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 09dc66f8..ebfc810b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,35 +5,51 @@ ### Python ```bash -# Run tests +# Run unit and integration tests python -m pytest tests/ -v -m "not live" -# Linting and formatting +# Linting and formatting checks python -m ruff check . -python -m black . +python -m ruff format --check . + +# Auto-format code +python -m ruff format . +python -m ruff check --fix . ``` ### Rust CLI ```bash cd cli + +# Run tests cargo test + +# Check clippy warnings cargo clippy -- -D warnings -cargo fmt + +# Format check +cargo fmt -- --check ``` ### Web UI ```bash cd web + +# Run linter npm run lint + +# Check TypeScript types npm run typecheck + +# Run Playwright E2E browser tests npx playwright test --project=desktop ``` ### Quality Gate -Run the full suite before submitting: +Always execute the local quality gate script to verify all requirements before committing: ```bash ./scripts/quality_gate.sh @@ -41,21 +57,21 @@ Run the full suite before submitting: ## Standards -- **Python**: Follow Black formatting and Ruff rules. Use type hints for public functions. -- **Rust**: Ensure `cargo clippy` and `cargo fmt` pass. -- **Commits**: Use [Conventional Commits](https://www.conventionalcommits.org/): +- **Python**: Adhere to Ruff formatting and linting rules. Use explicit type hints for all public functions. +- **Rust**: Ensure `cargo clippy` and `cargo fmt` pass without warnings. +- **Commits**: Follow Conventional Commits format (`type(scope): description`): - `feat:` new feature - `fix:` bug fix - - `docs:` documentation - - `chore:` maintenance - - `refactor:` code restructuring - - `test:` test updates -- **Branching**: Use `feat/`, `fix/`, `chore/`, or `docs/` prefixes. -- **File Size**: Source files must not exceed 500 lines. Split into sub-modules if they grow larger. + - `docs:` documentation updates + - `chore:` maintenance and dependency updates + - `refactor:` code restructuring without behavior changes + - `test:` test additions or modifications +- **Branching**: Prefix development branches with `feat/`, `fix/`, `chore/`, or `docs/`. +- **File Size Limit**: Source files must not exceed 500 lines. Refactor and partition into sub-modules if they exceed this limit. ## Pull Request Process -1. Update documentation for user-facing changes. -2. Add tests for new features or bug fixes. -3. Ensure the quality gate passes: `./scripts/quality_gate.sh`. -4. Update `AGENTS.md` if repository structure or skills change. +1. Document any user-facing changes in the appropriate docs/ guides. +2. Add unit or integration tests for new functionality. +3. Verify that the quality gate script runs successfully: `./scripts/quality_gate.sh`. +4. Update `AGENTS.md` if repository structure or skill definitions are modified. diff --git a/README.md b/README.md index a77b2ad2..c73aa598 100644 --- a/README.md +++ b/README.md @@ -1,107 +1,36 @@
-do-web-doc-resolver logo +do-web-doc-resolver logo # do-web-doc-resolver -**Resolve queries or URLs into compact, LLM-ready Markdown** — intelligent cascade routing across free and paid providers. -Zero-config by default: works out of the box with no API keys. +**Resolve queries or URLs into compact, LLM-ready Markdown** +An intelligent cascade routing engine across free, direct, paid, and browser-based providers. [![CI](https://github.com/d-oit/do-web-doc-resolver/actions/workflows/ci.yml/badge.svg)](https://github.com/d-oit/do-web-doc-resolver/actions) -[![Python](https://img.shields.io/badge/python-3.10%2B-3776ab?logo=python&logoColor=white)](https://www.python.org/) +[![Python](https://img.shields.io/badge/python-3.11%2B-3776ab?logo=python&logoColor=white)](https://www.python.org/) [![Rust](https://img.shields.io/badge/rust-stable-f74c00?logo=rust&logoColor=white)](https://www.rust-lang.org/) -[![Next.js](https://img.shields.io/badge/Next.js-15-black?logo=next.js)](https://nextjs.org/) +[![Next.js](https://img.shields.io/badge/Next.js-16-black?logo=next.js)](https://nextjs.org/) [![License: MIT](https://img.shields.io/badge/license-MIT-06b6d4.svg)](LICENSE) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md) -[**Live Demo**](https://web-eight-ivory-29.vercel.app) · [**Documentation**](docs/) · [**Report Bug**](https://github.com/d-oit/do-web-doc-resolver/issues) · [**Request Feature**](https://github.com/d-oit/do-web-doc-resolver/issues) +[Live Demo](https://web-eight-ivory-29.vercel.app) · [Documentation](docs/) · [Report Bug](https://github.com/d-oit/do-web-doc-resolver/issues) · [Request Feature](https://github.com/d-oit/do-web-doc-resolver/issues)
--- -## Why do-web-doc-resolver? +## What the Tool Does -- **Zero-key mode** — Works out of the box with no API keys; free providers are used by default -- **Intelligent cascade** — Routes through providers in priority order, stopping at the first successful result -- **Self-healing** — Circuit breakers and per-domain routing memory recover from failures automatically -- **LLM-optimized output** — Compact, deduplicated Markdown ready for direct injection into prompts -- **Three interfaces** — Python library, Rust CLI, and Next.js Web UI — one core, any workflow - ---- - -## Table of Contents - -- [Quick Start](#quick-start) -- [Architecture](#architecture) -- [Features](#features) -- [Installation](#installation) -- [Configuration](#configuration) -- [Usage](#usage) -- [Agent Skills](#agent-skills) -- [Data Layer](#data-layer) -- [Testing](#testing) -- [Repository Structure](#repository-structure) -- [Contributing](#contributing) -- [License](#license) - ---- - -## Quick Start - -```bash -# Clone and install (no API keys needed) -git clone https://github.com/d-oit/do-web-doc-resolver.git -cd do-web-doc-resolver -pip install -r requirements.txt - -# Resolve a URL -python -m scripts.cli "https://docs.example.com" - -# Resolve a search query (uses free providers) -python -m scripts.cli "your search query" -``` - -Or try the **[live demo →](https://web-eight-ivory-29.vercel.app)** - ---- - -## Architecture - -### Resolution Cascade - -The resolver uses a cascade pattern, trying providers in order until one succeeds: - -1. **Semantic Cache** (free, instant) — Cached results from previous queries -2. **Free Providers** (no key needed) — Exa MCP, Exa SDK, Tavily, DuckDuckGo -3. **Paid Providers** (API key required) — Serper, Mistral -4. **Fallback** (free) — Jina Reader, Firecrawl, Direct HTTP - -Query providers: Semantic Cache → Exa MCP → Exa SDK → Tavily → Serper → DuckDuckGo → Mistral -URL providers: Semantic Cache → llms.txt → Jina Reader → Firecrawl → Direct HTTP → Mistral Browser → DuckDuckGo - ---- - -## Features - -| Feature | Description | -|---|---| -| **Cascade Routing** | Automatic provider fallback with configurable priority order | -| **Semantic Cache** | In-memory similarity lookup with configurable TTL per provider | -| **Circuit Breakers** | Per-provider failure detection with automatic recovery | -| **Routing Memory** | Remembers which providers succeed for each domain | -| **Quality Scoring** | Ranks results by content density and relevance | -| **Multi-interface** | Python API, Rust CLI (`do-wdr`), and Next.js Web UI | -| **Zero-config** | Works without API keys using free providers by default | -| **LLM-ready output** | Compact Markdown optimized for prompt injection | +The web doc resolver translates complex web URLs or text search queries into clean, deduplicated, and token-efficient Markdown. This Markdown is structured for direct injection into Large Language Model (LLM) context windows or Retrieval-Augmented Generation (RAG) pipelines. It uses local semantic caching, heuristic quality scoring, circuit breakers, and per-domain routing memory to find the most efficient provider for any request. --- ## Installation -### Python (library + CLI) +### Python Library and CLI -Requires Python 3.10 or higher. +Install the Python core dependencies. Python 3.11 or higher is required. ```bash git clone https://github.com/d-oit/do-web-doc-resolver.git @@ -111,50 +40,28 @@ pip install -r requirements.txt ### Rust CLI (`do-wdr`) +Build the high-performance compiled binary. + ```bash cd cli cargo build --release -# Binary: cli/target/release/do-wdr +# Built binary is located at cli/target/release/do-wdr ``` ### Web UI (Next.js) +Install the web interface dependencies. + ```bash cd web npm install --legacy-peer-deps -npm run dev -# Open http://localhost:3000 ``` --- -## Configuration - -All API keys are **optional**. The tool works with zero configuration using free providers. - -| Variable | Provider | Required | Notes | -|---|---|---|---| -| `EXA_API_KEY` | Exa SDK | No | Enables Exa search with highlights | -| `TAVILY_API_KEY` | Tavily Search | No | Enables broad web search | -| `SERPER_API_KEY` | Serper (Google) | No | Enables Google search | -| `FIRECRAWL_API_KEY` | Firecrawl | No | Enables deep content extraction | -| `MISTRAL_API_KEY` | Mistral AI | No | Enables AI-powered search/browse | +## How to Run -```bash -# Linux/macOS -export EXA_API_KEY="your-key" - -# Windows PowerShell -$env:EXA_API_KEY="your-key" -``` - -Configuration file: [`config.toml`](config.toml) — routing thresholds, cache TTLs, rate limits. - ---- - -## Usage - -### Python API +### Python Library ```python from scripts.resolve import resolve @@ -171,195 +78,102 @@ print(result["content"]) ### Python CLI ```bash -python -m scripts.cli "your search query" +# Resolve a search query +python -m scripts.cli "Python subprocess" + +# Resolve a URL python -m scripts.cli "https://example.com" ``` -### Rust CLI (`do-wdr`) +### Rust CLI ```bash +# Resolve a URL or query ./cli/target/release/do-wdr resolve "https://docs.example.com" -./cli/target/release/do-wdr resolve "your search query" +./cli/target/release/do-wdr resolve "how to parse json in rust" ``` ### Web UI +Start the development server: + ```bash -cd web && npm run dev -# Open http://localhost:3000 and enter a URL or query +cd web +npm run dev +# Open http://localhost:3000 in your browser ``` --- -## Agent Skills - -The resolver ships as a skill for AI agents under `.agents/skills/`. It provides a self-contained `SKILL.md` with references, tests, and a portable Python module. +## The Cascade -| Skill | Interface | Purpose | -|---|---|---| -| [`do-web-doc-resolver`](.agents/skills/do-web-doc-resolver/) | Python | Full cascade resolver — importable module or CLI | -| [`do-wdr-cli`](.agents/skills/do-wdr-cli/) | Rust | Compiled `do-wdr` binary for fast resolution | +The resolver organizes providers into an escalation-based cascade, attempting free or low-cost static methods before falling back to heavier or paid scrapers and headless browser sessions. -### Using the Core Skill +### Query Resolution Cascade -```bash -# As a CLI (from project root) -python3 -m scripts.cli "https://docs.example.com" +1. **Semantic Cache**: Fast local SQLite + `sqlite-vec` vector lookup of similar previous queries. +2. **Exa MCP**: Desktop/local model-context-protocol search integration. +3. **Exa SDK**: Web search with highlights. +4. **Tavily**: Broad web search engine. +5. **Serper**: Google search API. +6. **DuckDuckGo**: Free HTML search fallback. +7. **Mistral Websearch**: LLM-augmented search routing. -# As a Python module -from scripts.resolve import resolve -result = resolve("your search query") -print(result["content"]) +### URL Resolution Cascade -# Via the Rust CLI -cd cli && cargo build --release -./target/release/do-wdr resolve "https://docs.example.com" -``` +1. **Semantic Cache**: Fast similarity lookup of previously cached page content. +2. **Document/Image parsers**: Local extraction of specialized extensions (Docling for `.pdf`/`.docx`/`.pptx`, OCR for `.png`/`.jpg`/`.jpeg`). +3. **llms.txt**: Reads static `.txt` files directly if published at the target domain root or path. +4. **Jina Reader**: Light markdown extraction API. +5. **Firecrawl**: Deep cloud content rendering API. +6. **Direct Fetch**: Local HTTP client utilizing `trafilatura` and `readability-lxml` parsing. +7. **Mistral Browser**: Headless JS-enabled browser rendering. +8. **Visual CLIP**: Multi-modal vision-based target validation. +9. **DuckDuckGo**: Domain-level search. +10. **Stealth**: anti-bot bypass tier. --- -## Data Layer - -The resolver uses two complementary storage systems — both in-memory by default, with optional persistent backends. - -### Semantic Cache +## Environment Variables Required -Provides similarity-based caching using local embeddings. Identifies semantically equivalent queries (not just exact matches) and returns cached results instantly. +All environment variables are optional. The tool defaults to free local direct fetching or cached results if no keys are set. -| Component | Detail | -|---|---| -| **Engine** | SQLite + [sqlite-vec](https://github.com/asg017/sqlite-vec) vector extension | -| **Embeddings** | `all-MiniLM-L6-v2` via sentence-transformers (~80MB, runs locally) | -| **Storage** | `~/.cache/do-web-doc-resolver/semantic/semantic_cache.db` | -| **Similarity** | Cosine distance, threshold `0.85` (configurable) | -| **Eviction** | LRU with max `10,000` entries (configurable) | -| **TTL** | Per-provider, 1–24 hours (see `config.toml`) | - -```bash -# Optional: install sqlite-vec for high-performance vector search -pip install sqlite-vec -``` - -**Configuration:** - -| Variable | Default | Description | +| Variable | Provider / Component | Purpose | |---|---|---| -| `DO_WDR_SEMANTIC_CACHE` | `1` | Set to `0` to disable | -| `DO_WDR_CACHE_THRESHOLD` | `0.85` | Minimum similarity for cache hits | -| `DO_WDR_CACHE_MAX_ENTRIES` | `10000` | Max entries before LRU eviction | - -### Routing Memory - -Learns which providers work best for each domain. Ranks providers by success rate, quality score, latency, and recency — so repeated requests to the same domain go to the fastest, most reliable provider first. - -| Component | Detail | -|---|---| -| **Storage** | In-memory (`defaultdict`), thread-safe | -| **Ranking** | Weighted: success rate × quality × recency / latency | -| **Decay** | Recency factor decays over 7 days | -| **Base score** | `0.5` for unknown provider/domain pairs | - -### Circuit Breakers - -Protects against cascading failures. When a provider fails 3 consecutive times, it is skipped for 5 minutes before retry. - -| Setting | Value | -|---|---| -| Failure threshold | 3 consecutive failures | -| Cooldown | 300 seconds (5 minutes) | -| Reset | Successful call resets failure count | - -### State Management - -All shared state is managed via a singleton `ResolverState` object (`scripts/state.py`): - -```python -from scripts.state import get_state - -state = get_state() -# state.circuit_breakers — CircuitBreakerRegistry -# state.routing_memory — RoutingMemory (per-domain learning) -# state.semantic_cache — SemanticCache (sqlite-vec) -``` +| `EXA_API_KEY` | Exa SDK | Activates Exa web search | +| `TAVILY_API_KEY` | Tavily Search | Activates Tavily web search | +| `SERPER_API_KEY` | Serper | Activates Google search fallback | +| `FIRECRAWL_API_KEY` | Firecrawl | Activates cloud JS rendering | +| `MISTRAL_API_KEY` | Mistral AI | Activates Mistral Search and Mistral Browser | +| `DO_WDR_SEMANTIC_CACHE` | Semantic Cache | Set to `0` to disable the SQLite semantic cache | +| `DO_WDR_CACHE_THRESHOLD` | Semantic Cache | Minimum similarity score (default `0.85`) | +| `DO_WDR_CACHE_MAX_ENTRIES`| Semantic Cache | Max entries before LRU eviction (default `10000`) | --- -## Testing +## How to Run Tests -### Python Suite +### Python Test Suite + +Run the unit and integration tests (excluding live external API calls): ```bash -python -m pytest tests/ -v -m "not live" +PYTHONPATH=. python -m pytest tests/ -v -m "not live" ``` -### Rust Suite +### Rust CLI Test Suite + +Run the native CLI tests: ```bash cd cli && cargo test ``` -### Web UI Suite - -```bash -cd web && npx playwright test --project=desktop -``` +### Web UI Test Suite -### Full Quality Gate +Run the Playwright E2E browser tests: ```bash -./scripts/quality_gate.sh -``` - ---- - -## Repository Structure - -```text -├── scripts/ # Python resolver core -│ ├── resolve.py # Main entry point -│ ├── _cascade.py # Cascade routing engine -│ ├── _query_resolve.py # Query resolution providers -│ ├── _url_resolve.py # URL resolution providers -│ ├── semantic_cache.py # SQLite-vec semantic cache -│ ├── routing_memory.py # Per-domain provider learning -│ ├── circuit_breaker.py # Provider failure protection -│ ├── state.py # Shared resolver state singleton -│ └── quality.py # Content quality scoring -├── cli/ # Rust CLI (do-wdr) -│ └── src/ -├── web/ # Next.js Web UI -│ └── app/ -├── tests/ # Python test suite -├── .agents/skills/ # Agent skill definitions -│ ├── do-web-doc-resolver/ # Core resolver skill -│ ├── do-wdr-cli/ # Rust CLI skill -│ ├── do-wdr-release/ # Release management -│ └── ... # 11 skills total -├── docs/ # Project documentation -├── agents-docs/ # Agent-specific reference -├── assets/ # Logo, screenshots, visual assets -├── config.toml # Routing, cache, and rate config -├── CONTRIBUTING.md # Contribution guidelines -├── LICENSE # MIT License -└── README.md # This file +cd web +npx playwright test --project=desktop ``` - ---- - -## Contributing - -Contributions are welcome! - -1. Fork the repository -2. Create a feature branch (`git checkout -b feat/my-feature`) -3. Add tests for new functionality -4. Run the quality gate: `./scripts/quality_gate.sh` -5. Submit a pull request - -See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines (Python linting, Rust clippy, Web typecheck). - ---- - -## License - -MIT License — see [LICENSE](LICENSE) for details. diff --git a/cli/src/resolver/query/mod.rs b/cli/src/resolver/query/mod.rs index eee3d699..eda18e80 100644 --- a/cli/src/resolver/query/mod.rs +++ b/cli/src/resolver/query/mod.rs @@ -54,6 +54,19 @@ impl QueryCascade { } } + /// Check if a query provider is available/configured + pub fn is_provider_available(&self, provider_type: ProviderType) -> bool { + match provider_type { + ProviderType::ExaMcp => self.exa_mcp.is_available(), + ProviderType::Exa => self.exa_sdk.is_available(), + ProviderType::Tavily => self.tavily.is_available(), + ProviderType::Serper => self.serper.is_available(), + ProviderType::DuckDuckGo => self.duckduckgo.is_available(), + ProviderType::MistralWebSearch => self.mistral_ws.is_available(), + _ => false, + } + } + /// Search using a specific provider pub async fn search_with_provider( &self, @@ -187,6 +200,14 @@ impl QueryCascade { .parse() .map_err(|e| ResolverError::Provider(format!("Invalid provider name: {}", e)))?; + if !self.is_provider_available(provider_type) { + tracing::debug!( + "Provider {} is not available/configured, skipping", + provider.name + ); + continue; + } + // Check negative cache { let nc = negative_cache.read().await; diff --git a/cli/src/resolver/url.rs b/cli/src/resolver/url.rs index 7ed9931c..bbe84c8f 100644 --- a/cli/src/resolver/url.rs +++ b/cli/src/resolver/url.rs @@ -51,6 +51,20 @@ impl UrlCascade { } } + /// Check if a URL provider is available/configured + pub fn is_provider_available(&self, provider_type: ProviderType) -> bool { + match provider_type { + ProviderType::LlmsTxt => self.llms_txt.is_available(), + ProviderType::Jina => self.jina.is_available(), + ProviderType::Docling => self.docling.is_available(), + ProviderType::Ocr => self.ocr.is_available(), + ProviderType::Firecrawl => self.firecrawl.is_available(), + ProviderType::DirectFetch => self.direct_fetch.is_available(), + ProviderType::MistralBrowser => self.mistral_browser.is_available(), + _ => false, + } + } + /// Check for document or image format and return provider type pub fn check_format(url: &str) -> Option { if url.ends_with(".pdf") || url.ends_with(".docx") || url.ends_with(".pptx") { @@ -226,6 +240,14 @@ impl UrlCascade { .parse() .map_err(|e| ResolverError::Provider(format!("Invalid provider name: {}", e)))?; + if !self.is_provider_available(provider_type) { + tracing::debug!( + "Provider {} is not available/configured, skipping", + provider.name + ); + continue; + } + // Check negative cache { let nc = negative_cache.read().await; diff --git a/docs/index.md b/docs/index.md index 954ebacc..4ef0b71e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,23 +1,23 @@ # Documentation Overview -Reference for the **do-web-doc-resolver** project. +Technical reference material for the **do-web-doc-resolver** project. ## Guides -- **[README](../README.md)**: Overview, installation, and quickstart. -- **[CONTRIBUTING](../CONTRIBUTING.md)**: Development workflow and standards. -- **[AGENTS](../AGENTS.md)**: Reference for AI agent integration. +- **[README](../README.md)**: Product capabilities, installation, usage, cascade routing, and testing. +- **[CONTRIBUTING](../CONTRIBUTING.md)**: Development workflows, coding standards, and linting guidelines. +- **[AGENTS](../AGENTS.md)**: AI agent integration protocol and execution constraints. ## Technical Reference -- **[Architecture](../agents-docs/OVERVIEW.md)**: Component breakdown and data flow. +- **[Architecture](../agents-docs/OVERVIEW.md)**: Core components and data flow pipelines. - **[Cascade Logic](https://github.com/d-oit/do-web-doc-resolver/blob/main/.agents/skills/do-web-doc-resolver/references/CASCADE.md)**: Query and URL resolution decision trees. -- **[Provider Details](https://github.com/d-oit/do-web-doc-resolver/blob/main/.agents/skills/do-web-doc-resolver/references/PROVIDERS.md)**: Provider capabilities and API details. -- **[Semantic Health](../agents-docs/SEMANTIC_HEALTH.md)**: Metrics and output quality standards. -- **[Standards](standards.md)**: LLM-readable document standards (2026). +- **[Provider Details](https://github.com/d-oit/do-web-doc-resolver/blob/main/.agents/skills/do-web-doc-resolver/references/PROVIDERS.md)**: Scraper capability tables and API specifications. +- **[Semantic Health](../agents-docs/SEMANTIC_HEALTH.md)**: Quality scoring and output validation. +- **[Standards](standards.md)**: LLM-readable Markdown document standards. ## Infrastructure -- **[Configuration](../agents-docs/CONFIG.md)**: Environment variables and config files. -- **[Deployment](../agents-docs/DEPLOYMENT.md)**: Vercel and CI/CD setup. -- **[Releases](../agents-docs/RELEASES.md)**: Versioning and release procedures. +- **[Configuration](../agents-docs/CONFIG.md)**: Routing thresholds, cache settings, and environment variables. +- **[Deployment](../agents-docs/DEPLOYMENT.md)**: Vercel integration and CI/CD workflow configurations. +- **[Releases](../agents-docs/RELEASES.md)**: Automated versioning and build procedures. diff --git a/docs/standards.md b/docs/standards.md index bb23e118..45bb006f 100644 --- a/docs/standards.md +++ b/docs/standards.md @@ -29,7 +29,7 @@ Standardized anchors facilitate content partitioning and citation mapping. To maximize RAG performance and minimize context window usage, all documents must adhere to extreme density requirements. - **Zero Filler**: Remove all conversational intros ("Certainly!", "I'd be happy to help"), transition theater ("In conclusion", "It is worth noting that"), and hollow affirmations. -- **AI-Slop Prohibition**: Aggressively prune marketing jargon and generic AI-generated superlatives. +- **AI-Slop Prohibition**: Aggressiveness in pruning marketing jargon and generic AI-generated superlatives. - **Prohibited Words**: - `seamlessly`, `robust`, `powerful`, `comprehensive`, `streamlined` - `leverage`, `revolutionize`, `game-changing`, `intuitive`