From dd4f8ec2d037e60b06e0b90c617c679959b2d982 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 5 Mar 2026 20:20:11 +0000 Subject: [PATCH] docs: audit and align all documentation to actual Python/FastAPI codebase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original planning docs described a C# ASP.NET Core + Python Flask dual-runtime architecture that was never implemented. The actual codebase is a single-process Python FastAPI application. Updated six documents to accurately reflect the implementation: - 03_functional_requirements: FR-ING-04 — embedding is in-process, not a Python worker subprocess - 04_system_requirements: replace entire C# backend section with Python/ FastAPI stack; remove .NET SDK prerequisite; fix performance attribution (PyMuPDF not PdfPig); update security/privacy and architecture constraints - 05_architecture: full rewrite — single-process FastAPI diagram; updated data-flow pipelines (no C#→Python HTTP calls); correct component table referencing actual source files; add missing /text-positions endpoint; remove non-existent Python worker API section and IPC bridge section - 06_scope_and_milestones: fix in-scope tech (PyMuPDF not PdfPig/C#); update M1/M2 milestone descriptions and gantt section labels - 07_repo_structure: replace C# project tree (QuickGuide.Api/, .sln, wwwroot/, python_worker/) with actual directory layout (src/main.py, src/services/, src/static/); update naming conventions and key dirs table - 08_risk_register: R-02 PdfPig→PyMuPDF; R-05 retitled and rewritten for single Python runtime; R-08 PdfPig→PyMuPDF; R-09 retitled and rewritten for FastAPI process startup failures https://claude.ai/code/session_01SPUUZCfAfzQ7Kna5JGCZSD --- docs/03_functional_requirements.md | 2 +- docs/04_system_requirements.md | 69 ++++++------- docs/05_architecture.md | 154 +++++++++++------------------ docs/06_scope_and_milestones.md | 16 +-- docs/07_repo_structure.md | 105 +++++++++----------- docs/08_risk_register.md | 20 ++-- 6 files changed, 153 insertions(+), 213 deletions(-) diff --git a/docs/03_functional_requirements.md b/docs/03_functional_requirements.md index c494fa6..2270d05 100644 --- a/docs/03_functional_requirements.md +++ b/docs/03_functional_requirements.md @@ -37,7 +37,7 @@ | **Description** | Each text chunk is converted to a 384-dimensional vector embedding using `all-MiniLM-L6-v2`. | | **Input** | Text chunk content. | | **Output** | Float32 embedding vector stored in the vector index and referenced in the database. | -| **Rules** | Embedding is generated via a Python worker subprocess. Batch size of 32 chunks for efficiency. | +| **Rules** | Embedding is generated in-process within the FastAPI application. Batch size of 32 chunks for efficiency. | ### FR-ING-05 — Progress Feedback | Field | Detail | diff --git a/docs/04_system_requirements.md b/docs/04_system_requirements.md index bf3f1a3..fb9c09b 100644 --- a/docs/04_system_requirements.md +++ b/docs/04_system_requirements.md @@ -1,6 +1,6 @@ # QuickGuide (QG) — System Requirements Document (SRD) -**Version:** 1.0 +**Version:** 1.1 **Date:** 2026-03-05 --- @@ -17,8 +17,8 @@ | Platform | Minimum Version | Notes | |---|---|---| | Windows | 10 | Primary development target | -| macOS | 12 Monterey | .NET 8 and Python supported | -| Linux | Ubuntu 22.04 / Fedora 38 | .NET 8 and Python supported | +| macOS | 12 Monterey | Python supported | +| Linux | Ubuntu 22.04 / Fedora 38 | Python supported | | Android / iOS | Any modern browser | Access via `http://localhost:8080` | > [!NOTE] @@ -26,54 +26,46 @@ ## 3 Software Dependencies -### 3.1 C# Backend (Primary — built for speed) +### 3.1 Python Backend (single process — handles all logic and ML) | Component | Technology | Version | Rationale | |---|---|---|---| -| **Runtime** | .NET 8 SDK | 8.0+ | Cross-platform, high-performance | -| **Web framework** | ASP.NET Core Minimal API | 8.0 | Fast HTTP, built-in DI, static file serving | -| **PDF parsing** | PdfPig | latest | Pure C#, no native dependencies, page-level text extraction | -| **Database** | SQLite via Microsoft.Data.Sqlite | latest | Zero-config, file-based, fast | -| **ORM** | Dapper | latest | Lightweight micro-ORM for C# | -| **JSON** | System.Text.Json | built-in | High-performance serialization | -| **Process management** | System.Diagnostics.Process | built-in | Launches Python worker subprocess | - -### 3.2 Python ML Worker (for embedding + vector search) - -| Component | Technology | Version | Rationale | -|---|---|---|---| -| **Runtime** | Python | 3.11+ | ML ecosystem support | -| **Embedding model** | sentence-transformers | latest | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly | -| **Vector index** | FAISS-cpu | latest | Efficient ANN search, widely supported | -| **HTTP server** | Flask | latest | Lightweight API for C# → Python communication | -| **Protocol** | HTTP on localhost:5001 | — | C# backend calls Python worker via REST | - -### 3.3 Frontend +| **Runtime** | Python | 3.11+ | ML ecosystem support; cross-platform | +| **Web framework** | FastAPI | 0.115.0+ | High-performance async API; built-in data validation | +| **ASGI server** | uvicorn[standard] | 0.30.0+ | Production-grade ASGI server for FastAPI | +| **PDF parsing** | PyMuPDF (fitz) | 1.24.0+ | Fast text extraction with character positions; no native dependencies | +| **File uploads** | python-multipart | 0.0.9+ | Multipart form-data support for FastAPI | +| **Async file I/O** | aiofiles | 24.1.0+ | Async file operations | +| **Database** | SQLite via Python stdlib `sqlite3` | built-in | Zero-config, file-based, fast | +| **Embedding model** | sentence-transformers | 3.0.0+ | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly | +| **Vector index** | faiss-cpu | 1.8.0+ | Efficient ANN search (numpy fallback if unavailable) | +| **Numerics** | numpy | 1.26.0–<2.0.0 | Vector operations and numpy fallback search | + +### 3.2 Frontend | Component | Technology | Rationale | |---|---|---| | **PDF rendering** | PDF.js (Mozilla) | Industry-standard browser PDF viewer | -| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served by C# backend | +| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served as static files by the FastAPI backend | | **Font** | Nunito (Google Fonts, self-hosted) | Warm, rounded, cozy aesthetic | -### 3.4 Build / Install Prerequisites +### 3.3 Build / Install Prerequisites | Tool | Version | Notes | |---|---|---| -| .NET SDK | 8.0+ | For building and running the C# backend | -| Python | 3.11+ | For the ML worker only | -| pip | bundled with Python | Installs sentence-transformers + FAISS | +| Python | 3.11+ | The only runtime required | +| pip | bundled with Python | Installs all dependencies from `requirements.txt` | | Git | any | For version control | > [!IMPORTANT] -> **Two runtimes required:** .NET 8 SDK (for the C# backend) and Python 3.11+ (for the ML embedding worker). Both are free and cross-platform. +> **One runtime required:** Python 3.11+ only. There is no separate .NET or C# dependency. ## 4 Performance Targets | Metric | Target | |---|---| -| PDF text extraction (1,000 pages) | ≤ 30 s (C# PdfPig) | -| Embedding generation (1,000 pages) | ≤ 90 s (Python worker) | +| PDF text extraction (1,000 pages) | ≤ 30 s (Python PyMuPDF) | +| Embedding generation (1,000 pages) | ≤ 90 s (sentence-transformers, batch=32) | | Total ingestion (1,000 pages) | ≤ 120 s | | Query → results | ≤ 3 s | | App start (backend + browser) | ≤ 5 s | @@ -84,22 +76,23 @@ - **No telemetry** — no external network calls (except first-time model download). - **Local-only storage** — PDFs, text, and embeddings stay on the device. -- **Localhost binding** — C# backend binds to `127.0.0.1:8080`; Python worker to `127.0.0.1:5001`. +- **Localhost binding** — FastAPI backend binds to `127.0.0.1:8080` by default (configurable via `QG_HOST` / `QG_PORT` env vars). - **No authentication** — single-user, local application. ## 6 Deployment Strategy | Platform | Delivery | |---|---| -| Windows | `qg.bat` double-click launcher → starts backend + opens browser | +| Windows | `qg.bat` double-click launcher → creates venv, installs deps, starts Python server, opens browser | | macOS / Linux | `qg.sh` launcher script | | Mobile | Access via browser on same network | -| Packaged (future) | Self-contained .NET publish + PyInstaller bundle | +| Packaged (future) | PyInstaller bundle for zero-install distribution | ## 7 Architecture Constraints -1. **C# for all non-ML logic** — PDF parsing, API, database, file management all in C#. -2. **Python only for ML** — embedding generation and vector search in a separate Python process. -3. **No external servers** — everything on localhost. -4. **Zero frontend build step** — HTML/CSS/JS served as static files by the C# backend. +1. **Python for all logic** — PDF parsing, API, database, file management, embeddings, and vector search all run in a single Python process. +2. **FastAPI as the web framework** — handles all HTTP routing, static file serving, and data validation. +3. **No external servers** — everything on localhost; no inter-process HTTP communication required. +4. **Zero frontend build step** — HTML/CSS/JS served as static files by the FastAPI backend. 5. **SQLite only** — no database server installation required. +6. **FAISS with numpy fallback** — if `faiss-cpu` is unavailable, flat cosine similarity via numpy is used automatically. diff --git a/docs/05_architecture.md b/docs/05_architecture.md index f03e729..36b381c 100644 --- a/docs/05_architecture.md +++ b/docs/05_architecture.md @@ -1,6 +1,6 @@ # QuickGuide (QG) — Architecture Document -**Version:** 1.0 +**Version:** 1.1 **Date:** 2026-03-05 --- @@ -12,24 +12,22 @@ graph TB subgraph UserDevice["User's Device"] Browser["Web Browser
(any platform)"] - subgraph CSharpBackend["C# Backend — ASP.NET Core 8"] - API["REST API
(Minimal API)"] - PdfSvc["PDF Parser
(PdfPig)"] + subgraph PythonBackend["Python Backend — FastAPI"] + API["REST API
(FastAPI routes)"] + PdfSvc["PDF Parser
(PyMuPDF / fitz)"] ChunkSvc["Text Chunker"] DocMgr["Document Manager"] HighlightSvc["Highlight Service"] - DbSvc["Database Service
(SQLite + Dapper)"] - end - - subgraph PythonWorker["Python ML Worker — Flask"] EmbedSvc["Embedding Service
(sentence-transformers)"] - VectorSvc["Vector Search
(FAISS)"] + VectorSvc["Vector Search
(FAISS / numpy)"] + DbSvc["Database Service
(SQLite)"] end subgraph DataLayer["Data Layer"] SQLite["SQLite DB
(qg.db)"] VectorIdx["FAISS Index Files
(data/vectors/)"] FileStore["PDF File Store
(data/pdfs/)"] + ModelCache["Model Cache
(data/models/)"] end subgraph Frontend["Static Frontend"] @@ -44,11 +42,10 @@ graph TB API --> ChunkSvc API --> DocMgr API --> HighlightSvc + API --> EmbedSvc + API --> VectorSvc API --> DbSvc - API -- "HTTP localhost:5001" --> EmbedSvc - API -- "HTTP localhost:5001" --> VectorSvc - PdfSvc --> FileStore ChunkSvc --> DbSvc DbSvc --> SQLite @@ -56,7 +53,7 @@ graph TB DocMgr --> SQLite HighlightSvc --> SQLite - EmbedSvc --> VectorIdx + EmbedSvc --> ModelCache VectorSvc --> VectorIdx ``` @@ -66,28 +63,26 @@ graph TB ```mermaid flowchart LR - A["User selects PDF"] --> B["C#: Copy to
data/pdfs/"] - B --> C["C#: Extract text
(PdfPig, per page)"] - C --> D["C#: Chunk text
(512 tok / 50 overlap)"] - D --> E["C# → Python:
POST /embed"] - E --> F["Python: Generate
embeddings
(sentence-transformers)"] - F --> G["Python: Add to
FAISS index"] - G --> H["C#: Store metadata
in SQLite"] - H --> I["Ready for Search ✓"] + A["User selects PDF"] --> B["FastAPI: Copy to
data/pdfs/"] + B --> C["PyMuPDF: Extract text
(per page, with char offsets)"] + C --> D["Chunker: Split text
(512 tok / 50 overlap)"] + D --> E["sentence-transformers:
Generate embeddings
(batch=32)"] + E --> F["FAISS: Add vectors
to per-document index"] + F --> G["SQLite: Store metadata,
chunks + status = ready"] + G --> H["Ready for Search ✓"] ``` ### 2.2 Search Flow ```mermaid flowchart LR - A["User types query"] --> B["C# API receives
search request"] - B --> C["C# → Python:
POST /search"] - C --> D["Python: Embed query
(sentence-transformers)"] - D --> E["Python: FAISS ANN
search (cosine sim)"] - E --> F["Python returns
top-K chunk IDs + scores"] - F --> G["C#: Fetch chunk text
from SQLite"] - G --> H["C# returns JSON
results to browser"] - H --> I["Browser: Navigate
PDF.js + Highlight"] + A["User types query"] --> B["FastAPI API receives
search request"] + B --> C["sentence-transformers:
Embed query (384-dim)"] + C --> D["FAISS (or numpy):
ANN search (cosine sim)"] + D --> E["Filter results:
similarity ≥ 0.30"] + E --> F["SQLite: Fetch
chunk text + page numbers"] + F --> G["FastAPI returns
JSON results to browser"] + G --> H["Browser: Navigate
PDF.js + Highlight"] ``` ### 2.3 Navigation & Highlighting Flow @@ -104,46 +99,41 @@ flowchart LR ## 3 Component Breakdown -### 3.1 C# Backend (ASP.NET Core 8 Minimal API) - -| File | Responsibility | -|---|---| -| `Program.cs` | App entry point, DI registration, route mapping, static file serving | -| `Services/PdfParserService.cs` | PdfPig text extraction — extracts text per page with character positions | -| `Services/TextChunkerService.cs` | Splits page text into overlapping token-windowed chunks | -| `Services/DocumentService.cs` | Document lifecycle: upload, list, delete, status tracking | -| `Services/SearchService.cs` | Orchestrates search: calls Python worker, merges with SQLite metadata | -| `Services/HighlightService.cs` | Highlight CRUD: create, read, update color, delete | -| `Services/PythonBridgeService.cs` | HTTP client that communicates with the Python ML worker | -| `Data/DatabaseService.cs` | SQLite connection, migrations, Dapper queries | -| `Models/` | C# record types for API requests/responses and DB entities | - -### 3.2 Python ML Worker (Flask) +### 3.1 Python Backend (FastAPI — single process, port 8080) | File | Responsibility | |---|---| -| `worker.py` | Flask app: `/embed`, `/search`, `/health` endpoints | -| `embedder.py` | Loads `all-MiniLM-L6-v2`, generates 384-dim embeddings | -| `vector_store.py` | FAISS index management: add vectors, search, save/load, delete | - -### 3.3 Frontend (HTML / CSS / Vanilla JS) +| `src/main.py` | App entry point, FastAPI routes, static file serving, browser auto-launch | +| `src/config.py` | All configuration: paths, server settings, model params, search thresholds | +| `src/models.py` | Pydantic request/response models (SearchRequest, HighlightCreate, etc.) | +| `src/database.py` | SQLite connection (thread-local, WAL mode), schema init, CRUD queries | +| `src/services/pdf_parser.py` | PyMuPDF text extraction per page; bounding-box search for highlighting | +| `src/services/chunker.py` | Splits page text into overlapping 512-token windows with 50-token overlap | +| `src/services/embedder.py` | Lazy-loads `all-MiniLM-L6-v2`; batch encodes text; normalizes to float32 | +| `src/services/vector_store.py` | Per-document FAISS index (or numpy fallback); add, search, save, load, delete | +| `src/services/search.py` | Semantic search orchestration: embed → vector search → filter → enrich | +| `src/services/documents.py` | Document lifecycle: upload, ingest pipeline, list, get, delete, progress tracking | +| `src/services/highlights.py` | Highlight CRUD: add, get, update color, delete | + +### 3.2 Frontend (HTML / CSS / Vanilla JS) | File | Responsibility | |---|---| -| `index.html` | App shell — layout, PDF viewer container, search panel | -| `css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows | -| `js/app.js` | API calls, search orchestration, state management, document selector | -| `js/pdfviewer.js` | PDF.js rendering, page navigation, text-layer highlight overlays, match jumping | +| `src/static/index.html` | App shell — layout, PDF viewer container, search panel, highlights tab | +| `src/static/css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows | +| `src/static/js/app.js` | API calls, search orchestration, PDF.js integration, state management, highlights | +| `src/static/img/qg-favicon.svg` | QG logo used as favicon and launcher icon | -### 3.4 Data Layer +### 3.3 Data Layer | Store | Technology | Contents | |---|---|---| -| `data/qg.db` | SQLite | Documents, text chunks, highlights, settings | -| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`) | +| `data/qg.db` | SQLite | Documents, text chunks, highlights | +| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`, or `.npy` fallback) | | `data/pdfs/` | File system | Original PDF files copied from user selections | +| `data/models/` | File system cache | Downloaded `all-MiniLM-L6-v2` model (~80 MB, cached after first run) | -## 4 API Endpoints (C# Backend — port 8080) +## 4 API Endpoints (FastAPI Backend — port 8080) | Method | Path | Description | |---|---|---| @@ -152,51 +142,25 @@ flowchart LR | `GET` | `/api/documents` | List all documents | | `GET` | `/api/documents/{id}` | Get document details | | `GET` | `/api/documents/{id}/status` | Get ingestion progress | -| `DELETE` | `/api/documents/{id}` | Delete document + data | +| `DELETE` | `/api/documents/{id}` | Delete document + all associated data | | `GET` | `/api/documents/{id}/pdf` | Serve PDF file for viewer | -| `POST` | `/api/search` | Semantic search (proxies to Python) | -| `POST` | `/api/highlights` | Add a highlight | -| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document | +| `GET` | `/api/documents/{id}/text-positions` | Search text on a page; returns bounding-box rects for precise highlighting | +| `POST` | `/api/search` | Semantic search (embed + FAISS + SQLite enrichment) | +| `POST` | `/api/highlights` | Add a highlight annotation | +| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document (optional `?page=N` filter) | | `PUT` | `/api/highlights/{id}` | Update highlight color | | `DELETE` | `/api/highlights/{id}` | Delete a highlight | -## 5 Python Worker API (port 5001) - -| Method | Path | Description | -|---|---|---| -| `GET` | `/health` | Health check + model status | -| `POST` | `/embed` | Generate embeddings for text chunks | -| `POST` | `/search` | Embed query + FAISS search → return chunk IDs + scores | -| `POST` | `/index/add` | Add vectors to a document's FAISS index | -| `POST` | `/index/delete` | Delete a document's FAISS index | - -## 6 Inter-Process Communication - -``` -┌─────────────────────────────────┐ HTTP (localhost:5001) ┌───────────────────────────┐ -│ C# Backend (ASP.NET Core) │ ──────────────────────────► │ Python Worker (Flask) │ -│ Port 8080 │ ◄────────────────────────── │ Port 5001 │ -│ │ JSON payloads │ │ -│ • PDF parsing (PdfPig) │ │ • sentence-transformers │ -│ • Text chunking │ │ • FAISS index management │ -│ • SQLite database │ │ │ -│ • File management │ │ │ -│ • Static file serving │ │ │ -└─────────────────────────────────┘ └───────────────────────────┘ -``` - -The C# backend **launches the Python worker** as a subprocess on startup and shuts it down on exit. Communication uses simple HTTP/JSON on localhost — no sockets, no message queues. - -## 7 Key Design Decisions +## 5 Key Design Decisions | Decision | Rationale | |---|---| -| **C# primary, Python secondary** | User requested C# for speed; Python only for ML tasks that lack C# equivalents | -| **ASP.NET Core Minimal API** | Less boilerplate than controllers; fast startup; built-in DI | -| **PdfPig over iTextSharp** | Pure C#, MIT licensed, good text extraction, no Java dependency | -| **Separate Python process** | Clean separation of concerns; C# backend stays fast; Python handles ML | -| **Flask over FastAPI (worker)** | Simpler for a small internal service; no async needed for batch embedding | -| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings | +| **Python single-process** | All logic — PDF parsing, API, database, embeddings, vector search — runs in one FastAPI process; no inter-process complexity | +| **FastAPI over Flask** | Async-native, built-in Pydantic validation, automatic OpenAPI docs, faster than Flask | +| **PyMuPDF over PdfPig** | Python-native, fast C++ core, returns character-level positions needed for highlight overlays | +| **sentence-transformers in-process** | Direct function calls instead of HTTP round-trips; simpler, lower latency | +| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings; numpy fallback ensures compatibility | | **SQLite over Postgres** | Zero install; single-file DB; perfect for local-first app | | **PDF.js over native viewer** | Works in every browser; supports text-layer highlighting | | **Vanilla JS over React** | Zero build step; served as static files; the app is simple enough | +| **Thread-local SQLite connections** | Avoids connection sharing across threads; WAL mode allows concurrent reads during writes | diff --git a/docs/06_scope_and_milestones.md b/docs/06_scope_and_milestones.md index c179b37..9ee9e9a 100644 --- a/docs/06_scope_and_milestones.md +++ b/docs/06_scope_and_milestones.md @@ -9,7 +9,7 @@ ### In Scope (v1.0) - Local PDF selection from file system, parsing, and managed storage -- Text extraction (PdfPig, C#) and vector embedding (sentence-transformers, Python) +- Text extraction (PyMuPDF, Python) and vector embedding (sentence-transformers, Python) - Semantic natural-language search via prompt box - Search result → one-click page navigation in PDF.js viewer - Auto-highlight matching phrases (yellow exact-word match) @@ -47,9 +47,9 @@ gantt section Planning M0 - Planning Docs & Scaffold :done, m0, 2026-03-05, 1d - section C# Backend + section Python Backend M1 - Core Backend & PDF Parsing :m1, after m0, 3d - M2 - Python Worker Integration :m2, after m1, 2d + M2 - Embeddings & Vector Search :m2, after m1, 2d section Frontend M3 - Web UI & PDF Viewer :m3, after m2, 4d @@ -66,13 +66,13 @@ gantt ### M0 — Planning & Scaffold ✅ All 8 planning documents finalized. Repository scaffolded with folder structure. -### M1 — Core C# Backend -**Deliverables:** ASP.NET Core project, PdfPig text extraction, text chunking, SQLite database, document CRUD API, static file serving. +### M1 — Core Python Backend +**Deliverables:** FastAPI project, PyMuPDF text extraction, text chunking, SQLite database, document CRUD API, static file serving. **Definition of Done:** Upload a PDF via API → text extracted and chunks stored in SQLite. -### M2 — Python Worker Integration -**Deliverables:** Python Flask worker, sentence-transformers embedding, FAISS index, C#→Python bridge service. -**DoD:** C# backend calls Python worker → embeddings generated → stored in FAISS. +### M2 — Embeddings & Vector Search +**Deliverables:** sentence-transformers embedding service, FAISS vector index (with numpy fallback), semantic search endpoint. +**DoD:** Upload a PDF → embeddings generated → stored in FAISS → semantic search returns ranked results. ### M3 — Web UI & PDF Viewer **Deliverables:** `index.html` with cozy UI, document selector, search prompt box, results list, PDF.js viewer, responsive layout. diff --git a/docs/07_repo_structure.md b/docs/07_repo_structure.md index 766fcf2..757aa7d 100644 --- a/docs/07_repo_structure.md +++ b/docs/07_repo_structure.md @@ -1,6 +1,6 @@ -# QuickGuide (QG) — Repository Structure Plan +# QuickGuide (QG) — Repository Structure -**Version:** 1.0 +**Version:** 1.1 **Date:** 2026-03-05 --- @@ -23,65 +23,50 @@ Quick_Guide/ │ ├── 07_repo_structure.md │ └── 08_risk_register.md │ -├── src/ -│ ├── QuickGuide.Api/ # C# ASP.NET Core project -│ │ ├── QuickGuide.Api.csproj # Project file with NuGet refs -│ │ ├── Program.cs # Entry point, DI, route mapping -│ │ ├── Models/ -│ │ │ ├── Document.cs # Document entity -│ │ │ ├── TextChunk.cs # Text chunk entity -│ │ │ ├── Highlight.cs # Highlight entity -│ │ │ └── ApiModels.cs # Request/response DTOs -│ │ ├── Services/ -│ │ │ ├── PdfParserService.cs # PdfPig text extraction -│ │ │ ├── TextChunkerService.cs # Token-windowed text splitting -│ │ │ ├── DocumentService.cs # Document lifecycle (upload, list, delete) -│ │ │ ├── SearchService.cs # Search orchestration (calls Python) -│ │ │ ├── HighlightService.cs # Highlight CRUD -│ │ │ └── PythonBridgeService.cs# HTTP client → Python worker -│ │ ├── Data/ -│ │ │ └── DatabaseService.cs # SQLite setup, migrations, Dapper queries -│ │ └── wwwroot/ # Static frontend files -│ │ ├── index.html # App shell -│ │ ├── css/ -│ │ │ └── app.css # Cozy design system -│ │ ├── js/ -│ │ │ ├── app.js # UI logic, API calls, state -│ │ │ └── pdfviewer.js # PDF.js integration + highlights -│ │ ├── lib/ -│ │ │ └── pdfjs/ # PDF.js library (vendored) -│ │ └── img/ -│ │ └── qg-logo.png # QG logo for favicon + launcher +├── src/ # All application source code +│ ├── main.py # FastAPI entry point, all route definitions +│ ├── config.py # Paths, server settings, model/search params +│ ├── models.py # Pydantic request/response models +│ ├── database.py # SQLite connection, schema init, CRUD queries │ │ -│ └── python_worker/ # Python ML microservice -│ ├── worker.py # Flask app: /embed, /search, /health -│ ├── embedder.py # sentence-transformers wrapper -│ ├── vector_store.py # FAISS index management -│ └── requirements.txt # Python dependencies +│ ├── services/ # Business logic layer +│ │ ├── __init__.py +│ │ ├── pdf_parser.py # PyMuPDF text extraction + bounding-box search +│ │ ├── chunker.py # Overlapping token-windowed text splitting +│ │ ├── embedder.py # sentence-transformers wrapper (lazy load + batch) +│ │ ├── vector_store.py # FAISS index management (with numpy fallback) +│ │ ├── search.py # Semantic search orchestration +│ │ ├── documents.py # Document lifecycle + ingestion pipeline +│ │ └── highlights.py # Highlight CRUD +│ │ +│ └── static/ # Frontend static files (served by FastAPI) +│ ├── index.html # App shell — layout, search panel, PDF viewer +│ ├── css/ +│ │ └── app.css # Cozy design system — warm colors, rounded corners +│ ├── js/ +│ │ └── app.js # UI logic, API calls, PDF.js integration, state +│ └── img/ +│ └── qg-favicon.svg # QG logo used as favicon and launcher icon │ ├── tests/ -│ ├── QuickGuide.Tests/ # C# xUnit test project -│ │ ├── QuickGuide.Tests.csproj -│ │ ├── PdfParserServiceTests.cs -│ │ ├── TextChunkerServiceTests.cs -│ │ └── SearchServiceTests.cs -│ └── python_worker/ # Python tests -│ ├── test_embedder.py -│ └── test_vector_store.py +│ ├── __init__.py +│ ├── test_chunker.py # Unit tests for text chunking logic +│ └── test_database.py # Unit tests for SQLite schema and cascade deletes │ ├── data/ # Created at runtime (gitignored) │ ├── qg.db # SQLite database │ ├── pdfs/ # User's uploaded PDF files -│ └── vectors/ # FAISS index files per document +│ ├── vectors/ # FAISS index files per document +│ └── models/ # Cached sentence-transformers model (~80 MB) │ ├── qg.bat # Windows launcher (double-click to start) ├── qg.sh # macOS/Linux launcher -├── QuickGuide.sln # .NET solution file +├── requirements.txt # Python package dependencies ├── README.md # Project overview + quickstart -├── QUICKSTART.md # Detailed setup instructions +├── QUICKSTART.md # Detailed setup instructions + troubleshooting ├── LICENSE # MIT License -├── .gitignore # Ignores data/, bin/, obj/, .venv/, etc. -└── .editorconfig # Consistent code style +├── .gitignore # Ignores data/, __pycache__/, .venv/, etc. +└── .editorconfig # Consistent code style across editors ``` ## 2 PDF Storage Location @@ -97,24 +82,22 @@ Quick_Guide/ | Element | Convention | Example | |---|---|---| -| C# files | PascalCase | `PdfParserService.cs` | -| C# classes | PascalCase | `DocumentService` | -| C# methods | PascalCase | `ExtractText()` | -| C# properties | PascalCase | `PageNumber` | -| Python modules | snake_case | `vector_store.py` | -| Python functions | snake_case | `generate_embeddings()` | +| Python modules | snake_case | `pdf_parser.py` | +| Python functions | snake_case | `extract_text()` | +| Python classes | PascalCase | `VectorStore` | +| Pydantic models | PascalCase | `SearchRequest` | | API routes | kebab-case paths | `/api/documents/upload` | | DB tables | snake_case | `text_chunks` | -| JS functions | camelCase | `loadPdf()` | +| JS functions | camelCase | `loadDocuments()` | | CSS classes | BEM-like `qg-` prefix | `qg-search-input` | ## 4 Key Directories Explained | Directory | Purpose | Gitignored? | |---|---|---| -| `src/QuickGuide.Api/` | All C# backend code | No | -| `src/python_worker/` | Python ML worker (embedding + vector search) | No | -| `src/QuickGuide.Api/wwwroot/` | Static frontend (HTML/CSS/JS) served by C# | No | -| `data/` | Runtime data (DB, PDFs, FAISS indexes) | **Yes** | -| `tests/` | All automated tests (C# + Python) | No | +| `src/` | All Python backend source code | No | +| `src/services/` | Modular business logic (parsing, embedding, search, highlights) | No | +| `src/static/` | Static frontend (HTML/CSS/JS) served by FastAPI | No | +| `data/` | Runtime data (DB, PDFs, FAISS indexes, model cache) | **Yes** | +| `tests/` | Automated Python unit tests | No | | `docs/` | Planning and reference documents | No | diff --git a/docs/08_risk_register.md b/docs/08_risk_register.md index 37569a2..3d1af81 100644 --- a/docs/08_risk_register.md +++ b/docs/08_risk_register.md @@ -33,7 +33,7 @@ | **Probability** | Medium | | **Impact** | Medium | | **Description** | A 2,000-page PDF with dense tables may exceed the 2-minute ingestion target. | -| **Mitigation** | C# PdfPig is fast for extraction. Batch embedding calls (32 chunks/batch). Background ingestion with progress updates. | +| **Mitigation** | Python PyMuPDF is fast for extraction. Batch embedding calls (32 chunks/batch). Background ingestion with progress updates. | | **Contingency** | Show realistic progress; allow the user to search partially-ingested documents. | ### R-03 🟡 — PDF.js Rendering on Mobile Browsers @@ -56,15 +56,15 @@ | **Mitigation** | Combine vector search with keyword matching (hybrid ranking). Tune chunk size and overlap. | | **Contingency** | Swap to a larger model (`bge-base-en-v1.5`, 110 MB) or add a BM25 keyword fallback. | -### R-05 🟡 — Dual Runtime Complexity (.NET + Python) +### R-05 🟡 — Setup Complexity (Python + Dependencies) | Field | Detail | |---|---| | **Category** | User Experience | | **Probability** | Medium | | **Impact** | Medium | -| **Description** | Users need both .NET 8 SDK and Python 3.11+ installed. This is more complex than a single-runtime app. | -| **Mitigation** | The `qg.bat` / `qg.sh` launcher checks for both and prints clear install instructions if missing. QUICKSTART.md has step-by-step setup. | -| **Contingency** | Provide a Docker Compose option for zero-install. Future: bundle with PyInstaller + self-contained .NET publish. | +| **Description** | Users need Python 3.11+ installed. First-time setup includes virtual environment creation and dependency installation (~80 MB model download). | +| **Mitigation** | The `qg.bat` / `qg.sh` launcher automates venv creation and pip install, printing clear instructions if Python is missing. QUICKSTART.md has step-by-step setup. | +| **Contingency** | Provide a Docker Compose option for zero-install. Future: bundle with PyInstaller for a single-executable distribution. | ### R-06 🟢 — FAISS Wheel Unavailable on Some Platforms | Field | Detail | @@ -91,18 +91,18 @@ | **Category** | Technical | | **Probability** | Low | | **Impact** | Medium | -| **Description** | Text extracted by PdfPig may not match PDF.js text layer character-for-character. | +| **Description** | Text extracted by PyMuPDF may not match PDF.js text layer character-for-character. | | **Mitigation** | Use fuzzy substring matching in the frontend when searching the PDF.js text layer. | -### R-09 🟢 — Python Worker Fails to Start +### R-09 🟢 — FastAPI Process Fails to Start | Field | Detail | |---|---| | **Category** | Technical | | **Probability** | Low | | **Impact** | High | -| **Description** | Python worker subprocess may fail (missing packages, wrong Python version). | -| **Mitigation** | C# backend health-checks the worker on startup with retries. Launcher script validates Python and pip before starting. | -| **Contingency** | Show a clear error in the UI: "ML worker not running — search disabled. Check terminal for details." | +| **Description** | The FastAPI Python process may fail to start (missing packages, wrong Python version, port already in use). | +| **Mitigation** | Launcher script validates Python and pip before starting. FastAPI startup errors are logged clearly in the terminal. | +| **Contingency** | Show a clear error in the UI if API calls fail. QUICKSTART.md documents common failure modes and fixes (e.g., `QG_PORT` env var for port conflicts). | ### R-10 🟢 — Scope Creep | Field | Detail |