diff --git a/docs/03_functional_requirements.md b/docs/03_functional_requirements.md
index c494fa6..2270d05 100644
--- a/docs/03_functional_requirements.md
+++ b/docs/03_functional_requirements.md
@@ -37,7 +37,7 @@
| **Description** | Each text chunk is converted to a 384-dimensional vector embedding using `all-MiniLM-L6-v2`. |
| **Input** | Text chunk content. |
| **Output** | Float32 embedding vector stored in the vector index and referenced in the database. |
-| **Rules** | Embedding is generated via a Python worker subprocess. Batch size of 32 chunks for efficiency. |
+| **Rules** | Embedding is generated in-process within the FastAPI application. Batch size of 32 chunks for efficiency. |
### FR-ING-05 — Progress Feedback
| Field | Detail |
diff --git a/docs/04_system_requirements.md b/docs/04_system_requirements.md
index bf3f1a3..fb9c09b 100644
--- a/docs/04_system_requirements.md
+++ b/docs/04_system_requirements.md
@@ -1,6 +1,6 @@
# QuickGuide (QG) — System Requirements Document (SRD)
-**Version:** 1.0
+**Version:** 1.1
**Date:** 2026-03-05
---
@@ -17,8 +17,8 @@
| Platform | Minimum Version | Notes |
|---|---|---|
| Windows | 10 | Primary development target |
-| macOS | 12 Monterey | .NET 8 and Python supported |
-| Linux | Ubuntu 22.04 / Fedora 38 | .NET 8 and Python supported |
+| macOS | 12 Monterey | Python supported |
+| Linux | Ubuntu 22.04 / Fedora 38 | Python supported |
| Android / iOS | Any modern browser | Access via `http://localhost:8080` |
> [!NOTE]
@@ -26,54 +26,46 @@
## 3 Software Dependencies
-### 3.1 C# Backend (Primary — built for speed)
+### 3.1 Python Backend (single process — handles all logic and ML)
| Component | Technology | Version | Rationale |
|---|---|---|---|
-| **Runtime** | .NET 8 SDK | 8.0+ | Cross-platform, high-performance |
-| **Web framework** | ASP.NET Core Minimal API | 8.0 | Fast HTTP, built-in DI, static file serving |
-| **PDF parsing** | PdfPig | latest | Pure C#, no native dependencies, page-level text extraction |
-| **Database** | SQLite via Microsoft.Data.Sqlite | latest | Zero-config, file-based, fast |
-| **ORM** | Dapper | latest | Lightweight micro-ORM for C# |
-| **JSON** | System.Text.Json | built-in | High-performance serialization |
-| **Process management** | System.Diagnostics.Process | built-in | Launches Python worker subprocess |
-
-### 3.2 Python ML Worker (for embedding + vector search)
-
-| Component | Technology | Version | Rationale |
-|---|---|---|---|
-| **Runtime** | Python | 3.11+ | ML ecosystem support |
-| **Embedding model** | sentence-transformers | latest | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly |
-| **Vector index** | FAISS-cpu | latest | Efficient ANN search, widely supported |
-| **HTTP server** | Flask | latest | Lightweight API for C# → Python communication |
-| **Protocol** | HTTP on localhost:5001 | — | C# backend calls Python worker via REST |
-
-### 3.3 Frontend
+| **Runtime** | Python | 3.11+ | ML ecosystem support; cross-platform |
+| **Web framework** | FastAPI | 0.115.0+ | High-performance async API; built-in data validation |
+| **ASGI server** | uvicorn[standard] | 0.30.0+ | Production-grade ASGI server for FastAPI |
+| **PDF parsing** | PyMuPDF (fitz) | 1.24.0+ | Fast text extraction with character positions; no native dependencies |
+| **File uploads** | python-multipart | 0.0.9+ | Multipart form-data support for FastAPI |
+| **Async file I/O** | aiofiles | 24.1.0+ | Async file operations |
+| **Database** | SQLite via Python stdlib `sqlite3` | built-in | Zero-config, file-based, fast |
+| **Embedding model** | sentence-transformers | 3.0.0+ | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly |
+| **Vector index** | faiss-cpu | 1.8.0+ | Efficient ANN search (numpy fallback if unavailable) |
+| **Numerics** | numpy | 1.26.0–<2.0.0 | Vector operations and numpy fallback search |
+
+### 3.2 Frontend
| Component | Technology | Rationale |
|---|---|---|
| **PDF rendering** | PDF.js (Mozilla) | Industry-standard browser PDF viewer |
-| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served by C# backend |
+| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served as static files by the FastAPI backend |
| **Font** | Nunito (Google Fonts, self-hosted) | Warm, rounded, cozy aesthetic |
-### 3.4 Build / Install Prerequisites
+### 3.3 Build / Install Prerequisites
| Tool | Version | Notes |
|---|---|---|
-| .NET SDK | 8.0+ | For building and running the C# backend |
-| Python | 3.11+ | For the ML worker only |
-| pip | bundled with Python | Installs sentence-transformers + FAISS |
+| Python | 3.11+ | The only runtime required |
+| pip | bundled with Python | Installs all dependencies from `requirements.txt` |
| Git | any | For version control |
> [!IMPORTANT]
-> **Two runtimes required:** .NET 8 SDK (for the C# backend) and Python 3.11+ (for the ML embedding worker). Both are free and cross-platform.
+> **One runtime required:** Python 3.11+ only. There is no separate .NET or C# dependency.
## 4 Performance Targets
| Metric | Target |
|---|---|
-| PDF text extraction (1,000 pages) | ≤ 30 s (C# PdfPig) |
-| Embedding generation (1,000 pages) | ≤ 90 s (Python worker) |
+| PDF text extraction (1,000 pages) | ≤ 30 s (Python PyMuPDF) |
+| Embedding generation (1,000 pages) | ≤ 90 s (sentence-transformers, batch=32) |
| Total ingestion (1,000 pages) | ≤ 120 s |
| Query → results | ≤ 3 s |
| App start (backend + browser) | ≤ 5 s |
@@ -84,22 +76,23 @@
- **No telemetry** — no external network calls (except first-time model download).
- **Local-only storage** — PDFs, text, and embeddings stay on the device.
-- **Localhost binding** — C# backend binds to `127.0.0.1:8080`; Python worker to `127.0.0.1:5001`.
+- **Localhost binding** — FastAPI backend binds to `127.0.0.1:8080` by default (configurable via `QG_HOST` / `QG_PORT` env vars).
- **No authentication** — single-user, local application.
## 6 Deployment Strategy
| Platform | Delivery |
|---|---|
-| Windows | `qg.bat` double-click launcher → starts backend + opens browser |
+| Windows | `qg.bat` double-click launcher → creates venv, installs deps, starts Python server, opens browser |
| macOS / Linux | `qg.sh` launcher script |
| Mobile | Access via browser on same network |
-| Packaged (future) | Self-contained .NET publish + PyInstaller bundle |
+| Packaged (future) | PyInstaller bundle for zero-install distribution |
## 7 Architecture Constraints
-1. **C# for all non-ML logic** — PDF parsing, API, database, file management all in C#.
-2. **Python only for ML** — embedding generation and vector search in a separate Python process.
-3. **No external servers** — everything on localhost.
-4. **Zero frontend build step** — HTML/CSS/JS served as static files by the C# backend.
+1. **Python for all logic** — PDF parsing, API, database, file management, embeddings, and vector search all run in a single Python process.
+2. **FastAPI as the web framework** — handles all HTTP routing, static file serving, and data validation.
+3. **No external servers** — everything on localhost; no inter-process HTTP communication required.
+4. **Zero frontend build step** — HTML/CSS/JS served as static files by the FastAPI backend.
5. **SQLite only** — no database server installation required.
+6. **FAISS with numpy fallback** — if `faiss-cpu` is unavailable, flat cosine similarity via numpy is used automatically.
diff --git a/docs/05_architecture.md b/docs/05_architecture.md
index f03e729..36b381c 100644
--- a/docs/05_architecture.md
+++ b/docs/05_architecture.md
@@ -1,6 +1,6 @@
# QuickGuide (QG) — Architecture Document
-**Version:** 1.0
+**Version:** 1.1
**Date:** 2026-03-05
---
@@ -12,24 +12,22 @@ graph TB
subgraph UserDevice["User's Device"]
Browser["Web Browser
(any platform)"]
- subgraph CSharpBackend["C# Backend — ASP.NET Core 8"]
- API["REST API
(Minimal API)"]
- PdfSvc["PDF Parser
(PdfPig)"]
+ subgraph PythonBackend["Python Backend — FastAPI"]
+ API["REST API
(FastAPI routes)"]
+ PdfSvc["PDF Parser
(PyMuPDF / fitz)"]
ChunkSvc["Text Chunker"]
DocMgr["Document Manager"]
HighlightSvc["Highlight Service"]
- DbSvc["Database Service
(SQLite + Dapper)"]
- end
-
- subgraph PythonWorker["Python ML Worker — Flask"]
EmbedSvc["Embedding Service
(sentence-transformers)"]
- VectorSvc["Vector Search
(FAISS)"]
+ VectorSvc["Vector Search
(FAISS / numpy)"]
+ DbSvc["Database Service
(SQLite)"]
end
subgraph DataLayer["Data Layer"]
SQLite["SQLite DB
(qg.db)"]
VectorIdx["FAISS Index Files
(data/vectors/)"]
FileStore["PDF File Store
(data/pdfs/)"]
+ ModelCache["Model Cache
(data/models/)"]
end
subgraph Frontend["Static Frontend"]
@@ -44,11 +42,10 @@ graph TB
API --> ChunkSvc
API --> DocMgr
API --> HighlightSvc
+ API --> EmbedSvc
+ API --> VectorSvc
API --> DbSvc
- API -- "HTTP localhost:5001" --> EmbedSvc
- API -- "HTTP localhost:5001" --> VectorSvc
-
PdfSvc --> FileStore
ChunkSvc --> DbSvc
DbSvc --> SQLite
@@ -56,7 +53,7 @@ graph TB
DocMgr --> SQLite
HighlightSvc --> SQLite
- EmbedSvc --> VectorIdx
+ EmbedSvc --> ModelCache
VectorSvc --> VectorIdx
```
@@ -66,28 +63,26 @@ graph TB
```mermaid
flowchart LR
- A["User selects PDF"] --> B["C#: Copy to
data/pdfs/"]
- B --> C["C#: Extract text
(PdfPig, per page)"]
- C --> D["C#: Chunk text
(512 tok / 50 overlap)"]
- D --> E["C# → Python:
POST /embed"]
- E --> F["Python: Generate
embeddings
(sentence-transformers)"]
- F --> G["Python: Add to
FAISS index"]
- G --> H["C#: Store metadata
in SQLite"]
- H --> I["Ready for Search ✓"]
+ A["User selects PDF"] --> B["FastAPI: Copy to
data/pdfs/"]
+ B --> C["PyMuPDF: Extract text
(per page, with char offsets)"]
+ C --> D["Chunker: Split text
(512 tok / 50 overlap)"]
+ D --> E["sentence-transformers:
Generate embeddings
(batch=32)"]
+ E --> F["FAISS: Add vectors
to per-document index"]
+ F --> G["SQLite: Store metadata,
chunks + status = ready"]
+ G --> H["Ready for Search ✓"]
```
### 2.2 Search Flow
```mermaid
flowchart LR
- A["User types query"] --> B["C# API receives
search request"]
- B --> C["C# → Python:
POST /search"]
- C --> D["Python: Embed query
(sentence-transformers)"]
- D --> E["Python: FAISS ANN
search (cosine sim)"]
- E --> F["Python returns
top-K chunk IDs + scores"]
- F --> G["C#: Fetch chunk text
from SQLite"]
- G --> H["C# returns JSON
results to browser"]
- H --> I["Browser: Navigate
PDF.js + Highlight"]
+ A["User types query"] --> B["FastAPI API receives
search request"]
+ B --> C["sentence-transformers:
Embed query (384-dim)"]
+ C --> D["FAISS (or numpy):
ANN search (cosine sim)"]
+ D --> E["Filter results:
similarity ≥ 0.30"]
+ E --> F["SQLite: Fetch
chunk text + page numbers"]
+ F --> G["FastAPI returns
JSON results to browser"]
+ G --> H["Browser: Navigate
PDF.js + Highlight"]
```
### 2.3 Navigation & Highlighting Flow
@@ -104,46 +99,41 @@ flowchart LR
## 3 Component Breakdown
-### 3.1 C# Backend (ASP.NET Core 8 Minimal API)
-
-| File | Responsibility |
-|---|---|
-| `Program.cs` | App entry point, DI registration, route mapping, static file serving |
-| `Services/PdfParserService.cs` | PdfPig text extraction — extracts text per page with character positions |
-| `Services/TextChunkerService.cs` | Splits page text into overlapping token-windowed chunks |
-| `Services/DocumentService.cs` | Document lifecycle: upload, list, delete, status tracking |
-| `Services/SearchService.cs` | Orchestrates search: calls Python worker, merges with SQLite metadata |
-| `Services/HighlightService.cs` | Highlight CRUD: create, read, update color, delete |
-| `Services/PythonBridgeService.cs` | HTTP client that communicates with the Python ML worker |
-| `Data/DatabaseService.cs` | SQLite connection, migrations, Dapper queries |
-| `Models/` | C# record types for API requests/responses and DB entities |
-
-### 3.2 Python ML Worker (Flask)
+### 3.1 Python Backend (FastAPI — single process, port 8080)
| File | Responsibility |
|---|---|
-| `worker.py` | Flask app: `/embed`, `/search`, `/health` endpoints |
-| `embedder.py` | Loads `all-MiniLM-L6-v2`, generates 384-dim embeddings |
-| `vector_store.py` | FAISS index management: add vectors, search, save/load, delete |
-
-### 3.3 Frontend (HTML / CSS / Vanilla JS)
+| `src/main.py` | App entry point, FastAPI routes, static file serving, browser auto-launch |
+| `src/config.py` | All configuration: paths, server settings, model params, search thresholds |
+| `src/models.py` | Pydantic request/response models (SearchRequest, HighlightCreate, etc.) |
+| `src/database.py` | SQLite connection (thread-local, WAL mode), schema init, CRUD queries |
+| `src/services/pdf_parser.py` | PyMuPDF text extraction per page; bounding-box search for highlighting |
+| `src/services/chunker.py` | Splits page text into overlapping 512-token windows with 50-token overlap |
+| `src/services/embedder.py` | Lazy-loads `all-MiniLM-L6-v2`; batch encodes text; normalizes to float32 |
+| `src/services/vector_store.py` | Per-document FAISS index (or numpy fallback); add, search, save, load, delete |
+| `src/services/search.py` | Semantic search orchestration: embed → vector search → filter → enrich |
+| `src/services/documents.py` | Document lifecycle: upload, ingest pipeline, list, get, delete, progress tracking |
+| `src/services/highlights.py` | Highlight CRUD: add, get, update color, delete |
+
+### 3.2 Frontend (HTML / CSS / Vanilla JS)
| File | Responsibility |
|---|---|
-| `index.html` | App shell — layout, PDF viewer container, search panel |
-| `css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows |
-| `js/app.js` | API calls, search orchestration, state management, document selector |
-| `js/pdfviewer.js` | PDF.js rendering, page navigation, text-layer highlight overlays, match jumping |
+| `src/static/index.html` | App shell — layout, PDF viewer container, search panel, highlights tab |
+| `src/static/css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows |
+| `src/static/js/app.js` | API calls, search orchestration, PDF.js integration, state management, highlights |
+| `src/static/img/qg-favicon.svg` | QG logo used as favicon and launcher icon |
-### 3.4 Data Layer
+### 3.3 Data Layer
| Store | Technology | Contents |
|---|---|---|
-| `data/qg.db` | SQLite | Documents, text chunks, highlights, settings |
-| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`) |
+| `data/qg.db` | SQLite | Documents, text chunks, highlights |
+| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`, or `.npy` fallback) |
| `data/pdfs/` | File system | Original PDF files copied from user selections |
+| `data/models/` | File system cache | Downloaded `all-MiniLM-L6-v2` model (~80 MB, cached after first run) |
-## 4 API Endpoints (C# Backend — port 8080)
+## 4 API Endpoints (FastAPI Backend — port 8080)
| Method | Path | Description |
|---|---|---|
@@ -152,51 +142,25 @@ flowchart LR
| `GET` | `/api/documents` | List all documents |
| `GET` | `/api/documents/{id}` | Get document details |
| `GET` | `/api/documents/{id}/status` | Get ingestion progress |
-| `DELETE` | `/api/documents/{id}` | Delete document + data |
+| `DELETE` | `/api/documents/{id}` | Delete document + all associated data |
| `GET` | `/api/documents/{id}/pdf` | Serve PDF file for viewer |
-| `POST` | `/api/search` | Semantic search (proxies to Python) |
-| `POST` | `/api/highlights` | Add a highlight |
-| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document |
+| `GET` | `/api/documents/{id}/text-positions` | Search text on a page; returns bounding-box rects for precise highlighting |
+| `POST` | `/api/search` | Semantic search (embed + FAISS + SQLite enrichment) |
+| `POST` | `/api/highlights` | Add a highlight annotation |
+| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document (optional `?page=N` filter) |
| `PUT` | `/api/highlights/{id}` | Update highlight color |
| `DELETE` | `/api/highlights/{id}` | Delete a highlight |
-## 5 Python Worker API (port 5001)
-
-| Method | Path | Description |
-|---|---|---|
-| `GET` | `/health` | Health check + model status |
-| `POST` | `/embed` | Generate embeddings for text chunks |
-| `POST` | `/search` | Embed query + FAISS search → return chunk IDs + scores |
-| `POST` | `/index/add` | Add vectors to a document's FAISS index |
-| `POST` | `/index/delete` | Delete a document's FAISS index |
-
-## 6 Inter-Process Communication
-
-```
-┌─────────────────────────────────┐ HTTP (localhost:5001) ┌───────────────────────────┐
-│ C# Backend (ASP.NET Core) │ ──────────────────────────► │ Python Worker (Flask) │
-│ Port 8080 │ ◄────────────────────────── │ Port 5001 │
-│ │ JSON payloads │ │
-│ • PDF parsing (PdfPig) │ │ • sentence-transformers │
-│ • Text chunking │ │ • FAISS index management │
-│ • SQLite database │ │ │
-│ • File management │ │ │
-│ • Static file serving │ │ │
-└─────────────────────────────────┘ └───────────────────────────┘
-```
-
-The C# backend **launches the Python worker** as a subprocess on startup and shuts it down on exit. Communication uses simple HTTP/JSON on localhost — no sockets, no message queues.
-
-## 7 Key Design Decisions
+## 5 Key Design Decisions
| Decision | Rationale |
|---|---|
-| **C# primary, Python secondary** | User requested C# for speed; Python only for ML tasks that lack C# equivalents |
-| **ASP.NET Core Minimal API** | Less boilerplate than controllers; fast startup; built-in DI |
-| **PdfPig over iTextSharp** | Pure C#, MIT licensed, good text extraction, no Java dependency |
-| **Separate Python process** | Clean separation of concerns; C# backend stays fast; Python handles ML |
-| **Flask over FastAPI (worker)** | Simpler for a small internal service; no async needed for batch embedding |
-| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings |
+| **Python single-process** | All logic — PDF parsing, API, database, embeddings, vector search — runs in one FastAPI process; no inter-process complexity |
+| **FastAPI over Flask** | Async-native, built-in Pydantic validation, automatic OpenAPI docs, faster than Flask |
+| **PyMuPDF over PdfPig** | Python-native, fast C++ core, returns character-level positions needed for highlight overlays |
+| **sentence-transformers in-process** | Direct function calls instead of HTTP round-trips; simpler, lower latency |
+| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings; numpy fallback ensures compatibility |
| **SQLite over Postgres** | Zero install; single-file DB; perfect for local-first app |
| **PDF.js over native viewer** | Works in every browser; supports text-layer highlighting |
| **Vanilla JS over React** | Zero build step; served as static files; the app is simple enough |
+| **Thread-local SQLite connections** | Avoids connection sharing across threads; WAL mode allows concurrent reads during writes |
diff --git a/docs/06_scope_and_milestones.md b/docs/06_scope_and_milestones.md
index c179b37..9ee9e9a 100644
--- a/docs/06_scope_and_milestones.md
+++ b/docs/06_scope_and_milestones.md
@@ -9,7 +9,7 @@
### In Scope (v1.0)
- Local PDF selection from file system, parsing, and managed storage
-- Text extraction (PdfPig, C#) and vector embedding (sentence-transformers, Python)
+- Text extraction (PyMuPDF, Python) and vector embedding (sentence-transformers, Python)
- Semantic natural-language search via prompt box
- Search result → one-click page navigation in PDF.js viewer
- Auto-highlight matching phrases (yellow exact-word match)
@@ -47,9 +47,9 @@ gantt
section Planning
M0 - Planning Docs & Scaffold :done, m0, 2026-03-05, 1d
- section C# Backend
+ section Python Backend
M1 - Core Backend & PDF Parsing :m1, after m0, 3d
- M2 - Python Worker Integration :m2, after m1, 2d
+ M2 - Embeddings & Vector Search :m2, after m1, 2d
section Frontend
M3 - Web UI & PDF Viewer :m3, after m2, 4d
@@ -66,13 +66,13 @@ gantt
### M0 — Planning & Scaffold ✅
All 8 planning documents finalized. Repository scaffolded with folder structure.
-### M1 — Core C# Backend
-**Deliverables:** ASP.NET Core project, PdfPig text extraction, text chunking, SQLite database, document CRUD API, static file serving.
+### M1 — Core Python Backend
+**Deliverables:** FastAPI project, PyMuPDF text extraction, text chunking, SQLite database, document CRUD API, static file serving.
**Definition of Done:** Upload a PDF via API → text extracted and chunks stored in SQLite.
-### M2 — Python Worker Integration
-**Deliverables:** Python Flask worker, sentence-transformers embedding, FAISS index, C#→Python bridge service.
-**DoD:** C# backend calls Python worker → embeddings generated → stored in FAISS.
+### M2 — Embeddings & Vector Search
+**Deliverables:** sentence-transformers embedding service, FAISS vector index (with numpy fallback), semantic search endpoint.
+**DoD:** Upload a PDF → embeddings generated → stored in FAISS → semantic search returns ranked results.
### M3 — Web UI & PDF Viewer
**Deliverables:** `index.html` with cozy UI, document selector, search prompt box, results list, PDF.js viewer, responsive layout.
diff --git a/docs/07_repo_structure.md b/docs/07_repo_structure.md
index 766fcf2..757aa7d 100644
--- a/docs/07_repo_structure.md
+++ b/docs/07_repo_structure.md
@@ -1,6 +1,6 @@
-# QuickGuide (QG) — Repository Structure Plan
+# QuickGuide (QG) — Repository Structure
-**Version:** 1.0
+**Version:** 1.1
**Date:** 2026-03-05
---
@@ -23,65 +23,50 @@ Quick_Guide/
│ ├── 07_repo_structure.md
│ └── 08_risk_register.md
│
-├── src/
-│ ├── QuickGuide.Api/ # C# ASP.NET Core project
-│ │ ├── QuickGuide.Api.csproj # Project file with NuGet refs
-│ │ ├── Program.cs # Entry point, DI, route mapping
-│ │ ├── Models/
-│ │ │ ├── Document.cs # Document entity
-│ │ │ ├── TextChunk.cs # Text chunk entity
-│ │ │ ├── Highlight.cs # Highlight entity
-│ │ │ └── ApiModels.cs # Request/response DTOs
-│ │ ├── Services/
-│ │ │ ├── PdfParserService.cs # PdfPig text extraction
-│ │ │ ├── TextChunkerService.cs # Token-windowed text splitting
-│ │ │ ├── DocumentService.cs # Document lifecycle (upload, list, delete)
-│ │ │ ├── SearchService.cs # Search orchestration (calls Python)
-│ │ │ ├── HighlightService.cs # Highlight CRUD
-│ │ │ └── PythonBridgeService.cs# HTTP client → Python worker
-│ │ ├── Data/
-│ │ │ └── DatabaseService.cs # SQLite setup, migrations, Dapper queries
-│ │ └── wwwroot/ # Static frontend files
-│ │ ├── index.html # App shell
-│ │ ├── css/
-│ │ │ └── app.css # Cozy design system
-│ │ ├── js/
-│ │ │ ├── app.js # UI logic, API calls, state
-│ │ │ └── pdfviewer.js # PDF.js integration + highlights
-│ │ ├── lib/
-│ │ │ └── pdfjs/ # PDF.js library (vendored)
-│ │ └── img/
-│ │ └── qg-logo.png # QG logo for favicon + launcher
+├── src/ # All application source code
+│ ├── main.py # FastAPI entry point, all route definitions
+│ ├── config.py # Paths, server settings, model/search params
+│ ├── models.py # Pydantic request/response models
+│ ├── database.py # SQLite connection, schema init, CRUD queries
│ │
-│ └── python_worker/ # Python ML microservice
-│ ├── worker.py # Flask app: /embed, /search, /health
-│ ├── embedder.py # sentence-transformers wrapper
-│ ├── vector_store.py # FAISS index management
-│ └── requirements.txt # Python dependencies
+│ ├── services/ # Business logic layer
+│ │ ├── __init__.py
+│ │ ├── pdf_parser.py # PyMuPDF text extraction + bounding-box search
+│ │ ├── chunker.py # Overlapping token-windowed text splitting
+│ │ ├── embedder.py # sentence-transformers wrapper (lazy load + batch)
+│ │ ├── vector_store.py # FAISS index management (with numpy fallback)
+│ │ ├── search.py # Semantic search orchestration
+│ │ ├── documents.py # Document lifecycle + ingestion pipeline
+│ │ └── highlights.py # Highlight CRUD
+│ │
+│ └── static/ # Frontend static files (served by FastAPI)
+│ ├── index.html # App shell — layout, search panel, PDF viewer
+│ ├── css/
+│ │ └── app.css # Cozy design system — warm colors, rounded corners
+│ ├── js/
+│ │ └── app.js # UI logic, API calls, PDF.js integration, state
+│ └── img/
+│ └── qg-favicon.svg # QG logo used as favicon and launcher icon
│
├── tests/
-│ ├── QuickGuide.Tests/ # C# xUnit test project
-│ │ ├── QuickGuide.Tests.csproj
-│ │ ├── PdfParserServiceTests.cs
-│ │ ├── TextChunkerServiceTests.cs
-│ │ └── SearchServiceTests.cs
-│ └── python_worker/ # Python tests
-│ ├── test_embedder.py
-│ └── test_vector_store.py
+│ ├── __init__.py
+│ ├── test_chunker.py # Unit tests for text chunking logic
+│ └── test_database.py # Unit tests for SQLite schema and cascade deletes
│
├── data/ # Created at runtime (gitignored)
│ ├── qg.db # SQLite database
│ ├── pdfs/ # User's uploaded PDF files
-│ └── vectors/ # FAISS index files per document
+│ ├── vectors/ # FAISS index files per document
+│ └── models/ # Cached sentence-transformers model (~80 MB)
│
├── qg.bat # Windows launcher (double-click to start)
├── qg.sh # macOS/Linux launcher
-├── QuickGuide.sln # .NET solution file
+├── requirements.txt # Python package dependencies
├── README.md # Project overview + quickstart
-├── QUICKSTART.md # Detailed setup instructions
+├── QUICKSTART.md # Detailed setup instructions + troubleshooting
├── LICENSE # MIT License
-├── .gitignore # Ignores data/, bin/, obj/, .venv/, etc.
-└── .editorconfig # Consistent code style
+├── .gitignore # Ignores data/, __pycache__/, .venv/, etc.
+└── .editorconfig # Consistent code style across editors
```
## 2 PDF Storage Location
@@ -97,24 +82,22 @@ Quick_Guide/
| Element | Convention | Example |
|---|---|---|
-| C# files | PascalCase | `PdfParserService.cs` |
-| C# classes | PascalCase | `DocumentService` |
-| C# methods | PascalCase | `ExtractText()` |
-| C# properties | PascalCase | `PageNumber` |
-| Python modules | snake_case | `vector_store.py` |
-| Python functions | snake_case | `generate_embeddings()` |
+| Python modules | snake_case | `pdf_parser.py` |
+| Python functions | snake_case | `extract_text()` |
+| Python classes | PascalCase | `VectorStore` |
+| Pydantic models | PascalCase | `SearchRequest` |
| API routes | kebab-case paths | `/api/documents/upload` |
| DB tables | snake_case | `text_chunks` |
-| JS functions | camelCase | `loadPdf()` |
+| JS functions | camelCase | `loadDocuments()` |
| CSS classes | BEM-like `qg-` prefix | `qg-search-input` |
## 4 Key Directories Explained
| Directory | Purpose | Gitignored? |
|---|---|---|
-| `src/QuickGuide.Api/` | All C# backend code | No |
-| `src/python_worker/` | Python ML worker (embedding + vector search) | No |
-| `src/QuickGuide.Api/wwwroot/` | Static frontend (HTML/CSS/JS) served by C# | No |
-| `data/` | Runtime data (DB, PDFs, FAISS indexes) | **Yes** |
-| `tests/` | All automated tests (C# + Python) | No |
+| `src/` | All Python backend source code | No |
+| `src/services/` | Modular business logic (parsing, embedding, search, highlights) | No |
+| `src/static/` | Static frontend (HTML/CSS/JS) served by FastAPI | No |
+| `data/` | Runtime data (DB, PDFs, FAISS indexes, model cache) | **Yes** |
+| `tests/` | Automated Python unit tests | No |
| `docs/` | Planning and reference documents | No |
diff --git a/docs/08_risk_register.md b/docs/08_risk_register.md
index 37569a2..3d1af81 100644
--- a/docs/08_risk_register.md
+++ b/docs/08_risk_register.md
@@ -33,7 +33,7 @@
| **Probability** | Medium |
| **Impact** | Medium |
| **Description** | A 2,000-page PDF with dense tables may exceed the 2-minute ingestion target. |
-| **Mitigation** | C# PdfPig is fast for extraction. Batch embedding calls (32 chunks/batch). Background ingestion with progress updates. |
+| **Mitigation** | Python PyMuPDF is fast for extraction. Batch embedding calls (32 chunks/batch). Background ingestion with progress updates. |
| **Contingency** | Show realistic progress; allow the user to search partially-ingested documents. |
### R-03 🟡 — PDF.js Rendering on Mobile Browsers
@@ -56,15 +56,15 @@
| **Mitigation** | Combine vector search with keyword matching (hybrid ranking). Tune chunk size and overlap. |
| **Contingency** | Swap to a larger model (`bge-base-en-v1.5`, 110 MB) or add a BM25 keyword fallback. |
-### R-05 🟡 — Dual Runtime Complexity (.NET + Python)
+### R-05 🟡 — Setup Complexity (Python + Dependencies)
| Field | Detail |
|---|---|
| **Category** | User Experience |
| **Probability** | Medium |
| **Impact** | Medium |
-| **Description** | Users need both .NET 8 SDK and Python 3.11+ installed. This is more complex than a single-runtime app. |
-| **Mitigation** | The `qg.bat` / `qg.sh` launcher checks for both and prints clear install instructions if missing. QUICKSTART.md has step-by-step setup. |
-| **Contingency** | Provide a Docker Compose option for zero-install. Future: bundle with PyInstaller + self-contained .NET publish. |
+| **Description** | Users need Python 3.11+ installed. First-time setup includes virtual environment creation and dependency installation (~80 MB model download). |
+| **Mitigation** | The `qg.bat` / `qg.sh` launcher automates venv creation and pip install, printing clear instructions if Python is missing. QUICKSTART.md has step-by-step setup. |
+| **Contingency** | Provide a Docker Compose option for zero-install. Future: bundle with PyInstaller for a single-executable distribution. |
### R-06 🟢 — FAISS Wheel Unavailable on Some Platforms
| Field | Detail |
@@ -91,18 +91,18 @@
| **Category** | Technical |
| **Probability** | Low |
| **Impact** | Medium |
-| **Description** | Text extracted by PdfPig may not match PDF.js text layer character-for-character. |
+| **Description** | Text extracted by PyMuPDF may not match PDF.js text layer character-for-character. |
| **Mitigation** | Use fuzzy substring matching in the frontend when searching the PDF.js text layer. |
-### R-09 🟢 — Python Worker Fails to Start
+### R-09 🟢 — FastAPI Process Fails to Start
| Field | Detail |
|---|---|
| **Category** | Technical |
| **Probability** | Low |
| **Impact** | High |
-| **Description** | Python worker subprocess may fail (missing packages, wrong Python version). |
-| **Mitigation** | C# backend health-checks the worker on startup with retries. Launcher script validates Python and pip before starting. |
-| **Contingency** | Show a clear error in the UI: "ML worker not running — search disabled. Check terminal for details." |
+| **Description** | The FastAPI Python process may fail to start (missing packages, wrong Python version, port already in use). |
+| **Mitigation** | Launcher script validates Python and pip before starting. FastAPI startup errors are logged clearly in the terminal. |
+| **Contingency** | Show a clear error in the UI if API calls fail. QUICKSTART.md documents common failure modes and fixes (e.g., `QG_PORT` env var for port conflicts). |
### R-10 🟢 — Scope Creep
| Field | Detail |