Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/03_functional_requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
| **Description** | Each text chunk is converted to a 384-dimensional vector embedding using `all-MiniLM-L6-v2`. |
| **Input** | Text chunk content. |
| **Output** | Float32 embedding vector stored in the vector index and referenced in the database. |
| **Rules** | Embedding is generated via a Python worker subprocess. Batch size of 32 chunks for efficiency. |
| **Rules** | Embedding is generated in-process within the FastAPI application. Batch size of 32 chunks for efficiency. |

### FR-ING-05 — Progress Feedback
| Field | Detail |
Expand Down
69 changes: 31 additions & 38 deletions docs/04_system_requirements.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# QuickGuide (QG) — System Requirements Document (SRD)

**Version:** 1.0
**Version:** 1.1
**Date:** 2026-03-05

---
Expand All @@ -17,63 +17,55 @@
| Platform | Minimum Version | Notes |
|---|---|---|
| Windows | 10 | Primary development target |
| macOS | 12 Monterey | .NET 8 and Python supported |
| Linux | Ubuntu 22.04 / Fedora 38 | .NET 8 and Python supported |
| macOS | 12 Monterey | Python supported |
| Linux | Ubuntu 22.04 / Fedora 38 | Python supported |
| Android / iOS | Any modern browser | Access via `http://localhost:8080` |

> [!NOTE]
> Mobile access is via the web browser — no native app install needed. The backend must be running on the same network or device.

## 3 Software Dependencies

### 3.1 C# Backend (Primary — built for speed)
### 3.1 Python Backend (single process — handles all logic and ML)

| Component | Technology | Version | Rationale |
|---|---|---|---|
| **Runtime** | .NET 8 SDK | 8.0+ | Cross-platform, high-performance |
| **Web framework** | ASP.NET Core Minimal API | 8.0 | Fast HTTP, built-in DI, static file serving |
| **PDF parsing** | PdfPig | latest | Pure C#, no native dependencies, page-level text extraction |
| **Database** | SQLite via Microsoft.Data.Sqlite | latest | Zero-config, file-based, fast |
| **ORM** | Dapper | latest | Lightweight micro-ORM for C# |
| **JSON** | System.Text.Json | built-in | High-performance serialization |
| **Process management** | System.Diagnostics.Process | built-in | Launches Python worker subprocess |

### 3.2 Python ML Worker (for embedding + vector search)

| Component | Technology | Version | Rationale |
|---|---|---|---|
| **Runtime** | Python | 3.11+ | ML ecosystem support |
| **Embedding model** | sentence-transformers | latest | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly |
| **Vector index** | FAISS-cpu | latest | Efficient ANN search, widely supported |
| **HTTP server** | Flask | latest | Lightweight API for C# → Python communication |
| **Protocol** | HTTP on localhost:5001 | — | C# backend calls Python worker via REST |

### 3.3 Frontend
| **Runtime** | Python | 3.11+ | ML ecosystem support; cross-platform |
| **Web framework** | FastAPI | 0.115.0+ | High-performance async API; built-in data validation |
| **ASGI server** | uvicorn[standard] | 0.30.0+ | Production-grade ASGI server for FastAPI |
| **PDF parsing** | PyMuPDF (fitz) | 1.24.0+ | Fast text extraction with character positions; no native dependencies |
| **File uploads** | python-multipart | 0.0.9+ | Multipart form-data support for FastAPI |
| **Async file I/O** | aiofiles | 24.1.0+ | Async file operations |
| **Database** | SQLite via Python stdlib `sqlite3` | built-in | Zero-config, file-based, fast |
| **Embedding model** | sentence-transformers | 3.0.0+ | `all-MiniLM-L6-v2`: 80 MB, 384-dim, CPU-friendly |
| **Vector index** | faiss-cpu | 1.8.0+ | Efficient ANN search (numpy fallback if unavailable) |
| **Numerics** | numpy | 1.26.0–<2.0.0 | Vector operations and numpy fallback search |

### 3.2 Frontend

| Component | Technology | Rationale |
|---|---|---|
| **PDF rendering** | PDF.js (Mozilla) | Industry-standard browser PDF viewer |
| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served by C# backend |
| **UI** | HTML / CSS / Vanilla JavaScript | Zero build step; served as static files by the FastAPI backend |
| **Font** | Nunito (Google Fonts, self-hosted) | Warm, rounded, cozy aesthetic |

### 3.4 Build / Install Prerequisites
### 3.3 Build / Install Prerequisites

| Tool | Version | Notes |
|---|---|---|
| .NET SDK | 8.0+ | For building and running the C# backend |
| Python | 3.11+ | For the ML worker only |
| pip | bundled with Python | Installs sentence-transformers + FAISS |
| Python | 3.11+ | The only runtime required |
| pip | bundled with Python | Installs all dependencies from `requirements.txt` |
| Git | any | For version control |

> [!IMPORTANT]
> **Two runtimes required:** .NET 8 SDK (for the C# backend) and Python 3.11+ (for the ML embedding worker). Both are free and cross-platform.
> **One runtime required:** Python 3.11+ only. There is no separate .NET or C# dependency.

## 4 Performance Targets

| Metric | Target |
|---|---|
| PDF text extraction (1,000 pages) | ≤ 30 s (C# PdfPig) |
| Embedding generation (1,000 pages) | ≤ 90 s (Python worker) |
| PDF text extraction (1,000 pages) | ≤ 30 s (Python PyMuPDF) |
| Embedding generation (1,000 pages) | ≤ 90 s (sentence-transformers, batch=32) |
| Total ingestion (1,000 pages) | ≤ 120 s |
| Query → results | ≤ 3 s |
| App start (backend + browser) | ≤ 5 s |
Expand All @@ -84,22 +76,23 @@

- **No telemetry** — no external network calls (except first-time model download).
- **Local-only storage** — PDFs, text, and embeddings stay on the device.
- **Localhost binding** — C# backend binds to `127.0.0.1:8080`; Python worker to `127.0.0.1:5001`.
- **Localhost binding** — FastAPI backend binds to `127.0.0.1:8080` by default (configurable via `QG_HOST` / `QG_PORT` env vars).
- **No authentication** — single-user, local application.

## 6 Deployment Strategy

| Platform | Delivery |
|---|---|
| Windows | `qg.bat` double-click launcher → starts backend + opens browser |
| Windows | `qg.bat` double-click launcher → creates venv, installs deps, starts Python server, opens browser |
| macOS / Linux | `qg.sh` launcher script |
| Mobile | Access via browser on same network |
| Packaged (future) | Self-contained .NET publish + PyInstaller bundle |
| Packaged (future) | PyInstaller bundle for zero-install distribution |

## 7 Architecture Constraints

1. **C# for all non-ML logic** — PDF parsing, API, database, file management all in C#.
2. **Python only for ML** — embedding generation and vector search in a separate Python process.
3. **No external servers** — everything on localhost.
4. **Zero frontend build step** — HTML/CSS/JS served as static files by the C# backend.
1. **Python for all logic** — PDF parsing, API, database, file management, embeddings, and vector search all run in a single Python process.
2. **FastAPI as the web framework** — handles all HTTP routing, static file serving, and data validation.
3. **No external servers** — everything on localhost; no inter-process HTTP communication required.
4. **Zero frontend build step** — HTML/CSS/JS served as static files by the FastAPI backend.
5. **SQLite only** — no database server installation required.
6. **FAISS with numpy fallback** — if `faiss-cpu` is unavailable, flat cosine similarity via numpy is used automatically.
154 changes: 59 additions & 95 deletions docs/05_architecture.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# QuickGuide (QG) — Architecture Document

**Version:** 1.0
**Version:** 1.1
**Date:** 2026-03-05

---
Expand All @@ -12,24 +12,22 @@ graph TB
subgraph UserDevice["User's Device"]
Browser["Web Browser<br/>(any platform)"]

subgraph CSharpBackend["C# Backend — ASP.NET Core 8"]
API["REST API<br/>(Minimal API)"]
PdfSvc["PDF Parser<br/>(PdfPig)"]
subgraph PythonBackend["Python Backend — FastAPI"]
API["REST API<br/>(FastAPI routes)"]
PdfSvc["PDF Parser<br/>(PyMuPDF / fitz)"]
ChunkSvc["Text Chunker"]
DocMgr["Document Manager"]
HighlightSvc["Highlight Service"]
DbSvc["Database Service<br/>(SQLite + Dapper)"]
end

subgraph PythonWorker["Python ML Worker — Flask"]
EmbedSvc["Embedding Service<br/>(sentence-transformers)"]
VectorSvc["Vector Search<br/>(FAISS)"]
VectorSvc["Vector Search<br/>(FAISS / numpy)"]
DbSvc["Database Service<br/>(SQLite)"]
end

subgraph DataLayer["Data Layer"]
SQLite["SQLite DB<br/>(qg.db)"]
VectorIdx["FAISS Index Files<br/>(data/vectors/)"]
FileStore["PDF File Store<br/>(data/pdfs/)"]
ModelCache["Model Cache<br/>(data/models/)"]
end

subgraph Frontend["Static Frontend"]
Expand All @@ -44,19 +42,18 @@ graph TB
API --> ChunkSvc
API --> DocMgr
API --> HighlightSvc
API --> EmbedSvc
API --> VectorSvc
API --> DbSvc

API -- "HTTP localhost:5001" --> EmbedSvc
API -- "HTTP localhost:5001" --> VectorSvc

PdfSvc --> FileStore
ChunkSvc --> DbSvc
DbSvc --> SQLite
DocMgr --> FileStore
DocMgr --> SQLite
HighlightSvc --> SQLite

EmbedSvc --> VectorIdx
EmbedSvc --> ModelCache
VectorSvc --> VectorIdx
```

Expand All @@ -66,28 +63,26 @@ graph TB

```mermaid
flowchart LR
A["User selects PDF"] --> B["C#: Copy to<br/>data/pdfs/"]
B --> C["C#: Extract text<br/>(PdfPig, per page)"]
C --> D["C#: Chunk text<br/>(512 tok / 50 overlap)"]
D --> E["C# → Python:<br/>POST /embed"]
E --> F["Python: Generate<br/>embeddings<br/>(sentence-transformers)"]
F --> G["Python: Add to<br/>FAISS index"]
G --> H["C#: Store metadata<br/>in SQLite"]
H --> I["Ready for Search ✓"]
A["User selects PDF"] --> B["FastAPI: Copy to<br/>data/pdfs/"]
B --> C["PyMuPDF: Extract text<br/>(per page, with char offsets)"]
C --> D["Chunker: Split text<br/>(512 tok / 50 overlap)"]
D --> E["sentence-transformers:<br/>Generate embeddings<br/>(batch=32)"]
E --> F["FAISS: Add vectors<br/>to per-document index"]
F --> G["SQLite: Store metadata,<br/>chunks + status = ready"]
G --> H["Ready for Search ✓"]
```

### 2.2 Search Flow

```mermaid
flowchart LR
A["User types query"] --> B["C# API receives<br/>search request"]
B --> C["C# → Python:<br/>POST /search"]
C --> D["Python: Embed query<br/>(sentence-transformers)"]
D --> E["Python: FAISS ANN<br/>search (cosine sim)"]
E --> F["Python returns<br/>top-K chunk IDs + scores"]
F --> G["C#: Fetch chunk text<br/>from SQLite"]
G --> H["C# returns JSON<br/>results to browser"]
H --> I["Browser: Navigate<br/>PDF.js + Highlight"]
A["User types query"] --> B["FastAPI API receives<br/>search request"]
B --> C["sentence-transformers:<br/>Embed query (384-dim)"]
C --> D["FAISS (or numpy):<br/>ANN search (cosine sim)"]
D --> E["Filter results:<br/>similarity ≥ 0.30"]
E --> F["SQLite: Fetch<br/>chunk text + page numbers"]
F --> G["FastAPI returns<br/>JSON results to browser"]
G --> H["Browser: Navigate<br/>PDF.js + Highlight"]
```

### 2.3 Navigation & Highlighting Flow
Expand All @@ -104,46 +99,41 @@ flowchart LR

## 3 Component Breakdown

### 3.1 C# Backend (ASP.NET Core 8 Minimal API)

| File | Responsibility |
|---|---|
| `Program.cs` | App entry point, DI registration, route mapping, static file serving |
| `Services/PdfParserService.cs` | PdfPig text extraction — extracts text per page with character positions |
| `Services/TextChunkerService.cs` | Splits page text into overlapping token-windowed chunks |
| `Services/DocumentService.cs` | Document lifecycle: upload, list, delete, status tracking |
| `Services/SearchService.cs` | Orchestrates search: calls Python worker, merges with SQLite metadata |
| `Services/HighlightService.cs` | Highlight CRUD: create, read, update color, delete |
| `Services/PythonBridgeService.cs` | HTTP client that communicates with the Python ML worker |
| `Data/DatabaseService.cs` | SQLite connection, migrations, Dapper queries |
| `Models/` | C# record types for API requests/responses and DB entities |

### 3.2 Python ML Worker (Flask)
### 3.1 Python Backend (FastAPI — single process, port 8080)

| File | Responsibility |
|---|---|
| `worker.py` | Flask app: `/embed`, `/search`, `/health` endpoints |
| `embedder.py` | Loads `all-MiniLM-L6-v2`, generates 384-dim embeddings |
| `vector_store.py` | FAISS index management: add vectors, search, save/load, delete |

### 3.3 Frontend (HTML / CSS / Vanilla JS)
| `src/main.py` | App entry point, FastAPI routes, static file serving, browser auto-launch |
| `src/config.py` | All configuration: paths, server settings, model params, search thresholds |
| `src/models.py` | Pydantic request/response models (SearchRequest, HighlightCreate, etc.) |
| `src/database.py` | SQLite connection (thread-local, WAL mode), schema init, CRUD queries |
| `src/services/pdf_parser.py` | PyMuPDF text extraction per page; bounding-box search for highlighting |
| `src/services/chunker.py` | Splits page text into overlapping 512-token windows with 50-token overlap |
| `src/services/embedder.py` | Lazy-loads `all-MiniLM-L6-v2`; batch encodes text; normalizes to float32 |
| `src/services/vector_store.py` | Per-document FAISS index (or numpy fallback); add, search, save, load, delete |
| `src/services/search.py` | Semantic search orchestration: embed → vector search → filter → enrich |
| `src/services/documents.py` | Document lifecycle: upload, ingest pipeline, list, get, delete, progress tracking |
| `src/services/highlights.py` | Highlight CRUD: add, get, update color, delete |

### 3.2 Frontend (HTML / CSS / Vanilla JS)

| File | Responsibility |
|---|---|
| `index.html` | App shell — layout, PDF viewer container, search panel |
| `css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows |
| `js/app.js` | API calls, search orchestration, state management, document selector |
| `js/pdfviewer.js` | PDF.js rendering, page navigation, text-layer highlight overlays, match jumping |
| `src/static/index.html` | App shell — layout, PDF viewer container, search panel, highlights tab |
| `src/static/css/app.css` | Cozy design system — warm colors, rounded corners, soft shadows |
| `src/static/js/app.js` | API calls, search orchestration, PDF.js integration, state management, highlights |
| `src/static/img/qg-favicon.svg` | QG logo used as favicon and launcher icon |

### 3.4 Data Layer
### 3.3 Data Layer

| Store | Technology | Contents |
|---|---|---|
| `data/qg.db` | SQLite | Documents, text chunks, highlights, settings |
| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`) |
| `data/qg.db` | SQLite | Documents, text chunks, highlights |
| `data/vectors/` | FAISS index files | Embedding vectors per document (`.faiss` + `.map`, or `.npy` fallback) |
| `data/pdfs/` | File system | Original PDF files copied from user selections |
| `data/models/` | File system cache | Downloaded `all-MiniLM-L6-v2` model (~80 MB, cached after first run) |

## 4 API Endpoints (C# Backend — port 8080)
## 4 API Endpoints (FastAPI Backend — port 8080)

| Method | Path | Description |
|---|---|---|
Expand All @@ -152,51 +142,25 @@ flowchart LR
| `GET` | `/api/documents` | List all documents |
| `GET` | `/api/documents/{id}` | Get document details |
| `GET` | `/api/documents/{id}/status` | Get ingestion progress |
| `DELETE` | `/api/documents/{id}` | Delete document + data |
| `DELETE` | `/api/documents/{id}` | Delete document + all associated data |
| `GET` | `/api/documents/{id}/pdf` | Serve PDF file for viewer |
| `POST` | `/api/search` | Semantic search (proxies to Python) |
| `POST` | `/api/highlights` | Add a highlight |
| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document |
| `GET` | `/api/documents/{id}/text-positions` | Search text on a page; returns bounding-box rects for precise highlighting |
| `POST` | `/api/search` | Semantic search (embed + FAISS + SQLite enrichment) |
| `POST` | `/api/highlights` | Add a highlight annotation |
| `GET` | `/api/documents/{id}/highlights` | Get highlights for a document (optional `?page=N` filter) |
| `PUT` | `/api/highlights/{id}` | Update highlight color |
| `DELETE` | `/api/highlights/{id}` | Delete a highlight |

## 5 Python Worker API (port 5001)

| Method | Path | Description |
|---|---|---|
| `GET` | `/health` | Health check + model status |
| `POST` | `/embed` | Generate embeddings for text chunks |
| `POST` | `/search` | Embed query + FAISS search → return chunk IDs + scores |
| `POST` | `/index/add` | Add vectors to a document's FAISS index |
| `POST` | `/index/delete` | Delete a document's FAISS index |

## 6 Inter-Process Communication

```
┌─────────────────────────────────┐ HTTP (localhost:5001) ┌───────────────────────────┐
│ C# Backend (ASP.NET Core) │ ──────────────────────────► │ Python Worker (Flask) │
│ Port 8080 │ ◄────────────────────────── │ Port 5001 │
│ │ JSON payloads │ │
│ • PDF parsing (PdfPig) │ │ • sentence-transformers │
│ • Text chunking │ │ • FAISS index management │
│ • SQLite database │ │ │
│ • File management │ │ │
│ • Static file serving │ │ │
└─────────────────────────────────┘ └───────────────────────────┘
```

The C# backend **launches the Python worker** as a subprocess on startup and shuts it down on exit. Communication uses simple HTTP/JSON on localhost — no sockets, no message queues.

## 7 Key Design Decisions
## 5 Key Design Decisions

| Decision | Rationale |
|---|---|
| **C# primary, Python secondary** | User requested C# for speed; Python only for ML tasks that lack C# equivalents |
| **ASP.NET Core Minimal API** | Less boilerplate than controllers; fast startup; built-in DI |
| **PdfPig over iTextSharp** | Pure C#, MIT licensed, good text extraction, no Java dependency |
| **Separate Python process** | Clean separation of concerns; C# backend stays fast; Python handles ML |
| **Flask over FastAPI (worker)** | Simpler for a small internal service; no async needed for batch embedding |
| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings |
| **Python single-process** | All logic — PDF parsing, API, database, embeddings, vector search — runs in one FastAPI process; no inter-process complexity |
| **FastAPI over Flask** | Async-native, built-in Pydantic validation, automatic OpenAPI docs, faster than Flask |
| **PyMuPDF over PdfPig** | Python-native, fast C++ core, returns character-level positions needed for highlight overlays |
| **sentence-transformers in-process** | Direct function calls instead of HTTP round-trips; simpler, lower latency |
| **FAISS over custom ANN** | Battle-tested, optimized C++ core, excellent Python bindings; numpy fallback ensures compatibility |
| **SQLite over Postgres** | Zero install; single-file DB; perfect for local-first app |
| **PDF.js over native viewer** | Works in every browser; supports text-layer highlighting |
| **Vanilla JS over React** | Zero build step; served as static files; the app is simple enough |
| **Thread-local SQLite connections** | Avoids connection sharing across threads; WAL mode allows concurrent reads during writes |
Loading