Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ OAuth hosts in production must set `JVSPATIAL_OAUTH_KEY_ENCRYPTION_KEY` from a s

### 11. Cross-context persistence

The in-memory matcher and PostgreSQL pushdown treat both missing and JSON `null` as absent for `$exists`; MongoDB retains its native field-presence semantics. `graph_transaction` isolates request identity state and invalidates touched entities in the parent cache after commit or rollback. Index setup must be scoped to the database instance so one store cannot suppress another store's indexes. Preserve these contracts when changing query translation, caches, or transactions.
The in-memory matcher and PostgreSQL pushdown treat both missing and JSON `null` as absent for `$exists`; MongoDB retains its native field-presence semantics. `graph_transaction` isolates request identity state, never returns outer-context cached entities inside a transaction, and invalidates touched entities in the parent cache after commit or rollback. Native transaction handles retain the shared read contract on their own connection/session; unsupported backends must continue to fail capability checks. Index setup must be scoped to the database instance so one store cannot suppress another store's indexes. Preserve these contracts when changing query translation, caches, or transactions.

Composite keyset continuation includes every sort value and direction plus ID. Never apply LIMIT before residual filtering or unsupported ordering. A configured fallback scan budget raises on exhaustion rather than returning a partial result. Bulk cache entries require confirmed outcomes; ambiguous writes are invalidated. Preserve JSON path locks/atomic durability and streaming handle cleanup on cancellation.

Expand Down Expand Up @@ -126,7 +126,7 @@ See [graph-projection.md](docs/md/graph-projection.md) and SPEC §6.9.
| Subclass lookup returns wrong class | Two unrelated classes share `__name__` | Set `__entity_name__: ClassVar[Optional[str]] = "Distinct"` on one |
| Tests pass locally, fail under serverless mode | Forgot to test both modes | Use `reset_serverless_mode_cache()` between tests |
| 401 with valid token | Auth state on wrong database | Auth is **always** on prime DB; do not relocate |
| Slow query never logged | `slow_query_ms` not configured | Set in `create_database(observe=True, slow_query_ms=N)` |
| Slow query absent from logs | Successful operations are deliberately silent | Set `observe=True` with a metrics recorder; inspect the slow-operation counter |
| Walker visits the same node forever | Protection disabled | Re-enable `protection_enabled=True` |
| `JVSPATIAL_FOO` env var ignored | Not in allowlist | Add to `jvspatial/env_adapter.py` allowlist |
| SQLite query falls back to in-memory filter | Operator not yet pushed down by `SQLiteTranslator` | Check `jvspatial/db/_sqlite_translate.py` for supported operators |
Expand Down
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.2.2] - 2026-10-13

### Fixed

- Transaction reads now expose `find_one`, `count`, `find_ids`, `find_many`,
`find_iter` and incident-edge counts through the shared transaction contract.
PostgreSQL uses its held connection for native reads; MongoDB fallbacks retain
their session. Unsupported backends still advertise no native transactions.
- PostgreSQL transaction operations serialize access to their connection,
reject use after completion, retain tenant scope, and bootstrap new collections
within the same transaction without acquiring a second connection. Collection
bootstrap caches are promoted only after commit.
- Graph transactions no longer reuse entities bound to an outer context or
populate shared process caches with uncommitted data. Parent-cache invalidation
remains in place after commit and rollback.

- Graph transaction activation now fails closed when a deployment returns no
transaction handle. Shared transaction contexts roll back on cancellation,
and MongoDB commit/rollback await asynchronous session cleanup.

### Changed

- Successful database observations produce no log records, including slow calls
and DEBUG-enabled runs. Failures emit structured ERROR records; operation and
slow-call metrics remain available. Successful calls also skip log-payload
construction and message formatting.
- Updated transaction, paging, observability and backend support documentation;
corrected stale specification and roadmap claims and refreshed the docs index.

## [0.2.1] - 2026-10-12

### Fixed
Expand Down
2 changes: 1 addition & 1 deletion PRD.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jvspatial is an **async-first, serverless-compatible, object-spatial Python libr

It is *not*:

- A graph database engine. jvspatial layers a graph model on top of existing backends (MongoDB, SQLite, JSON files, DynamoDB) and unifies query semantics across them.
- A graph database engine. jvspatial layers a graph model on top of existing backends (PostgreSQL, MongoDB, SQLite, JSON files, DynamoDB) and unifies query semantics across them.
- A web framework. It builds on FastAPI; it does not replace it.
- A workflow engine. Walkers traverse data; they are not a generic task orchestrator.
- A UI library. Visualization helpers exist (DOT, Mermaid), but rendering is out of scope.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ server = Server(
- [Custom Database Guide](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/custom-database-guide.md) - Implementing custom database backends
- [Graph Visualization](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/graph-visualization.md) - Export graphs in DOT/Mermaid formats
- [Pagination](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/pagination.md) - ObjectPager usage
- [Observability](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/observability.md) - Structured DB logging, slow-query threshold, MetricsRecorder, OpenTelemetry adapter
- [Observability](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/observability.md) - Failure-only DB logging, operation/slow-call metrics, OpenTelemetry adapter
- [Performance Benchmarks](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/benchmarks.md) - Regression-detection bench suite (pytest-benchmark + CI workflow)
- [API Stability](https://github.com/TrueSelph/jvspatial/blob/main/docs/md/stability.md) - Public/internal/experimental tiers and deprecation policy

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Recent waves have systematically eliminated constant-time-comparison gaps, fail-

- `find_many`, `bulk_save`, native `count()` pushdown landed in 0.0.7. Continue:
- Native bulk-update equivalents on backends that support them (Mongo `bulk_write`, SQLite `executemany UPDATE`).
- Continue surfacing slow-query telemetry via the observability layer; tune `slow_query_ms` defaults per backend.
- Surface slow-query counts and durations through observability metrics; successful operations remain silent in application logs. Tune `slow_query_ms` for workload-specific metrics.

### 1.3 Observability (active)

Expand Down Expand Up @@ -53,7 +53,7 @@ No migration framework. Field removals or renames break existing records silentl

### 2.2 Multi-database transactions are best-effort only

Only MongoDB supports ACID transactions. Cross-backend operations have no built-in coordination.
PostgreSQL and topology-qualified MongoDB support native ACID transactions. Cross-backend operations have no built-in coordination; JSON, SQLite and DynamoDB do not offer native graph transactions.

- **Decision needed**: document the limit explicitly in handler patterns or build an opt-in saga primitive.
- **Location**: `jvspatial/db/transaction.py`, `jvspatial/db/database.py:48+`.
Expand Down
44 changes: 40 additions & 4 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ cases are documented rather than normalized:

Adapters declare capabilities as class attributes:

- `supports_transactions: bool` — `True` for MongoDB (replica set); `False` for SQLite (best-effort), JSON, DynamoDB.
- `supports_transactions: bool` — the adapter-level flag is `True` for PostgreSQL and MongoDB and `False` for SQLite, JSON and DynamoDB. MongoDB also requires a successful `await db.is_transactional()` topology probe; its static flag alone is not deployment qualification. Unsupported backends do not offer multi-write atomicity.

Node adjacency always lives in the edge collection (indexed on `source` /
`target` by `Edge.get_indexes`). There is no `edge_ids_mode` toggle and no
Expand All @@ -292,9 +292,33 @@ Callers branching on capabilities should test the flag, not the adapter class.
| SQLite | `jvspatial/db/sqlite.py` | No (single-conn fsync) | `aiosqlite`; Mongo→SQL via `SQLiteTranslator` |
| MongoDB | `jvspatial/db/mongodb.py` | Yes | Motor default; opt-in PyMongo Async; native bulk/cursor/compound ops |
| DynamoDB | `jvspatial/db/dynamodb.py` | No | `aioboto3`; throttle-retry; `BatchGetItem` chunks of 100 |
| PostgreSQL | `jvspatial/db/postgres.py` | Yes | Held-connection ACID transactions, RLS tenant scope and native query paging |

For a file-backed SQLite database reused on a different event loop, the adapter closes the old `aiosqlite` connection and its worker thread before opening a connection on the new loop. An in-memory SQLite database retains its existing connection so its contents are not lost (`jvspatial/db/sqlite.py`).

### 4.3.1 Transaction read and cache contract

`Transaction` supplies `find_one`, `count`, `find_ids`, `find_many`, `find_iter`
and `count_incident_edges` alongside CRUD. All fallback reads route through the
transaction handle, not its outer database. PostgreSQL overrides native reads
on its held connection, serializes concurrent operations, rejects completed
handles and keeps tenant scope and collection bootstrap on that connection.
MongoDB inherited reads retain the native session; fallback count/projection/
bulk work may be less efficient than native aggregation. JSON, SQLite and
DynamoDB remain unsupported for native graph transactions. If an adapter cannot
open a handle, `graph_transaction` raises `TransactionUnavailable` without
binding an ordinary context. `transaction_context` rolls back on cancellation;
MongoDB completion awaits native session cleanup for both driver shapes.

`graph_transaction` uses a separate request identity map. It does not read outer
process-cache entities or write transaction-local entities into the shared
cache. Reads from an explicit different context do not reuse a request identity
entry bound to the transaction. Touched parent cache keys are invalidated after
both commit and rollback. Collection DDL cache entries are promoted only after
commit; rollback permits subsequent bootstrap.
Sources: `jvspatial/db/transaction.py`, `jvspatial/db/postgres.py`,
`jvspatial/core/context.py`. See [Transactions](docs/md/transactions.md).

### 4.4 Atomic IO (JSON)

`jvspatial/db/_atomic.py` provides crash-safe writes: temp file → `fsync` → `rename` → `fsync(directory)`. Per-file mutex via `PathLockManager` (`_path_locks.py`) serializes concurrent writes to the same record. Bounded LRU prevents lock-table growth.
Expand Down Expand Up @@ -841,14 +865,26 @@ TTL is per-cache-instance; default `None` (cache until eviction). Invalidation i

### 14.1 MetricsRecorder protocol

`MetricsRecorder` defines `record_db_op`, `record_hook`, `record_cache_event`. Implementations may push to Prometheus, StatsD, or any sink. The library provides:
`MetricsRecorder` defines `record_duration`, `increment_counter` and `record_value`. Implementations may push to Prometheus, StatsD, or any sink. The library provides:

- `NullRecorder` — default; no-op.
- `NullMetricsRecorder` — default; no-op.
- OpenTelemetry adapter (`observability/otel.py`) — optional, requires `[otel]` extra.

### 14.2 Structured DB logging

`jvspatial/db/_observable.py` — wraps a `Database` to emit `{backend, op, collection, duration_ms, success, result_count}`. Successful routine timings use DEBUG; failed operations and durations at or above the configurable `slow_query_ms` threshold use WARNING. Log filtering does not disable metrics. Residual scans add execution mode, partial-pushdown status and examined/decoded work; observed PostgreSQL calls add pool wait, and retry helpers add driver attempts/retry count. Iterator active duration is separate from total consumer-inclusive duration. These are driver/candidate metrics, not physical protocol round trips or server rows scanned.
`create_database(observe=True)` wraps any backend and records metrics for each
wrapped operation. Successful operations produce no log record at any logger
level, including calls reaching `slow_query_ms`; those calls increment a slow
metric. Failures emit ERROR with the existing `{backend, op, collection,
duration_ms, success, result_count}` field schema where applicable. Log filtering
does not disable metrics. Successful calls skip logging payload construction
and message formatting. Residual scans add execution mode, partial-pushdown
status and examined/decoded work; observed PostgreSQL calls add pool wait, and
retry helpers add driver attempts/retry count. Iterator active duration is
separate from total consumer-inclusive duration. These are driver/candidate
metrics, not physical protocol round trips or server rows scanned.
Source: `jvspatial/db/_observable.py`.


### 14.3 GraphContext PerformanceMonitor

Expand Down
21 changes: 12 additions & 9 deletions docs/md/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This directory holds detailed how-to and reference documentation. For higher-lev
- [AGENTS.md](../../AGENTS.md) — agent maintenance guide
- [CHANGELOG.md](../../CHANGELOG.md) — release history

The documents below answer *how* to use each subsystem. Every link resolves; entries marked **NEW** were added since the previous index revision.
The documents below answer *how* to use each subsystem. Every link resolves; entries marked were added since the previous index revision.

---

Expand Down Expand Up @@ -57,11 +57,11 @@ The documents below answer *how* to use each subsystem. Every link resolves; ent
| Document | What's in it |
|---|---|
| [Authentication](authentication.md) | JWT, API key, refresh-token flows. |
| [API Keys](api-keys.md) | **NEW** Key management endpoints and hashing model. |
| [API Keys](api-keys.md) | Key management endpoints and hashing model. |
| [Password Migration Guide](password-migration-guide.md) | bcrypt upgrade path. |
| [Security Review](security-review.md) | **NEW** Audit findings and resolved fixes. |
| [Security Operational Notes](security-operational-notes.md) | **NEW** Runtime security guidance. |
| [Production Deployment](production-deployment.md) | **NEW** Hardening checklist for production. |
| [Security Review](security-review.md) | Audit findings and resolved fixes. |
| [Security Operational Notes](security-operational-notes.md) | Runtime security guidance. |
| [Production Deployment](production-deployment.md) | Hardening checklist for production. |

## Database and Storage

Expand All @@ -70,6 +70,9 @@ The documents below answer *how* to use each subsystem. Every link resolves; ent
| [MongoDB Query Interface](mongodb-query-interface.md) | Mongo-style operators across all backends. |
| [Custom Database Guide](custom-database-guide.md) | Implementing and registering a new `Database` adapter. |
| [DynamoDB Guide](dynamodb-guide.md) | DynamoDB-specific setup and limits. |
| [PostgreSQL Guide](postgres-guide.md) | Connection setup, native transactions, tenant scope, pooling and vector support. |
| [Transactions](transactions.md) | Shared transaction reads, backend support, commit/rollback and cache isolation. |
| [Cursor Pagination](cursor-pagination.md) | Deterministic ID/keyset pages, continuation and iterator semantics. |
| [File Storage Architecture](file-storage-architecture.md) | Storage interface, security layer, version model. |
| [File Storage Usage](file-storage-usage.md) | Upload, list, download, versioning. |
| [Caching](caching.md) | Memory / Redis / layered caches; read-through DB wrapper. |
Expand All @@ -79,11 +82,11 @@ The documents below answer *how* to use each subsystem. Every link resolves; ent

| Document | What's in it |
|---|---|
| [Observability](observability.md) | **NEW** Structured DB op logs, `MetricsRecorder`, slow-query threshold, OTEL adapter. |
| [Observability](observability.md) | Failure-only DB logs, per-operation/slow-call metrics and OpenTelemetry adapter. |
| [Logging Service](logging-service.md) | Persistent application logging. |
| [Custom Log Levels](custom-log-levels.md) | Adding domain levels (`AUDIT`, `SECURITY`, …). |
| [Benchmarks](benchmarks.md) | **NEW** Regression-detection bench suite. |
| [Serverless Mode](serverless-mode.md) | **NEW** Detection precedence, mode-dependent defaults, LWA env. |
| [Benchmarks](benchmarks.md) | Regression-detection bench suite. |
| [Serverless Mode](serverless-mode.md) | Detection precedence, mode-dependent defaults, LWA env. |
| [Scheduler](scheduler.md) | In-process scheduler and decorators. |
| [Webhook Architecture](webhook-architecture.md) | Webhook routing, signature verification. |
| [Webhooks Quickstart](webhooks-quickstart.md) | Common webhook patterns. |
Expand All @@ -97,7 +100,7 @@ The documents below answer *how* to use each subsystem. Every link resolves; ent
| [Design Decisions](design-decisions.md) | Philosophy and tradeoffs. |
| [Module Responsibility Matrix](module-responsibility-matrix.md) | Which package owns which concern. |
| [Import Patterns](import-patterns.md) | Stable vs internal imports. |
| [Stability](stability.md) | **NEW** Public / internal / experimental tiers + deprecation policy. |
| [Stability](stability.md) | Public / internal / experimental tiers + deprecation policy. |
| [Optimization](optimization.md) | Performance tuning: `neighborhood`, walker prefetch, Postgres paths, fast deserialize. |

## Configuration
Expand Down
Loading
Loading