Skip to content

Latest commit

 

History

History
342 lines (254 loc) · 16.8 KB

api.md

File metadata and controls

342 lines (254 loc) · 16.8 KB

API

Public Python surfaces useful for automation, tests, and plugins. Import from the installed db_storage_manager package.

Package metadata

from db_storage_manager import __version__, __author__

Configuration

from db_storage_manager.config import (
    APP_NAME,
    APP_VERSION,
    USER_DATA_DIR,
    CONNECTIONS_FILE,
    SETTINGS_FILE,
    DEFAULT_SETTINGS,
    DEFAULT_PORT,
    DEFAULT_TIMEOUT,
)

Connection factory

from db_storage_manager.db import (
    ConnectionConfig,
    DatabaseConnectionFactory,
)

cfg = ConnectionConfig(
    id="local-1",
    name="demo",
    type="sqlite",
    database="/tmp/demo.db",
)
conn = DatabaseConnectionFactory.create_connection(cfg)

All driver methods are async:

import asyncio

async def run():
    await conn.connect()
    result = await conn.execute_query("SELECT 1 AS n", safe_mode=True)
    analysis = await conn.analyze_storage()
    await conn.disconnect()
    return result, analysis

asyncio.run(run())

Abstract methods on DatabaseConnection: connect, disconnect, analyze_storage, execute_query, get_schema, create_backup, restore_backup, plus test_connection.

REST companion (local HTTP)

python -m db_storage_manager.api --host 127.0.0.1 --port 8765
Method Path Notes
GET /api/v1/health Liveness (no auth)
GET /api/v1/info Version + native backend
GET /api/v1/connections SecureStore profiles; passwords redacted
GET /api/v1/connections/import/platforms OAuth / env import guides (no secrets)
POST /api/v1/connections/import/env Body { "path": "/path/.env", "save": false } — parse profiles (redacted); save: true writes SecureStore
POST /api/v1/connections/import/ssh Body { "path": "/path/KEY-or-dir", "save": false } — SSH bastion profile from key + sidecars
GET /api/v1/backups Local backups (?backup_dir= optional)
POST /api/v1/backups/{id}/verify SHA-256 verify
POST /api/v1/backups/{id}/dry-run Checksum + payload dry-run (no live DB write)
GET / Minimal Web UI

Set DSM_API_TOKEN and/or create a SecureStore pairing token (db-storage-manager-pair) to require Bearer auth on all routes except health. See cli.md and privacy-companion.md.

Companion export (Python)

from db_storage_manager.export import export_companion_bundle

bundle = export_companion_bundle()  # passwords redacted
# export_companion_bundle(include_secrets=True, consent=True)  # desktop only

Native performance helpers

from db_storage_manager.native import (
    backend_info,
    is_available,
    gzip_compress_file,
    gzip_decompress_file,
    zstd_compress_file,
    zstd_decompress_file,
    encrypt_file,
    decrypt_file,
    sha256_file,
    sha256_files_parallel,
    aggregate_f64,
    hash_rows_parallel,
)

backend_info()  # {'available': True, 'language': 'rust', ...} or Python fallback
# encrypt_file / decrypt_file use DSMENC1 (AES-256-GCM + PBKDF2); native ↔ Python compatible

Install a Release wheel or build with maturin (see installation.md). Backup verification:

from db_storage_manager.backups.manager import BackupManager

manager = BackupManager()
manager.verify_backup_checksum(backup_info)
manager.verify_backups_parallel([backup_info, ...])
manager.dry_run_restore(backup_info)  # checksum + payload checks; never writes to a live DB
manager.dry_run_restore(backup_info, encryption_key="passphrase")  # encrypted backups

verify_* results include ok, status (pass / fail / missing_checksum / not_found), and digests. Local backups persist SHA-256 in a .meta.json sidecar next to the backup file. Passphrases are never written to sidecars.

Secure storage

from db_storage_manager.security.store import SecureStore

store = SecureStore()
store.save_connections([...])
connections = store.get_connections()
store.set_settings({...})
settings = store.get_settings()

Monitoring

from db_storage_manager.monitoring import (
    AlertManager,
    DatabaseMonitor,
)

monitor = DatabaseMonitor(connection)  # async DatabaseConnection
data = await monitor.poll_once()
# data["metrics"], data["health"], data["alerts"], data["alertCounts"]

manager = AlertManager()
manager.check_metrics({"activeConnections": 80, "cacheHitRatio": 55})
manager.acknowledge_by_id(manager.get_active_alerts()[0].id)
manager.acknowledge_all()
counts = manager.get_counts()  # critical / warning / info / total

DatabaseMonitor.poll_once() collects metrics, runs health checks, evaluates thresholds (with dedupe for active alerts), and returns get_dashboard_data().

SSH

from db_storage_manager.ssh import (
    SSHTunnel,
    SSHTunnelError,
    TunnelManager,
    normalize_ssh_config,
    validate_ssh_config,
)

manager = TunnelManager()
tunnel = manager.create_tunnel_sync(
    "conn-id",
    ssh_config={
        "enabled": True,
        "ssh_host": "bastion.example.com",
        "ssh_port": 22,
        "ssh_username": "deploy",
        "ssh_password": None,
        "ssh_key_path": "~/.ssh/id_ed25519",
        "remote_host": "localhost",
        "remote_port": 5432,
    },
)
endpoint = tunnel.get_local_endpoint()  # {"host": "127.0.0.1", "port": ...}
manager.close_tunnel_sync("conn-id")

SSHTunnelError carries clear messages for missing fields, bad keys, auth failures, timeouts, and refused connections.

GUI entry (advanced)

from db_storage_manager.main import main
from db_storage_manager.gui.main_window import MainWindow

Prefer the documented CLI entry for end users. GUI classes expect a running QApplication.

Plugins and extensions

Public SDK (db_storage_manager.plugins.sdk)

Stable authoring surface (API version 1.0):

from db_storage_manager.plugins.sdk import (
    PLUGIN_API_VERSION,
    Plugin,
    PluginMetadata,
    PluginType,
    load_manifest,
    validate_manifest,
    ManifestError,
)

A third-party plugin is a directory containing plugin.json plus an entry module:

{
  "id": "hello_echo",
  "name": "Hello Echo",
  "version": "1.0.0",
  "author": "VoxHash Technologies",
  "description": "Example SDK plugin",
  "plugin_type": "custom",
  "api_version": "1.0",
  "entry": "plugin:HelloEchoPlugin",
  "dependencies": []
}

entry is module:ClassName relative to the plugin root. The class must subclass Plugin. See examples/plugins/hello_echo/.

Catalog and provider plugins stay on this API. They are opt-in (install and enable; a folder on disk does not run). Data comes from a real catalog or the operator's provider API. No credentials are bundled. When the engine, plan or role cannot answer, return plugins.sdk.not_available(reason) (status not_available). Do not guess a pass or a failure. CATALOG_PLUGIN_RULES is the same contract.

LocalPluginMarketplace.install_from_path installs one directory when it contains plugin.json. When the selected folder does not, and its immediate subfolders do, it installs those children. A licence key applies only when a single directory was selected. A parent install does not reuse one key. The GUI asks before installing several and prompts for each premium licence.

Plugins are opt-in. They call real catalogs or provider APIs with credentials the operator supplies; none are bundled. When a plan, role, engine, or provider call cannot answer the question, return plugins.sdk.not_available(reason) (status is not_available). Do not guess pass or fail. Installing a parent folder (discover_plugin_roots) installs each child that has plugin.json. A single plugin directory still installs that one plugin. A premium child needs its own licence; one key is not applied to every product in the parent.

Premium plugins (optional license block)

A plugin without a license block is free. To sell a plugin through LicenseChain, create a product in the LicenseChain dashboard (or run python scripts/licensechain_products.py path/to/plugin --apply) and add its id:

{
  "license": {
    "provider": "licensechain",
    "product_id": "<LicenseChain product UUID>",
    "pricing": { "currency": "USD", "monthly": 1.99, "yearly": 16.00, "one_time": 24.00 }
  }
}
  • product_id is required; purchase_url defaults to https://pay.licensechain.app/checkout/<product_id>; provider must be licensechain. Unknown or malformed values reject the manifest (fail closed)
  • Provisioned LicenseChain product ids for the bundle and each first-party premium plugin are in db_storage_manager.plugins.pricing.PREMIUM_PRODUCT_IDS (monthly, yearly, one-time per entry)
  • A licence for any bundle product (PREMIUM_PRODUCT_IDS["bundle"]: monthly, yearly or one-time) also unlocks every plugin whose product_id is one of the first-party ids in PREMIUM_PRODUCT_IDS; see plugins.pricing.accepted_product_ids. A plugin-only key unlocks only that plugin, and third-party product_ids are matched exactly
  • pricing is display-only (shown in the licence prompt); LicenseChain holds the authoritative price. Options: monthly, yearly, one_time, positive amounts, 3-letter uppercase currency. First-party plugins stay at or under plugins.sdk.PRICING_TIERS["plugin"]
  • The host checks the licence before importing plugin code (install), on enable and on every execution. Plugins never see the licence key. Built-in plugins ignore the block
  • Python: LocalPluginMarketplace.activate_license(plugin_id, key, email=None), deactivate_license, license_status, require_license; install_from_path(path, license_key=..., license_email=...). A missing licence raises PluginLicenseRequired

PITR Verify (pitr-verify)

First premium plugin. It is not a built-in: install the directory from db_storage_manager.premium_plugins.bundled_plugin_dir("pitr-verify"). Installing requires plugins:manage. execute requires monitoring:read.

license.product_id is plugins.pricing.PREMIUM_PRODUCT_IDS["pitr-verify"]["monthly"] (products live on the current LicenseChain seller account; do not run licensechain_products.py --apply again for this plugin).

Schema drift (schema-drift)

Second premium plugin. Install db_storage_manager.premium_plugins.bundled_plugin_dir("schema-drift"). Installing requires plugins:manage. execute requires monitoring:read.

It snapshots SQLite (sqlite_master / PRAGMA table_info), PostgreSQL (information_schema.columns in public) or MySQL (information_schema.columns for the current database) and diffs that against a baseline dict or a second connection. license.product_id is PREMIUM_PRODUCT_IDS["schema-drift"]["monthly"].

plugin = load_plugin_from_directory(bundled_plugin_dir("schema-drift"))
report = plugin.execute(connection, other=other_connection)
from db_storage_manager.plugins.loader import load_plugin_from_directory
from db_storage_manager.premium_plugins import bundled_plugin_dir

plugin = load_plugin_from_directory(bundled_plugin_dir("pitr-verify"))
report = plugin.execute(connection, expected_retention_days=7)

report["status"] is ok, fail or not_available. pitr_enabled is true, false, or null when the check could not be read (permission denied, provider 401/403, or a field the API did not return). Self-hosted PostgreSQL is decided from wal_level, archive_mode, archive_command / archive_library and pg_stat_archiver. That view does not include how long WAL is retained, so expected_retention_days on a self-hosted server is not_available rather than a guessed match.

When connection.config.extra["cloud_provider"] is set, retention comes from the provider and the catalog is attached as extra checks. Supported values: aws_rds (Aurora PostgreSQL uses DescribeDBClusters), gcp_cloudsql, azure_postgres (flexible server; resource_group is required), supabase (DSM_SUPABASE_ACCESS_TOKEN), neon (NEON_API_KEY). cloud_instance_id is the instance, server, project ref or project id. Credentials are the operator's (SecureStore / environment); none are bundled. azure_sql is reported not_available because it is SQL Server.

RLS audit (rls-audit)

PostgreSQL only. Reads pg_class (relrowsecurity), pg_policies and pg_proc.prosecdef. A table with row security off, a permissive policy whose expression is true (or missing), and a SECURITY DEFINER function outside pg_catalog / information_schema are findings. SQLite and MySQL return not_available. Install bundled_plugin_dir("rls-audit"). Running requires monitoring:read. license.product_id is PREMIUM_PRODUCT_IDS["rls-audit"]["monthly"].

Pooler health (pooler-health)

Pass the direct connection and, when you have one, the pooled connection. The plugin times SELECT 1, reads max_connections and pg_stat_activity, and tries SHOW pool_mode. It does not open a host you did not pass. No pooled connection means status partial. Running requires monitoring:read. license.product_id is PREMIUM_PRODUCT_IDS["pooler-health"]["monthly"].

Encrypted export (encrypted-export)

Exports one table (csv, sql, or parquet) and wraps the file in DSMENC1. redact names columns whose non-null values become REDACTED before encryption. The plaintext file is removed. Parquet is not_available when pandas has no parquet engine. Running requires backups:create. license.product_id is PREMIUM_PRODUCT_IDS["encrypted-export"]["monthly"].

report = plugin.execute(connection, table="people", dest="people.csv.enc", passphrase="…", format="csv", redact=["secret"])

report["status"] is exported, fail or not_available. The passphrase is not part of the report.

Extension inventory (extension-inventory)

PostgreSQL pg_available_extensions. Installed xml2, intagg, or tsearch2 is a finding (deprecated or removed in the PostgreSQL manual). An installed version other than default_version is listed as outdated. SQLite and MySQL are not_available. Running requires monitoring:read. license.product_id is PREMIUM_PRODUCT_IDS["extension-inventory"]["monthly"].

Multi-cloud discovery (multi-cloud-discovery)

Lists DigitalOcean (GET /v2/databases), Supabase (GET /v1/projects, host db.<ref>.supabase.co), Neon (projects plus endpoints), and Aiven (GET /v1/project/{project}/service, Authorization: aivenv1). Tokens are the operator's environment or connection extra. Passwords and URIs are dropped. A missing token or HTTP 401/403 is not_available for that provider. Discover Cloud calls execute only when the plugin is installed and licensed. license.product_id is PREMIUM_PRODUCT_IDS["multi-cloud-discovery"]["monthly"].

Trusted sources (trusted-sources-check)

Compares an allow list with the session client IP (inet_client_addr() or the MySQL process-list host). It does not call an IP-echo service. Neon documents an empty allowed_ips.ips as open. Supabase entitlement: disallowed is not_available. An empty DigitalOcean rule list is partial. 0.0.0.0/0 is a finding. Running requires monitoring:read. license.product_id is PREMIUM_PRODUCT_IDS["trusted-sources-check"]["monthly"].

Maintenance window (maintenance-window-watch)

alerts is the feed. PostgreSQL final dates are from https://www.postgresql.org/support/versioning/. MySQL dates are Oracle extended-support ends (8.0 ended 2026-04-30, 8.4 ends 2032-04-30, 9.7 ends 2034-04-30). A version absent from those tables is not_available. Pending DigitalOcean maintenance_window or RDS PendingMaintenanceActions add warning alerts. MariaDB is not compared to the MySQL calendar. Running requires monitoring:read. license.product_id is PREMIUM_PRODUCT_IDS["maintenance-window-watch"]["monthly"].

Local marketplace

from pathlib import Path
from db_storage_manager.plugins.marketplace import LocalPluginMarketplace

mp = LocalPluginMarketplace()
mp.list_installed()  # builtins + user installs
mp.install_from_path(Path("examples/plugins/hello_echo"))
mp.install_from_url("https://example.com/my-plugin.zip")  # zip or tar.gz
mp.set_enabled("hello_echo", False)
mp.uninstall("hello_echo")  # user installs only
manager = mp.load_into_manager()
manager.execute_plugin("hello_echo", message="hi")

Registry files live under ~/.config/db-storage-manager/plugins/ (registry.json, installed/). Enable/disable flags are stored in SecureStore settings key plugins. There is no remote catalog with placeholder listings; a future public store can publish the same plugin.json layout.