Skip to content

feat(connections): list external databases and read and write them - #157

Open
Upd4ting wants to merge 7 commits into
mainfrom
feat/external-databases
Open

Upd4ting wants to merge 7 commits into
mainfrom
feat/external-databases

Conversation

@Upd4ting

@Upd4ting Upd4ting commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Why

dms-database only showed the project's own database. It now also lists every external database the connect flow added, so users see all their data in one place. Each external database is read and edited through its own interface-database binding, with the same browse layer as the own database. Edits stay off until the owner turns them on for that database.

⚠ sensitive: this writes to a customer's external database, and it adds a stored table, connection_settings.

What changes

  • Connections list. GET /api/database/connections returns the own database first, then each external connection with id, kind, label, engine, writes and unsupported.

    • An external connection is a named @antelopejs/interface-database-catalog connection that has an @antelopejs/interface-database connection with the same id on this module's entry.
    • engine and label come from the catalog's Info(). When the catalog refuses, outside development, engine is null and label falls back to the connection id.
  • Description from the catalog. GET /<id>/relations returns ListRelations and GET /<id>/relations/<schema>.<name> returns DescribeRelations. A catalog refusal comes back as 409 with its reason, and an unknown relation as 404.

  • Reads and writes through the binding.

    • GET /<id>/rows, GET /<id>/row and PUT /<id>/row take the Data browser's query parameters.
    • They resolve a Database handle (src/service/database.ts): the own database, or require(GetInterfaceInstance(...).path) for an external one.
    • browse.ts, declaredFields.ts and the new editRow in rowEdit.ts take that handle. The existing /api/database/browse/* routes pass the own database and behave as before.
  • Writes toggle.

    • PUT /<id>/writes with { "enabled": boolean } sets it.
    • The setting is stored in connection_settings, keyed by connection id, and starts off. While it is off, an edit is refused with 403.
  • Unsupported features. An external connection reports migrations, schemaEdits, indexes, diagram, queryConsole, instances and rowReferences. Its tables have no instances, so a request on one that names filter_instance gets a 400.

  • Module context. Core answers a module's named connections only in that module's own context, and an API request reaches the controller in the API module's context. The module records its id at construct(), and the lookups and binding loads run under RunWithResponsibleModule.

  • Minimal UI. A new Connections page (/modules/database/connections) shows:

    • the list, with an engine badge and an External badge;
    • the writes switch;
    • the unsupported features as disabled badges;
    • the catalog relations, and the first 25 rows of the relation you pick.

    The designed switcher and external view come after the design session.

  • Docs. The new page docs/4.usage/6.external-databases.md, plus one line in the README.

The config shape ANT-2044 must write

The connect flow (ANT-2044, dms-assistant) must add these entries to the dms-database module entry, next to what it already writes for the provider and the tables module:

"dms-database": {
  importOverrides: [
    { interface: "@antelopejs/interface-database", source: "mongodb" },               // own database: must come first
    { interface: "@antelopejs/interface-database", source: "postgres-erp", id: "erp" },
    { interface: "@antelopejs/interface-database-catalog", source: "postgres-erp", id: "erp" },
  ],
},
  • The first interface-database entry pins the default binding. Without the own-database entry first, dms-database's own tables would move to the external database.
  • One id for both connections. It must be the same as the catalog id the other consumers (dms-builder, dms-ai) use.
  • Tables come from the tables module. The tables dms-database reads are the ones the tables module declares, pinned to the same provider. Core shares the interface-database@<provider> copy, so Schema.list() on the named binding lists them.
  • Removing a connection. dms-builder's RemoveConnection removes a named connection from every module's importOverrides, which covers both named entries. The own-database pin stays.

Proof

The new tests boot a real DMS with:

  • an in-memory MongoDB;
  • an external MongoDB module, mongodb-erp, in mode: "external", on a seeded erp database;
  • the tables module test-fixtures/erp-tables;
  • the connections above. See src/test/harness.ts.

Before, on main with this harness. The only browse API cannot reach the external table:

  [integration] external database before ANT-2043
    1) reads the external customers table

  1) ... reads the external customers table:
     Error: the object {
  "body": "Unknown schema: erp"
  "status": 404
} was thrown

Before the module-context fix. Over HTTP, the list holds only the own database:

  1) [integration] external connections over HTTP
       lists, reads and gates writes to an external connection on a request the API module serves:
      AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  {
    connections: [
      { engine: null, id: 'own', ... },
-     { engine: 'mongodb', id: 'erp', kind: 'external', label: 'erp', ...

After:

  [integration] external connections over HTTP
    ✔ lists, reads and gates writes to an external connection on a request the API module serves

  [integration] external connections: listing and reads
    ✔ lists the own database, then each external connection with its engine, writes off
    ✔ reads an external table through the connection's binding, which the own browse API cannot reach
    ✔ describes an external relation from the connection's catalog
    ✔ refuses an unknown connection, a writes toggle on the own database and an instance on an external one

  [integration] external connections: writes
    ✔ refuses an external write while its writes are off, then writes through the binding once they are on

  6 passing

Other results:

  • The full backend suite passes: 145 tests.
  • pnpm test:layer passes: 168 tests, including two for the Connections page.
  • pnpm run checks passes: 12 of 12.
flowchart LR
  UI[Connections page] -->|GET/PUT /api/database/connections/:id/...| C[DatabaseConnectionsController]
  C -->|own| OWN[OWN_DATABASE: Schema.list]
  C -->|external id| R[resolveDatabase / resolveCatalog]
  R -->|RunWithResponsibleModule dms-database| L[GetInterfaceInstance + require path]
  L --> DB[interface-database@provider: Schema.list, queries]
  L --> CAT[interface-database-catalog@provider: Info, ListRelations, DescribeRelations]
  C -->|PUT row| W{writes on?}
  W -->|no| X[403]
  W -->|yes| E[editRow through the binding]
  E --> DB
Loading

Review

Two adversarial reviewers, on Opus and Sonnet, attacked the change. Both findings are fixed, and each fix has its own test:

  • PUT /own/row edited across all instances, which the old edit route refuses. The guard now lives in editRow.
  • Over HTTP, no external connection was visible, because of the module context described above.

They also probed these, and the code held:

  • the writes gate, including after toggling it on and off again;
  • confusion between the own and the external database, with same-named schemas and own tables requested through /erp/rows;
  • _id edits;
  • pagination, sort, search and filters on an external table;
  • the refusal statuses.

Follow-ups

  • An edit that names a column the external table lacks comes back as the provider's raw ExternalWriteError instead of a mapped 4xx.

…rough their binding

dms-database enumerates the named interface-database-catalog connections that
have an interface-database connection of the same id, and serves the own
database and each external one under /api/database/connections. Rows are read
and edited through the browse layer, which now takes the Database a request
names. Writes to an external connection stay off until its toggle, stored in
connection_settings, turns them on.
The guard lived in the own browse route only, so PUT
/api/database/connections/own/row edited whichever instance held the id.
editRow now refuses it for every caller.
…quests

Core answers a module's connections in that module's context, and a request
reaches the controller in the API module's, where dms-database's named
connections are invisible. The module records itself at construct and the
lookups and binding loads run as it.
Lists the project database and every external connection, toggles an
external connection's writes, and shows its catalog relations and their
first rows. Also moves the interface-database-catalog and test fixture
ranges to the >=x.y.z <1.0.0 shape lint requires.
… what its state does

The switch's accessible name stayed on its first state, and the hint said edits
are refused while writes were on.
@Upd4ting
Upd4ting marked this pull request as ready for review November 11, 2026 21:22

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant