Repository navigation
Enhance export features, refactor capabilities, and improve auth pages - #975
Merged
Merged
Conversation
TimescaleE2ETest creates a hypertable with SQL, lists it as a table with its columns and reads its rows back on PostgreSQL 16 and 17 images. The connector pages list the verified versions.
ParadeDBE2ETest creates a table with a BM25 index, lists it with its columns and searches it on PostgreSQL 16 and 17 images. The connector pages list the verified versions.
Redis now lists databases, tables and columns like every other source, so the query page tree needs no separate key browsing. This reverts commit d47d84d.
getMetaData() lists the numbered databases (CONFIG GET databases, 16 when refused), the keys of each as tables and key, type and value as columns, through the SQL driver foundation. A connection opens other databases on demand, and SELECT reads a key qualified with its database (`2`.`user:1`), as the workbench writes it, and honours LIMIT.
…pace The plugin lists databases, tables and columns from the driver's JDBC metadata and declares the capabilities ZooKeeper does, so the workbench shows the database picker and table tree; the separate key browser and its Lua scripts are gone. RedisE2ETest checks the listing and the workbench's qualified SELECT on Redis 6.2 and 7.4.
Neo4j reported only query:execute, so it had no tree. The plugin lists the database it connects to, node labels as tables and their properties as columns from the JDBC driver's metadata, and answers the version with Cypher. Neo4jE2ETest verifies it on Neo4j 4.4 and 5.
Constraints and indexes need index:read, foreign keys and usage need foreign-key:read, so a source like Redis shows only the basic tab, and the primary and unique rows leave the basic panel when indexes are not declared.
The JDBC metadata fallback read IS_AUTOINCREMENT from getColumns(), a JDBC 4.1 column that drivers such as MonetDB's do not return, so listing a table's columns failed. Treat a missing column as not auto-increment.
MonetDB has no version() function, so the default validator failed and the source could not be checked. Read monet_version instead.
MonetDBE2ETest creates a schema, table and rows with SQL, lists them as a database, table and columns and reads the rows back. The new connector page lists the verified versions.
Each query opens and closes its own connection, and H2 drops an in-memory database when its last connection closes, so tables created by one query were gone in the next. Connect with DB_CLOSE_DELAY=-1 to jdbc:h2:mem:<database>.
H2 upper-cases unquoted aliases, so the information_schema queries returned rows keyed OBJECT_NAME and the tree showed nothing. Read them from JDBC metadata, which keys rows as the web reads them.
H2E2ETest runs the in-memory engine in process: a schema, table and rows created with SQL are listed and read back by later queries. H2 runs inside the server, so the connector page lists the bundled version and says why no container is needed.
H2 upper-cases unquoted aliases, so the information_schema queries returned rows keyed OBJECT_NAME and the tree showed nothing. Read them from JDBC metadata, which keys rows as the web reads them.
H2TCPE2ETest starts an H2 TCP server in process, creates a schema, table and rows with SQL, lists them and reads the rows back. The new connector page lists the verified version.
The host field holds a directory, but the URL was built as host:port and the JDBC URL guard rejects a path as a host, so no DuckDB source could connect. Build jdbc:duckdb:<directory>/<database> instead, checking the directory for URL option characters and the file name with the guard.
DuckDB reports tables as BASE TABLE, which JDBC metadata lookups for TABLE miss, so the tree showed no tables. Read them from information_schema.tables.
DuckDBE2ETest opens a database file in a temporary directory, creates a schema, table and rows with SQL, lists them and reads the rows back. DuckDB runs inside the server, so the connector page lists the bundled version and says why no container is needed.
Structure compare queried MySQL's information_schema.COLUMNS directly, so it failed for every source without it, such as Redis keys. It reads the columns from the plugin's getColumns now, by the standard result names, like the rest of the workbench.
The guides still described HTTP adapters and plugin modules that no longer exist. They now cover the rules every plugin follows (JDBC only, one metadata management, capabilities as a contract, e2e tests), the plugin and service classes, the two ways to list databases, tables and columns, writing a driver on metaforge-driver-sql, SPI registration, capabilities.json, the e2e test, local development and connector docs.
26ai names its product Oracle AI Database, so the validator's LIKE 'Oracle Database%' matched no row and the source check failed. Match 'Oracle%Database%' instead.
OracleE2ETest creates a user, table and rows with SQL on Oracle Free images, lists them as a database, table and columns and reads the rows back. The new connector page lists the verified versions.
Elasticsearch SQL has no version() function, so the default validator failed and no source could be checked. Select a constant, as other sources without a version function do.
…and 8.19 ElasticSearch8xE2ETest writes an index over HTTP, lists it as a table of the cluster with its fields and queries it with SQL. The new connector page lists the verified versions and the license the JDBC driver needs.
InfluxQL needs FROM in every SELECT, so the validator SELECT 1 failed and no source could be checked. SHOW DIAGNOSTICS answers without a database and starts with the server build.
InfluxDB has no information_schema and its driver reports retention policies as schemas, so the tree showed one empty database. List them with SHOW DATABASES, SHOW MEASUREMENTS and SHOW TAG/FIELD KEYS, keyed as the web reads them.
InfluxdbE2ETest writes points over HTTP, lists them as a database, measurement, tags and fields and reads them back with InfluxQL. The connector pages list the verified versions.
TrinoE2ETest creates a schema, table and rows in the memory catalog, lists them as a database, table and columns and reads the rows back. The new connector page lists the verified versions.
IoTDB has no version() function, so the default validator failed and no source could be checked.
The driver's schema metadata throws once no database is left, and it names devices without their path. List databases, devices (by full path, as queries name them) and measurements with SHOW DATABASES, SHOW DEVICES and SHOW TIMESERIES.
The server and the store catalogue read from cdn.north.devlive.org, the Tencent COS bucket, but releases uploaded to Aliyun OSS, so a new version stayed invisible until it was synced by hand. Plugin packages, the server distribution (the build job's, which embeds the web UI) and metadata.json now go to COS through a pinned COSCLI, as in ToolForge, and the server distribution is attached to the GitHub release.
The packaged application.properties still used the legacy stats.devlive.org alias.
The fault e2e cut the proxy once and kept losing the race against the sync: the source read is materialized before the throttled write on two of the engine pairs, and the writeRetries rewrote the failed batch on a fresh connection, so the runs kept finishing successfully. The test now cuts for good (down=true rejects every reconnect while dropAll severs the live connections), pins writeRetries to zero so the loss surfaces as the retryable failure the test asserts, and accepts that an immediate cut leaves no checkpoint because nothing was written yet. ChaosProxy gains the open connection count and a per-buffer throttle the test uses to stretch the source reads.
- findProjectRoot parsed the file: classpath URL as a relative File, which anchored the search at the working directory and made every module resolve as the project root - LocalSchedulerModuleTest then looked for the scheduler plugin under the test module directory and failed on an empty Optional - trust only multi-module root markers (mvnw, .git) for the root check, cover kotlin module layouts when skipping module dirs, and give the scheduler test an explicit scan depth
- findProjectRoot parsed the file: classpath URL as a relative File, which anchored the search at the working directory and made every module resolve as the project root - LocalSchedulerModuleTest then looked for the scheduler plugin under the test module directory and failed on an empty Optional - trust only multi-module root markers (mvnw, .git) for the root check, cover kotlin module layouts when skipping module dirs, and give the scheduler test an explicit scan depth
- the backend suite log grows past 100 MB and the Actions log stream truncates it around 30-100 MB, so every marathon failure surfaced without its stack trace - upload the per-module surefire reports on failure; they carry the exact failing test and exception regardless of log size
- the 1 hour suite finished with BUILD SUCCESS yet the step exited 127 five seconds later: the background pruner loop still held the step's stdout/stderr pipes and poisoned the final exit code - fully detach the loop (stdout/stderr/stdin) and reap it with wait after killing, so the step result is the Maven result alone
- the suite died mid-marathon with no surefire failures: engine containers (mongo first) got SIGKILLed with exit code 137 and the surefire JVM was reaped right after, which is the kernel OOM killer running out of the 7 GB runner RAM - enable an 8 GB swapfile before the suite so those peaks no longer trigger random OOM deaths; the runner has about 90 GB free disk
The flags after the surefire goal were wrapped onto their own lines inside the block scalar, so the shell ran "-Dcheckstyle.skip ..." as a command and the step died with exit 127 right after a green BUILD SUCCESS. The -pl/-amd and -DargLine options never took effect: every run tested the whole reactor with the default heap instead. Join the invocation into a single command.
Testcontainers marks an image as locally available and never re-checks it, so the every-5-minutes "prune -af" loop deleting postgres:16 between its pull and a later container start surfaced as "No such image" 404 in MetaForgeExecutorFaultE2ETest. Gate the aggressive prune on the free disk dropping below 35 GB and keep only the dangling-layer cleanup on the regular cadence.
Sources also return functions and procedures from getTables; the query tree showed them as tables. Views now sit in their own folder.
getTables listed every built-in function (over 1700) under each database.
A SELECT over a large table read every row into the server and sent them all to the browser. Statements run from the query page (ad hoc, history, snippets) are now bounded twice: for engines whose dialect ends with LIMIT, a single query without its own row clause gets LIMIT cap+1 written into the SQL, and the JDBC statement asks the driver for at most cap+1 rows. The adapter returns the first cap rows and marks the response as truncated, and the result panel says so. The cap defaults to 1000 and is set under System settings > Runtime > Query (1 to 100000). Reports, datasets and syncs still read whole results.
The v3.00.6 node answers logins with "Public key is not initialized yet, please try again later" for a while after its web port starts answering, and the validator call asserted immediately. Route it through the retrying ok() overload, which now also retries that transient marker alongside Presto/Trino's "No nodes available". Verified locally against both v2.00.18 and v3.00.6 containers.
core-0015 and core-0019 were edited after installations had already run them, so those databases fail startup with a checksum validation error. Accept the pre-edit checksums with validCheckSum and add core-0020, which renames the old checkpoint column to checkpoint_payload (guarded by columnExists, so fresh installs skip it). Verified on an H2 database built from the pre-edit changelog and against the five-database matrix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one)
Changelog entry (Details of this change)
Affected version
latest version