Add pgjdbc / DBeaver / DBX catalog metadata interception - #108
Open
lijiajun wants to merge 13 commits into
Open
Conversation
- 字段列表(01_columns): 新增 handle_dbeaver_columns_query_if_match, 从 attrelid=quote_ident(sch)||\x27.\x27||quote_ident(tbl) 提取表名, 用 PRAGMA table_info 生成 11 列; 原逻辑落到 verbatim SQLite 执行, pg_attribute.attrelid 是数字 OID 串与 sch.tbl 永远不匹配 -> 0 行 - 索引列表(02_indexes): 新增 v40_get_index_list + v40_extract_index_table, 指纹 pg_index&&array_agg&&!_pg_expandarray, 用 PRAGMA index_list/index_info 生成 9 列; v38 getIndexInfo 要求 _pg_expandarray 不命中此形态 - table=None 时返回 9 列空结构, 供 extended Describe 阶段宣告列数(v29i truth probe 不触发 realigning 短路), 提取到字面量表名时返回真实数据 - 部署验证: simple + extended(参数化 =public =users) 双路径均返回 1 行 9 列, Describe==Execute 列数一致
…p is_pg_catalog_object The v40+v41 rebase onto erans erans#87/erans#88 left extended.rs structurally broken (cherry-pick shifted a 520->21 line hunk, orphaning an if/else chain). erans now serves catalog columns natively from SQLite views, so the user's hardcoded catalog columns are redundant. Revert extended.rs to erans base and re-add only the self-contained is_pg_catalog_object helper that src/query/executor.rs depends on.
Rebasing v40+v41 onto erans erans#87/erans#88 left several user-added symbols unresolved because conflicted files were resolved by taking erans' version: - translator/mod.rs: declare the user-added escape/ilike/limit translator modules (files already present, only the mod declarations were dropped). - translator/schema_prefix_translator.rs: restore user's SchemaPrefixTranslator (strip_public_prefix + translate_query wired to LimitTranslator). - catalog/query_interceptor.rs: re-add CatalogInterceptor::parse_pg_type_modifiers and fix the v23_* test calls to use Self::parse_select (it is an assoc fn, not a free fn).
…regression) Rebase (commit 7dbae72) reverted extended.rs to erans' clean version and dropped the v29i catalog truth probe. Without it, Describe emitted NoData (0 columns) for catalog queries whose Parse phase cannot infer the result shape -- e.g. the DBX/JetBrains object-list query -- so DataGrip rendered an empty table list even though Execute returned 16 rows. Re-add v29i_fields_from_columns + v29i_probe_catalog_columns and call the probe at the top of handle_describe (statement path): when the announced field count disagrees with what CatalogInterceptor actually returns, we realign and send a correct RowDescription (memoised per query text). Verified on dune (v45, md5 0171742ed0a25c2bf4790b5ebc7326c1): an extended-protocol probe of the object-list query now returns 9 columns / 16 rows; the server log shows 'v29i truth probe: announced 0 fields but the catalog returns 9 -> realigning'.
pgsqlite 的 pg_inherits 视图缺 ihparent 列，dbx 点开 Tables 节点发的查询 (LEFT JOIN pg_inherits i ... pc.oid = i.ihparent + LIMIT -1 OFFSET) 报 'no such column: i.ihparent' -> 表列表空白。新增 v47_dbx_table_list handler， 把继承链去掉 (SQLite 无继承, parent_schema/parent_name 恒 NULL)，保留 table_name/table_type/table_comment 三列，列序严格对齐。 含此前未提交的 v45(schema-list)/v46(extension-list)。 Verified: 真实 Tables 查询 simple+extended 均返回 16 行 5 列，Describe==Execute 列数一致；v40/v41/v42/v43/v45/v46 回归无破坏。
dbx 走 extended 协议且用 Describe(Portal) 获取列信息，而 truth probe 只在 Describe(Statement) 运行。Parse 阶段对 catalog 查询故意设 fd_len=0 (handle_parse 526行 Vec::new())，导致 Describe(Portal) 直接发 NoData(0列)， dbx 据此把 Execute 的 5 列 DataRow 按 0 列丢弃 -> 表列表/对象列表空白。 修复：Parse 阶段对 catalog 查询调用 v29i_probe_catalog_columns 探测真实列数 并写入 field_descriptions，使 Describe(S)/Describe(Portal) 从一开始就拿到 正确列数，不再依赖 Describe(Statement) 的 realigning 补救。 Verified: 模拟 dbx 完整流程 Parse->Describe(S)->Bind->Describe(P)->Execute， Describe(S)=Describe(P)=Execute 字段数=5，16 行；simple 路径 16 行 5 列。 已部署 v47fix (md5 2f2a2ee8...)。
v29i_probe_catalog_columns 把 $1..$32 替换成 NULL 后执行探测。dbx 表列表 查询含 LIMIT CAST($4 AS BIGINT) OFFSET CAST($5 AS BIGINT)，参数化替换后 变成 LIMIT CAST(NULL AS BIGINT)，在 SQLite 里是非法语法 (LIMIT must be an integer)，导致整条探测失败、返回 None，Parse 阶段 fd_len 仍为 0 -> Describe(Portal) 发 NoData(0列) -> 客户端把数据按 0 列丢弃 -> 表列表空白。 列数与行数无关，探测前将 LIMIT/OFFSET NULL 修正为合法形态 (LIMIT 0 / 移除 OFFSET) 再探测。 已部署 v47fix2 (md5 0de88805...)。验证: 真参数化探针 Parse->Describe(S)->Bind->Describe(P)->Execute 三阶段列数均=5, 16 行; dbx 重连后自动发表列表查询, 服务端 execute_select 正常无报错。
dbx/DataGrip 只走 Describe(Portal) 取列数，且 Parse 阶段 catalog 探测未落到 statement（field_descriptions 恒为 0），导致 Describe(Portal) 发 NoData(0列)，客户端把 Execute 的 DataRow 按 0 列丢弃 -> 表列表/对象列表等节点空白。 在 Describe(Portal) field_descriptions 为空且为 catalog 查询时， 复用 v29i_probe_catalog_columns 探测真实列数并发送 RowDescription， 与 Describe(Statement) 的 truth probe 行为对齐。已用真实 Parse->Bind->Describe(Portal)->Execute 探针验证：Describe(Portal) 宣告 5 列、Execute 返回 16 行。部署 tag=v47fix3 md5=c0f11d40...
dbx/DataGrip 表列表查询用 LIMIT CAST($4 AS BIGINT) OFFSET CAST($5 AS BIGINT) 的参数写法，但 analyze_select_params 只识别 $N::type，漏掉了 CAST($N AS TYPE)，导致 $4/$5 被默认推断成 text(25) 而非 int8(20)。 服务端 Describe(S) 返回全 text 的参数类型，pgjdbc 拿到错误类型后在 Bind 阶段序列化参数时报 'error serializing parameter N'。 补上 CAST($N AS TYPE) 正则识别。验证：传空 param_types 触发推断， ParameterDescription 从 [25,25,25,25,25] 恢复为 [25,25,25,20,20]。 部署 tag=v47fix4 md5=f25c42c7185b11f15ed161b5892dab99
…such table dbx 点开表发 SELECT * FROM "public"."bath_records"，而 extended 协议的 handle_execute 走的是 extended.rs 自己的 execute_select(4623)，它直接 db.query 不经过 executor.rs 那条带 strip_public_prefix 的路径，SQLite 报 'no such table: public.bath_records'。 在 execute_select 入口补 strip_public_prefix（与 executor.rs PATCH v27 对齐）， 去掉 public. schema 前缀。验证：SELECT * FROM "public"."bath_records" 从报 no such table 变为正常返回 9 列(空表 0 行)，users 表正常返回 2 行。 部署 tag=v47fix5 md5=34c364d3b6d3c4a0fd96d20c7cd0b40d
dbx 点开表发外键查询(getImportedKeys, information_schema, 7 列)，但 Parse 阶段 field_descriptions 被错算成 3 列。dbx 走 Describe(Portal) 取列数， v47fix3 只在 field_descriptions 为空时补探测，没覆盖'非空但列数错'的情况， 导致 Describe(P) 发 3 列 RowDescription、Execute 发 7 列 DataRow，列数错位， pgjdbc 报 'unexpected message from server'。 将 catalog truth probe 前移到 Describe(Portal) 入口：对 catalog 查询探测真实 列数，与 field_descriptions 不一致则 realign。验证：外键查询 Describe(P) 从 3 列修正为 7 列。部署 tag=v47fix6 md5=560c5b864c3451db2230eef0793ec88f
dbx 点开表发 SELECT * FROM "public"."bath_records"（非 catalog），Parse 阶段 translated_for_analysis 没 strip public -> field_descriptions 测试查询失败 -> field_descriptions 空 -> Describe(P) 发 NoData(0列)，而 Execute 阶段 execute_select (v47fix5) strip 后返回 9 列 -> 列数错位 -> pgjdbc 'unexpected message from server'。 在 handle_parse 翻译链路补 strip_public_prefix（与 v47fix5 对齐），让 Parse 阶段 field_descriptions 正确算出 9 列。验证：Describe(P) 从 NoData 变为 9 列 RowDescription。 部署 tag=v47fix7 md5=a11fb86ddc99a7b89a7b5d84ef69c86a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pgsqlite exposes a SQLite database over the PostgreSQL wire protocol. GUI/IDE clients (DBeaver, DataGrip/DBX, and any tool driven by the PostgreSQL JDBC driver
pgjdbc) issue a fixed set ofDatabaseMetaDatacatalog queries (getColumns, getIndexInfo, getPrimaryKeys, getSchemas, getImported/ExportedKeys, getCrossReference, plus the01_columns/02_indexes/03_attrdefqueries DBeaver/DBX send). These templated queries don't always map cleanly onto SQLite, so the clients show empty column lists, missing indexes, or blank default-value labels.This PR adds a
CatalogInterceptorlayer that recognizes those known pgjdbc/DBeaver/DBX templates by query fingerprint and answers them directly from SQLitePRAGMAs, while leaving everything else to the existing translator.What's added
catalog/query_interceptor.rs— central interceptor withis_catalog_queryfingerprinting andv38_jdbc_metadata()that dispatches the known templates. Result columns are aligned to the exact projection width the client expects (critical for the extended-protocol Describe/Execute path, otherwiseunexpected message).intercept_query:handle_dbeaver_columns_query_if_match—01_columnsquery → column list inpg_attributeshape (fixes blank DBeaver column list).v40_get_index_list—02_indexesquery → index list built fromPRAGMA index_list+index_info(array_agg shape).v42attrdef handling —03_attrdefquery → column default labels synthesized fromPRAGMA table_infodflt_value(fixes blank default-value label in DBX).translator/escape_string_translator.rs,ilike_translator.rs,limit_translator.rs,schema_prefix_translator.rs) — escape-string literal handling, ILIKE, LIMIT, andpublic.schema-prefix stripping so client-generated SQL is rewritten into SQLite-compatible form.executor.rs,extended.rs,session/state.rs,db_handler.rs,config.rs,main.rs,functions/system_functions.rs,constraint_populator.rs,pg_settings.rs,protocol/codec.rs,types/datetime_utils.rs,unnest_translator.rs.Notes / scope
erans/pgsqlitemain(includes \dt reports "Did not find any tables" on a database that has tables #87/information_schema.tables lists pgsqlite's materialized pg_* and information_schema_* relations #88). Catalog columns that erans now serves natively from SQLite views (information_schema.*,pg_namespace,pg_class) are left to erans; this PR focuses on the pgjdbc/DBeaver/DBX templates that still need explicit handling.armv7-unknown-linux-musleabihf(the deployment target) with 0 errors.Test plan
psql/ psycopg2 workflows still behave as before.