diff --git a/CLAUDE.md b/CLAUDE.md index 0a7571d2..26e2c13e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,12 +85,13 @@ When modifying `__pgsqlite_*` tables: 2. Define migration with version, name, description, up/down SQL, and dependencies 3. Update Current Migrations list below -### Current Migrations (v1-v28) +### Current Migrations (v1-v29) - v1-v10: Initial schema, ENUM, DateTime, Arrays, Full-Text Search, catalog tables - v15-v19: pg_depend, pg_proc, pg_description, pg_roles/pg_user, pg_stats - v20-v25: information_schema support (routines, views, referential_constraints, check_constraints, triggers), pg_tablespace - v26-v27: Enhanced pg_attribute, pg_proc type fixes - v28: pg_class full column parity; internal relations moved to pg_catalog/information_schema namespaces +- v29: Namespace-aware information_schema.tables/.columns served from SQLite views ## Major Features diff --git a/docs/superpowers/plans/2026-08-11-information-schema-namespace.md b/docs/superpowers/plans/2026-08-11-information-schema-namespace.md new file mode 100644 index 00000000..6e0afb62 --- /dev/null +++ b/docs/superpowers/plans/2026-08-11-information-schema-namespace.md @@ -0,0 +1,1719 @@ +# Namespace-aware `information_schema` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `information_schema.tables` and `information_schema.columns` report pgsqlite's own catalog relations under `pg_catalog` / `information_schema` instead of as user tables in `public`, by serving both from SQLite views rather than hand-rolled Rust handlers. + +**Architecture:** An exact-name registry of the relations pgsqlite's migrations create is exposed to SQL as a UDF. Migration v29 rebuilds the two `information_schema` views to use it, plus four more UDFs that resolve a column's PostgreSQL type from `__pgsqlite_schema`. `SchemaPrefixTranslator` then rewrites `information_schema.tables` → `information_schema_tables` so the existing fall-through path in `CatalogInterceptor` executes the query against the view, and the Rust handlers are deleted. + +**Tech Stack:** Rust, rusqlite (scalar UDFs via `create_scalar_function`), SQLite views, tokio-postgres for integration tests. + +**Spec:** `docs/superpowers/specs/2026-08-11-information-schema-namespace-design.md` +**Issue:** [#88](https://github.com/erans/pgsqlite/issues/88). Follow-up: [#102](https://github.com/erans/pgsqlite/issues/102). + +## Global Constraints + +- **Pre-commit checklist** (from `CLAUDE.md`) before every commit: `cargo check` (no errors or warnings), `cargo clippy`, `cargo build`, `cargo test`. +- **Never infer types from column names.** Only explicit PG type declarations, `__pgsqlite_schema`, `PRAGMA table_info`, or explicit casts. +- **Datetime storage is INTEGER** (microseconds/days since epoch). A catalog surface must never report `integer` for a `TIMESTAMP` column — that leaks storage detail. +- **`LIKE '__pgsqlite_%'` is wrong** — `_` is a LIKE wildcard, so it matches unrelated names. New SQL uses `substr(name, 1, 11) <> '__pgsqlite_'`, the form #85 established. +- **Do not modify `pg_class` or `pg_attribute`.** Both are deferred to #102. If a task seems to need them, stop and re-read the spec's Non-goals. +- **Migration v29 is unreleased** across these tasks, so Tasks 3 and 5 both edit it. Any local test database created between those tasks must be deleted, or v29 will be recorded as applied and the second half will not run. Integration tests create a fresh DB per run and are unaffected. +- **`Migration::down` is never executed.** There is no rollback path anywhere in `src/migration/` — the field is stored and never read. v29 supplies a `down` to match repo convention, but do not write a test that exercises it and do not claim the migration was verified reversible. +- Namespace OIDs are fixed by the v28 `pg_namespace` view: `11` = `pg_catalog`, `2200` = `public`, `13000` = `information_schema`. + +## File Structure + +**Created:** +- `src/catalog/internal_relations.rs` — the exact-name list of relations pgsqlite's migrations create, and `relnamespace()` over it. No SQL, no I/O; a pure lookup table so it can be unit-tested and read at a glance. +- `src/catalog/column_type_info.rs` — `pg_column_info()`, the PG-type-string → `information_schema` column metadata mapping moved out of the handler being deleted. +- `tests/information_schema_namespace_test.rs` — integration tests for #88. + +**Modified:** +- `src/catalog/mod.rs` — declare the two new modules. +- `src/functions/catalog_functions.rs` — five new UDFs wrapping the two new modules. +- `src/migration/registry.rs` — migration v29; register it in the `MIGRATIONS` block. +- `src/translator/schema_prefix_translator.rs` — rewrite `information_schema.{tables,columns}` to the underscore view names. +- `src/catalog/query_interceptor.rs` — delete two dispatch arms, two handler functions, and one private helper. +- `src/session/db_handler.rs`, `src/query/extended.rs` — delete the now-unreachable duplicate handlers. +- `tests/migration_test.rs` — v29 added to the expected migration list. +- `CLAUDE.md` — Current Migrations list. + +The two new `src/catalog/` modules are deliberately separate: one answers "is this relation ours?", the other "what does this type look like to `information_schema`?". Neither knows about the other, and both are pure functions over their input, which is what makes the UDF wrappers trivial. + +--- + +### Task 1: Internal-relation registry and `__pgsqlite_relnamespace` + +**Files:** +- Create: `src/catalog/internal_relations.rs` +- Modify: `src/catalog/mod.rs:17` (add module declaration) +- Modify: `src/functions/catalog_functions.rs` (add UDF at end of `register_catalog_functions`) +- Test: unit tests inline in `src/catalog/internal_relations.rs` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `pgsqlite::catalog::internal_relations::relnamespace(name: &str) -> i32` + - `pgsqlite::catalog::internal_relations::INTERNAL_PG_CATALOG_RELATIONS: &[&str]` + - `pgsqlite::catalog::internal_relations::INTERNAL_INFORMATION_SCHEMA_RELATIONS: &[&str]` + - SQL function `__pgsqlite_relnamespace(name TEXT) -> INTEGER` returning `11`, `13000`, `2200`, or `NULL` for `NULL` input. + +- [ ] **Step 1: Write the failing test** + +Create `src/catalog/internal_relations.rs` containing only the test module for now: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn internal_pg_relations_map_to_pg_catalog() { + assert_eq!(relnamespace("pg_class"), PG_CATALOG_NAMESPACE); + assert_eq!(relnamespace("pg_constraint"), PG_CATALOG_NAMESPACE); + assert_eq!(relnamespace("idx_enum_values_label"), PG_CATALOG_NAMESPACE); + } + + #[test] + fn internal_information_schema_relations_map_to_information_schema() { + assert_eq!(relnamespace("information_schema_tables"), INFORMATION_SCHEMA_NAMESPACE); + assert_eq!(relnamespace("information_schema_columns"), INFORMATION_SCHEMA_NAMESPACE); + } + + #[test] + fn user_tables_map_to_public() { + assert_eq!(relnamespace("customers"), PUBLIC_NAMESPACE); + } + + /// The reason this is an exact-name list rather than a `LIKE 'pg\_%'` test: + /// a user table whose name merely starts with `pg_` is not ours. See #102. + #[test] + fn user_tables_named_like_catalog_relations_map_to_public() { + assert_eq!(relnamespace("pg_myreport"), PUBLIC_NAMESPACE); + assert_eq!(relnamespace("information_schema_export"), PUBLIC_NAMESPACE); + assert_eq!(relnamespace("idx_customers_email"), PUBLIC_NAMESPACE); + } + + #[test] + fn lists_have_no_duplicates_and_do_not_overlap() { + let mut all: Vec<&str> = INTERNAL_PG_CATALOG_RELATIONS + .iter() + .chain(INTERNAL_INFORMATION_SCHEMA_RELATIONS.iter()) + .copied() + .collect(); + let total = all.len(); + all.sort_unstable(); + all.dedup(); + assert_eq!(all.len(), total, "duplicate entry in the internal relation lists"); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test --lib internal_relations` +Expected: FAIL — compile error, `relnamespace` and the constants are not defined. + +- [ ] **Step 3: Write the implementation** + +Prepend to `src/catalog/internal_relations.rs`, above the test module: + +```rust +//! Exact names of the relations pgsqlite's own migrations create. +//! +//! Prefix matching is not sufficient in either direction: it claims user tables +//! that merely start with `pg_`, and it misses the `idx_*` indexes migrations +//! create on `__pgsqlite_*` tables. Both failure modes are user-visible, so this +//! list is exact and is guarded against drift by +//! `tests/information_schema_namespace_test.rs::internal_relation_list_matches_migrated_database`. +//! +//! Adding a catalog relation to `src/migration/registry.rs` means adding it here. + +/// Namespace OIDs, fixed by the `pg_namespace` view in migration v28. +pub const PG_CATALOG_NAMESPACE: i32 = 11; +pub const PUBLIC_NAMESPACE: i32 = 2200; +pub const INFORMATION_SCHEMA_NAMESPACE: i32 = 13000; + +/// Relations pgsqlite creates that PostgreSQL keeps in `pg_catalog`. +pub const INTERNAL_PG_CATALOG_RELATIONS: &[&str] = &[ + // Real tables + "pg_attrdef", + "pg_constraint", + "pg_depend", + "pg_index", + // Views + "pg_am", + "pg_attribute", + "pg_class", + "pg_database", + "pg_description", + "pg_enum", + "pg_foreign_data_wrapper", + "pg_namespace", + "pg_proc", + "pg_roles", + "pg_stat_activity", + "pg_stat_all_indexes", + "pg_stat_all_tables", + "pg_stat_database", + "pg_stat_user_indexes", + "pg_stat_user_tables", + "pg_type", + "pg_user", + // Indexes on __pgsqlite_* tables. Not named __pgsqlite_*, so no prefix + // filter catches them; they leak into \di today (#102). + "idx_array_types_table", + "idx_comments_lookup", + "idx_datetime_cache_table", + "idx_enum_values_label", + "idx_enum_values_type", + "idx_fts_metadata_table", + "idx_numeric_constraints_table", + "idx_string_constraints_table", +]; + +/// Relations pgsqlite creates that PostgreSQL keeps in `information_schema`. +pub const INTERNAL_INFORMATION_SCHEMA_RELATIONS: &[&str] = &[ + "information_schema_columns", + "information_schema_key_column_usage", + "information_schema_referential_constraints", + "information_schema_schemata", + "information_schema_table_constraints", + "information_schema_tables", +]; + +/// Namespace OID for a relation name. Anything unlisted is a user relation. +/// +/// Unlisted defaults to `public` deliberately: the failure mode is showing an +/// internal relation, never hiding a user's table. +pub fn relnamespace(name: &str) -> i32 { + if INTERNAL_PG_CATALOG_RELATIONS.contains(&name) { + PG_CATALOG_NAMESPACE + } else if INTERNAL_INFORMATION_SCHEMA_RELATIONS.contains(&name) { + INFORMATION_SCHEMA_NAMESPACE + } else { + PUBLIC_NAMESPACE + } +} +``` + +Add to `src/catalog/mod.rs` after line 17 (`pub mod constraint_populator;`): + +```rust +pub mod internal_relations; +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cargo test --lib internal_relations` +Expected: PASS, 5 tests. + +- [ ] **Step 5: Register the UDF** + +In `src/functions/catalog_functions.rs`, inside `register_catalog_functions`, before its final `Ok(())`: + +```rust + // __pgsqlite_relnamespace(name) - namespace OID for a relation. + // Lets catalog views ask "is this relation ours?" by exact name rather than + // by prefix. Returns NULL for NULL input so a view row degrades rather than + // failing the query. + conn.create_scalar_function( + "__pgsqlite_relnamespace", + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let name: Option = ctx.get(0)?; + Ok(name.map(|n| crate::catalog::internal_relations::relnamespace(&n))) + }, + )?; +``` + +- [ ] **Step 6: Verify the UDF is reachable from SQL** + +Add to the test module in `src/catalog/internal_relations.rs`: + +```rust + #[test] + fn udf_is_callable_from_sql() { + let conn = rusqlite::Connection::open_in_memory().unwrap(); + crate::functions::register_all_functions(&conn).unwrap(); + + let ns: i32 = conn + .query_row("SELECT __pgsqlite_relnamespace('pg_class')", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, PG_CATALOG_NAMESPACE); + + let ns: i32 = conn + .query_row("SELECT __pgsqlite_relnamespace('pg_myreport')", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, PUBLIC_NAMESPACE); + + let ns: Option = conn + .query_row("SELECT __pgsqlite_relnamespace(NULL)", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, None); + } +``` + +Run: `cargo test --lib internal_relations` +Expected: PASS, 6 tests. + +- [ ] **Step 7: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings, all tests pass. + +- [ ] **Step 8: Commit** + +```bash +git add src/catalog/internal_relations.rs src/catalog/mod.rs src/functions/catalog_functions.rs +git commit -m "feat: exact-name registry of pgsqlite's internal catalog relations (#88) + +Prefix matching fails both ways: 'pg\_%' claims a user table named +pg_myreport, and nothing catches the 8 idx_* indexes migrations create on +__pgsqlite_* tables. Exact list of all 36, exposed to SQL as +__pgsqlite_relnamespace(name) for the catalog views to use." +``` + +--- + +### Task 2: `pg_column_info` and the four type UDFs + +**Files:** +- Create: `src/catalog/column_type_info.rs` +- Modify: `src/catalog/mod.rs` (add module declaration) +- Modify: `src/functions/catalog_functions.rs` (four UDFs) +- Test: unit tests inline in `src/catalog/column_type_info.rs` + +**Interfaces:** +- Consumes: nothing. +- Produces: + - `pgsqlite::catalog::column_type_info::PgColumnInfo { data_type: String, character_maximum_length: Option, numeric_precision: Option, numeric_scale: Option }` + - `pgsqlite::catalog::column_type_info::pg_column_info(pg_type: &str) -> PgColumnInfo` + - SQL functions `__pgsqlite_pg_data_type(TEXT) -> TEXT`, `__pgsqlite_char_max_length(TEXT) -> INTEGER`, `__pgsqlite_numeric_precision(TEXT) -> INTEGER`, `__pgsqlite_numeric_scale(TEXT) -> INTEGER`. + +This is a move-and-fix of the private `map_sqlite_type_to_pg_column_info` at `src/catalog/query_interceptor.rs:2190`, which Task 6 deletes. The move is what keeps `character_maximum_length`, `numeric_precision` and `numeric_scale` populated once the handler is gone. The fixes are the rows that measurement showed wrong: `SERIAL`, `TIMESTAMPTZ`, `TIME`/`TIMETZ`, and array types. + +- [ ] **Step 1: Write the failing test** + +Create `src/catalog/column_type_info.rs` containing only the test module: + +```rust +#[cfg(test)] +mod tests { + use super::*; + + fn dt(pg_type: &str) -> String { + pg_column_info(pg_type).data_type + } + + /// Types the deleted handler already got right. These must not regress. + #[test] + fn preserves_correct_handler_behavior() { + assert_eq!(dt("INTEGER"), "integer"); + assert_eq!(dt("TEXT"), "text"); + assert_eq!(dt("BOOLEAN"), "boolean"); + assert_eq!(dt("UUID"), "uuid"); + assert_eq!(dt("JSONB"), "jsonb"); + assert_eq!(dt("NUMERIC(10,2)"), "numeric"); + assert_eq!(dt("VARCHAR(50)"), "character varying"); + assert_eq!(dt("TIMESTAMP"), "timestamp without time zone"); + assert_eq!(dt("DATE"), "date"); + assert_eq!(dt("BLOB"), "bytea"); + } + + /// Types the deleted handler got wrong. Measured in the spec. + #[test] + fn fixes_types_the_handler_reported_as_text() { + assert_eq!(dt("SERIAL"), "integer"); + assert_eq!(dt("BIGSERIAL"), "bigint"); + assert_eq!(dt("TIMESTAMPTZ"), "timestamp with time zone"); + assert_eq!(dt("TIMESTAMP WITH TIME ZONE"), "timestamp with time zone"); + assert_eq!(dt("TIMETZ"), "time with time zone"); + assert_eq!(dt("TIME WITH TIME ZONE"), "time with time zone"); + assert_eq!(dt("TIME"), "time without time zone"); + } + + #[test] + fn array_types_report_array() { + assert_eq!(dt("TEXT[]"), "ARRAY"); + assert_eq!(dt("INTEGER[]"), "ARRAY"); + assert_eq!(dt("NUMERIC(10,2)[]"), "ARRAY"); + } + + #[test] + fn character_maximum_length_comes_from_the_modifier() { + assert_eq!(pg_column_info("VARCHAR(50)").character_maximum_length, Some(50)); + assert_eq!(pg_column_info("CHAR(8)").character_maximum_length, Some(8)); + assert_eq!(pg_column_info("VARCHAR").character_maximum_length, None); + assert_eq!(pg_column_info("TEXT").character_maximum_length, None); + } + + #[test] + fn numeric_precision_and_scale_come_from_the_modifier() { + let info = pg_column_info("NUMERIC(10,2)"); + assert_eq!(info.numeric_precision, Some(10)); + assert_eq!(info.numeric_scale, Some(2)); + + let info = pg_column_info("DECIMAL(38)"); + assert_eq!(info.numeric_precision, Some(38)); + assert_eq!(info.numeric_scale, Some(0)); + + let info = pg_column_info("NUMERIC"); + assert_eq!(info.numeric_precision, None); + assert_eq!(info.numeric_scale, None); + } + + /// PostgreSQL reports precision and scale for integer types too. + #[test] + fn integer_types_report_binary_precision() { + let info = pg_column_info("INTEGER"); + assert_eq!(info.numeric_precision, Some(32)); + assert_eq!(info.numeric_scale, Some(0)); + + let info = pg_column_info("BIGINT"); + assert_eq!(info.numeric_precision, Some(64)); + assert_eq!(info.numeric_scale, Some(0)); + } + + /// Text is the safe fallback: matches both the deleted handler's final + /// branch and the view's `ELSE 'text'`. ENUM types land here (#88 non-goal). + #[test] + fn unknown_types_fall_back_to_text() { + assert_eq!(dt("my_enum_type"), "text"); + assert_eq!(dt(""), "text"); + } + + #[test] + fn case_is_insensitive() { + assert_eq!(dt("varchar(50)"), "character varying"); + assert_eq!(dt("TimestampTZ"), "timestamp with time zone"); + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test --lib column_type_info` +Expected: FAIL — compile error, `pg_column_info` is not defined. + +- [ ] **Step 3: Write the implementation** + +Prepend to `src/catalog/column_type_info.rs`: + +```rust +//! PostgreSQL type string to `information_schema` column metadata. +//! +//! Moved out of `CatalogInterceptor::map_sqlite_type_to_pg_column_info` when the +//! `information_schema.columns` handler was deleted, so the views could keep +//! reporting `character_maximum_length`, `numeric_precision` and `numeric_scale`. +//! +//! Input is the declared PostgreSQL type from `__pgsqlite_schema.pg_type` +//! (`VARCHAR(50)`, `NUMERIC(10,2)`, `TIMESTAMPTZ`), falling back to the SQLite +//! declared type for databases created outside pgsqlite. Deriving from the +//! string rather than an OID is deliberate — an OID cannot carry the modifier. + +/// The four `information_schema.columns` fields that depend on a column's type. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PgColumnInfo { + pub data_type: String, + pub character_maximum_length: Option, + pub numeric_precision: Option, + pub numeric_scale: Option, +} + +impl PgColumnInfo { + fn plain(data_type: &str) -> Self { + Self { + data_type: data_type.to_string(), + character_maximum_length: None, + numeric_precision: None, + numeric_scale: None, + } + } + + fn integral(data_type: &str, precision: i32) -> Self { + Self { + data_type: data_type.to_string(), + character_maximum_length: None, + numeric_precision: Some(precision), + numeric_scale: Some(0), + } + } +} + +/// Split `NUMERIC(10,2)` into `("NUMERIC", ["10", "2"])`. +fn split_modifier(upper: &str) -> (&str, Vec<&str>) { + match (upper.find('('), upper.rfind(')')) { + (Some(open), Some(close)) if close > open => { + let base = upper[..open].trim(); + let params = upper[open + 1..close] + .split(',') + .map(|p| p.trim()) + .collect(); + (base, params) + } + _ => (upper.trim(), Vec::new()), + } +} + +pub fn pg_column_info(pg_type: &str) -> PgColumnInfo { + let upper = pg_type.trim().to_uppercase(); + + // Array types report `ARRAY`, with the element type in udt_name in real + // PostgreSQL. Checked before modifiers so `NUMERIC(10,2)[]` is an array. + if upper.ends_with("[]") { + return PgColumnInfo::plain("ARRAY"); + } + + let (base, params) = split_modifier(&upper); + let param = |i: usize| params.get(i).and_then(|p| p.parse::().ok()); + + match base { + "VARCHAR" | "CHARACTER VARYING" => PgColumnInfo { + data_type: "character varying".to_string(), + character_maximum_length: param(0), + numeric_precision: None, + numeric_scale: None, + }, + "CHAR" | "CHARACTER" | "BPCHAR" => PgColumnInfo { + data_type: "character".to_string(), + character_maximum_length: param(0), + numeric_precision: None, + numeric_scale: None, + }, + "NUMERIC" | "DECIMAL" => PgColumnInfo { + data_type: "numeric".to_string(), + character_maximum_length: None, + numeric_precision: param(0), + // A precision with no scale means scale 0; no precision means + // unconstrained, where PostgreSQL reports NULL for both. + numeric_scale: param(1).or(param(0).map(|_| 0)), + }, + + "SMALLINT" | "INT2" | "SMALLSERIAL" => PgColumnInfo::integral("smallint", 16), + "INTEGER" | "INT" | "INT4" | "SERIAL" => PgColumnInfo::integral("integer", 32), + "BIGINT" | "INT8" | "BIGSERIAL" => PgColumnInfo::integral("bigint", 64), + + "REAL" | "FLOAT4" => PgColumnInfo { + data_type: "real".to_string(), + character_maximum_length: None, + numeric_precision: Some(24), + numeric_scale: None, + }, + "DOUBLE PRECISION" | "FLOAT8" | "FLOAT" | "DOUBLE" => PgColumnInfo { + data_type: "double precision".to_string(), + character_maximum_length: None, + numeric_precision: Some(53), + numeric_scale: None, + }, + + "TEXT" => PgColumnInfo::plain("text"), + "BYTEA" | "BLOB" => PgColumnInfo::plain("bytea"), + "BOOLEAN" | "BOOL" => PgColumnInfo::plain("boolean"), + "UUID" => PgColumnInfo::plain("uuid"), + "JSON" => PgColumnInfo::plain("json"), + "JSONB" => PgColumnInfo::plain("jsonb"), + "MONEY" => PgColumnInfo::plain("money"), + "INTERVAL" => PgColumnInfo::plain("interval"), + + "DATE" => PgColumnInfo::plain("date"), + "TIME" | "TIME WITHOUT TIME ZONE" => PgColumnInfo::plain("time without time zone"), + "TIMETZ" | "TIME WITH TIME ZONE" => PgColumnInfo::plain("time with time zone"), + "TIMESTAMP" | "DATETIME" | "TIMESTAMP WITHOUT TIME ZONE" => { + PgColumnInfo::plain("timestamp without time zone") + } + "TIMESTAMPTZ" | "TIMESTAMP WITH TIME ZONE" => { + PgColumnInfo::plain("timestamp with time zone") + } + + // Unknown types, including ENUMs, report text. Matches the deleted + // handler's final branch and the view's previous `ELSE 'text'`. + _ => PgColumnInfo::plain("text"), + } +} +``` + +Add to `src/catalog/mod.rs`: + +```rust +pub mod column_type_info; +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cargo test --lib column_type_info` +Expected: PASS, 8 tests. + +- [ ] **Step 5: Register the four UDFs** + +In `src/functions/catalog_functions.rs`, after the `__pgsqlite_relnamespace` registration: + +```rust + // Column type metadata for the information_schema.columns view. Four + // scalar functions rather than one, because a SQLite scalar UDF returns a + // single value and the view needs four fields from the same input. + macro_rules! register_column_info_fn { + ($name:literal, $field:ident, $ty:ty) => { + conn.create_scalar_function( + $name, + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let pg_type: Option = ctx.get(0)?; + let value: Option<$ty> = pg_type.and_then(|t| { + crate::catalog::column_type_info::pg_column_info(&t).$field.into() + }); + Ok(value) + }, + )?; + }; + } + + conn.create_scalar_function( + "__pgsqlite_pg_data_type", + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let pg_type: Option = ctx.get(0)?; + Ok(pg_type.map(|t| crate::catalog::column_type_info::pg_column_info(&t).data_type)) + }, + )?; + + register_column_info_fn!("__pgsqlite_char_max_length", character_maximum_length, i32); + register_column_info_fn!("__pgsqlite_numeric_precision", numeric_precision, i32); + register_column_info_fn!("__pgsqlite_numeric_scale", numeric_scale, i32); +``` + +Note on the macro: `pg_column_info(...).$field` is already `Option`, so `.into()` is an identity conversion that lets `and_then` accept it. If the compiler objects, replace `.into()` with nothing and use `.and_then(|t| pg_column_info(&t).$field)` directly — that is the intended semantics either way: `NULL` in gives `NULL` out, and a type with no modifier gives `NULL`. + +- [ ] **Step 6: Verify the UDFs are reachable from SQL** + +Add to the test module in `src/catalog/column_type_info.rs`: + +```rust + #[test] + fn udfs_are_callable_from_sql() { + let conn = rusqlite::Connection::open_in_memory().unwrap(); + crate::functions::register_all_functions(&conn).unwrap(); + + let (dt, len): (String, Option) = conn + .query_row( + "SELECT __pgsqlite_pg_data_type('VARCHAR(50)'), __pgsqlite_char_max_length('VARCHAR(50)')", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(dt, "character varying"); + assert_eq!(len, Some(50)); + + let (p, s): (Option, Option) = conn + .query_row( + "SELECT __pgsqlite_numeric_precision('NUMERIC(10,2)'), __pgsqlite_numeric_scale('NUMERIC(10,2)')", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!((p, s), (Some(10), Some(2))); + + let dt: Option = conn + .query_row("SELECT __pgsqlite_pg_data_type(NULL)", [], |r| r.get(0)) + .unwrap(); + assert_eq!(dt, None); + } +``` + +Run: `cargo test --lib column_type_info` +Expected: PASS, 9 tests. + +- [ ] **Step 7: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings, all tests pass. + +- [ ] **Step 8: Commit** + +```bash +git add src/catalog/column_type_info.rs src/catalog/mod.rs src/functions/catalog_functions.rs +git commit -m "feat: pg_column_info and type UDFs for information_schema.columns (#88) + +Moves map_sqlite_type_to_pg_column_info out of the handler about to be +deleted, so the view can keep populating character_maximum_length, +numeric_precision and numeric_scale. Fixes the rows measurement showed +wrong: SERIAL, TIMESTAMPTZ, TIME/TIMETZ and array types all reported text." +``` + +--- + +### Task 3: Migration v29 — rebuild both `information_schema` views + +**Files:** +- Modify: `src/migration/registry.rs:37` (register v29) and end of file (define it) +- Modify: `tests/migration_test.rs:26,34` (expected migration list and version) +- Test: `tests/migration_test.rs` + +**Interfaces:** +- Consumes: `__pgsqlite_relnamespace`, `__pgsqlite_pg_data_type`, `__pgsqlite_char_max_length`, `__pgsqlite_numeric_precision`, `__pgsqlite_numeric_scale` from Tasks 1 and 2. +- Produces: SQLite views `information_schema_tables` and `information_schema_columns` with corrected `table_schema` and column metadata. Column order and names are unchanged from v14 except where noted below. + +Two column-order notes, both deliberate: + +- `information_schema_tables` moves `is_insertable_into` from position 5 to position 10, where PostgreSQL puts it. The v14 view had it at 5; the Rust handler being deleted had it at 10. Matching the handler keeps `SELECT *` results stable for existing clients and matches real PostgreSQL. +- `information_schema_columns` keeps v14's 44 columns in v14's order, which already matches the handler and PostgreSQL. + +**Both view bodies below were prototyped against a migrated database before this plan was written**, with SQL stand-ins for the UDFs. Confirmed: the lateral `JOIN pragma_table_info(m.name)` resolves, the `pg_namespace` join partitions relations 22 `pg_catalog` / 6 `information_schema` / 2 user, `COALESCE(s.pg_type, p.type)` yields the declared PostgreSQL types (`SERIAL`, `NUMERIC(10,2)`, `TEXT[]`, `TIMESTAMPTZ`) that `pg_column_info` expects, and `p.pk > 0` gives `is_nullable = 'NO'` for `SERIAL PRIMARY KEY`. If a query against these views returns nothing, suspect the UDFs or `trusted_schema`, not the join shape. + +One environment note for manual poking: pgsqlite runs SQLite in WAL mode, so `cp foo.db bar.db` silently loses recent writes. Use `sqlite3 foo.db ".backup bar.db"`. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/migration_test.rs`: + +```rust +#[test] +fn test_v29_information_schema_views_are_namespace_aware() { + let temp_dir = TempDir::new().unwrap(); + let db_path = temp_dir.path().join("v29.db"); + + let conn = Connection::open(&db_path).unwrap(); + let mut runner = MigrationRunner::new(conn); + runner.run_pending_migrations().unwrap(); + let conn = runner.into_connection(); + + // The views call UDFs, which the migration runner's connection does not have. + pgsqlite::functions::register_all_functions(&conn).unwrap(); + conn.pragma_update(None, "trusted_schema", true).unwrap(); + + conn.execute_batch( + "CREATE TABLE customers (id INTEGER PRIMARY KEY, name VARCHAR(50), amount NUMERIC(10,2));", + ) + .unwrap(); + + // Internal relations are not in public. + let internal_in_public: i32 = conn + .query_row( + "SELECT COUNT(*) FROM information_schema_tables \ + WHERE table_schema = 'public' AND table_name LIKE 'pg\\_%' ESCAPE '\\'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(internal_in_public, 0, "pg_* relations still reported in public"); + + let pg_class_schema: String = conn + .query_row( + "SELECT table_schema FROM information_schema_tables WHERE table_name = 'pg_class'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(pg_class_schema, "pg_catalog"); + + let is_tables_schema: String = conn + .query_row( + "SELECT table_schema FROM information_schema_tables \ + WHERE table_name = 'information_schema_tables'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(is_tables_schema, "information_schema"); + + // The user table is the only thing in public. + let public_tables: Vec = conn + .prepare("SELECT table_name FROM information_schema_tables WHERE table_schema = 'public' ORDER BY table_name") + .unwrap() + .query_map([], |r| r.get(0)) + .unwrap() + .collect::>() + .unwrap(); + assert_eq!(public_tables, vec!["customers".to_string()]); + + // Column metadata the old view lost. + let (dt, len): (String, Option) = conn + .query_row( + "SELECT data_type, character_maximum_length FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'name'", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(dt, "character varying"); + assert_eq!(len, Some(50)); + + let (p, s): (Option, Option) = conn + .query_row( + "SELECT numeric_precision, numeric_scale FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'amount'", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!((p, s), (Some(10), Some(2))); + + // INTEGER PRIMARY KEY is NOT NULL, which the v14 view got wrong. + let nullable: String = conn + .query_row( + "SELECT is_nullable FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'id'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(nullable, "NO"); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test --test migration_test test_v29_information_schema_views_are_namespace_aware` +Expected: FAIL — `assert_eq!(internal_in_public, 0)` reports a non-zero count, because v14's view hardcodes `'public'`. + +- [ ] **Step 3: Register the migration** + +In `src/migration/registry.rs`, after line 37 (`register_v28_pg_class_full_columns(&mut registry);`): + +```rust + register_v29_information_schema_namespace(&mut registry); +``` + +- [ ] **Step 4: Write the migration** + +Append to `src/migration/registry.rs`: + +```rust +/// Version 29: Namespace-aware information_schema views (#88) +/// +/// The views are rebuilt rather than patched. `information_schema_tables` +/// derives `table_schema` from `__pgsqlite_relnamespace` rather than hardcoding +/// `'public'`, and reads the UDF directly rather than `pg_class.relnamespace` so +/// it does not inherit v28's `LIKE 'pg\_%'` heuristic (#102). +/// +/// `information_schema_columns` reads sqlite_master, pragma_table_info and +/// __pgsqlite_schema directly rather than layering on pg_attribute, which keeps +/// this migration off a view every ORM reads and is what makes column_default, +/// is_nullable, character_maximum_length and numeric precision/scale reachable. +fn register_v29_information_schema_namespace(registry: &mut BTreeMap) { + registry.insert(29, Migration { + version: 29, + name: "information_schema_namespace", + description: "Namespace-aware information_schema.tables and .columns served from SQLite", + up: MigrationAction::SqlBatch(&[ + r#"DROP VIEW IF EXISTS information_schema_tables"#, + r#"DROP VIEW IF EXISTS information_schema_columns"#, + + // Column order matches PostgreSQL: is_insertable_into is 10th, not + // 5th as in v14. The deleted Rust handler already used this order. + r#" + CREATE VIEW information_schema_tables AS + SELECT + 'main' as table_catalog, + n.nspname as table_schema, + c.relname as table_name, + CASE c.relkind + WHEN 'r' THEN 'BASE TABLE' + WHEN 'v' THEN 'VIEW' + ELSE 'UNKNOWN' + END as table_type, + NULL as self_referencing_column_name, + NULL as reference_generation, + NULL as user_defined_type_catalog, + NULL as user_defined_type_schema, + NULL as user_defined_type_name, + CASE c.relkind WHEN 'r' THEN 'YES' ELSE 'NO' END as is_insertable_into, + 'NO' as is_typed, + NULL as commit_action + FROM pg_class c + JOIN pg_namespace n ON n.oid = __pgsqlite_relnamespace(c.relname) + WHERE c.relkind IN ('r', 'v') + "#, + + // 44 columns in v14's order, which matches PostgreSQL. + // substr(...) <> '__pgsqlite_' rather than LIKE '__pgsqlite_%': + // `_` is a LIKE wildcard, so the LIKE form matches unrelated names. + r#" + CREATE VIEW information_schema_columns AS + SELECT + 'main' as table_catalog, + n.nspname as table_schema, + m.name as table_name, + p.name as column_name, + p.cid + 1 as ordinal_position, + p.dflt_value as column_default, + CASE WHEN p."notnull" = 1 OR p.pk > 0 THEN 'NO' ELSE 'YES' END as is_nullable, + __pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type)) as data_type, + __pgsqlite_char_max_length(COALESCE(s.pg_type, p.type)) as character_maximum_length, + __pgsqlite_char_max_length(COALESCE(s.pg_type, p.type)) as character_octet_length, + __pgsqlite_numeric_precision(COALESCE(s.pg_type, p.type)) as numeric_precision, + CASE + WHEN __pgsqlite_numeric_precision(COALESCE(s.pg_type, p.type)) IS NULL THEN NULL + ELSE 10 + END as numeric_precision_radix, + __pgsqlite_numeric_scale(COALESCE(s.pg_type, p.type)) as numeric_scale, + NULL as datetime_precision, + NULL as interval_type, + NULL as interval_precision, + NULL as character_set_catalog, + NULL as character_set_schema, + NULL as character_set_name, + NULL as collation_catalog, + NULL as collation_schema, + NULL as collation_name, + NULL as domain_catalog, + NULL as domain_schema, + NULL as domain_name, + 'main' as udt_catalog, + 'pg_catalog' as udt_schema, + __pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type)) as udt_name, + NULL as scope_catalog, + NULL as scope_schema, + NULL as scope_name, + NULL as maximum_cardinality, + p.cid + 1 as dtd_identifier, + 'NO' as is_self_referencing, + 'NO' as is_identity, + NULL as identity_generation, + NULL as identity_start, + NULL as identity_increment, + NULL as identity_maximum, + NULL as identity_minimum, + 'NO' as identity_cycle, + 'NEVER' as is_generated, + NULL as generation_expression, + 'YES' as is_updatable + FROM sqlite_master m + JOIN pragma_table_info(m.name) p + LEFT JOIN __pgsqlite_schema s + ON s.table_name = m.name AND s.column_name = p.name + JOIN pg_namespace n ON n.oid = __pgsqlite_relnamespace(m.name) + WHERE m.type = 'table' + AND m.name NOT LIKE 'sqlite_%' + AND substr(m.name, 1, 11) <> '__pgsqlite_' + "#, + + r#" + UPDATE __pgsqlite_metadata + SET value = '29', updated_at = strftime('%s', 'now') + WHERE key = 'schema_version'; + "#, + ]), + down: Some(MigrationAction::SqlBatch(&[ + r#"DROP VIEW IF EXISTS information_schema_tables"#, + r#"DROP VIEW IF EXISTS information_schema_columns"#, + + // Restore the v14 views: copied verbatim from + // register_v14_information_schema_views's `up`. + r#" + CREATE VIEW IF NOT EXISTS information_schema_tables AS + SELECT + 'main' as table_catalog, + 'public' as table_schema, + relname as table_name, + CASE relkind + WHEN 'r' THEN 'BASE TABLE' + WHEN 'v' THEN 'VIEW' + ELSE 'UNKNOWN' + END as table_type, + 'YES' as is_insertable_into, + NULL as self_referencing_column_name, + NULL as reference_generation, + NULL as user_defined_type_catalog, + NULL as user_defined_type_schema, + NULL as user_defined_type_name, + 'NO' as is_typed, + 'NO' as commit_action + FROM pg_class + WHERE relkind IN ('r', 'v'); + "#, + + // NOTE TO IMPLEMENTER: this `down` is never executed. No rollback + // path exists in src/migration/ -- `Migration::down` is stored and + // never read. It is populated to match repo convention so a future + // rollback runner finds it, not because it can be tested. + r#" + CREATE VIEW IF NOT EXISTS information_schema_columns AS + SELECT + 'main' as table_catalog, + 'public' as table_schema, + c.relname as table_name, + a.attname as column_name, + a.attnum as ordinal_position, + NULL as column_default, + CASE WHEN a.attnotnull = 't' THEN 'NO' ELSE 'YES' END as is_nullable, + CASE a.atttypid + WHEN 23 THEN 'integer' + WHEN 25 THEN 'text' + WHEN 700 THEN 'real' + WHEN 701 THEN 'double precision' + WHEN 17 THEN 'bytea' + WHEN 1043 THEN 'character varying' + WHEN 1042 THEN 'character' + WHEN 16 THEN 'boolean' + WHEN 1082 THEN 'date' + WHEN 1083 THEN 'time without time zone' + WHEN 1114 THEN 'timestamp without time zone' + WHEN 1184 THEN 'timestamp with time zone' + WHEN 1700 THEN 'numeric' + ELSE 'text' + END as data_type, + NULL as character_maximum_length, + NULL as character_octet_length, + NULL as numeric_precision, + NULL as numeric_precision_radix, + NULL as numeric_scale, + NULL as datetime_precision, + NULL as interval_type, + NULL as interval_precision, + NULL as character_set_catalog, + NULL as character_set_schema, + NULL as character_set_name, + NULL as collation_catalog, + NULL as collation_schema, + NULL as collation_name, + NULL as domain_catalog, + NULL as domain_schema, + NULL as domain_name, + NULL as udt_catalog, + NULL as udt_schema, + NULL as udt_name, + NULL as scope_catalog, + NULL as scope_schema, + NULL as scope_name, + NULL as maximum_cardinality, + NULL as dtd_identifier, + 'NO' as is_self_referencing, + 'NO' as is_identity, + NULL as identity_generation, + NULL as identity_start, + NULL as identity_increment, + NULL as identity_maximum, + NULL as identity_minimum, + NULL as identity_cycle, + 'NO' as is_generated, + NULL as generation_expression, + 'NO' as is_updatable + FROM pg_class c + JOIN pg_attribute a ON c.oid = a.attrelid + WHERE c.relkind = 'r' + AND a.attnum > 0; + "#, + + r#" + UPDATE __pgsqlite_metadata + SET value = '28', updated_at = strftime('%s', 'now') + WHERE key = 'schema_version'; + "#, + ])), + dependencies: vec![28], + }); +} +``` + +- [ ] **Step 5: Update the migration list test** + +In `tests/migration_test.rs`, change line 26 to append `29` to the expected vector, and line 34 to expect version `"29"`: + +```rust + assert_eq!(applied, vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]); +``` + +```rust + assert_eq!(version, "29"); +``` + +Also search the file for any other hardcoded `28` and update it: + +Run: `grep -n '\b28\b' tests/migration_test.rs` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `cargo test --test migration_test` +Expected: PASS, all tests including the new one. + +- [ ] **Step 7: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings, all tests pass. Some `information_schema` integration tests still exercise the Rust handlers and should be unaffected — the views are not routed to yet. + +- [ ] **Step 8: Commit** + +```bash +git add src/migration/registry.rs tests/migration_test.rs +git commit -m "feat: migration v29 rebuilds information_schema views namespace-aware (#88) + +table_schema now comes from __pgsqlite_relnamespace rather than a hardcoded +'public'. columns reads sqlite_master/pragma_table_info/__pgsqlite_schema +directly instead of pg_attribute, which keeps v29 off a view every ORM reads +and recovers column_default, is_nullable for INTEGER PRIMARY KEY, +character_maximum_length and numeric precision/scale. + +Views are not routed to yet -- the Rust handlers still intercept." +``` + +--- + +### Task 4: Route `information_schema.tables` to the view + +**Files:** +- Modify: `src/translator/schema_prefix_translator.rs:11-44,76-93` +- Modify: `src/catalog/query_interceptor.rs:728-731` (delete arm), `:1844-1935` (delete handler) +- Modify: `src/session/db_handler.rs:1174-1176`, `:1650-1652`, `:2662-2729` (delete duplicates) +- Modify: `src/query/extended.rs:2262` (delete branch) +- Create: `tests/information_schema_namespace_test.rs` + +**Interfaces:** +- Consumes: the `information_schema_tables` view from Task 3. +- Produces: `information_schema.tables` queries executing as SQL against the view. No Rust API surface. + +- [ ] **Step 1: Write the failing test** + +Create `tests/information_schema_namespace_test.rs`: + +```rust +mod common; +use common::setup_test_server; + +/// Issue #88: information_schema.tables listed pgsqlite's own catalog relations +/// as user tables in public, so Django inspectdb and SQLAlchemy automap would +/// generate models for pg_constraint and friends. +#[tokio::test] +async fn internal_relations_are_not_reported_in_public() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query("CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT)") + .await + .unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["customers".to_string()]); +} + +#[tokio::test] +async fn internal_relations_report_their_real_schema() { + let server = setup_test_server().await; + let client = &server.client; + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"pg_class"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "pg_catalog"); + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"information_schema_tables"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "information_schema"); +} + +/// The deleted handler ignored ORDER BY entirely. +#[tokio::test] +async fn order_by_is_respected() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE zebra (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE alpha (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE middle (id INTEGER)").await.unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["alpha".to_string(), "middle".to_string(), "zebra".to_string()]); +} + +/// The deleted handler returned zero rows for any aggregate. +#[tokio::test] +async fn aggregates_work() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE a (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE b (id INTEGER)").await.unwrap(); + + let row = client + .query_one( + "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, i64>(0), 2); +} + +/// The deleted handler supported equality on table_name and nothing else. +#[tokio::test] +async fn non_table_name_predicates_filter() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE t (id INTEGER)").await.unwrap(); + client.simple_query("CREATE VIEW v AS SELECT id FROM t").await.unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables \ + WHERE table_schema = 'public' AND table_type = 'VIEW'", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["v".to_string()]); +} + +/// #102 divergence, asserted so it is documented rather than discovered. +/// pg_class still uses v28's LIKE 'pg\_%' heuristic, so a user table named +/// pg_* is misfiled there while information_schema.tables gets it right. +/// When #102 lands, the second assertion flips to "public". +#[tokio::test] +async fn user_table_named_like_a_catalog_relation_is_visible() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query("CREATE TABLE pg_myreport (id INTEGER PRIMARY KEY)") + .await + .unwrap(); + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"pg_myreport"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "public", "#88: exact-name matching"); + + let rows = client + .query( + "SELECT n.nspname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace \ + WHERE c.relname = $1", + &[&"pg_myreport"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!( + rows[0].get::<_, &str>(0), + "pg_catalog", + "known divergence until #102 lands; flip this to 'public' there" + ); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test --test information_schema_namespace_test` +Expected: FAIL. `internal_relations_are_not_reported_in_public` reports 29 names instead of 1, because the Rust handler still intercepts. + +- [ ] **Step 3: Add the rewrite to SchemaPrefixTranslator** + +In `src/translator/schema_prefix_translator.rs`, inside `translate_query` after the `catalog_functions` loop (line 40) and before the `debug!`: + +```rust + // information_schema relations exist as SQLite views with underscores. + // Rewriting here routes them to the views through the interceptor's + // fall-through path, the same way pg_catalog.* reaches pg_class. + // Only the two relations served by views; the rest still have handlers. + result = result.replace("information_schema.tables", "information_schema_tables"); + result = result.replace("information_schema.columns", "information_schema_columns"); + result = result.replace("INFORMATION_SCHEMA.TABLES", "information_schema_tables"); + result = result.replace("INFORMATION_SCHEMA.COLUMNS", "information_schema_columns"); +``` + +In `translate_object_name`, replace the comment at line 91 and add the AST path: + +```rust + if schema_name == "pg_catalog" { + // Replace with just the table name + name.0 = vec![table.clone()]; + } else if schema_name == "information_schema" { + // The two relations served by SQLite views are rewritten to + // their underscore names; the rest keep their Rust handlers. + let table_name = match table { + ObjectNamePart::Identifier(ident) => ident.value.to_lowercase(), + }; + if table_name == "tables" || table_name == "columns" { + let mut ident = match table { + ObjectNamePart::Identifier(ident) => ident.clone(), + }; + ident.value = format!("information_schema_{table_name}"); + name.0 = vec![ObjectNamePart::Identifier(ident)]; + } + } +``` + +- [ ] **Step 4: Add translator unit tests** + +Add to the `tests` module at the bottom of `src/translator/schema_prefix_translator.rs`: + +```rust + #[test] + fn test_information_schema_tables_rewrite() { + let query = "SELECT table_name FROM information_schema.tables ORDER BY 1"; + let translated = SchemaPrefixTranslator::translate_query(query); + assert_eq!(translated, "SELECT table_name FROM information_schema_tables ORDER BY 1"); + } + + #[test] + fn test_information_schema_columns_rewrite() { + let query = "SELECT * FROM information_schema.columns"; + let translated = SchemaPrefixTranslator::translate_query(query); + assert_eq!(translated, "SELECT * FROM information_schema_columns"); + } + + /// Relations that still have Rust handlers must not be rewritten -- there is + /// no `information_schema_routines` view to fall through to. + #[test] + fn test_other_information_schema_relations_are_untouched() { + for relation in ["routines", "views", "triggers", "check_constraints"] { + let query = format!("SELECT * FROM information_schema.{relation}"); + assert_eq!(SchemaPrefixTranslator::translate_query(&query), query); + } + } + + /// `table_constraints` shares a prefix with `tables` -- verify no collision. + #[test] + fn test_table_constraints_is_not_caught_by_the_tables_rewrite() { + let query = "SELECT * FROM information_schema.table_constraints"; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } +``` + +Run: `cargo test --lib schema_prefix_translator` +Expected: PASS. + +- [ ] **Step 5: Delete the `tables` handler and its dispatch** + +In `src/catalog/query_interceptor.rs`, delete the arm at lines 728-731: + +```rust + // Handle information_schema.tables queries + if table_name.contains("information_schema.tables") { + return Some(Ok(Self::handle_information_schema_tables_query(select, &db).await)); + } +``` + +and delete the whole `handle_information_schema_tables_query` function (lines 1844-1935, from `async fn handle_information_schema_tables_query` through its closing brace). + +In `src/session/db_handler.rs`, delete the two dispatch checks: + +```rust + if lower_query.contains("information_schema.tables") { + return self.handle_information_schema_tables_query(query, session_id).await; + } +``` + +(at lines 1174-1176 and 1650-1652) and the whole `handle_information_schema_tables_query` method (lines 2662-2729). + +In `src/query/extended.rs`, delete the `else if query.contains("information_schema.tables")` branch beginning at line 2262 — read the surrounding `if`/`else if` chain first and keep it well-formed. + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `cargo test --test information_schema_namespace_test` +Expected: PASS, 6 tests. + +If `internal_relations_report_their_real_schema` fails with "no such function: __pgsqlite_relnamespace", the query reached a connection without UDFs registered — check `src/session/connection_manager.rs:80`. + +If it fails with "no such table: information_schema_tables", the rewrite did not fire; log the translated query at `src/catalog/query_interceptor.rs:97`. + +- [ ] **Step 7: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings. `cargo clippy` may now flag `extract_table_name_filters` as dead code if the columns handler was its only other caller — leave it until Task 5, which deletes that caller too. + +- [ ] **Step 8: Commit** + +```bash +git add -A +git commit -m "fix: serve information_schema.tables from SQLite, not a Rust handler (#88) + +Closes the headline defect: internal relations now report pg_catalog and +information_schema for table_schema instead of appearing as user tables in +public. + +Also fixes three defects the handler carried: ORDER BY was ignored, +aggregates returned zero rows, and any predicate other than equality on +table_name was silently dropped." +``` + +--- + +### Task 5: Route `information_schema.columns` to the view + +**Files:** +- Modify: `src/catalog/query_interceptor.rs:733-740` (delete arm), `:1937-2190` (delete handler), `:2190-2245` (delete `map_sqlite_type_to_pg_column_info`) +- Modify: `tests/information_schema_namespace_test.rs` (add tests) + +**Interfaces:** +- Consumes: the `information_schema_columns` view from Task 3; the rewrite added in Task 4 already covers `information_schema.columns`. +- Produces: `information_schema.columns` queries executing as SQL against the view. + +- [ ] **Step 1: Write the failing test** + +Add to `tests/information_schema_namespace_test.rs`: + +```rust +/// The type fidelity table from the spec. The deleted handler got id, ts and +/// tags wrong; the pre-v29 view got six of eight wrong, including reporting +/// `integer` for TIMESTAMPTZ, which leaked the INTEGER datetime storage. +#[tokio::test] +async fn column_types_are_reported_faithfully() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query( + "CREATE TABLE fidelity ( + id SERIAL PRIMARY KEY, + amount NUMERIC(10,2), + uid UUID, + doc JSONB, + tags TEXT[], + ts TIMESTAMPTZ, + flag BOOLEAN, + nick VARCHAR(50) + )", + ) + .await + .unwrap(); + + let rows = client + .query( + "SELECT column_name, data_type FROM information_schema.columns \ + WHERE table_name = 'fidelity' ORDER BY ordinal_position", + &[], + ) + .await + .unwrap(); + + let got: Vec<(String, String)> = rows + .iter() + .map(|r| (r.get::<_, String>(0), r.get::<_, String>(1))) + .collect(); + + assert_eq!( + got, + vec![ + ("id".to_string(), "integer".to_string()), + ("amount".to_string(), "numeric".to_string()), + ("uid".to_string(), "uuid".to_string()), + ("doc".to_string(), "jsonb".to_string()), + ("tags".to_string(), "ARRAY".to_string()), + ("ts".to_string(), "timestamp with time zone".to_string()), + ("flag".to_string(), "boolean".to_string()), + ("nick".to_string(), "character varying".to_string()), + ] + ); +} + +/// The four columns the handler populated and the pre-v29 view left NULL. +/// Django and SQLAlchemy both read these. +#[tokio::test] +async fn column_modifiers_and_defaults_are_reported() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query( + "CREATE TABLE widgets ( + id INTEGER PRIMARY KEY, + name VARCHAR(50), + price NUMERIC(10,2), + active BOOLEAN DEFAULT true + )", + ) + .await + .unwrap(); + + let row = client + .query_one( + "SELECT character_maximum_length FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'name'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, Option>(0), Some(50)); + + let row = client + .query_one( + "SELECT numeric_precision, numeric_scale FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'price'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, Option>(0), Some(10)); + assert_eq!(row.get::<_, Option>(1), Some(2)); + + let row = client + .query_one( + "SELECT column_default FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'active'", + &[], + ) + .await + .unwrap(); + assert!( + row.get::<_, Option>(0).is_some(), + "column_default should be populated for DEFAULT true" + ); + + // INTEGER PRIMARY KEY is NOT NULL in PostgreSQL. pragma_table_info reports + // notnull=0 for it, so the view must also consider pk. + let row = client + .query_one( + "SELECT is_nullable FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'id'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, &str>(0), "NO"); +} + +#[tokio::test] +async fn columns_of_internal_relations_are_not_in_public() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE t (id INTEGER)").await.unwrap(); + + let rows = client + .query( + "SELECT DISTINCT table_name FROM information_schema.columns \ + WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["t".to_string()]); +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cargo test --test information_schema_namespace_test column_types_are_reported_faithfully` +Expected: FAIL — `id` reports `text` and `ts` reports `text`, because the Rust handler still intercepts. + +- [ ] **Step 3: Delete the `columns` handler and its dispatch** + +In `src/catalog/query_interceptor.rs`, delete the arm at lines 733-740: + +```rust + // Handle information_schema.columns queries + if table_name.contains("information_schema.columns") { + if let Some(ref session_state) = session { + return Some(Self::handle_information_schema_columns_query_with_session(select, &db, &session_state.id).await); + } else { + return None; + } + } +``` + +Delete `handle_information_schema_columns_query_with_session` (starts line 1937) in full, and `map_sqlite_type_to_pg_column_info` (starts line 2190) in full — the latter now lives in `src/catalog/column_type_info.rs`. + +Then check whether any other handler still calls the helpers those two used: + +Run: `grep -n "extract_selected_columns\|extract_table_name_filters" src/catalog/query_interceptor.rs` + +Keep them if other handlers call them; delete any that are now unused. `cargo clippy` will report dead code either way. + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cargo test --test information_schema_namespace_test` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Run the pre-existing information_schema tests** + +Run: `cargo test --test information_schema_test --test information_schema_comprehensive_test` +Expected: PASS. These assert `data_type` for `INTEGER`, `VARCHAR`, `DECIMAL(10,2)`, `BOOLEAN`, `TIMESTAMP`, and `is_nullable = 'NO'` for `INTEGER PRIMARY KEY` — all covered by the new view. + +If `test_information_schema_columns` fails on a column count, the view is emitting rows for a table the handler filtered. Compare its `WHERE` clause against the handler's `type='table' AND name NOT LIKE 'sqlite_%' AND substr(name,1,11) <> '__pgsqlite_'`. + +- [ ] **Step 6: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings, all tests pass. + +- [ ] **Step 7: Commit** + +```bash +git add -A +git commit -m "fix: serve information_schema.columns from SQLite, not a Rust handler (#88) + +Types are now resolved from __pgsqlite_schema through pg_column_info, so +SERIAL reports integer and TIMESTAMPTZ reports timestamp with time zone +instead of text. character_maximum_length, numeric precision/scale, +column_default and is_nullable for INTEGER PRIMARY KEY are preserved from +the deleted handler rather than regressing to NULL." +``` + +--- + +### Task 6: Drift guard, docs, and full-suite verification + +**Files:** +- Modify: `tests/information_schema_namespace_test.rs` (drift test) +- Modify: `CLAUDE.md` (Current Migrations list) + +**Interfaces:** +- Consumes: everything from Tasks 1-5. +- Produces: nothing new; this task closes the loop. + +- [ ] **Step 1: Write the drift test** + +Add to `tests/information_schema_namespace_test.rs`: + +```rust +/// The internal-relation list is maintained by hand, so it can drift when a +/// migration adds a catalog relation. Fail here rather than silently leaking +/// the new relation into information_schema.tables as a user table. +#[test] +fn internal_relation_list_matches_migrated_database() { + use pgsqlite::catalog::internal_relations::{ + INTERNAL_INFORMATION_SCHEMA_RELATIONS, INTERNAL_PG_CATALOG_RELATIONS, + }; + use pgsqlite::migration::MigrationRunner; + use rusqlite::Connection; + use tempfile::TempDir; + + let temp_dir = TempDir::new().unwrap(); + let db_path = temp_dir.path().join("drift.db"); + let conn = Connection::open(&db_path).unwrap(); + let mut runner = MigrationRunner::new(conn); + runner.run_pending_migrations().unwrap(); + let conn = runner.into_connection(); + + let mut in_database: Vec = conn + .prepare( + "SELECT name FROM sqlite_master \ + WHERE name NOT LIKE 'sqlite_%' AND substr(name, 1, 11) <> '__pgsqlite_' \ + ORDER BY name", + ) + .unwrap() + .query_map([], |r| r.get(0)) + .unwrap() + .collect::>() + .unwrap(); + in_database.sort(); + + let mut in_list: Vec = INTERNAL_PG_CATALOG_RELATIONS + .iter() + .chain(INTERNAL_INFORMATION_SCHEMA_RELATIONS.iter()) + .map(|s| s.to_string()) + .collect(); + in_list.sort(); + + let missing: Vec<&String> = in_database.iter().filter(|n| !in_list.contains(n)).collect(); + assert!( + missing.is_empty(), + "migrations create relations absent from src/catalog/internal_relations.rs: {missing:?}. \ + Add them, or they will be reported as user tables in information_schema.tables." + ); + + let stale: Vec<&String> = in_list.iter().filter(|n| !in_database.contains(n)).collect(); + assert!( + stale.is_empty(), + "src/catalog/internal_relations.rs lists relations no migration creates: {stale:?}" + ); +} +``` + +- [ ] **Step 2: Run the drift test** + +Run: `cargo test --test information_schema_namespace_test internal_relation_list_matches_migrated_database` +Expected: PASS. If it fails listing relations, add exactly those names to the correct list in `src/catalog/internal_relations.rs` — `information_schema_*` names go in `INTERNAL_INFORMATION_SCHEMA_RELATIONS`, everything else in `INTERNAL_PG_CATALOG_RELATIONS`. + +- [ ] **Step 3: Update CLAUDE.md** + +In the `### Current Migrations (v1-v28)` section, change the heading to `(v1-v29)` and append: + +```markdown +- v29: Namespace-aware information_schema.tables/.columns served from SQLite views +``` + +- [ ] **Step 4: Run the full Rust suite** + +Run: `cargo test` +Expected: PASS. Pay attention to `orm_constraint_discovery_test` and `permission_functions_test`, which query `information_schema` and were not touched directly. + +- [ ] **Step 5: Run the SQLAlchemy suite** + +Run: `./tests/python/run_sqlalchemy_tests.sh` +Expected: 8/8 pass. This is the strongest signal that ORM introspection still works after the handler deletions. + +If the script needs a driver argument or a virtualenv that is not present, note that in the commit message rather than skipping silently. + +- [ ] **Step 6: Manual verification against the issue's reproduction** + +```bash +cargo build +rm -f /tmp/t88.sqlite* +./target/debug/pgsqlite --database /tmp/t88.sqlite --port 5599 & +sleep 3 +psql "host=127.0.0.1 port=5599 user=postgres dbname=main gssencmode=disable" \ + -c "CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT);" +psql "host=127.0.0.1 port=5599 user=postgres dbname=main gssencmode=disable" \ + -tA -c "SELECT table_schema, table_name FROM information_schema.tables ORDER BY 1,2;" +psql "host=127.0.0.1 port=5599 user=postgres dbname=main gssencmode=disable" -c "\dt" +kill %1 +``` + +Expected: `customers` is the only row with `table_schema = public`; internal relations report `pg_catalog` or `information_schema`; the list is sorted; `\dt` still shows only `customers`. + +- [ ] **Step 7: Pre-commit checklist** + +Run: `cargo check && cargo clippy && cargo build && cargo test` +Expected: no errors, no new warnings, all tests pass. + +- [ ] **Step 8: Commit** + +```bash +git add -A +git commit -m "test: guard the internal-relation list against migration drift (#88) + +A future migration that adds a catalog relation without updating +internal_relations.rs would silently leak it into information_schema.tables +as a user table. Fail in CI instead. + +Closes #88." +``` + +--- + +## Post-implementation + +- File the two follow-ups the spec names but does not track yet: view columns in `information_schema.columns` / `pg_attribute`, and the umbrella audit of the six remaining `information_schema` Rust handlers (`key_column_usage`, `table_constraints`, `referential_constraints`, `routines`, `views`, `schemata`) for the same `ORDER BY` / aggregate / predicate defects. +- [#102](https://github.com/erans/pgsqlite/issues/102) is already filed and now unblocked: both its changes depend on `__pgsqlite_relnamespace`, which Task 1 delivers. diff --git a/docs/superpowers/specs/2026-08-11-information-schema-namespace-design.md b/docs/superpowers/specs/2026-08-11-information-schema-namespace-design.md new file mode 100644 index 00000000..1890d5d3 --- /dev/null +++ b/docs/superpowers/specs/2026-08-11-information-schema-namespace-design.md @@ -0,0 +1,341 @@ +# Namespace-aware `information_schema` served from SQLite + +Issue: [#88](https://github.com/erans/pgsqlite/issues/88) +Date: 2026-08-11 +Status: Approved, ready for implementation planning + +## Problem + +`information_schema.tables` reports pgsqlite's own materialized catalog +relations as user tables in the `public` schema: + +``` +$ psql -tA -c "SELECT table_schema, table_name FROM information_schema.tables ORDER BY 1,2" +public|pg_am +public|pg_attrdef +public|pg_class +... +public|information_schema_tables +public|information_schema_columns +... +public|customers +``` + +28 internal relations alongside the one user table. Django `inspectdb` and +SQLAlchemy `automap` would generate models for `pg_constraint` and friends. + +In real PostgreSQL these rows *do* exist — `information_schema.tables` on a +stock cluster returns ~180 rows — but under `table_schema = 'pg_catalog'` and +`table_schema = 'information_schema'`. Clients filter on `table_schema`; that +is the mechanism that makes them invisible, not omission. + +### Root cause + +Not the handler cited in the issue. `src/catalog/query_interceptor.rs:2005` sits +inside the *columns* handler. The live path is +`handle_information_schema_tables_query` at +`src/catalog/query_interceptor.rs:1844`, dispatched from the arm at +`src/catalog/query_interceptor.rs:729`. It reads: + +```rust +db.query("SELECT name, type FROM sqlite_master WHERE type IN ('table', 'view') \ + AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%'") +``` + +— no catalog filtering at all — and emits `Some("public".to_string().into_bytes())` +for every row's `table_schema`. + +Migration v28 (from #87) already assigns `pg_%` relations to namespace 11 and +`information_schema_%` to 13000 in the `pg_class` view, which is why `\dt` is +clean. The `information_schema_tables` *view* at +`src/migration/registry.rs:1815` selects from `pg_class` but hardcodes +`'public' as table_schema`, discarding that work — and the Rust handler bypasses +both. + +### The handlers are stale, not just unfiltered + +The same class of defect #87 found in `PgClassHandler`. Measured against the +Rust handler on a fresh database: + +| Query | Result | +| --- | --- | +| `SELECT table_name FROM information_schema.tables ORDER BY 1` | 29 rows, **unsorted** — `ORDER BY` ignored | +| `SELECT count(*) FROM information_schema.tables WHERE table_schema='public'` | **0 rows** — aggregates unsupported | +| `SELECT count(*) FROM information_schema.tables WHERE table_type='BASE TABLE'` | **0 rows** | +| `SELECT DISTINCT table_name FROM information_schema.columns` | 60 rows, **`DISTINCT` ignored** | + +`WHERE` support extends only to equality on `table_name` +(`extract_table_name_filters`); every other predicate is silently dropped rather +than erroring. Any fix that keeps the handler inherits all of this. + +### Why the `columns` half is harder + +`information_schema_columns` reads `pg_attribute`, whose `atttypid` is a `CASE` +over the *SQLite* declared type (`src/migration/registry.rs:2604`), not the +PostgreSQL type recorded in `__pgsqlite_schema`. The two paths are differently +wrong, measured on +`CREATE TABLE fidelity (id SERIAL PRIMARY KEY, amount NUMERIC(10,2), uid UUID, +doc JSONB, tags TEXT[], ts TIMESTAMPTZ, flag BOOLEAN, nick VARCHAR(50))`: + +| column | declared | Rust handler | SQLite view | correct | +| --- | --- | --- | --- | --- | +| `id` | `SERIAL` | `text` ✗ | `integer` ✓ | `integer` | +| `amount` | `NUMERIC(10,2)` | `numeric` ✓ | `text` ✗ | `numeric` | +| `uid` | `UUID` | `uuid` ✓ | `text` ✗ | `uuid` | +| `doc` | `JSONB` | `jsonb` ✓ | `text` ✗ | `jsonb` | +| `tags` | `TEXT[]` | `text` ✗ | `text` ✗ | `ARRAY` | +| `ts` | `TIMESTAMPTZ` | `text` ✗ | `integer` ✗✗ | `timestamp with time zone` | +| `flag` | `BOOLEAN` | `boolean` ✓ | `integer` ✗ | `boolean` | +| `nick` | `VARCHAR(50)` | `character varying` ✓ | `text` ✗ | `character varying` | + +Routing `columns` to the view as-is would regress six of eight columns, and +`ts → integer` would leak the INTEGER datetime storage into a client-visible +surface. `pg_attribute` must become type-accurate first. The authoritative +mapping already exists as `SchemaTypeMapper::pg_type_string_to_oid` +(`src/types/schema_type_mapper.rs:112`); it simply is not reachable from SQL. + +### Prefix matching is not sufficient + +v28 identifies internal relations with `name LIKE 'pg\_%' ESCAPE '\'`. A user +table named `pg_myreport` is therefore filed under `pg_catalog` — today that +only hides it from `\dt`. If `information_schema.tables` derived `table_schema` +by joining `pg_namespace` on `pg_class.relnamespace`, it would inherit the +misfiling and hide that table from ORM introspection too — a regression on +current behavior, where it is at least visible. The `information_schema` views +therefore call the exact-name UDF directly rather than reading +`pg_class.relnamespace`; correcting `pg_class` itself is a follow-up. + +The prefix test also *under*-matches. Migrations create eight indexes named +`idx_*`, which no `__pgsqlite_%` or `pg_%` filter catches: + +``` +$ psql -c "\di" + Schema | Name | Type | Owner | Table +--------+-------------------------------+-------+----------+------- + public | idx_array_types_table | index | postgres | + public | idx_comments_lookup | index | postgres | + ... 8 rows +``` + +All eight are `CREATE INDEX IF NOT EXISTS` statements in +`src/migration/registry.rs`, and all eight leak into `\di` and ORM index +reflection today. Indexes are not visible through `information_schema.tables` +(which selects `relkind IN ('r','v')`), so this leak is fixed by the `pg_class` +follow-up, not here — but the registry this design introduces is what makes that +fix a one-line change. + +## Design + +Make the SQLite views the single source of truth, delete the Rust handlers, and +identify internal relations by exact name rather than by prefix. + +### Component 1: internal-relation registry + +New module `src/catalog/internal_relations.rs` holding a `const` list of the +relation names pgsqlite's migrations create — 4 tables, 24 views, 8 indexes — +each tagged with its namespace OID (11 `pg_catalog`, 13000 `information_schema`). + +This is the authoritative answer to "is this ours?". It supersedes the +`LIKE 'pg\_%'` heuristic for the `information_schema` views in this change, and +for `pg_class` in the follow-up. It is a static list because the migration +registry is static; adding a catalog relation to a future migration means adding +a line here, and the drift test below fails loudly if the two disagree. + +Exposed to SQL as one UDF in `src/functions/catalog_functions.rs`: + +- `__pgsqlite_relnamespace(name TEXT) -> INTEGER` — returns `11`, `13000`, or + `2200` for anything not in the list. + +### Component 2: type-resolution UDFs + +The handler being deleted resolves a column's PostgreSQL type through +`map_sqlite_type_to_pg_column_info` (`src/catalog/query_interceptor.rs:2190`), +a private function returning +`(data_type, character_maximum_length, numeric_precision, numeric_scale)` from +the declared type string. Three of those four are `NULL` in the current view, so +deleting the handler without replacing them would regress `VARCHAR(50)` to no +length and `NUMERIC(10,2)` to no precision — both read by Django and SQLAlchemy. + +Move that function to `src/catalog/column_type_info.rs` as +`pub fn pg_column_info(pg_type: &str) -> PgColumnInfo`, fixing the gaps the +measurement above exposed (`SERIAL`, `TIMESTAMPTZ`, `TIME`/`TIMETZ`, array +suffixes), and expose it as four UDFs in `src/functions/catalog_functions.rs`: + +- `__pgsqlite_pg_data_type(pg_type TEXT) -> TEXT` — the SQL-standard `data_type` + spelling: `integer`, `character varying`, `timestamp with time zone`, `ARRAY` +- `__pgsqlite_char_max_length(pg_type TEXT) -> INTEGER | NULL` +- `__pgsqlite_numeric_precision(pg_type TEXT) -> INTEGER | NULL` +- `__pgsqlite_numeric_scale(pg_type TEXT) -> INTEGER | NULL` + +Deriving from the type string rather than an OID is deliberate: an OID cannot +carry the `(50)` or `(10,2)` modifier those last three columns need. + +All five UDFs register through `functions::register_all_functions` +(`src/functions/mod.rs:20`), so every connection gets them, including the one +`db.query()` uses for the fall-through path. + +### Component 3: migration v29 + +Two view redefinitions, following the v28 pattern of `DROP VIEW` + `CREATE VIEW` +with a `down` that restores the v14 definitions verbatim. `pg_class` and +`pg_attribute` are both deliberately left alone — see Non-goals. + +- **`information_schema_tables`** — `table_schema` becomes + `__pgsqlite_relnamespace(relname)` resolved through `pg_namespace.nspname`, + replacing `'public' as table_schema`. It reads the UDF directly rather than + `pg_class.relnamespace`, so it does not inherit v28's prefix heuristic. +- **`information_schema_columns`** — rebuilt on `sqlite_master`, + `pragma_table_info`, and `__pgsqlite_schema` directly, the same sources + `pg_attribute` reads, rather than layering on `pg_attribute`. This keeps v29 + off a view every ORM reads for column reflection, and is what makes the four + recovered columns reachable: + + | column | source | + | --- | --- | + | `table_schema` | `__pgsqlite_relnamespace(m.name)` → `pg_namespace.nspname` | + | `column_default` | `pragma_table_info.dflt_value` | + | `is_nullable` | `pragma_table_info.notnull` **or** `pk` — an `INTEGER PRIMARY KEY` is `NOT NULL`, which the current view gets wrong and `tests/information_schema_test.rs:183` asserts | + | `data_type`, `udt_name` | `__pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type))` | + | `character_maximum_length`, `character_octet_length` | `__pgsqlite_char_max_length(...)` | + | `numeric_precision`, `numeric_scale` | `__pgsqlite_numeric_precision/scale(...)` | + + `COALESCE(s.pg_type, p.type)` preserves behavior for tables with no + `__pgsqlite_schema` row (databases created outside pgsqlite). + +`information_schema_tables` continues to select `FROM pg_class` for the relation +list and `relkind`; only the namespace derivation bypasses it. + +### Component 4: routing + +`SchemaPrefixTranslator::translate_query` +(`src/translator/schema_prefix_translator.rs:11`) already strips `pg_catalog.` +and the interceptor executes the rewritten query via `db.query()` when no +handler claims it (`src/catalog/query_interceptor.rs:198`) — the mechanism that +carries `pg_class` to SQLite post-#87. Line 91's comment, *"Don't remove +information_schema prefix - it's handled by query interceptor"*, is the thing +being changed. + +Add `information_schema.tables → information_schema_tables` and +`information_schema.columns → information_schema_columns`, then delete: + +- the dispatch arms at `src/catalog/query_interceptor.rs:729` and `:734` +- `handle_information_schema_tables_query` and + `handle_information_schema_columns_query_with_session` +- the unreachable duplicates at `src/session/db_handler.rs:1174`, `:1650`, + `:2663` and the branch at `src/query/extended.rs:2262` + +The JOIN-rewrite path at `src/catalog/query_interceptor.rs:275` already performs +the same substitution and stays as-is; after this change both paths converge on +the same views instead of disagreeing. + +### Data flow + +``` +psql + └─ executor + └─ CatalogInterceptor::intercept + ├─ SchemaPrefixTranslator information_schema.tables + │ → information_schema_tables + ├─ handle_catalog_query → None (no arm matches) + └─ db.query(translated) + └─ SQLite view + ├─ pg_class (relation list, relkind) + └─ __pgsqlite_relnamespace(relname) → pg_namespace.nspname +``` + +### Error handling + +- `__pgsqlite_relnamespace` returns `2200` for any unlisted name, so an + unrecognized relation is treated as a user table — the safe direction, since + the failure mode is showing an internal relation rather than hiding a user's. +- `__pgsqlite_pg_data_type` returns `text` for unmapped type strings, matching + both the current view's `ELSE 'text'` and the handler's final fallback. +- `__pgsqlite_char_max_length`, `__pgsqlite_numeric_precision` and + `__pgsqlite_numeric_scale` return `NULL` when the type carries no modifier — + `NULL` is what `information_schema` specifies for a type without one. +- All five UDFs return `NULL` on `NULL` input rather than erroring, so a view + row with a missing `__pgsqlite_schema` entry degrades instead of failing the + query. +- Migration `down` restores the v14 `information_schema_*` definitions verbatim. + `pg_class` and `pg_attribute` are untouched by v29, so their v28 and v26 + definitions survive both directions. + +## Testing + +TDD throughout, starting from a test that fails on the current tree. + +1. **Regression test for #88** — `information_schema.tables` reports + `pg_catalog` / `information_schema` for internal relations and `public` only + for user tables; `WHERE table_schema='public'` returns exactly the user + tables. +2. **Handler defects** — `ORDER BY` sorts, `count(*)` returns a count, + `WHERE table_type='BASE TABLE'` filters, `DISTINCT` deduplicates. All four + fail today. +3. **Type fidelity** — the eight-column table above, asserting the "correct" + column of that table. +4. **Recovered columns** — `VARCHAR(50)` reports + `character_maximum_length = 50`, `NUMERIC(10,2)` reports precision 10 / + scale 2, `DEFAULT true` reports a non-`NULL` `column_default`, and an + `INTEGER PRIMARY KEY` reports `is_nullable = 'NO'`. These guard the four + columns the handler populates and the current view does not. +5. **Exact-name matching** — a user table named `pg_myreport` reports + `table_schema='public'` in `information_schema.tables`. The same test asserts + it is still misfiled in `pg_class` under v28's heuristic, documenting the + known divergence until the follow-up lands; that assertion flips there. +6. **No regressions** — `tests/information_schema_test.rs`, + `tests/information_schema_comprehensive_test.rs`, + `tests/orm_constraint_discovery_test.rs`, `tests/permission_functions_test.rs` + and the SQLAlchemy suite stay green. +7. **Migration `down`** — v29 supplies one restoring the v14 views verbatim, per + repo convention. It cannot be tested: no rollback path exists anywhere in + `src/migration/`, and `Migration::down` is never executed. The convention is + worth following so a future rollback runner finds it populated, but the plan + should not claim a round-trip test it cannot run. +8. **Registry drift** — the internal-relation list matches exactly what a + migrated-to-head database contains, so a future migration that adds a catalog + relation without updating the list fails here rather than silently leaking it. + +## Non-goals + +- **`pg_class` namespace assignment.** v28's `LIKE 'pg\_%'` heuristic stays, so a + user table named `pg_myreport` remains misfiled in `pg_class` and the eight + `idx_*` indexes keep leaking into `\di`. Split out to keep this change off the + path of `\dt`, `\di`, and every ORM introspection query one week after #87 + landed there. The registry and UDF this design introduces make the follow-up a + one-expression swap. Tracked as + [#102](https://github.com/erans/pgsqlite/issues/102). +- **`pg_attribute.atttypid`.** Still derived from the SQLite declared type, so + `\d` and ORM column reflection keep reporting `text` for `NUMERIC`, `UUID` and + `JSONB`. `information_schema_columns` no longer reads `pg_attribute`, so #88 is + fully fixed without touching it — and the same blast-radius argument that split + `pg_class` out applies here. Folded into + [#102](https://github.com/erans/pgsqlite/issues/102). +- **Columns of views.** `information_schema.columns` reports nothing for a view, + since the rebuilt view keeps the handler's `type = 'table'` restriction — true + today on both paths. Follow-up issue. +- **ENUM columns.** `data_type` reports `text` rather than `USER-DEFINED` with + `udt_name` set to the enum type. Unchanged from today; a scalar UDF cannot + reach `EnumMetadata`, which needs the connection. +- **Stripping the `information_schema_` prefix** from `table_name`, so + `information_schema.tables` reports `information_schema_tables` rather than + PostgreSQL's `tables`. Cosmetic; no client filters on it. +- **The remaining `information_schema` Rust handlers** — + `key_column_usage`, `table_constraints`, `referential_constraints`, `routines`, + `views`, `schemata` keep their handlers and very likely share these defects. + Out of scope; worth an umbrella issue once this lands and the pattern is + proven twice. + +## Follow-ups + +1. **`pg_class` exact-name namespacing + `pg_attribute.atttypid`** — + [#102](https://github.com/erans/pgsqlite/issues/102). Swap v28's prefix `CASE` + for `__pgsqlite_relnamespace(name)`, and resolve `atttypid` from + `__pgsqlite_schema`. Fixes the `pg_myreport` misfiling, the eight leaking + `idx_*` indexes, and `\d` type reporting; removes the divergence test 5 + documents. +2. **View columns in `information_schema.columns` and `pg_attribute`** — extend + beyond `type = 'table'`. To be filed. +3. **Umbrella: remaining `information_schema` handlers** — audit the other six + for the same `ORDER BY` / aggregate / predicate defects. To be filed once this + lands. + diff --git a/src/catalog/column_type_info.rs b/src/catalog/column_type_info.rs new file mode 100644 index 00000000..787950f5 --- /dev/null +++ b/src/catalog/column_type_info.rs @@ -0,0 +1,252 @@ +//! PostgreSQL type string to `information_schema` column metadata. +//! +//! Moved out of `CatalogInterceptor::map_sqlite_type_to_pg_column_info` when the +//! `information_schema.columns` handler was deleted, so the views could keep +//! reporting `character_maximum_length`, `numeric_precision` and `numeric_scale`. +//! +//! Input is the declared PostgreSQL type from `__pgsqlite_schema.pg_type` +//! (`VARCHAR(50)`, `NUMERIC(10,2)`, `TIMESTAMPTZ`), falling back to the SQLite +//! declared type for databases created outside pgsqlite. Deriving from the +//! string rather than an OID is deliberate — an OID cannot carry the modifier. + +/// The four `information_schema.columns` fields that depend on a column's type. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PgColumnInfo { + pub data_type: String, + pub character_maximum_length: Option, + pub numeric_precision: Option, + pub numeric_scale: Option, +} + +impl PgColumnInfo { + fn plain(data_type: &str) -> Self { + Self { + data_type: data_type.to_string(), + character_maximum_length: None, + numeric_precision: None, + numeric_scale: None, + } + } + + fn integral(data_type: &str, precision: i32) -> Self { + Self { + data_type: data_type.to_string(), + character_maximum_length: None, + numeric_precision: Some(precision), + numeric_scale: Some(0), + } + } +} + +/// Split `NUMERIC(10,2)` into `("NUMERIC", ["10", "2"])`. +fn split_modifier(upper: &str) -> (&str, Vec<&str>) { + match (upper.find('('), upper.rfind(')')) { + (Some(open), Some(close)) if close > open => { + let base = upper[..open].trim(); + let params = upper[open + 1..close] + .split(',') + .map(|p| p.trim()) + .collect(); + (base, params) + } + _ => (upper.trim(), Vec::new()), + } +} + +pub fn pg_column_info(pg_type: &str) -> PgColumnInfo { + let upper = pg_type.trim().to_uppercase(); + + // Array types report `ARRAY`, with the element type in udt_name in real + // PostgreSQL. Checked before modifiers so `NUMERIC(10,2)[]` is an array. + if upper.ends_with("[]") { + return PgColumnInfo::plain("ARRAY"); + } + + let (base, params) = split_modifier(&upper); + let param = |i: usize| params.get(i).and_then(|p| p.parse::().ok()); + + match base { + "VARCHAR" | "CHARACTER VARYING" => PgColumnInfo { + data_type: "character varying".to_string(), + character_maximum_length: param(0), + numeric_precision: None, + numeric_scale: None, + }, + "CHAR" | "CHARACTER" | "BPCHAR" => PgColumnInfo { + data_type: "character".to_string(), + character_maximum_length: param(0), + numeric_precision: None, + numeric_scale: None, + }, + "NUMERIC" | "DECIMAL" => PgColumnInfo { + data_type: "numeric".to_string(), + character_maximum_length: None, + numeric_precision: param(0), + // A precision with no scale means scale 0; no precision means + // unconstrained, where PostgreSQL reports NULL for both. + numeric_scale: param(1).or(param(0).map(|_| 0)), + }, + + "SMALLINT" | "INT2" | "SMALLSERIAL" => PgColumnInfo::integral("smallint", 16), + "INTEGER" | "INT" | "INT4" | "SERIAL" => PgColumnInfo::integral("integer", 32), + "BIGINT" | "INT8" | "BIGSERIAL" => PgColumnInfo::integral("bigint", 64), + + "REAL" | "FLOAT4" => PgColumnInfo { + data_type: "real".to_string(), + character_maximum_length: None, + numeric_precision: Some(24), + numeric_scale: None, + }, + "DOUBLE PRECISION" | "FLOAT8" | "FLOAT" | "DOUBLE" => PgColumnInfo { + data_type: "double precision".to_string(), + character_maximum_length: None, + numeric_precision: Some(53), + numeric_scale: None, + }, + + "TEXT" => PgColumnInfo::plain("text"), + "BYTEA" | "BLOB" => PgColumnInfo::plain("bytea"), + "BOOLEAN" | "BOOL" => PgColumnInfo::plain("boolean"), + "UUID" => PgColumnInfo::plain("uuid"), + "JSON" => PgColumnInfo::plain("json"), + "JSONB" => PgColumnInfo::plain("jsonb"), + "MONEY" => PgColumnInfo::plain("money"), + "INTERVAL" => PgColumnInfo::plain("interval"), + + "DATE" => PgColumnInfo::plain("date"), + "TIME" | "TIME WITHOUT TIME ZONE" => PgColumnInfo::plain("time without time zone"), + "TIMETZ" | "TIME WITH TIME ZONE" => PgColumnInfo::plain("time with time zone"), + "TIMESTAMP" | "DATETIME" | "TIMESTAMP WITHOUT TIME ZONE" => { + PgColumnInfo::plain("timestamp without time zone") + } + "TIMESTAMPTZ" | "TIMESTAMP WITH TIME ZONE" => { + PgColumnInfo::plain("timestamp with time zone") + } + + // Unknown types, including ENUMs, report text. Matches the deleted + // handler's final branch and the view's previous `ELSE 'text'`. + _ => PgColumnInfo::plain("text"), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn dt(pg_type: &str) -> String { + pg_column_info(pg_type).data_type + } + + /// Types the deleted handler already got right. These must not regress. + #[test] + fn preserves_correct_handler_behavior() { + assert_eq!(dt("INTEGER"), "integer"); + assert_eq!(dt("TEXT"), "text"); + assert_eq!(dt("BOOLEAN"), "boolean"); + assert_eq!(dt("UUID"), "uuid"); + assert_eq!(dt("JSONB"), "jsonb"); + assert_eq!(dt("NUMERIC(10,2)"), "numeric"); + assert_eq!(dt("VARCHAR(50)"), "character varying"); + assert_eq!(dt("TIMESTAMP"), "timestamp without time zone"); + assert_eq!(dt("DATE"), "date"); + assert_eq!(dt("BLOB"), "bytea"); + } + + /// Types the deleted handler got wrong. Measured in the spec. + #[test] + fn fixes_types_the_handler_reported_as_text() { + assert_eq!(dt("SERIAL"), "integer"); + assert_eq!(dt("BIGSERIAL"), "bigint"); + assert_eq!(dt("TIMESTAMPTZ"), "timestamp with time zone"); + assert_eq!(dt("TIMESTAMP WITH TIME ZONE"), "timestamp with time zone"); + assert_eq!(dt("TIMETZ"), "time with time zone"); + assert_eq!(dt("TIME WITH TIME ZONE"), "time with time zone"); + assert_eq!(dt("TIME"), "time without time zone"); + } + + #[test] + fn array_types_report_array() { + assert_eq!(dt("TEXT[]"), "ARRAY"); + assert_eq!(dt("INTEGER[]"), "ARRAY"); + assert_eq!(dt("NUMERIC(10,2)[]"), "ARRAY"); + } + + #[test] + fn character_maximum_length_comes_from_the_modifier() { + assert_eq!(pg_column_info("VARCHAR(50)").character_maximum_length, Some(50)); + assert_eq!(pg_column_info("CHAR(8)").character_maximum_length, Some(8)); + assert_eq!(pg_column_info("VARCHAR").character_maximum_length, None); + assert_eq!(pg_column_info("TEXT").character_maximum_length, None); + } + + #[test] + fn numeric_precision_and_scale_come_from_the_modifier() { + let info = pg_column_info("NUMERIC(10,2)"); + assert_eq!(info.numeric_precision, Some(10)); + assert_eq!(info.numeric_scale, Some(2)); + + let info = pg_column_info("DECIMAL(38)"); + assert_eq!(info.numeric_precision, Some(38)); + assert_eq!(info.numeric_scale, Some(0)); + + let info = pg_column_info("NUMERIC"); + assert_eq!(info.numeric_precision, None); + assert_eq!(info.numeric_scale, None); + } + + /// PostgreSQL reports precision and scale for integer types too. + #[test] + fn integer_types_report_binary_precision() { + let info = pg_column_info("INTEGER"); + assert_eq!(info.numeric_precision, Some(32)); + assert_eq!(info.numeric_scale, Some(0)); + + let info = pg_column_info("BIGINT"); + assert_eq!(info.numeric_precision, Some(64)); + assert_eq!(info.numeric_scale, Some(0)); + } + + /// Text is the safe fallback: matches both the deleted handler's final + /// branch and the view's `ELSE 'text'`. ENUM types land here (#88 non-goal). + #[test] + fn unknown_types_fall_back_to_text() { + assert_eq!(dt("my_enum_type"), "text"); + assert_eq!(dt(""), "text"); + } + + #[test] + fn case_is_insensitive() { + assert_eq!(dt("varchar(50)"), "character varying"); + assert_eq!(dt("TimestampTZ"), "timestamp with time zone"); + } + + #[test] + fn udfs_are_callable_from_sql() { + let conn = rusqlite::Connection::open_in_memory().unwrap(); + crate::functions::register_all_functions(&conn).unwrap(); + + let (dt, len): (String, Option) = conn + .query_row( + "SELECT __pgsqlite_pg_data_type('VARCHAR(50)'), __pgsqlite_char_max_length('VARCHAR(50)')", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(dt, "character varying"); + assert_eq!(len, Some(50)); + + let (p, s): (Option, Option) = conn + .query_row( + "SELECT __pgsqlite_numeric_precision('NUMERIC(10,2)'), __pgsqlite_numeric_scale('NUMERIC(10,2)')", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!((p, s), (Some(10), Some(2))); + + let dt: Option = conn + .query_row("SELECT __pgsqlite_pg_data_type(NULL)", [], |r| r.get(0)) + .unwrap(); + assert_eq!(dt, None); + } +} diff --git a/src/catalog/internal_relations.rs b/src/catalog/internal_relations.rs new file mode 100644 index 00000000..3e87bd96 --- /dev/null +++ b/src/catalog/internal_relations.rs @@ -0,0 +1,142 @@ +//! Exact names of the relations pgsqlite's own migrations create. +//! +//! Prefix matching is not sufficient in either direction: it claims user tables +//! that merely start with `pg_`, and it misses the `idx_*` indexes migrations +//! create on `__pgsqlite_*` tables. Both failure modes are user-visible, so this +//! list is exact and is guarded against drift by +//! `tests/information_schema_namespace_test.rs::internal_relation_list_matches_migrated_database`. +//! +//! Adding a catalog relation to `src/migration/registry.rs` means adding it here. + +/// Namespace OIDs, fixed by the `pg_namespace` view in migration v28. +pub const PG_CATALOG_NAMESPACE: i32 = 11; +pub const PUBLIC_NAMESPACE: i32 = 2200; +pub const INFORMATION_SCHEMA_NAMESPACE: i32 = 13000; + +/// Relations pgsqlite creates that PostgreSQL keeps in `pg_catalog`. +pub const INTERNAL_PG_CATALOG_RELATIONS: &[&str] = &[ + // Real tables + "pg_attrdef", + "pg_constraint", + "pg_depend", + "pg_index", + // Views + "pg_am", + "pg_attribute", + "pg_class", + "pg_database", + "pg_description", + "pg_enum", + "pg_foreign_data_wrapper", + "pg_namespace", + "pg_proc", + "pg_roles", + "pg_stat_activity", + "pg_stat_all_indexes", + "pg_stat_all_tables", + "pg_stat_database", + "pg_stat_user_indexes", + "pg_stat_user_tables", + "pg_type", + "pg_user", + // Indexes on __pgsqlite_* tables. Not named __pgsqlite_*, so no prefix + // filter catches them; they leak into \di today (#102). + "idx_array_types_table", + "idx_comments_lookup", + "idx_datetime_cache_table", + "idx_enum_values_label", + "idx_enum_values_type", + "idx_fts_metadata_table", + "idx_numeric_constraints_table", + "idx_string_constraints_table", +]; + +/// Relations pgsqlite creates that PostgreSQL keeps in `information_schema`. +pub const INTERNAL_INFORMATION_SCHEMA_RELATIONS: &[&str] = &[ + "information_schema_columns", + "information_schema_key_column_usage", + "information_schema_referential_constraints", + "information_schema_schemata", + "information_schema_table_constraints", + "information_schema_tables", +]; + +/// Namespace OID for a relation name. Anything unlisted is a user relation. +/// +/// Unlisted defaults to `public` deliberately: the failure mode is showing an +/// internal relation, never hiding a user's table. +pub fn relnamespace(name: &str) -> i32 { + if INTERNAL_PG_CATALOG_RELATIONS.contains(&name) { + PG_CATALOG_NAMESPACE + } else if INTERNAL_INFORMATION_SCHEMA_RELATIONS.contains(&name) { + INFORMATION_SCHEMA_NAMESPACE + } else { + PUBLIC_NAMESPACE + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn internal_pg_relations_map_to_pg_catalog() { + assert_eq!(relnamespace("pg_class"), PG_CATALOG_NAMESPACE); + assert_eq!(relnamespace("pg_constraint"), PG_CATALOG_NAMESPACE); + assert_eq!(relnamespace("idx_enum_values_label"), PG_CATALOG_NAMESPACE); + } + + #[test] + fn internal_information_schema_relations_map_to_information_schema() { + assert_eq!(relnamespace("information_schema_tables"), INFORMATION_SCHEMA_NAMESPACE); + assert_eq!(relnamespace("information_schema_columns"), INFORMATION_SCHEMA_NAMESPACE); + } + + #[test] + fn user_tables_map_to_public() { + assert_eq!(relnamespace("customers"), PUBLIC_NAMESPACE); + } + + /// The reason this is an exact-name list rather than a `LIKE 'pg\_%'` test: + /// a user table whose name merely starts with `pg_` is not ours. See #102. + #[test] + fn user_tables_named_like_catalog_relations_map_to_public() { + assert_eq!(relnamespace("pg_myreport"), PUBLIC_NAMESPACE); + assert_eq!(relnamespace("information_schema_export"), PUBLIC_NAMESPACE); + assert_eq!(relnamespace("idx_customers_email"), PUBLIC_NAMESPACE); + } + + #[test] + fn lists_have_no_duplicates_and_do_not_overlap() { + let mut all: Vec<&str> = INTERNAL_PG_CATALOG_RELATIONS + .iter() + .chain(INTERNAL_INFORMATION_SCHEMA_RELATIONS.iter()) + .copied() + .collect(); + let total = all.len(); + all.sort_unstable(); + all.dedup(); + assert_eq!(all.len(), total, "duplicate entry in the internal relation lists"); + } + + #[test] + fn udf_is_callable_from_sql() { + let conn = rusqlite::Connection::open_in_memory().unwrap(); + crate::functions::register_all_functions(&conn).unwrap(); + + let ns: i32 = conn + .query_row("SELECT __pgsqlite_relnamespace('pg_class')", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, PG_CATALOG_NAMESPACE); + + let ns: i32 = conn + .query_row("SELECT __pgsqlite_relnamespace('pg_myreport')", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, PUBLIC_NAMESPACE); + + let ns: Option = conn + .query_row("SELECT __pgsqlite_relnamespace(NULL)", [], |r| r.get(0)) + .unwrap(); + assert_eq!(ns, None); + } +} diff --git a/src/catalog/mod.rs b/src/catalog/mod.rs index 23e312cc..38156acd 100644 --- a/src/catalog/mod.rs +++ b/src/catalog/mod.rs @@ -15,5 +15,7 @@ pub mod pg_settings; pub mod system_functions; pub mod where_evaluator; pub mod constraint_populator; +pub mod internal_relations; +pub mod column_type_info; pub use query_interceptor::CatalogInterceptor; \ No newline at end of file diff --git a/src/catalog/query_interceptor.rs b/src/catalog/query_interceptor.rs index c095ed6f..54a829ca 100644 --- a/src/catalog/query_interceptor.rs +++ b/src/catalog/query_interceptor.rs @@ -725,20 +725,6 @@ impl CatalogInterceptor { return Some(Ok(Self::handle_information_schema_schemata_query(select, &db).await)); } - // Handle information_schema.tables queries - if table_name.contains("information_schema.tables") { - return Some(Ok(Self::handle_information_schema_tables_query(select, &db).await)); - } - - // Handle information_schema.columns queries - if table_name.contains("information_schema.columns") { - if let Some(ref session_state) = session { - return Some(Self::handle_information_schema_columns_query_with_session(select, &db, &session_state.id).await); - } else { - return None; - } - } - // Handle information_schema.key_column_usage queries if table_name.contains("information_schema.key_column_usage") { if let Some(ref session_state) = session { @@ -1841,411 +1827,6 @@ impl CatalogInterceptor { } } - async fn handle_information_schema_tables_query(select: &Select, db: &DbHandler) -> DbResponse { - debug!("Handling information_schema.tables query"); - - // Get list of tables and views from SQLite - let tables_response = match db.query("SELECT name, type FROM sqlite_master WHERE type IN ('table', 'view') AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%'").await { - Ok(response) => response, - Err(_) => return DbResponse { - columns: vec!["table_name".to_string()], - rows: vec![], - rows_affected: 0, - }, - }; - - // Define information_schema.tables columns (enhanced with all PostgreSQL standard columns) - let all_columns = vec![ - "table_catalog".to_string(), - "table_schema".to_string(), - "table_name".to_string(), - "table_type".to_string(), - "self_referencing_column_name".to_string(), - "reference_generation".to_string(), - "user_defined_type_catalog".to_string(), - "user_defined_type_schema".to_string(), - "user_defined_type_name".to_string(), - "is_insertable_into".to_string(), - "is_typed".to_string(), - "commit_action".to_string(), - ]; - - // Extract selected columns - let (selected_columns, column_indices) = Self::extract_selected_columns(select, &all_columns); - - // Check for WHERE clause filtering - let table_filters = if let Some(ref where_clause) = select.selection { - Self::extract_table_name_filters(where_clause) - } else { - Vec::new() - }; - - // Build rows - let mut rows = Vec::new(); - for table_row in &tables_response.rows { - if table_row.len() >= 2 - && let (Some(Some(table_name_bytes)), Some(Some(table_type_bytes))) = - (table_row.first(), table_row.get(1)) { - let table_name = String::from_utf8_lossy(table_name_bytes).to_string(); - let sqlite_type = String::from_utf8_lossy(table_type_bytes).to_string(); - - // Apply WHERE clause filtering if present - if !table_filters.is_empty() && !table_filters.contains(&table_name) { - continue; - } - - // Map SQLite type to PostgreSQL table_type - let table_type = match sqlite_type.as_str() { - "table" => "BASE TABLE", - "view" => "VIEW", - _ => "BASE TABLE", // Default fallback - }; - - // Determine if table is insertable (views are not) - let is_insertable = if table_type == "VIEW" { "NO" } else { "YES" }; - - // Create full row with all columns - let full_row: Vec>> = vec![ - Some("main".to_string().into_bytes()), // table_catalog - Some("public".to_string().into_bytes()), // table_schema - Some(table_name.into_bytes()), // table_name - Some(table_type.to_string().into_bytes()), // table_type - None, // self_referencing_column_name - None, // reference_generation - None, // user_defined_type_catalog - None, // user_defined_type_schema - None, // user_defined_type_name - Some(is_insertable.to_string().into_bytes()), // is_insertable_into - Some("NO".to_string().into_bytes()), // is_typed - None, // commit_action - ]; - - // Project only the requested columns - let projected_row: Vec>> = column_indices.iter() - .map(|&idx| full_row[idx].clone()) - .collect(); - - rows.push(projected_row); - } - } - - let rows_count = rows.len(); - DbResponse { - columns: selected_columns, - rows, - rows_affected: rows_count, - } - } - - pub async fn handle_information_schema_columns_query_with_session(select: &Select, db: &DbHandler, session_id: &Uuid) -> Result { - debug!("Handling information_schema.columns query"); - - // Define information_schema.columns columns (PostgreSQL standard) - let all_columns = vec![ - "table_catalog".to_string(), - "table_schema".to_string(), - "table_name".to_string(), - "column_name".to_string(), - "ordinal_position".to_string(), - "column_default".to_string(), - "is_nullable".to_string(), - "data_type".to_string(), - "character_maximum_length".to_string(), - "character_octet_length".to_string(), - "numeric_precision".to_string(), - "numeric_precision_radix".to_string(), - "numeric_scale".to_string(), - "datetime_precision".to_string(), - "interval_type".to_string(), - "interval_precision".to_string(), - "character_set_catalog".to_string(), - "character_set_schema".to_string(), - "character_set_name".to_string(), - "collation_catalog".to_string(), - "collation_schema".to_string(), - "collation_name".to_string(), - "domain_catalog".to_string(), - "domain_schema".to_string(), - "domain_name".to_string(), - "udt_catalog".to_string(), - "udt_schema".to_string(), - "udt_name".to_string(), - "scope_catalog".to_string(), - "scope_schema".to_string(), - "scope_name".to_string(), - "maximum_cardinality".to_string(), - "dtd_identifier".to_string(), - "is_self_referencing".to_string(), - "is_identity".to_string(), - "identity_generation".to_string(), - "identity_start".to_string(), - "identity_increment".to_string(), - "identity_maximum".to_string(), - "identity_minimum".to_string(), - "identity_cycle".to_string(), - "is_generated".to_string(), - "generation_expression".to_string(), - "is_updatable".to_string(), - ]; - - // Extract selected columns - let (selected_columns, column_indices) = Self::extract_selected_columns(select, &all_columns); - - // Check for WHERE clause filtering - let table_filter = if let Some(ref where_clause) = select.selection { - Self::extract_table_name_filter(where_clause) - } else { - None - }; - - // Get list of tables from SQLite - let tables_query = if let Some(table_name) = &table_filter { - format!("SELECT name FROM sqlite_master WHERE type='table' AND name = '{}' AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%'", table_name) - } else { - "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%'".to_string() - }; - - let tables_response = match db.connection_manager().execute_with_session(session_id, |conn| { - let mut stmt = conn.prepare(&tables_query)?; - let mut rows = Vec::new(); - let mut query_rows = stmt.query([])?; - while let Some(row) = query_rows.next()? { - let name: String = row.get(0)?; - rows.push(vec![Some(name.into_bytes())]); - } - Ok(DbResponse { - columns: vec!["name".to_string()], - rows, - rows_affected: 0, - }) - }) { - Ok(response) => response, - Err(_) => return Ok(DbResponse { - columns: selected_columns, - rows: vec![], - rows_affected: 0, - }), - }; - - let mut rows = Vec::new(); - - // Process each table - for table_row in &tables_response.rows { - if let Some(Some(table_name_bytes)) = table_row.first() { - let table_name = String::from_utf8_lossy(table_name_bytes).to_string(); - - // Get column information using PRAGMA table_info - let pragma_query = format!("PRAGMA table_info({})", table_name); - let table_info_response = match db.connection_manager().execute_with_session(session_id, |conn| { - let mut stmt = conn.prepare(&pragma_query)?; - let mut rows = Vec::new(); - let mut query_rows = stmt.query([])?; - while let Some(row) = query_rows.next()? { - let cid: i32 = row.get(0)?; - let name: String = row.get(1)?; - let type_name: String = row.get(2)?; - let not_null: i32 = row.get(3)?; - let default_value: Option = row.get(4)?; - let pk: i32 = row.get(5)?; - - let row_data = vec![ - Some(cid.to_string().into_bytes()), - Some(name.into_bytes()), - Some(type_name.into_bytes()), - Some(not_null.to_string().into_bytes()), - default_value.map(|v| v.into_bytes()), - Some(pk.to_string().into_bytes()), - ]; - rows.push(row_data); - } - Ok(DbResponse { - columns: vec!["cid".to_string(), "name".to_string(), "type".to_string(), "notnull".to_string(), "dflt_value".to_string(), "pk".to_string()], - rows, - rows_affected: 0, - }) - }) { - Ok(response) => response, - Err(_) => continue, - }; - - // Process each column - for (ordinal, column_row) in table_info_response.rows.iter().enumerate() { - if column_row.len() >= 6 - && let (Some(Some(name_bytes)), Some(Some(type_bytes)), Some(Some(notnull_bytes)), default_opt, Some(Some(pk_bytes))) = - (column_row.get(1), column_row.get(2), column_row.get(3), column_row.get(4), column_row.get(5)) { - - let column_name = String::from_utf8_lossy(name_bytes).to_string(); - let sqlite_type = String::from_utf8_lossy(type_bytes).to_string(); - let not_null = String::from_utf8_lossy(notnull_bytes) == "1"; - let default_value = match default_opt { - Some(Some(default_bytes)) => String::from_utf8_lossy(default_bytes), - _ => "".into(), - }; - let is_primary_key = String::from_utf8_lossy(pk_bytes) == "1"; - - // First try to get type from __pgsqlite_schema, then fall back to SQLite type - let pg_type = match db.connection_manager().execute_with_session(session_id, |conn| { - // Check if __pgsqlite_schema exists first - let table_exists: i32 = conn.query_row( - "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='__pgsqlite_schema'", - [], - |row| row.get(0) - ).unwrap_or(0); - - if table_exists > 0 { - let query = "SELECT pg_type FROM __pgsqlite_schema WHERE table_name = ? AND column_name = ?"; - let mut stmt = conn.prepare(query)?; - let result = stmt.query_row([&table_name, &column_name], |row| { - row.get::<_, String>(0) - }); - Ok(result) - } else { - Err(rusqlite::Error::InvalidPath("__pgsqlite_schema not found".into())) - } - }) { - Ok(Ok(stored_type)) => stored_type, - _ => sqlite_type.clone(), - }; - - // Map type to PostgreSQL type - let (pg_data_type, char_max_length, numeric_precision, numeric_scale) = - Self::map_sqlite_type_to_pg_column_info(&pg_type); - - // Determine nullability - let is_nullable = if not_null || is_primary_key { "NO" } else { "YES" }; - - // Handle default value - let column_default = if default_value.is_empty() || default_value == "NULL" { - None - } else { - Some(default_value.to_string().into_bytes()) - }; - - let full_row: Vec>> = vec![ - Some("main".to_string().into_bytes()), // table_catalog - Some("public".to_string().into_bytes()), // table_schema - Some(table_name.clone().into_bytes()), // table_name - Some(column_name.clone().into_bytes()), // column_name - Some((ordinal + 1).to_string().into_bytes()), // ordinal_position (1-based) - column_default, // column_default - Some(is_nullable.to_string().into_bytes()), // is_nullable - Some(pg_data_type.clone().into_bytes()), // data_type - char_max_length.map(|v| v.to_string().into_bytes()), // character_maximum_length - char_max_length.map(|v| v.to_string().into_bytes()), // character_octet_length - numeric_precision.map(|v| v.to_string().into_bytes()), // numeric_precision - numeric_precision.map(|_| "10".to_string().into_bytes()), // numeric_precision_radix - numeric_scale.map(|v| v.to_string().into_bytes()), // numeric_scale - None, // datetime_precision - None, // interval_type - None, // interval_precision - None, // character_set_catalog - None, // character_set_schema - None, // character_set_name - None, // collation_catalog - None, // collation_schema - None, // collation_name - None, // domain_catalog - None, // domain_schema - None, // domain_name - Some("main".to_string().into_bytes()), // udt_catalog - Some("pg_catalog".to_string().into_bytes()), // udt_schema - Some(pg_data_type.clone().into_bytes()), // udt_name - None, // scope_catalog - None, // scope_schema - None, // scope_name - None, // maximum_cardinality - Some((ordinal + 1).to_string().into_bytes()), // dtd_identifier - Some("NO".to_string().into_bytes()), // is_self_referencing - Some("NO".to_string().into_bytes()), // is_identity - None, // identity_generation - None, // identity_start - None, // identity_increment - None, // identity_maximum - None, // identity_minimum - Some("NO".to_string().into_bytes()), // identity_cycle - Some("NEVER".to_string().into_bytes()), // is_generated - None, // generation_expression - Some("YES".to_string().into_bytes()), // is_updatable - ]; - - // Project only the requested columns - let projected_row: Vec>> = column_indices.iter() - .map(|&idx| full_row[idx].clone()) - .collect(); - - rows.push(projected_row); - } - } - } - } - - let rows_affected = rows.len(); - Ok(DbResponse { - columns: selected_columns, - rows, - rows_affected, - }) - } - - fn map_sqlite_type_to_pg_column_info(sqlite_type: &str) -> (String, Option, Option, Option) { - let sqlite_type_upper = sqlite_type.to_uppercase(); - - // Handle parametric types like VARCHAR(255), DECIMAL(10,2) - if let Some(paren_pos) = sqlite_type_upper.find('(') { - let base_type = &sqlite_type_upper[..paren_pos]; - let params_str = &sqlite_type_upper[paren_pos+1..]; - if let Some(close_paren) = params_str.find(')') { - let params_str = ¶ms_str[..close_paren]; - let params: Vec<&str> = params_str.split(',').map(|s| s.trim()).collect(); - - match base_type { - "VARCHAR" | "CHAR" | "CHARACTER VARYING" => { - let length = params.first().and_then(|p| p.parse().ok()).unwrap_or(255); - return ("character varying".to_string(), Some(length), None, None); - }, - "DECIMAL" | "NUMERIC" => { - let precision = params.first().and_then(|p| p.parse().ok()).unwrap_or(10); - let scale = params.get(1).and_then(|p| p.parse().ok()).unwrap_or(0); - return ("numeric".to_string(), None, Some(precision), Some(scale)); - }, - _ => {} - } - } - } - - // Handle base types - match sqlite_type_upper.as_str() { - "INTEGER" | "INT" => ("integer".to_string(), None, Some(32), Some(0)), - "BIGINT" => ("bigint".to_string(), None, Some(64), Some(0)), - "SMALLINT" => ("smallint".to_string(), None, Some(16), Some(0)), - "REAL" | "FLOAT" => ("real".to_string(), None, Some(24), None), - "DOUBLE" | "DOUBLE PRECISION" => ("double precision".to_string(), None, Some(53), None), - "TEXT" => ("text".to_string(), None, None, None), - "BLOB" => ("bytea".to_string(), None, None, None), - "BOOLEAN" | "BOOL" => ("boolean".to_string(), None, None, None), - "DATE" => ("date".to_string(), None, None, None), - "TIME" => ("time without time zone".to_string(), None, None, None), - "TIMESTAMP" | "DATETIME" => ("timestamp without time zone".to_string(), None, None, None), - "UUID" => ("uuid".to_string(), None, None, None), - "JSON" => ("json".to_string(), None, None, None), - "JSONB" => ("jsonb".to_string(), None, None, None), - "VARCHAR" | "CHARACTER VARYING" => ("character varying".to_string(), None, None, None), - "CHAR" | "CHARACTER" => ("character".to_string(), None, None, None), - _ => { - // Default fallback for unknown types - if sqlite_type_upper.contains("CHAR") || sqlite_type_upper.contains("TEXT") { - ("text".to_string(), None, None, None) - } else if sqlite_type_upper.contains("INT") { - ("integer".to_string(), None, Some(32), Some(0)) - } else if sqlite_type_upper.contains("REAL") || sqlite_type_upper.contains("FLOAT") { - ("real".to_string(), None, Some(24), None) - } else { - ("text".to_string(), None, None, None) - } - } - } - } - pub async fn handle_information_schema_key_column_usage_query(select: &Select, db: &DbHandler, session_id: &Uuid) -> Result { debug!("Handling information_schema.key_column_usage query"); @@ -2640,69 +2221,6 @@ impl CatalogInterceptor { None } - fn extract_table_name_filters(where_clause: &Expr) -> Vec { - match where_clause { - Expr::BinaryOp { left, op, right } => { - // Handle "table_name = 'value'" - if let (Expr::Identifier(ident), sqlparser::ast::BinaryOperator::Eq, Expr::Value(value_with_span)) = - (left.as_ref(), op, right.as_ref()) - && ident.value.to_lowercase() == "table_name" - && let sqlparser::ast::Value::SingleQuotedString(value) = &value_with_span.value { - return vec![value.clone()]; - } - // Handle "'value' = table_name" (reversed) - if let (Expr::Value(value_with_span), sqlparser::ast::BinaryOperator::Eq, Expr::Identifier(ident)) = - (left.as_ref(), op, right.as_ref()) - && ident.value.to_lowercase() == "table_name" - && let sqlparser::ast::Value::SingleQuotedString(value) = &value_with_span.value { - return vec![value.clone()]; - } - // Handle compound identifiers like "information_schema.tables.table_name = 'value'" - if let (Expr::CompoundIdentifier(parts), sqlparser::ast::BinaryOperator::Eq, Expr::Value(value_with_span)) = - (left.as_ref(), op, right.as_ref()) - && let Some(last_part) = parts.last() - && last_part.value.to_lowercase() == "table_name" - && let sqlparser::ast::Value::SingleQuotedString(value) = &value_with_span.value { - return vec![value.clone()]; - } - } - Expr::InList { expr, list, negated } => { - // Handle "table_name IN ('value1', 'value2')" - if !negated { - if let Expr::Identifier(ident) = expr.as_ref() - && ident.value.to_lowercase() == "table_name" { - let mut values = Vec::new(); - for item in list { - if let Expr::Value(value_with_span) = item - && let sqlparser::ast::Value::SingleQuotedString(value) = &value_with_span.value { - values.push(value.clone()); - } - } - return values; - } - // Handle compound identifiers in IN clause - if let Expr::CompoundIdentifier(parts) = expr.as_ref() - && let Some(last_part) = parts.last() - && last_part.value.to_lowercase() == "table_name" { - let mut values = Vec::new(); - for item in list { - if let Expr::Value(value_with_span) = item - && let sqlparser::ast::Value::SingleQuotedString(value) = &value_with_span.value { - values.push(value.clone()); - } - } - return values; - } - } - } - Expr::Nested(inner) => { - return Self::extract_table_name_filters(inner); - } - _ => {} - } - Vec::new() - } - pub async fn handle_information_schema_routines_query(select: &Select, _db: &DbHandler) -> Result { debug!("Handling information_schema.routines query"); diff --git a/src/functions/catalog_functions.rs b/src/functions/catalog_functions.rs index 47140407..3d72c102 100644 --- a/src/functions/catalog_functions.rs +++ b/src/functions/catalog_functions.rs @@ -46,6 +46,53 @@ pub fn register_catalog_functions(conn: &Connection) -> Result<()> { }, )?; + // __pgsqlite_relnamespace(name) - namespace OID for a relation. + // Lets catalog views ask "is this relation ours?" by exact name rather than + // by prefix. Returns NULL for NULL input so a view row degrades rather than + // failing the query. + conn.create_scalar_function( + "__pgsqlite_relnamespace", + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let name: Option = ctx.get(0)?; + Ok(name.map(|n| crate::catalog::internal_relations::relnamespace(&n))) + }, + )?; + + // Column type metadata for the information_schema.columns view. Four + // scalar functions rather than one, because a SQLite scalar UDF returns a + // single value and the view needs four fields from the same input. + macro_rules! register_column_info_fn { + ($name:literal, $field:ident) => { + conn.create_scalar_function( + $name, + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let pg_type: Option = ctx.get(0)?; + let value: Option = pg_type + .and_then(|t| crate::catalog::column_type_info::pg_column_info(&t).$field); + Ok(value) + }, + )?; + }; + } + + conn.create_scalar_function( + "__pgsqlite_pg_data_type", + 1, + FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + |ctx| { + let pg_type: Option = ctx.get(0)?; + Ok(pg_type.map(|t| crate::catalog::column_type_info::pg_column_info(&t).data_type)) + }, + )?; + + register_column_info_fn!("__pgsqlite_char_max_length", character_maximum_length); + register_column_info_fn!("__pgsqlite_numeric_precision", numeric_precision); + register_column_info_fn!("__pgsqlite_numeric_scale", numeric_scale); + debug!("Catalog functions registered successfully"); Ok(()) } diff --git a/src/migration/registry.rs b/src/migration/registry.rs index 9f40b424..fe8621ec 100644 --- a/src/migration/registry.rs +++ b/src/migration/registry.rs @@ -35,6 +35,7 @@ lazy_static! { register_v26_enhanced_pg_attribute_support(&mut registry); register_v27_fix_pg_proc_types(&mut registry); register_v28_pg_class_full_columns(&mut registry); + register_v29_information_schema_namespace(&mut registry); registry }; @@ -3030,3 +3031,234 @@ fn register_v28_pg_class_full_columns(registry: &mut BTreeMap) { dependencies: vec![27], }); } + +/// Version 29: Namespace-aware information_schema views (#88) +/// +/// The views are rebuilt rather than patched. `information_schema_tables` +/// derives `table_schema` from `__pgsqlite_relnamespace` rather than hardcoding +/// `'public'`, and reads the UDF directly rather than `pg_class.relnamespace` so +/// it does not inherit v28's `LIKE 'pg\_%'` heuristic (#102). +/// +/// `information_schema_columns` reads sqlite_master, pragma_table_info and +/// __pgsqlite_schema directly rather than layering on pg_attribute, which keeps +/// this migration off a view every ORM reads and is what makes column_default, +/// is_nullable, character_maximum_length and numeric precision/scale reachable. +fn register_v29_information_schema_namespace(registry: &mut BTreeMap) { + registry.insert(29, Migration { + version: 29, + name: "information_schema_namespace", + description: "Namespace-aware information_schema.tables and .columns served from SQLite", + up: MigrationAction::SqlBatch(&[ + r#"DROP VIEW IF EXISTS information_schema_tables"#, + r#"DROP VIEW IF EXISTS information_schema_columns"#, + + // Column order matches PostgreSQL: is_insertable_into is 10th, not + // 5th as in v14. The deleted Rust handler already used this order. + r#" + CREATE VIEW information_schema_tables AS + SELECT + 'main' as table_catalog, + n.nspname as table_schema, + c.relname as table_name, + CASE c.relkind + WHEN 'r' THEN 'BASE TABLE' + WHEN 'v' THEN 'VIEW' + ELSE 'UNKNOWN' + END as table_type, + NULL as self_referencing_column_name, + NULL as reference_generation, + NULL as user_defined_type_catalog, + NULL as user_defined_type_schema, + NULL as user_defined_type_name, + CASE c.relkind WHEN 'r' THEN 'YES' ELSE 'NO' END as is_insertable_into, + 'NO' as is_typed, + NULL as commit_action + FROM pg_class c + JOIN pg_namespace n ON n.oid = __pgsqlite_relnamespace(c.relname) + WHERE c.relkind IN ('r', 'v') + "#, + + // 44 columns in v14's order, which matches PostgreSQL. + // substr(...) <> '__pgsqlite_' rather than LIKE '__pgsqlite_%': + // `_` is a LIKE wildcard, so the LIKE form matches unrelated names. + r#" + CREATE VIEW information_schema_columns AS + SELECT + 'main' as table_catalog, + n.nspname as table_schema, + m.name as table_name, + p.name as column_name, + p.cid + 1 as ordinal_position, + p.dflt_value as column_default, + CASE WHEN p."notnull" = 1 OR p.pk > 0 THEN 'NO' ELSE 'YES' END as is_nullable, + __pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type)) as data_type, + __pgsqlite_char_max_length(COALESCE(s.pg_type, p.type)) as character_maximum_length, + __pgsqlite_char_max_length(COALESCE(s.pg_type, p.type)) as character_octet_length, + __pgsqlite_numeric_precision(COALESCE(s.pg_type, p.type)) as numeric_precision, + CASE __pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type)) + WHEN 'smallint' THEN 2 + WHEN 'integer' THEN 2 + WHEN 'bigint' THEN 2 + WHEN 'real' THEN 2 + WHEN 'double precision' THEN 2 + WHEN 'numeric' THEN 10 + ELSE NULL + END as numeric_precision_radix, + __pgsqlite_numeric_scale(COALESCE(s.pg_type, p.type)) as numeric_scale, + NULL as datetime_precision, + NULL as interval_type, + NULL as interval_precision, + NULL as character_set_catalog, + NULL as character_set_schema, + NULL as character_set_name, + NULL as collation_catalog, + NULL as collation_schema, + NULL as collation_name, + NULL as domain_catalog, + NULL as domain_schema, + NULL as domain_name, + 'main' as udt_catalog, + 'pg_catalog' as udt_schema, + __pgsqlite_pg_data_type(COALESCE(s.pg_type, p.type)) as udt_name, + NULL as scope_catalog, + NULL as scope_schema, + NULL as scope_name, + NULL as maximum_cardinality, + p.cid + 1 as dtd_identifier, + 'NO' as is_self_referencing, + 'NO' as is_identity, + NULL as identity_generation, + NULL as identity_start, + NULL as identity_increment, + NULL as identity_maximum, + NULL as identity_minimum, + 'NO' as identity_cycle, + 'NEVER' as is_generated, + NULL as generation_expression, + 'YES' as is_updatable + FROM sqlite_master m + JOIN pragma_table_info(m.name) p + LEFT JOIN __pgsqlite_schema s + ON s.table_name = m.name AND s.column_name = p.name + JOIN pg_namespace n ON n.oid = __pgsqlite_relnamespace(m.name) + WHERE m.type = 'table' + AND m.name NOT LIKE 'sqlite_%' + AND substr(m.name, 1, 11) <> '__pgsqlite_' + "#, + + r#" + UPDATE __pgsqlite_metadata + SET value = '29', updated_at = strftime('%s', 'now') + WHERE key = 'schema_version'; + "#, + ]), + down: Some(MigrationAction::SqlBatch(&[ + r#"DROP VIEW IF EXISTS information_schema_tables"#, + r#"DROP VIEW IF EXISTS information_schema_columns"#, + + // Restore the v14 views: copied verbatim from + // register_v14_information_schema_views's `up`. + r#" + CREATE VIEW IF NOT EXISTS information_schema_tables AS + SELECT + 'main' as table_catalog, + 'public' as table_schema, + relname as table_name, + CASE relkind + WHEN 'r' THEN 'BASE TABLE' + WHEN 'v' THEN 'VIEW' + ELSE 'UNKNOWN' + END as table_type, + 'YES' as is_insertable_into, + NULL as self_referencing_column_name, + NULL as reference_generation, + NULL as user_defined_type_catalog, + NULL as user_defined_type_schema, + NULL as user_defined_type_name, + 'NO' as is_typed, + 'NO' as commit_action + FROM pg_class + WHERE relkind IN ('r', 'v'); + "#, + + // NOTE TO IMPLEMENTER: this `down` is never executed. No rollback + // path exists in src/migration/ -- `Migration::down` is stored and + // never read. It is populated to match repo convention so a future + // rollback runner finds it, not because it can be tested. + r#" + CREATE VIEW IF NOT EXISTS information_schema_columns AS + SELECT + 'main' as table_catalog, + 'public' as table_schema, + c.relname as table_name, + a.attname as column_name, + a.attnum as ordinal_position, + NULL as column_default, + CASE WHEN a.attnotnull = 't' THEN 'NO' ELSE 'YES' END as is_nullable, + CASE a.atttypid + WHEN 23 THEN 'integer' + WHEN 25 THEN 'text' + WHEN 700 THEN 'real' + WHEN 701 THEN 'double precision' + WHEN 17 THEN 'bytea' + WHEN 1043 THEN 'character varying' + WHEN 1042 THEN 'character' + WHEN 16 THEN 'boolean' + WHEN 1082 THEN 'date' + WHEN 1083 THEN 'time without time zone' + WHEN 1114 THEN 'timestamp without time zone' + WHEN 1184 THEN 'timestamp with time zone' + WHEN 1700 THEN 'numeric' + ELSE 'text' + END as data_type, + NULL as character_maximum_length, + NULL as character_octet_length, + NULL as numeric_precision, + NULL as numeric_precision_radix, + NULL as numeric_scale, + NULL as datetime_precision, + NULL as interval_type, + NULL as interval_precision, + NULL as character_set_catalog, + NULL as character_set_schema, + NULL as character_set_name, + NULL as collation_catalog, + NULL as collation_schema, + NULL as collation_name, + NULL as domain_catalog, + NULL as domain_schema, + NULL as domain_name, + NULL as udt_catalog, + NULL as udt_schema, + NULL as udt_name, + NULL as scope_catalog, + NULL as scope_schema, + NULL as scope_name, + NULL as maximum_cardinality, + NULL as dtd_identifier, + 'NO' as is_self_referencing, + 'NO' as is_identity, + NULL as identity_generation, + NULL as identity_start, + NULL as identity_increment, + NULL as identity_maximum, + NULL as identity_minimum, + NULL as identity_cycle, + 'NO' as is_generated, + NULL as generation_expression, + 'NO' as is_updatable + FROM pg_class c + JOIN pg_attribute a ON c.oid = a.attrelid + WHERE c.relkind = 'r' + AND a.attnum > 0; + "#, + + r#" + UPDATE __pgsqlite_metadata + SET value = '28', updated_at = strftime('%s', 'now') + WHERE key = 'schema_version'; + "#, + ])), + dependencies: vec![28], + }); +} diff --git a/src/query/extended.rs b/src/query/extended.rs index cc0d9323..fd1cb4ff 100644 --- a/src/query/extended.rs +++ b/src/query/extended.rs @@ -2260,7 +2260,10 @@ impl ExtendedQueryHandler { }); } } else if query.contains("information_schema.tables") { - // Return all information_schema.tables columns + // information_schema.tables now executes against the + // information_schema_tables SQLite view (#88), but Describe + // still needs static metadata for SELECT * -- this mirrors + // the view's column list from migration v29 exactly. let all_columns = vec![ ("table_catalog", PgType::Text.to_oid()), ("table_schema", PgType::Text.to_oid()), @@ -2412,6 +2415,20 @@ impl ExtendedQueryHandler { let type_oid = Self::get_catalog_column_type(&name, query); (name, type_oid) } + sqlparser::ast::Expr::Function(func) => { + let fname = func.name.to_string().to_lowercase(); + // PostgreSQL names an unaliased aggregate column after + // the function itself, e.g. `count(*)` -> "count". + // Scoped to information_schema (#88 fall-through routing) + // so pg_depend/pg_class COUNT(*) callers, which read the + // result as text today, are unaffected. + let type_oid = if fname == "count" && query.contains("information_schema") { + PgType::Int8.to_oid() + } else { + PgType::Text.to_oid() + }; + (fname, type_oid) + } _ => ("?column?".to_string(), PgType::Text.to_oid()), } } @@ -2424,6 +2441,14 @@ impl ExtendedQueryHandler { let name = parts.last().map(|p| p.value.to_lowercase()).unwrap_or_else(|| "?column?".to_string()); Self::get_catalog_column_type(&name, query) } + sqlparser::ast::Expr::Function(func) => { + let fname = func.name.to_string().to_lowercase(); + if fname == "count" && query.contains("information_schema") { + PgType::Int8.to_oid() + } else { + PgType::Text.to_oid() + } + } _ => PgType::Text.to_oid(), }; (alias.value.clone(), type_oid) diff --git a/src/session/db_handler.rs b/src/session/db_handler.rs index bf3f2c73..749b6689 100644 --- a/src/session/db_handler.rs +++ b/src/session/db_handler.rs @@ -1171,27 +1171,6 @@ impl DbHandler { // For catalog queries, we need to use the catalog interceptor // This requires an Arc, but we can't create a cyclic Arc here // Instead, let's handle specific queries directly for now - if lower_query.contains("information_schema.tables") { - return self.handle_information_schema_tables_query(query, session_id).await; - } - - if lower_query.contains("information_schema.columns") { - // Use the catalog interceptor for columns queries with session-based execution - use crate::catalog::query_interceptor::CatalogInterceptor; - let parsed_query = sqlparser::parser::Parser::parse_sql(&sqlparser::dialect::PostgreSqlDialect {}, query); - if let Ok(mut statements) = parsed_query - && let Some(sqlparser::ast::Statement::Query(query_ast)) = statements.pop() - && let Some(select) = query_ast.body.as_select() { - return CatalogInterceptor::handle_information_schema_columns_query_with_session(select, self, session_id).await; - } - // Fallback to empty response if parsing fails - return Ok(DbResponse { - columns: vec!["table_name".to_string(), "column_name".to_string()], - rows: vec![], - rows_affected: 0, - }); - } - if lower_query.contains("information_schema.key_column_usage") { use crate::catalog::query_interceptor::CatalogInterceptor; let parsed_query = sqlparser::parser::Parser::parse_sql(&sqlparser::dialect::PostgreSqlDialect {}, query); @@ -1647,27 +1626,6 @@ impl DbHandler { // For catalog queries, we need to use the catalog interceptor // This requires an Arc, but we can't create a cyclic Arc here // Instead, let's handle specific queries directly for now - if lower_query.contains("information_schema.tables") { - return self.handle_information_schema_tables_query(query, session_id).await; - } - - if lower_query.contains("information_schema.columns") { - // Use the catalog interceptor for columns queries with session-based execution - use crate::catalog::query_interceptor::CatalogInterceptor; - let parsed_query = sqlparser::parser::Parser::parse_sql(&sqlparser::dialect::PostgreSqlDialect {}, query); - if let Ok(mut statements) = parsed_query - && let Some(sqlparser::ast::Statement::Query(query_ast)) = statements.pop() - && let Some(select) = query_ast.body.as_select() { - return CatalogInterceptor::handle_information_schema_columns_query_with_session(select, self, session_id).await; - } - // Fallback to empty response if parsing fails - return Ok(DbResponse { - columns: vec!["table_name".to_string(), "column_name".to_string()], - rows: vec![], - rows_affected: 0, - }); - } - if lower_query.contains("information_schema.key_column_usage") { use crate::catalog::query_interceptor::CatalogInterceptor; let parsed_query = sqlparser::parser::Parser::parse_sql(&sqlparser::dialect::PostgreSqlDialect {}, query); @@ -2659,77 +2617,6 @@ impl DbHandler { &self.statement_cache_optimizer } - /// Handle information_schema.tables query - async fn handle_information_schema_tables_query(&self, query: &str, session_id: &Uuid) -> Result { - debug!("Handling information_schema.tables query: {}", query); - - // Check if this is a simple table_name only query - if query.contains("SELECT table_name") && !query.contains("table_catalog") { - // Simple query - just return table names - let tables_query = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%' ORDER BY name"; - - return self.connection_manager.execute_with_session(session_id, |conn| { - let mut stmt = conn.prepare(tables_query)?; - let rows: Result, _> = stmt.query_map([], |row| { - let table_name: String = row.get(0)?; - Ok(vec![Some(table_name.into_bytes())]) - })?.collect(); - - Ok(DbResponse { - columns: vec!["table_name".to_string()], - rows: rows?, - rows_affected: 0, - }) - }); - } - - // Full information_schema.tables query - let tables_query = "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' AND name NOT LIKE '__pgsqlite_%' ORDER BY name"; - - self.connection_manager.execute_with_session(session_id, |conn| { - let mut stmt = conn.prepare(tables_query)?; - let rows: Result, _> = stmt.query_map([], |row| { - let table_name: String = row.get(0)?; - // Return full information_schema.tables row - Ok(vec![ - Some("main".to_string().into_bytes()), // table_catalog - Some("public".to_string().into_bytes()), // table_schema - Some(table_name.into_bytes()), // table_name - Some("BASE TABLE".to_string().into_bytes()), // table_type - None, // self_referencing_column_name - None, // reference_generation - None, // user_defined_type_catalog - None, // user_defined_type_schema - None, // user_defined_type_name - None, // is_insertable_into - None, // is_typed - None, // commit_action - ]) - })?.collect(); - - Ok(DbResponse { - columns: vec![ - "table_catalog".to_string(), - "table_schema".to_string(), - "table_name".to_string(), - "table_type".to_string(), - "self_referencing_column_name".to_string(), - "reference_generation".to_string(), - "user_defined_type_catalog".to_string(), - "user_defined_type_schema".to_string(), - "user_defined_type_name".to_string(), - "is_insertable_into".to_string(), - "is_typed".to_string(), - "commit_action".to_string(), - ], - rows: rows?, - rows_affected: 0, - }) - }) - } - - - /// Handle SQLAlchemy table existence check query /// This optimizes the complex JOIN query by doing a simple table lookup async fn handle_table_existence_query(&self, query: &str, session_id: &Uuid) -> Result { diff --git a/src/translator/schema_prefix_translator.rs b/src/translator/schema_prefix_translator.rs index 51fb0461..9c92e39f 100644 --- a/src/translator/schema_prefix_translator.rs +++ b/src/translator/schema_prefix_translator.rs @@ -6,26 +6,93 @@ use tracing::debug; /// but SQLite doesn't support schemas, so we need to strip the prefix pub struct SchemaPrefixTranslator; +/// Case-insensitively replace every occurrence of `needle` with `replacement`, +/// skipping SQL string literals (`'...'`) and quoted identifiers (`"..."`). +/// +/// A blind `str::replace` rewrites matches inside literals, so +/// `SELECT 'information_schema.tables'` used to come back corrupted and a +/// `WHERE msg = 'see pg_catalog.pg_class'` used to stop matching stored rows. +/// Matching is also case-insensitive because the catalog interceptor's gate is, +/// so spellings like `Information_Schema.Tables` reach this translator and must +/// not fall through untranslated to SQLite. +/// +/// `needle` must be ASCII. Single left-to-right scan; everything outside a +/// replaced span is preserved byte for byte. +/// +/// INVARIANT: callers must strip SQL comments first. This scanner does not +/// recognize `--` or `/* */`, so an apostrophe inside a comment (`-- don't`) +/// would leave it believing the rest of the query is one long string literal +/// and silently skip every real qualifier after it. `strip_sql_comments` runs +/// ahead of this translator on both entry paths (`query::executor` and +/// `query::extended`) and is itself literal-aware, which is what makes that +/// unreachable today. +fn replace_outside_literals(input: &str, needle: &str, replacement: &str) -> String { + debug_assert!(needle.is_ascii(), "needle must be ASCII for case-insensitive matching"); + if needle.is_empty() || input.len() < needle.len() { + return input.to_string(); + } + + let bytes = input.as_bytes(); + let needle = needle.as_bytes(); + let mut out = String::with_capacity(input.len()); + // Everything in `input[..copied]` has already been emitted into `out`. + let mut copied = 0usize; + let mut i = 0usize; + + while i < bytes.len() { + let quote = bytes[i]; + if quote == b'\'' || quote == b'"' { + // Skip the whole quoted span, honoring the SQL doubled-quote escape + // ('it''s' is one literal). Unterminated spans run to end of input, + // which leaves them untouched -- the parser rejects them later. + i += 1; + while i < bytes.len() { + if bytes[i] == quote { + if bytes.get(i + 1) == Some("e) { + i += 2; + continue; + } + i += 1; + break; + } + i += 1; + } + continue; + } + + if bytes.len() - i >= needle.len() && bytes[i..i + needle.len()].eq_ignore_ascii_case(needle) + { + out.push_str(&input[copied..i]); + out.push_str(replacement); + i += needle.len(); + copied = i; + continue; + } + + i += 1; + } + + out.push_str(&input[copied..]); + out +} + impl SchemaPrefixTranslator { /// Translate a query string by removing schema prefixes pub fn translate_query(query: &str) -> String { - // Simple string replacement approach for known pg_catalog tables let mut result = query.to_string(); - + // List of known pg_catalog tables that we have views for let catalog_tables = [ - "pg_class", "pg_namespace", "pg_attribute", "pg_type", + "pg_class", "pg_namespace", "pg_attribute", "pg_type", "pg_constraint", "pg_index", "pg_attrdef", "pg_am", "pg_enum", "pg_range" ]; - + for table in &catalog_tables { // Replace pg_catalog.table with just table - result = result.replace(&format!("pg_catalog.{table}"), table); - // Also handle uppercase - result = result.replace(&format!("PG_CATALOG.{}", table.to_uppercase()), table); + result = replace_outside_literals(&result, &format!("pg_catalog.{table}"), table); } - + // Also remove schema prefix from functions let catalog_functions = [ "pg_table_is_visible", "pg_get_userbyid", "pg_get_constraintdef", @@ -33,16 +100,22 @@ impl SchemaPrefixTranslator { "current_database", "current_schema", "current_user", "session_user", "pg_backend_pid", "pg_is_in_recovery", "current_schemas" ]; - + for func in &catalog_functions { - result = result.replace(&format!("pg_catalog.{func}"), func); - result = result.replace(&format!("PG_CATALOG.{}", func.to_uppercase()), func); + result = replace_outside_literals(&result, &format!("pg_catalog.{func}"), func); } - + + // information_schema relations exist as SQLite views with underscores. + // Rewriting here routes them to the views through the interceptor's + // fall-through path, the same way pg_catalog.* reaches pg_class. + // Only the two relations served by views; the rest still have handlers. + result = replace_outside_literals(&result, "information_schema.tables", "information_schema_tables"); + result = replace_outside_literals(&result, "information_schema.columns", "information_schema_columns"); + debug!("Schema prefix translation: {} -> {}", query, result); result } - + /// Translate an AST by removing schema prefixes pub fn translate_statement(stmt: &mut Statement) -> Result<(), sqlparser::parser::ParserError> { match stmt { @@ -87,8 +160,20 @@ impl SchemaPrefixTranslator { if schema_name == "pg_catalog" { // Replace with just the table name name.0 = vec![table.clone()]; + } else if schema_name == "information_schema" { + // The two relations served by SQLite views are rewritten to + // their underscore names; the rest keep their Rust handlers. + let table_name = match table { + ObjectNamePart::Identifier(ident) => ident.value.to_lowercase(), + }; + if table_name == "tables" || table_name == "columns" { + let mut ident = match table { + ObjectNamePart::Identifier(ident) => ident.clone(), + }; + ident.value = format!("information_schema_{table_name}"); + name.0 = vec![ObjectNamePart::Identifier(ident)]; + } } - // Don't remove information_schema prefix - it's handled by query interceptor } } } @@ -117,4 +202,112 @@ mod tests { let translated = SchemaPrefixTranslator::translate_query(query); assert_eq!(translated, "SELECT * FROM pg_class c JOIN pg_namespace n ON c.relnamespace = n.oid"); } + + #[test] + fn test_information_schema_tables_rewrite() { + let query = "SELECT table_name FROM information_schema.tables ORDER BY 1"; + let translated = SchemaPrefixTranslator::translate_query(query); + assert_eq!(translated, "SELECT table_name FROM information_schema_tables ORDER BY 1"); + } + + #[test] + fn test_information_schema_columns_rewrite() { + let query = "SELECT * FROM information_schema.columns"; + let translated = SchemaPrefixTranslator::translate_query(query); + assert_eq!(translated, "SELECT * FROM information_schema_columns"); + } + + /// Relations that still have Rust handlers must not be rewritten -- there is + /// no `information_schema_routines` view to fall through to. + #[test] + fn test_other_information_schema_relations_are_untouched() { + for relation in ["routines", "views", "triggers", "check_constraints"] { + let query = format!("SELECT * FROM information_schema.{relation}"); + assert_eq!(SchemaPrefixTranslator::translate_query(&query), query); + } + } + + /// `table_constraints` shares a prefix with `tables` -- verify no collision. + #[test] + fn test_table_constraints_is_not_caught_by_the_tables_rewrite() { + let query = "SELECT * FROM information_schema.table_constraints"; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } + + /// The catalog interceptor gates on a lowercased query, so mixed-case + /// spellings reach the translator and must not fall through to SQLite. + #[test] + fn test_mixed_case_schema_qualifiers_are_rewritten() { + for query in [ + "SELECT * FROM Information_Schema.Tables", + "SELECT * FROM information_schema.TABLES", + "SELECT * FROM INFORMATION_SCHEMA.tables", + "SELECT * FROM INFORMATION_SCHEMA.TABLES", + ] { + assert_eq!( + SchemaPrefixTranslator::translate_query(query), + "SELECT * FROM information_schema_tables", + "failed for {query}" + ); + } + + assert_eq!( + SchemaPrefixTranslator::translate_query("SELECT * FROM Information_Schema.Columns"), + "SELECT * FROM information_schema_columns" + ); + assert_eq!( + SchemaPrefixTranslator::translate_query("SELECT * FROM Pg_Catalog.Pg_Class"), + "SELECT * FROM pg_class" + ); + } + + /// A blind `str::replace` rewrote matches inside string literals, silently + /// corrupting stored SQL text and breaking equality comparisons. + #[test] + fn test_string_literals_are_not_rewritten() { + for query in [ + "SELECT 'information_schema.tables' AS s", + "SELECT 'information_schema.columns' AS s", + "SELECT 'pg_catalog.pg_class' AS s", + "INSERT INTO notes (msg) VALUES ('see information_schema.tables for details')", + "SELECT * FROM notes WHERE msg = 'see pg_catalog.pg_class for details'", + ] { + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } + } + + /// `'it''s'` is one literal, not two -- a naive scan would treat the text + /// after the doubled quote as unquoted and rewrite it. + #[test] + fn test_doubled_quote_escape_keeps_the_literal_intact() { + let query = "SELECT 'it''s information_schema.tables' AS s"; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } + + /// Double quotes are identifier quoting in PostgreSQL, so a quoted + /// identifier is one literal name and never a schema qualifier. + #[test] + fn test_quoted_identifiers_are_not_rewritten() { + let query = r#"SELECT * FROM "information_schema.tables""#; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + + let query = r#"SELECT * FROM "pg_catalog.pg_class""#; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } + + /// Text outside the replaced spans, including non-ASCII, is preserved. + #[test] + fn test_rewrites_outside_literals_still_happen_alongside_literals() { + let query = "SELECT 'information_schema.tables' AS lbl FROM information_schema.tables WHERE table_name = 'naïve.pg_catalog.pg_class'"; + assert_eq!( + SchemaPrefixTranslator::translate_query(query), + "SELECT 'information_schema.tables' AS lbl FROM information_schema_tables WHERE table_name = 'naïve.pg_catalog.pg_class'" + ); + } + + #[test] + fn test_unterminated_literal_is_left_alone() { + let query = "SELECT 'information_schema.tables"; + assert_eq!(SchemaPrefixTranslator::translate_query(query), query); + } } \ No newline at end of file diff --git a/tests/information_schema_namespace_test.rs b/tests/information_schema_namespace_test.rs new file mode 100644 index 00000000..797ad99b --- /dev/null +++ b/tests/information_schema_namespace_test.rs @@ -0,0 +1,540 @@ +mod common; +use common::setup_test_server; + +/// Issue #88: information_schema.tables listed pgsqlite's own catalog relations +/// as user tables in public, so Django inspectdb and SQLAlchemy automap would +/// generate models for pg_constraint and friends. +#[tokio::test] +async fn internal_relations_are_not_reported_in_public() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query("CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT)") + .await + .unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["customers".to_string()]); +} + +#[tokio::test] +async fn internal_relations_report_their_real_schema() { + let server = setup_test_server().await; + let client = &server.client; + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"pg_class"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "pg_catalog"); + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"information_schema_tables"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "information_schema"); +} + +/// The deleted handler ignored ORDER BY entirely. +#[tokio::test] +async fn order_by_is_respected() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE zebra (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE alpha (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE middle (id INTEGER)").await.unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["alpha".to_string(), "middle".to_string(), "zebra".to_string()]); +} + +/// The deleted handler returned zero rows for any aggregate. +#[tokio::test] +async fn aggregates_work() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE a (id INTEGER)").await.unwrap(); + client.simple_query("CREATE TABLE b (id INTEGER)").await.unwrap(); + + let row = client + .query_one( + "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, i64>(0), 2); +} + +/// The deleted handler supported equality on table_name and nothing else. +#[tokio::test] +async fn non_table_name_predicates_filter() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE t (id INTEGER)").await.unwrap(); + client.simple_query("CREATE VIEW v AS SELECT id FROM t").await.unwrap(); + + let rows = client + .query( + "SELECT table_name FROM information_schema.tables \ + WHERE table_schema = 'public' AND table_type = 'VIEW'", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["v".to_string()]); +} + +/// #102 divergence, asserted so it is documented rather than discovered. +/// pg_class still uses v28's LIKE 'pg\_%' heuristic, so a user table named +/// pg_* is misfiled there while information_schema.tables gets it right. +/// When #102 lands, the second assertion flips to "public". +#[tokio::test] +async fn user_table_named_like_a_catalog_relation_is_visible() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query("CREATE TABLE pg_myreport (id INTEGER PRIMARY KEY)") + .await + .unwrap(); + + let rows = client + .query( + "SELECT table_schema FROM information_schema.tables WHERE table_name = $1", + &[&"pg_myreport"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].get::<_, &str>(0), "public", "#88: exact-name matching"); + + let rows = client + .query( + "SELECT n.nspname FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace \ + WHERE c.relname = $1", + &[&"pg_myreport"], + ) + .await + .unwrap(); + assert_eq!(rows.len(), 1); + assert_eq!( + rows[0].get::<_, &str>(0), + "pg_catalog", + "known divergence until #102 lands; flip this to 'public' there" + ); +} + +/// The type fidelity table from the spec. The deleted handler got id, ts and +/// tags wrong; the pre-v29 view got six of eight wrong, including reporting +/// `integer` for TIMESTAMPTZ, which leaked the INTEGER datetime storage. +#[tokio::test] +async fn column_types_are_reported_faithfully() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query( + "CREATE TABLE fidelity ( + id SERIAL PRIMARY KEY, + amount NUMERIC(10,2), + uid UUID, + doc JSONB, + tags TEXT[], + ts TIMESTAMPTZ, + flag BOOLEAN, + nick VARCHAR(50) + )", + ) + .await + .unwrap(); + + let rows = client + .query( + "SELECT column_name, data_type FROM information_schema.columns \ + WHERE table_name = 'fidelity' ORDER BY ordinal_position", + &[], + ) + .await + .unwrap(); + + let got: Vec<(String, String)> = rows + .iter() + .map(|r| (r.get::<_, String>(0), r.get::<_, String>(1))) + .collect(); + + assert_eq!( + got, + vec![ + ("id".to_string(), "integer".to_string()), + ("amount".to_string(), "numeric".to_string()), + ("uid".to_string(), "uuid".to_string()), + ("doc".to_string(), "jsonb".to_string()), + ("tags".to_string(), "ARRAY".to_string()), + ("ts".to_string(), "timestamp with time zone".to_string()), + ("flag".to_string(), "boolean".to_string()), + ("nick".to_string(), "character varying".to_string()), + ] + ); +} + +/// The four columns the handler populated and the pre-v29 view left NULL. +/// Django and SQLAlchemy both read these. +#[tokio::test] +async fn column_modifiers_and_defaults_are_reported() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query( + "CREATE TABLE widgets ( + id INTEGER PRIMARY KEY, + name VARCHAR(50), + price NUMERIC(10,2), + active BOOLEAN DEFAULT true + )", + ) + .await + .unwrap(); + + let row = client + .query_one( + "SELECT character_maximum_length FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'name'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, Option>(0), Some(50)); + + let row = client + .query_one( + "SELECT numeric_precision, numeric_scale FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'price'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, Option>(0), Some(10)); + assert_eq!(row.get::<_, Option>(1), Some(2)); + + let row = client + .query_one( + "SELECT column_default FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'active'", + &[], + ) + .await + .unwrap(); + assert!( + row.get::<_, Option>(0).is_some(), + "column_default should be populated for DEFAULT true" + ); + + // INTEGER PRIMARY KEY is NOT NULL in PostgreSQL. pragma_table_info reports + // notnull=0 for it, so the view must also consider pk. + let row = client + .query_one( + "SELECT is_nullable FROM information_schema.columns \ + WHERE table_name = 'widgets' AND column_name = 'id'", + &[], + ) + .await + .unwrap(); + assert_eq!(row.get::<_, &str>(0), "NO"); +} + +#[tokio::test] +async fn columns_of_internal_relations_are_not_in_public() { + let server = setup_test_server().await; + let client = &server.client; + + client.simple_query("CREATE TABLE t (id INTEGER)").await.unwrap(); + + let rows = client + .query( + "SELECT DISTINCT table_name FROM information_schema.columns \ + WHERE table_schema = 'public' ORDER BY table_name", + &[], + ) + .await + .unwrap(); + + let names: Vec = rows.iter().map(|r| r.get::<_, String>(0)).collect(); + assert_eq!(names, vec!["t".to_string()]); +} + +/// The internal-relation list is maintained by hand, so it can drift when a +/// migration adds a catalog relation. Fail here rather than silently leaking +/// the new relation into information_schema.tables as a user table. +#[test] +fn internal_relation_list_matches_migrated_database() { + use pgsqlite::catalog::internal_relations::{ + INTERNAL_INFORMATION_SCHEMA_RELATIONS, INTERNAL_PG_CATALOG_RELATIONS, + }; + use pgsqlite::migration::MigrationRunner; + use rusqlite::Connection; + use tempfile::TempDir; + + let temp_dir = TempDir::new().unwrap(); + let db_path = temp_dir.path().join("drift.db"); + let conn = Connection::open(&db_path).unwrap(); + let mut runner = MigrationRunner::new(conn); + runner.run_pending_migrations().unwrap(); + let conn = runner.into_connection(); + + let mut in_database: Vec = conn + .prepare( + "SELECT name FROM sqlite_master \ + WHERE name NOT LIKE 'sqlite_%' AND substr(name, 1, 11) <> '__pgsqlite_' \ + ORDER BY name", + ) + .unwrap() + .query_map([], |r| r.get(0)) + .unwrap() + .collect::>() + .unwrap(); + in_database.sort(); + + let mut in_list: Vec = INTERNAL_PG_CATALOG_RELATIONS + .iter() + .chain(INTERNAL_INFORMATION_SCHEMA_RELATIONS.iter()) + .map(|s| s.to_string()) + .collect(); + in_list.sort(); + + let missing: Vec<&String> = in_database.iter().filter(|n| !in_list.contains(n)).collect(); + assert!( + missing.is_empty(), + "migrations create relations absent from src/catalog/internal_relations.rs: {missing:?}. \ + Add them, or they will be reported as user tables in information_schema.tables." + ); + + let stale: Vec<&String> = in_list.iter().filter(|n| !in_database.contains(n)).collect(); + assert!( + stale.is_empty(), + "src/catalog/internal_relations.rs lists relations no migration creates: {stale:?}" + ); +} + +fn simple_query_column(messages: &[tokio_postgres::SimpleQueryMessage]) -> Vec { + let mut values: Vec = messages + .iter() + .filter_map(|m| match m { + tokio_postgres::SimpleQueryMessage::Row(r) => Some(r.get(0).unwrap().to_string()), + _ => None, + }) + .collect(); + values.sort(); + values +} + +/// The catalog interceptor gates on a lowercased query, so a mixed-case +/// qualifier enters the catalog path. Before the case-insensitive rewrite it +/// fell through untranslated and died with "no such table". The Rust handler +/// this branch deleted dispatched on a lowercased name, so every spelling +/// worked; JDBC-derived tooling emits `INFORMATION_SCHEMA.tables`. +#[tokio::test] +async fn mixed_case_schema_qualifiers_route_to_the_views() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query("CREATE TABLE gadgets (id INTEGER PRIMARY KEY, label TEXT)") + .await + .unwrap(); + + let baseline = simple_query_column( + &client + .simple_query( + "SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'", + ) + .await + .unwrap(), + ); + assert_eq!(baseline, vec!["gadgets".to_string()]); + + for query in [ + "SELECT table_name FROM Information_Schema.Tables WHERE table_schema = 'public'", + "SELECT table_name FROM information_schema.TABLES WHERE table_schema = 'public'", + "SELECT table_name FROM INFORMATION_SCHEMA.tables WHERE table_schema = 'public'", + "SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = 'public'", + ] { + let messages = client + .simple_query(query) + .await + .unwrap_or_else(|e| panic!("{query} failed: {e}")); + assert_eq!(simple_query_column(&messages), baseline, "mismatch for {query}"); + } + + let baseline = simple_query_column( + &client + .simple_query( + "SELECT column_name FROM information_schema.columns WHERE table_name = 'gadgets'", + ) + .await + .unwrap(), + ); + assert_eq!(baseline, vec!["id".to_string(), "label".to_string()]); + + for query in [ + "SELECT column_name FROM Information_Schema.Columns WHERE table_name = 'gadgets'", + "SELECT column_name FROM information_schema.COLUMNS WHERE table_name = 'gadgets'", + "SELECT column_name FROM INFORMATION_SCHEMA.columns WHERE table_name = 'gadgets'", + ] { + let messages = client + .simple_query(query) + .await + .unwrap_or_else(|e| panic!("{query} failed: {e}")); + assert_eq!(simple_query_column(&messages), baseline, "mismatch for {query}"); + } +} + +/// The schema-prefix translator used a blind `str::replace`, which rewrote +/// matches inside string literals. Any table holding SQL text -- an audit log, +/// a migration history, a notes table -- got corrupted values and silently +/// wrong equality results. +#[tokio::test] +async fn string_literals_containing_schema_qualifiers_survive() { + let server = setup_test_server().await; + let client = &server.client; + + for literal in [ + "information_schema.tables", + "information_schema.columns", + "pg_catalog.pg_class", + ] { + let row = client + .query_one(&format!("SELECT '{literal}' AS s"), &[]) + .await + .unwrap_or_else(|e| panic!("selecting '{literal}' failed: {e}")); + assert_eq!(row.get::<_, &str>(0), literal); + } + + client + .simple_query("CREATE TABLE notes (id INTEGER PRIMARY KEY, msg TEXT)") + .await + .unwrap(); + + for (id, msg) in [ + (1i32, "see information_schema.tables for details"), + (2i32, "see pg_catalog.pg_class for details"), + ] { + client + .execute("INSERT INTO notes (id, msg) VALUES ($1, $2)", &[&id, &msg]) + .await + .unwrap(); + + // simple_query, not query_one: the catalog interceptor's gate is a + // substring test over the whole query text, so a literal mentioning + // pg_catalog sends even this count down the catalog path, which types + // the result as text. Pre-existing and orthogonal to literal handling. + let messages = client + .simple_query(&format!("SELECT count(*) FROM notes WHERE msg = '{msg}'")) + .await + .unwrap(); + assert_eq!( + simple_query_column(&messages), + vec!["1".to_string()], + "round-trip failed for {msg:?}" + ); + + let row = client + .query_one("SELECT msg FROM notes WHERE id = $1", &[&id]) + .await + .unwrap(); + assert_eq!(row.get::<_, &str>(0), msg); + } + + // The SQL-standard doubled-quote escape: 'it''s ...' is one literal. + let row = client + .query_one("SELECT 'it''s information_schema.tables' AS s", &[]) + .await + .unwrap(); + assert_eq!(row.get::<_, &str>(0), "it's information_schema.tables"); +} + +/// PostgreSQL reports radix 2 for the binary-precision numeric types and 10 +/// only for numeric/decimal. v29 first reported 10 for every typed precision. +#[tokio::test] +async fn numeric_precision_radix_matches_postgresql() { + let server = setup_test_server().await; + let client = &server.client; + + client + .simple_query( + "CREATE TABLE radix_probe ( + i INTEGER, + b BIGINT, + s SMALLINT, + d DOUBLE PRECISION, + n NUMERIC(10,2), + t TEXT + )", + ) + .await + .unwrap(); + + let rows = client + .query( + "SELECT column_name, numeric_precision, numeric_precision_radix \ + FROM information_schema.columns WHERE table_name = 'radix_probe' \ + ORDER BY ordinal_position", + &[], + ) + .await + .unwrap(); + + let got: Vec<(String, Option, Option)> = rows + .iter() + .map(|r| { + ( + r.get::<_, String>(0), + r.get::<_, Option>(1), + r.get::<_, Option>(2), + ) + }) + .collect(); + + assert_eq!( + got, + vec![ + ("i".to_string(), Some(32), Some(2)), + ("b".to_string(), Some(64), Some(2)), + ("s".to_string(), Some(16), Some(2)), + ("d".to_string(), Some(53), Some(2)), + ("n".to_string(), Some(10), Some(10)), + ("t".to_string(), None, None), + ] + ); +} diff --git a/tests/migration_test.rs b/tests/migration_test.rs index 25b2239c..2259fae7 100644 --- a/tests/migration_test.rs +++ b/tests/migration_test.rs @@ -23,7 +23,7 @@ fn test_fresh_database_migration() { // Should apply all migrations assert_eq!(applied.len(), MIGRATIONS.len()); - assert_eq!(applied, vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]); + assert_eq!(applied, vec![1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]); // Verify schema version let conn = runner.into_connection(); @@ -32,7 +32,7 @@ fn test_fresh_database_migration() { [], |row| row.get(0) ).unwrap(); - assert_eq!(version, "28"); + assert_eq!(version, "29"); // Now check should pass let runner2 = MigrationRunner::new(conn); @@ -68,7 +68,7 @@ fn test_idempotent_migrations() { let conn = Connection::open(&db_path).unwrap(); let mut runner = MigrationRunner::new(conn); let applied = runner.run_pending_migrations().unwrap(); - assert_eq!(applied.len(), 28); + assert_eq!(applied.len(), 29); drop(runner); // Second run - should apply nothing @@ -109,8 +109,8 @@ fn test_existing_schema_detection() { let mut runner = MigrationRunner::new(conn); let applied = runner.run_pending_migrations().unwrap(); - // Should recognize existing schema as version 1 and only apply versions 2-28 - assert_eq!(applied.len(), 27); + // Should recognize existing schema as version 1 and only apply versions 2-29 + assert_eq!(applied.len(), 28); assert_eq!(applied[0], 2); assert_eq!(applied[1], 3); assert_eq!(applied[2], 4); @@ -124,6 +124,7 @@ fn test_existing_schema_detection() { assert_eq!(applied[10], 12); assert_eq!(applied[25], 27); assert_eq!(applied[26], 28); + assert_eq!(applied[27], 29); // Verify final version let conn = runner.into_connection(); @@ -132,7 +133,7 @@ fn test_existing_schema_detection() { [], |row| row.get(0) ).unwrap(); - assert_eq!(version, "28"); + assert_eq!(version, "29"); // Now check should pass let runner2 = MigrationRunner::new(conn); @@ -157,7 +158,7 @@ fn test_migration_history() { .unwrap() .collect::, _>>().unwrap(); - assert_eq!(migrations.len(), 28); + assert_eq!(migrations.len(), 29); assert_eq!(migrations[0], (1, "initial_schema".to_string(), "completed".to_string())); assert_eq!(migrations[1], (2, "enum_type_support".to_string(), "completed".to_string())); assert_eq!(migrations[2], (3, "datetime_timezone_support".to_string(), "completed".to_string())); @@ -220,4 +221,97 @@ fn test_check_up_to_date_database() { // Check should pass for up-to-date database assert!(runner.check_schema_version().is_ok()); +} + +#[test] +fn test_v29_information_schema_views_are_namespace_aware() { + let temp_dir = TempDir::new().unwrap(); + let db_path = temp_dir.path().join("v29.db"); + + let conn = Connection::open(&db_path).unwrap(); + let mut runner = MigrationRunner::new(conn); + runner.run_pending_migrations().unwrap(); + let conn = runner.into_connection(); + + // The views call UDFs, which the migration runner's connection does not have. + pgsqlite::functions::register_all_functions(&conn).unwrap(); + conn.pragma_update(None, "trusted_schema", true).unwrap(); + + conn.execute_batch( + "CREATE TABLE customers (id INTEGER PRIMARY KEY, name VARCHAR(50), amount NUMERIC(10,2));", + ) + .unwrap(); + + // Internal relations are not in public. + let internal_in_public: i32 = conn + .query_row( + "SELECT COUNT(*) FROM information_schema_tables \ + WHERE table_schema = 'public' AND table_name LIKE 'pg\\_%' ESCAPE '\\'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(internal_in_public, 0, "pg_* relations still reported in public"); + + let pg_class_schema: String = conn + .query_row( + "SELECT table_schema FROM information_schema_tables WHERE table_name = 'pg_class'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(pg_class_schema, "pg_catalog"); + + let is_tables_schema: String = conn + .query_row( + "SELECT table_schema FROM information_schema_tables \ + WHERE table_name = 'information_schema_tables'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(is_tables_schema, "information_schema"); + + // The user table is the only thing in public. + let public_tables: Vec = conn + .prepare("SELECT table_name FROM information_schema_tables WHERE table_schema = 'public' ORDER BY table_name") + .unwrap() + .query_map([], |r| r.get(0)) + .unwrap() + .collect::>() + .unwrap(); + assert_eq!(public_tables, vec!["customers".to_string()]); + + // Column metadata the old view lost. + let (dt, len): (String, Option) = conn + .query_row( + "SELECT data_type, character_maximum_length FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'name'", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!(dt, "character varying"); + assert_eq!(len, Some(50)); + + let (p, s): (Option, Option) = conn + .query_row( + "SELECT numeric_precision, numeric_scale FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'amount'", + [], + |r| Ok((r.get(0)?, r.get(1)?)), + ) + .unwrap(); + assert_eq!((p, s), (Some(10), Some(2))); + + // INTEGER PRIMARY KEY is NOT NULL, which the v14 view got wrong. + let nullable: String = conn + .query_row( + "SELECT is_nullable FROM information_schema_columns \ + WHERE table_name = 'customers' AND column_name = 'id'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(nullable, "NO"); } \ No newline at end of file