diff --git a/README.md b/README.md index 5e40ddc..5d583de 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,16 @@ codec), and it is shipped as a single self-contained WebAssembly module — the | overrides | a list, see [below](#override-option) | Yes | Overrides the generated (RBS) type of specific columns / parameters. | | moduleName | default: `Codegen` (e.g. `PgCodegen`) | Yes | The Ruby module the generated code lives in, e.g. `MyApp::Db`. | | preparedStatements | default: `true`
values: `false`,`true` | Yes | `pg` only: `false` sends the SQL with every call (`exec_params`) - needed behind pgbouncer in transaction mode. | +| rename | an object, e.g. `{authors: Writer, bio: biography}` | Yes | Renames generated identifiers: a table name (or `schema.table`) gives the model class, a column name gives the row / model attribute. | +| exactTableNames | default: `false`
values: `false`,`true` | Yes | `true` keeps table names as they are for model classes (`authors` => `Authors`) instead of singularizing them. | + +Unknown options are rejected (with a hint for near misses such as `generate_types`). Options can also be set once for +all `sql:` blocks in the top-level `options:` of `sqlc.yaml`, under the plugin's name (`options: {ruby: {moduleName: MyApp::Db}}`). +The `options:` of a codegen entry win over these, and the `overrides` of both apply. A driver used with another +`engine:` (e.g. `pg` with `engine: mysql`) is an error. + +Table, column and enum comments (`COMMENT ON ...` in PostgreSQL, `COMMENT '...'` in MySQL) become doc comments of the +generated models (column comments in the RBS). Unknown option names (e.g. a typo such as `moduleNmae`) and `preparedStatements` with a driver other than `pg` are errors. @@ -31,11 +41,18 @@ overrides: ruby_type: type: "Symbol" notNull: true - - column: "*:" # every query + - column: "*:" # every query and model ruby_type: type: "Pathname" notNull: false + - column: "." # a table column (".
." too): its model and query columns + ruby_type: + type: "Symbol" + - db_type: "uuid" # every column / parameter of a database type + ruby_type: + type: "String" ``` +The first match wins, in this order: `:`, `
.`, `*:`, `db_type`. ### Configuration ```yaml @@ -123,6 +140,8 @@ end | `:exec` | ✅ | ✅ | ✅ | | | `:execrows` | ✅ | ✅ | ✅ | returns the number of affected rows | | `:execlastid` | ✅ | ✅ | ✅ | PostgreSQL / SQLite need a `RETURNING ` clause (the value of that column is returned); MySQL returns `client.last_id` | +| `:execresult` | ✅ | ✅ | ✅ | returns `ExecResult` (`rows_affected`, `last_insert_id`; `last_insert_id` is `nil` on PostgreSQL, use `RETURNING`) | +| `:batchexec` / `:batchone` / `:batchmany` | ✅ | ✅ | ✅ | take an `Array` of `Args` and run the prepared statement once per element on one connection; return `nil` / `Array[Row?]` / `Array[Array[Row]]` | | `:copyfrom` | ✅ | ✅ | ✅ | takes an `Array` of `Args` (a generated `Data` class). PostgreSQL: `COPY ... FROM STDIN`. MySQL: `LOAD DATA LOCAL INFILE` from a temporary CSV - needs `local_infile: true` in the `Mysql2::Client` params and `local_infile=1` on the server. SQLite: batched multi-row `INSERT` | ### Macros @@ -139,6 +158,8 @@ end * **Nullability:** nullable columns / parameters are typed `T?` in the `.rbs`. * **Arrays (PostgreSQL):** `Array[T]` for columns and parameters (`Array[Array[T]]` for `T[][]`); arrays of types without a decoder (`uuid[]`, enum arrays, ...) are `Array[String]`. * **Names:** a parameter repeated in a query gets a suffix (`id:`, `id_2:`); result columns named like a keyword or a method every row has (`class`, `hash`, `end`, ...) get a trailing `_` (`class_`); a table `authors` next to a table `author` keeps its plural model name (`Authors`). Query names that would replace a generated method (`Transaction`, `Initialize`, ...) are rejected. +* **Other PostgreSQL types:** geometric, range / multirange, full-text, `hstore`, `ltree`, `timetz`, `reg*` and composite types are `String`s (the text the pg gem returns); domains and types the plugin does not know are `untyped` instead of failing the generation. +* **SQLite declared types** (`VARCHAR(255)`, `DECIMAL(10,5)`, `UNSIGNED BIG INT`, ...) follow SQLite's [type affinity](https://www.sqlite.org/datatype3.html#determination_of_column_affinity). * Types follow the Ruby classes the drivers actually return (e.g. `BigDecimal` for `numeric`/`decimal`, `Date`, `Time`, `IPAddr`; `json`/`jsonb` are parsed, so `untyped`). Not applicable to the Ruby plugin (they exist in the C# plugin because of its runtime): `withAsyncSuffix` / `withCancellationToken` (Ruby has no async/await: use the Fiber scheduler, see above), `useDapper`, target framework / csproj / namespace options (`moduleName` is the equivalent of the namespace). diff --git a/docs/Architecture.md b/docs/Architecture.md index 381b96b..851d678 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -20,7 +20,7 @@ stdout ◄─ Proto (encode) ◄─ GenerateResponse (query_sql.rb, query_sql.rb |---|---| | `exe/plugin.rb` | entry point. Errors go to stderr with exit code 1 (sqlc shows them); `SQLC_GEN_RUBY_DEBUG=1` adds a backtrace | | `proto.rb`, `plugin_pb.rb` | a ~150 line protobuf wire codec and the subset of sqlc's `codegen.proto` we read. Field numbers must match upstream. `google-protobuf` is a C extension and cannot run inside ruby.wasm | -| `options.rb`, `ruby_version.rb` | parse + validate the `options:` JSON (`driver`, `rubyVersion`, `overrides`, ...) | +| `options.rb`, `ruby_version.rb` | parse + validate the `options:` JSON (`driver`, `rubyVersion`, `overrides`, `rename`, ...; unknown keys are errors) | | `generator.rb` | the orchestrator: one file per query file (`query_sql.rb`), plus `models.rb`, `connection.rb`, `Gemfile`; builds the module / class / `Data` skeleton | | `method_gen.rb` | the code of one query: signature (keyword arguments), SQL constant, command (`:one`, `:many`, `:exec`, `:execrows`, `:execlastid`, `:copyfrom`), `sqlc.slice`, bang finder, and the RBS signature | | `row_layout.rb` | turns result columns into row attributes; `sqlc.embed` collapses a table's columns into one model attribute | diff --git a/examples/authors/mysql/query.sql b/examples/authors/mysql/query.sql index 3f25a8c..13e6835 100644 --- a/examples/authors/mysql/query.sql +++ b/examples/authors/mysql/query.sql @@ -89,6 +89,18 @@ SELECT * FROM extended_bios WHERE bio_type = ? LIMIT 1; -- name: TruncateExtendedBios :exec DELETE FROM extended_bios; +-- name: CreateAuthorResult :execresult +INSERT INTO authors (name, bio) VALUES (?, ?); + +-- name: CreateAuthorsBatch :batchexec +INSERT INTO authors (name, bio) VALUES (?, ?); + +-- name: GetAuthorBatch :batchone +SELECT * FROM authors WHERE id = ?; + +-- name: ListAuthorsByNameBatch :batchmany +SELECT * FROM authors WHERE name = ? ORDER BY id; + -- name: CountRegressionsByText :one SELECT count(*) FROM regressions WHERE `order` = sqlc.arg(term) OR note = sqlc.arg(term); diff --git a/examples/authors/postgresql/query.sql b/examples/authors/postgresql/query.sql index 8d76c5d..1ee4431 100644 --- a/examples/authors/postgresql/query.sql +++ b/examples/authors/postgresql/query.sql @@ -124,6 +124,28 @@ SELECT * FROM authors WHERE id > $1 AND id < $2 ORDER BY id; +-- name: InsertSpecialTypes :exec +INSERT INTO postgres_special_types (c_timetz, c_tsvector, c_point, c_int4range, c_oid, c_dimensions, c_positive) +VALUES ($1, $2, $3, $4, $5, $6, $7); + +-- name: GetSpecialTypes :one +SELECT * FROM postgres_special_types LIMIT 1; + +-- name: TruncateSpecialTypes :exec +TRUNCATE TABLE postgres_special_types; + +-- name: CreateAuthorResult :execresult +INSERT INTO authors (name, bio) VALUES ($1, $2); + +-- name: CreateAuthorsBatch :batchexec +INSERT INTO authors (name, bio) VALUES ($1, $2); + +-- name: GetAuthorBatch :batchone +SELECT * FROM authors WHERE id = $1; + +-- name: ListAuthorsByNameBatch :batchmany +SELECT * FROM authors WHERE name = $1 ORDER BY id; + -- name: CreateRegression :one INSERT INTO regressions (id, "order", payload, doc, bio_types, uids) VALUES ($1, $2, $3, $4, $5, $6) RETURNING *; diff --git a/examples/authors/postgresql/schema.sql b/examples/authors/postgresql/schema.sql index be5a234..8905d68 100644 --- a/examples/authors/postgresql/schema.sql +++ b/examples/authors/postgresql/schema.sql @@ -65,6 +65,24 @@ CREATE TABLE postgres_array_types ( c_integer_matrix INTEGER [][] ); +CREATE TYPE extended.dimensions AS (width INTEGER, height INTEGER); + +CREATE DOMAIN positive_int AS INTEGER CHECK (VALUE > 0); + +CREATE TABLE postgres_special_types ( + c_timetz TIMETZ, + c_tsvector TSVECTOR, + c_point POINT, + c_int4range INT4RANGE, + c_oid OID, + c_dimensions extended.dimensions, + c_positive positive_int +); + +COMMENT ON TABLE authors IS 'People who wrote at least one book.'; +COMMENT ON COLUMN authors.bio IS 'Short biography, NULL when unknown.'; +COMMENT ON TYPE extended.bio_type IS 'Kinds of biographies.'; + CREATE TABLE regressions ( id bigint PRIMARY KEY, "order" text NOT NULL, diff --git a/examples/authors/sqlite/query.sql b/examples/authors/sqlite/query.sql index 7dd6f3b0..a439cba 100644 --- a/examples/authors/sqlite/query.sql +++ b/examples/authors/sqlite/query.sql @@ -74,6 +74,27 @@ SELECT FROM authors INNER JOIN books ON authors.id = books.author_id WHERE books.name = ?; +-- name: CreateAuthorResult :execresult +INSERT INTO authors (name, bio) VALUES (?, ?); + +-- name: CreateAuthorsBatch :batchexec +INSERT INTO authors (name, bio) VALUES (?, ?); + +-- name: GetAuthorBatch :batchone +SELECT * FROM authors WHERE id = ?; + +-- name: ListAuthorsByNameBatch :batchmany +SELECT * FROM authors WHERE name = ? ORDER BY id; + +-- name: InsertDeclaredTypes :exec +INSERT INTO declared_types (c_varchar, c_decimal, c_unsigned_big_int, c_native, c_double) VALUES (?, ?, ?, ?, ?); + +-- name: GetDeclaredTypes :one +SELECT * FROM declared_types LIMIT 1; + +-- name: TruncateDeclaredTypes :exec +DELETE FROM declared_types; + -- name: CreateRegression :exec INSERT INTO regressions ("order", price) VALUES (?, ?) RETURNING id; diff --git a/examples/authors/sqlite/schema.sql b/examples/authors/sqlite/schema.sql index e8ebd30..4b72f2e 100644 --- a/examples/authors/sqlite/schema.sql +++ b/examples/authors/sqlite/schema.sql @@ -13,6 +13,14 @@ CREATE TABLE books ( FOREIGN KEY (author_id) REFERENCES authors (id) ON DELETE CASCADE ); +CREATE TABLE declared_types ( + c_varchar VARCHAR(255), + c_decimal DECIMAL(10, 5), + c_unsigned_big_int UNSIGNED BIG INT, + c_native NATIVE CHARACTER(70), + c_double DOUBLE PRECISION +); + CREATE TABLE regressions ( id INTEGER PRIMARY KEY AUTOINCREMENT, "order" VARCHAR(255) NOT NULL, diff --git a/examples/mysql2/query_sql.rb b/examples/mysql2/query_sql.rb index 27ebb40..f940cf5 100644 --- a/examples/mysql2/query_sql.rb +++ b/examples/mysql2/query_sql.rb @@ -187,6 +187,37 @@ class GetFirstExtendedBioByTypeRow < Data.define(:author_name, :name, :bio_type, DELETE FROM extended_bios SQL + CREATE_AUTHOR_RESULT_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES (?, ?) + SQL + + CREATE_AUTHORS_BATCH_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES (?, ?) + SQL + + class CreateAuthorsBatchArgs < Data.define(:name, :bio) + end + + GET_AUTHOR_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE id = ? + SQL + + class GetAuthorBatchRow < Data.define(:id, :name, :bio) + end + + class GetAuthorBatchArgs < Data.define(:id) + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE name = ? ORDER BY id + SQL + + class ListAuthorsByNameBatchRow < Data.define(:id, :name, :bio) + end + + class ListAuthorsByNameBatchArgs < Data.define(:name) + end + COUNT_REGRESSIONS_BY_TEXT_SQL = <<~'SQL' SELECT count(*) FROM regressions WHERE `order` = ? OR note = ? SQL @@ -260,6 +291,9 @@ def transaction(isolation: nil) end end + class ExecResult < Data.define(:rows_affected, :last_insert_id) + end + # Returns the author with the given id, or nil when there is none. def get_author(id:) with_client('get_author') do |client| @@ -610,6 +644,67 @@ def truncate_extended_bios end end + def create_author_result(name:, bio:) + with_client('create_author_result') do |client| + query_params = [name, bio] + stmt = client.prepare(CREATE_AUTHOR_RESULT_SQL) + begin + stmt.execute(*query_params) + ExecResult.new(stmt.affected_rows, client.last_id) + ensure + stmt.close + end + end + end + + def create_authors_batch(rows) + with_client('create_authors_batch') do |client| + stmt = client.prepare(CREATE_AUTHORS_BATCH_SQL) + begin + rows.each do |args| + query_params = [args.name, args.bio] + stmt.execute(*query_params) + end + nil + ensure + stmt.close + end + end + end + + def get_author_batch(rows) + with_client('get_author_batch') do |client| + stmt = client.prepare(GET_AUTHOR_BATCH_SQL) + begin + rows.map do |args| + query_params = [args.id] + result = stmt.execute(*query_params, as: :array) + row = result.first + row && GetAuthorBatchRow.new(row[0], row[1], row[2]) + end + ensure + stmt.close + end + end + end + + def list_authors_by_name_batch(rows) + with_client('list_authors_by_name_batch') do |client| + stmt = client.prepare(LIST_AUTHORS_BY_NAME_BATCH_SQL) + begin + rows.map do |args| + query_params = [args.name] + result = stmt.execute(*query_params, as: :array) + result.map do |row| + ListAuthorsByNameBatchRow.new(row[0], row[1], row[2]) + end + end + ensure + stmt.close + end + end + end + def count_regressions_by_text(term:) with_client('count_regressions_by_text') do |client| query_params = [term, term] diff --git a/examples/mysql2/query_sql.rbs b/examples/mysql2/query_sql.rbs index ce57615..1ccc945 100644 --- a/examples/mysql2/query_sql.rbs +++ b/examples/mysql2/query_sql.rbs @@ -195,6 +195,44 @@ module Mysql2Codegen TRUNCATE_EXTENDED_BIOS_SQL: String + CREATE_AUTHOR_RESULT_SQL: String + + CREATE_AUTHORS_BATCH_SQL: String + + class CreateAuthorsBatchArgs + attr_reader name: String + attr_reader bio: String? + def initialize: ( String name, String? bio) -> void + end + + GET_AUTHOR_BATCH_SQL: String + + class GetAuthorBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class GetAuthorBatchArgs + attr_reader id: Integer + def initialize: ( Integer id) -> void + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL: String + + class ListAuthorsByNameBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class ListAuthorsByNameBatchArgs + attr_reader name: String + def initialize: ( String name) -> void + end + COUNT_REGRESSIONS_BY_TEXT_SQL: String class CountRegressionsByTextRow @@ -229,6 +267,12 @@ module Mysql2Codegen def transaction: [T] (?isolation: (:read_uncommitted | :read_committed | :repeatable_read | :serializable)?) { (self) -> T } -> T + class ExecResult + attr_reader rows_affected: Integer + attr_reader last_insert_id: Integer? + def initialize: ( Integer rows_affected, Integer? last_insert_id) -> void + end + def get_author: (id: Integer) -> GetAuthorRow? def get_author!: (id: Integer) -> GetAuthorRow @@ -277,6 +321,14 @@ module Mysql2Codegen def truncate_extended_bios: -> void + def create_author_result: (name: String, bio: String?) -> ExecResult + + def create_authors_batch: (Array[CreateAuthorsBatchArgs] rows) -> void + + def get_author_batch: (Array[GetAuthorBatchArgs] rows) -> Array[GetAuthorBatchRow?] + + def list_authors_by_name_batch: (Array[ListAuthorsByNameBatchArgs] rows) -> Array[Array[ListAuthorsByNameBatchRow]] + def count_regressions_by_text: (term: String) -> CountRegressionsByTextRow? def count_regressions_by_text!: (term: String) -> CountRegressionsByTextRow diff --git a/examples/pg/models.rb b/examples/pg/models.rb index fd04101..02758eb 100644 --- a/examples/pg/models.rb +++ b/examples/pg/models.rb @@ -3,6 +3,7 @@ # auto-generated by sqlc - do not edit module PgCodegen + # Kinds of biographies. module ExtendedBioType AUTOBIOGRAPHY = 'Autobiography' BIOGRAPHY = 'Biography' @@ -10,6 +11,7 @@ module ExtendedBioType ALL = [AUTOBIOGRAPHY, BIOGRAPHY, MEMOIR].freeze end + # People who wrote at least one book. class Author < Data.define(:id, :name, :bio) end @@ -53,6 +55,17 @@ class PostgresExtraType < Data.define( class PostgresArrayType < Data.define(:c_uuid_array, :c_bio_type_array, :c_integer_matrix) end + class PostgresSpecialType < Data.define( + :c_timetz, + :c_tsvector, + :c_point, + :c_int4range, + :c_oid, + :c_dimensions, + :c_positive + ) + end + class Regression < Data.define( :id, :order, diff --git a/examples/pg/models.rbs b/examples/pg/models.rbs index 5644e4d..69c5313 100644 --- a/examples/pg/models.rbs +++ b/examples/pg/models.rbs @@ -1,5 +1,6 @@ # auto-generated by sqlc - do not edit module PgCodegen + # Kinds of biographies. module ExtendedBioType AUTOBIOGRAPHY: String BIOGRAPHY: String @@ -7,9 +8,11 @@ module PgCodegen ALL: Array[String] end + # People who wrote at least one book. class Author attr_reader id: Integer attr_reader name: String + # Short biography, NULL when unknown. attr_reader bio: String? def initialize: ( Integer id, String name, String? bio) -> void end @@ -91,6 +94,25 @@ module PgCodegen def initialize: ( Array[String]? c_uuid_array, Array[String]? c_bio_type_array, Array[Array[Integer]]? c_integer_matrix) -> void end + class PostgresSpecialType + attr_reader c_timetz: String? + attr_reader c_tsvector: String? + attr_reader c_point: String? + attr_reader c_int4range: String? + attr_reader c_oid: Integer? + attr_reader c_dimensions: String? + attr_reader c_positive: untyped + def initialize: ( + String? c_timetz, + String? c_tsvector, + String? c_point, + String? c_int4range, + Integer? c_oid, + String? c_dimensions, + untyped c_positive + ) -> void + end + class Regression attr_reader id: Integer attr_reader order: String diff --git a/examples/pg/query_sql.rb b/examples/pg/query_sql.rb index 605aebc..de218d9 100644 --- a/examples/pg/query_sql.rb +++ b/examples/pg/query_sql.rb @@ -239,6 +239,61 @@ class ListAuthorsWithOptionalBookRow < Data.define(:author, :book) class ListAuthorsBetweenIdsRow < Data.define(:id, :name, :bio) end + INSERT_SPECIAL_TYPES_SQL = <<~'SQL' + INSERT INTO postgres_special_types (c_timetz, c_tsvector, c_point, c_int4range, c_oid, c_dimensions, c_positive) + VALUES ($1, $2, $3, $4, $5, $6, $7) + SQL + + GET_SPECIAL_TYPES_SQL = <<~'SQL' + SELECT c_timetz, c_tsvector, c_point, c_int4range, c_oid, c_dimensions, c_positive FROM postgres_special_types LIMIT 1 + SQL + + class GetSpecialTypesRow < Data.define( + :c_timetz, + :c_tsvector, + :c_point, + :c_int4range, + :c_oid, + :c_dimensions, + :c_positive + ) + end + + TRUNCATE_SPECIAL_TYPES_SQL = <<~'SQL' + TRUNCATE TABLE postgres_special_types + SQL + + CREATE_AUTHOR_RESULT_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES ($1, $2) + SQL + + CREATE_AUTHORS_BATCH_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES ($1, $2) + SQL + + class CreateAuthorsBatchArgs < Data.define(:name, :bio) + end + + GET_AUTHOR_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE id = $1 + SQL + + class GetAuthorBatchRow < Data.define(:id, :name, :bio) + end + + class GetAuthorBatchArgs < Data.define(:id) + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE name = $1 ORDER BY id + SQL + + class ListAuthorsByNameBatchRow < Data.define(:id, :name, :bio) + end + + class ListAuthorsByNameBatchArgs < Data.define(:name) + end + CREATE_REGRESSION_SQL = <<~'SQL' INSERT INTO regressions (id, "order", payload, doc, bio_types, uids) VALUES ($1, $2, $3, $4, $5, $6) RETURNING id, "order", payload, doc, bio_types, uids SQL @@ -327,6 +382,9 @@ def transaction(isolation: nil) end end + class ExecResult < Data.define(:rows_affected, :last_insert_id) + end + # Returns the author with the given id, or nil when there is none. def get_author(id:) with_client('get_author') do |client| @@ -756,6 +814,119 @@ def list_authors_between_ids(id:, id_2:) end end + def insert_special_types(c_timetz:, c_tsvector:, c_point:, c_int4range:, c_oid:, c_dimensions:, c_positive:) + with_client('insert_special_types') do |client| + query_params = [ + c_timetz, + c_tsvector, + c_point, + c_int4range, + c_oid, + c_dimensions, + c_positive + ] + unless client.prepared_statements.include?('query_sql_insert_special_types') + client.prepare('query_sql_insert_special_types', INSERT_SPECIAL_TYPES_SQL) + client.prepared_statements.add('query_sql_insert_special_types') + end + client.exec_prepared('query_sql_insert_special_types', query_params) + nil + end + end + + def get_special_types + with_client('get_special_types') do |client| + unless client.prepared_statements.include?('query_sql_get_special_types') + client.prepare('query_sql_get_special_types', GET_SPECIAL_TYPES_SQL) + client.prepared_statements.add('query_sql_get_special_types') + end + result = client.exec_prepared('query_sql_get_special_types') + row = result.first + row && GetSpecialTypesRow.new( + row['c_timetz'], + row['c_tsvector'], + row['c_point'], + row['c_int4range'], + row['c_oid'], + row['c_dimensions'], + row['c_positive'] + ) + end + end + + # Like #get_special_types, but raises NotFound when there is no row. + def get_special_types!(...) + get_special_types(...) || raise(NotFound, 'GetSpecialTypes returned no row') + end + + def truncate_special_types + with_client('truncate_special_types') do |client| + unless client.prepared_statements.include?('query_sql_truncate_special_types') + client.prepare('query_sql_truncate_special_types', TRUNCATE_SPECIAL_TYPES_SQL) + client.prepared_statements.add('query_sql_truncate_special_types') + end + client.exec_prepared('query_sql_truncate_special_types') + nil + end + end + + def create_author_result(name:, bio:) + with_client('create_author_result') do |client| + query_params = [name, bio] + unless client.prepared_statements.include?('query_sql_create_author_result') + client.prepare('query_sql_create_author_result', CREATE_AUTHOR_RESULT_SQL) + client.prepared_statements.add('query_sql_create_author_result') + end + result = client.exec_prepared('query_sql_create_author_result', query_params) + ExecResult.new(result.cmd_tuples, nil) + end + end + + def create_authors_batch(rows) + with_client('create_authors_batch') do |client| + unless client.prepared_statements.include?('query_sql_create_authors_batch') + client.prepare('query_sql_create_authors_batch', CREATE_AUTHORS_BATCH_SQL) + client.prepared_statements.add('query_sql_create_authors_batch') + end + rows.each do |args| + query_params = [args.name, args.bio] + client.exec_prepared('query_sql_create_authors_batch', query_params) + end + nil + end + end + + def get_author_batch(rows) + with_client('get_author_batch') do |client| + unless client.prepared_statements.include?('query_sql_get_author_batch') + client.prepare('query_sql_get_author_batch', GET_AUTHOR_BATCH_SQL) + client.prepared_statements.add('query_sql_get_author_batch') + end + rows.map do |args| + query_params = [args.id] + result = client.exec_prepared('query_sql_get_author_batch', query_params) + row = result.first + row && GetAuthorBatchRow.new(row['id'], row['name'], row['bio']) + end + end + end + + def list_authors_by_name_batch(rows) + with_client('list_authors_by_name_batch') do |client| + unless client.prepared_statements.include?('query_sql_list_authors_by_name_batch') + client.prepare('query_sql_list_authors_by_name_batch', LIST_AUTHORS_BY_NAME_BATCH_SQL) + client.prepared_statements.add('query_sql_list_authors_by_name_batch') + end + rows.map do |args| + query_params = [args.name] + result = client.exec_prepared('query_sql_list_authors_by_name_batch', query_params) + result.map do |row| + ListAuthorsByNameBatchRow.new(row['id'], row['name'], row['bio']) + end + end + end + end + def create_regression(id:, order:, payload:, doc:, bio_types:, uids:) with_client('create_regression') do |client| query_params = [ diff --git a/examples/pg/query_sql.rbs b/examples/pg/query_sql.rbs index fdbbe0c..7d2cea6 100644 --- a/examples/pg/query_sql.rbs +++ b/examples/pg/query_sql.rbs @@ -228,6 +228,69 @@ module PgCodegen def initialize: ( Integer id, String name, String? bio) -> void end + INSERT_SPECIAL_TYPES_SQL: String + + GET_SPECIAL_TYPES_SQL: String + + class GetSpecialTypesRow + attr_reader c_timetz: String? + attr_reader c_tsvector: String? + attr_reader c_point: String? + attr_reader c_int4range: String? + attr_reader c_oid: Integer? + attr_reader c_dimensions: String? + attr_reader c_positive: untyped + def initialize: ( + String? c_timetz, + String? c_tsvector, + String? c_point, + String? c_int4range, + Integer? c_oid, + String? c_dimensions, + untyped c_positive + ) -> void + end + + TRUNCATE_SPECIAL_TYPES_SQL: String + + CREATE_AUTHOR_RESULT_SQL: String + + CREATE_AUTHORS_BATCH_SQL: String + + class CreateAuthorsBatchArgs + attr_reader name: String + attr_reader bio: String? + def initialize: ( String name, String? bio) -> void + end + + GET_AUTHOR_BATCH_SQL: String + + class GetAuthorBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class GetAuthorBatchArgs + attr_reader id: Integer + def initialize: ( Integer id) -> void + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL: String + + class ListAuthorsByNameBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class ListAuthorsByNameBatchArgs + attr_reader name: String + def initialize: ( String name) -> void + end + CREATE_REGRESSION_SQL: String class CreateRegressionRow @@ -293,6 +356,12 @@ module PgCodegen def transaction: [T] (?isolation: (:read_uncommitted | :read_committed | :repeatable_read | :serializable)?) { (self) -> T } -> T + class ExecResult + attr_reader rows_affected: Integer + attr_reader last_insert_id: Integer? + def initialize: ( Integer rows_affected, Integer? last_insert_id) -> void + end + def get_author: (id: Integer) -> GetAuthorRow? def get_author!: (id: Integer) -> GetAuthorRow @@ -361,6 +430,22 @@ module PgCodegen def list_authors_between_ids: (id: Integer, id_2: Integer) -> Array[ListAuthorsBetweenIdsRow] + def insert_special_types: (c_timetz: String?, c_tsvector: String?, c_point: String?, c_int4range: String?, c_oid: Integer?, c_dimensions: String?, c_positive: untyped) -> void + + def get_special_types: -> GetSpecialTypesRow? + + def get_special_types!: -> GetSpecialTypesRow + + def truncate_special_types: -> void + + def create_author_result: (name: String, bio: String?) -> ExecResult + + def create_authors_batch: (Array[CreateAuthorsBatchArgs] rows) -> void + + def get_author_batch: (Array[GetAuthorBatchArgs] rows) -> Array[GetAuthorBatchRow?] + + def list_authors_by_name_batch: (Array[ListAuthorsByNameBatchArgs] rows) -> Array[Array[ListAuthorsByNameBatchRow]] + def create_regression: (id: Integer, order: String, payload: String?, doc: untyped, bio_types: Array[String]?, uids: Array[String]?) -> CreateRegressionRow? def create_regression!: (id: Integer, order: String, payload: String?, doc: untyped, bio_types: Array[String]?, uids: Array[String]?) -> CreateRegressionRow diff --git a/examples/sqlite3/models.rb b/examples/sqlite3/models.rb index 7df9524..3d30bfb 100644 --- a/examples/sqlite3/models.rb +++ b/examples/sqlite3/models.rb @@ -9,6 +9,15 @@ class Author < Data.define(:id, :name, :bio) class Book < Data.define(:id, :name, :author_id, :description) end + class DeclaredType < Data.define( + :c_varchar, + :c_decimal, + :c_unsigned_big_int, + :c_native, + :c_double + ) + end + class Regression < Data.define(:id, :order, :price) end end diff --git a/examples/sqlite3/models.rbs b/examples/sqlite3/models.rbs index afe0090..10e489a 100644 --- a/examples/sqlite3/models.rbs +++ b/examples/sqlite3/models.rbs @@ -15,6 +15,21 @@ module Sqlite3Codegen def initialize: ( Integer id, String name, Integer author_id, String? description) -> void end + class DeclaredType + attr_reader c_varchar: String? + attr_reader c_decimal: Float? + attr_reader c_unsigned_big_int: Integer? + attr_reader c_native: String? + attr_reader c_double: Float? + def initialize: ( + String? c_varchar, + Float? c_decimal, + Integer? c_unsigned_big_int, + String? c_native, + Float? c_double + ) -> void + end + class Regression attr_reader id: Integer attr_reader order: String diff --git a/examples/sqlite3/query_sql.rb b/examples/sqlite3/query_sql.rb index e693317..5b3b136 100644 --- a/examples/sqlite3/query_sql.rb +++ b/examples/sqlite3/query_sql.rb @@ -127,6 +127,58 @@ class GetDuplicateAuthorsRow < Data.define(:author, :author2) class GetAuthorsByBookNameRow < Data.define(:id, :name, :bio, :book) end + CREATE_AUTHOR_RESULT_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES (?, ?) + SQL + + CREATE_AUTHORS_BATCH_SQL = <<~'SQL' + INSERT INTO authors (name, bio) VALUES (?, ?) + SQL + + class CreateAuthorsBatchArgs < Data.define(:name, :bio) + end + + GET_AUTHOR_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE id = ? + SQL + + class GetAuthorBatchRow < Data.define(:id, :name, :bio) + end + + class GetAuthorBatchArgs < Data.define(:id) + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL = <<~'SQL' + SELECT id, name, bio FROM authors WHERE name = ? ORDER BY id + SQL + + class ListAuthorsByNameBatchRow < Data.define(:id, :name, :bio) + end + + class ListAuthorsByNameBatchArgs < Data.define(:name) + end + + INSERT_DECLARED_TYPES_SQL = <<~'SQL' + INSERT INTO declared_types (c_varchar, c_decimal, c_unsigned_big_int, c_native, c_double) VALUES (?, ?, ?, ?, ?) + SQL + + GET_DECLARED_TYPES_SQL = <<~'SQL' + SELECT c_varchar, c_decimal, c_unsigned_big_int, c_native, c_double FROM declared_types LIMIT 1 + SQL + + class GetDeclaredTypesRow < Data.define( + :c_varchar, + :c_decimal, + :c_unsigned_big_int, + :c_native, + :c_double + ) + end + + TRUNCATE_DECLARED_TYPES_SQL = <<~'SQL' + DELETE FROM declared_types + SQL + CREATE_REGRESSION_SQL = <<~'SQL' INSERT INTO regressions ("order", price) VALUES (?, ?) RETURNING id SQL @@ -198,6 +250,9 @@ def transaction(mode: nil) end end + class ExecResult < Data.define(:rows_affected, :last_insert_id) + end + # Returns the author with the given id, or nil when there is none. def get_author(id:) with_client('get_author') do |client| @@ -432,6 +487,122 @@ def get_authors_by_book_name(name:) end end + def create_author_result(name:, bio:) + with_client('create_author_result') do |client| + query_params = [name, bio] + stmt = client.prepare(CREATE_AUTHOR_RESULT_SQL) + begin + stmt.execute(*query_params) + ExecResult.new(client.changes, client.last_insert_row_id) + ensure + stmt.close + end + end + end + + def create_authors_batch(rows) + with_client('create_authors_batch') do |client| + stmt = client.prepare(CREATE_AUTHORS_BATCH_SQL) + begin + rows.each do |args| + query_params = [args.name, args.bio] + stmt.execute(*query_params) + end + nil + ensure + stmt.close + end + end + end + + def get_author_batch(rows) + with_client('get_author_batch') do |client| + stmt = client.prepare(GET_AUTHOR_BATCH_SQL) + begin + rows.map do |args| + query_params = [args.id] + result = stmt.execute(*query_params) + row = result.first + row && GetAuthorBatchRow.new(row[0], row[1], row[2]) + end + ensure + stmt.close + end + end + end + + def list_authors_by_name_batch(rows) + with_client('list_authors_by_name_batch') do |client| + stmt = client.prepare(LIST_AUTHORS_BY_NAME_BATCH_SQL) + begin + rows.map do |args| + query_params = [args.name] + result = stmt.execute(*query_params) + result.map do |row| + ListAuthorsByNameBatchRow.new(row[0], row[1], row[2]) + end + end + ensure + stmt.close + end + end + end + + def insert_declared_types(c_varchar:, c_decimal:, c_unsigned_big_int:, c_native:, c_double:) + with_client('insert_declared_types') do |client| + query_params = [ + c_varchar, + c_decimal, + c_unsigned_big_int, + c_native, + c_double + ] + stmt = client.prepare(INSERT_DECLARED_TYPES_SQL) + begin + stmt.execute(*query_params) + nil + ensure + stmt.close + end + end + end + + def get_declared_types + with_client('get_declared_types') do |client| + stmt = client.prepare(GET_DECLARED_TYPES_SQL) + begin + result = stmt.execute + row = result.first + row && GetDeclaredTypesRow.new( + row[0], + row[1], + row[2], + row[3], + row[4] + ) + ensure + stmt.close + end + end + end + + # Like #get_declared_types, but raises NotFound when there is no row. + def get_declared_types!(...) + get_declared_types(...) || raise(NotFound, 'GetDeclaredTypes returned no row') + end + + def truncate_declared_types + with_client('truncate_declared_types') do |client| + stmt = client.prepare(TRUNCATE_DECLARED_TYPES_SQL) + begin + stmt.execute + nil + ensure + stmt.close + end + end + end + def create_regression(order:, price:) with_client('create_regression') do |client| query_params = [order, price] diff --git a/examples/sqlite3/query_sql.rbs b/examples/sqlite3/query_sql.rbs index 4cb7912..26c6770 100644 --- a/examples/sqlite3/query_sql.rbs +++ b/examples/sqlite3/query_sql.rbs @@ -108,6 +108,65 @@ module Sqlite3Codegen def initialize: ( Integer id, String name, String? bio, Book book) -> void end + CREATE_AUTHOR_RESULT_SQL: String + + CREATE_AUTHORS_BATCH_SQL: String + + class CreateAuthorsBatchArgs + attr_reader name: String + attr_reader bio: String? + def initialize: ( String name, String? bio) -> void + end + + GET_AUTHOR_BATCH_SQL: String + + class GetAuthorBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class GetAuthorBatchArgs + attr_reader id: Integer + def initialize: ( Integer id) -> void + end + + LIST_AUTHORS_BY_NAME_BATCH_SQL: String + + class ListAuthorsByNameBatchRow + attr_reader id: Integer + attr_reader name: String + attr_reader bio: String? + def initialize: ( Integer id, String name, String? bio) -> void + end + + class ListAuthorsByNameBatchArgs + attr_reader name: String + def initialize: ( String name) -> void + end + + INSERT_DECLARED_TYPES_SQL: String + + GET_DECLARED_TYPES_SQL: String + + class GetDeclaredTypesRow + attr_reader c_varchar: String? + attr_reader c_decimal: Float? + attr_reader c_unsigned_big_int: Integer? + attr_reader c_native: String? + attr_reader c_double: Float? + def initialize: ( + String? c_varchar, + Float? c_decimal, + Integer? c_unsigned_big_int, + String? c_native, + Float? c_double + ) -> void + end + + TRUNCATE_DECLARED_TYPES_SQL: String + CREATE_REGRESSION_SQL: String class CreateRegressionRow @@ -154,6 +213,12 @@ module Sqlite3Codegen def transaction: [T] (?mode: (:deferred | :immediate | :exclusive)?) { (self) -> T } -> T + class ExecResult + attr_reader rows_affected: Integer + attr_reader last_insert_id: Integer? + def initialize: ( Integer rows_affected, Integer? last_insert_id) -> void + end + def get_author: (id: Integer) -> GetAuthorRow? def get_author!: (id: Integer) -> GetAuthorRow @@ -188,6 +253,22 @@ module Sqlite3Codegen def get_authors_by_book_name: (name: String) -> Array[GetAuthorsByBookNameRow] + def create_author_result: (name: String, bio: String?) -> ExecResult + + def create_authors_batch: (Array[CreateAuthorsBatchArgs] rows) -> void + + def get_author_batch: (Array[GetAuthorBatchArgs] rows) -> Array[GetAuthorBatchRow?] + + def list_authors_by_name_batch: (Array[ListAuthorsByNameBatchArgs] rows) -> Array[Array[ListAuthorsByNameBatchRow]] + + def insert_declared_types: (c_varchar: String?, c_decimal: Float?, c_unsigned_big_int: Integer?, c_native: String?, c_double: Float?) -> void + + def get_declared_types: -> GetDeclaredTypesRow? + + def get_declared_types!: -> GetDeclaredTypesRow + + def truncate_declared_types: -> void + def create_regression: (order: String, price: Float?) -> void def update_regression_prices: (price: Float?) -> Integer diff --git a/lib/sqlc_gen_ruby/catalog.rb b/lib/sqlc_gen_ruby/catalog.rb index fd9b68b..0382935 100644 --- a/lib/sqlc_gen_ruby/catalog.rb +++ b/lib/sqlc_gen_ruby/catalog.rb @@ -7,24 +7,35 @@ module SqlcGenRuby class Catalog INTERNAL_SCHEMAS = %w[pg_catalog information_schema pg_temp].freeze - Table = Struct.new(:schema, :name, :columns, :model_name) + Table = Struct.new(:schema, :name, :columns, :model_name, :comment) attr_reader :tables, :enums - def initialize(proto_catalog) + # rename: { db name => Ruby name } (tables -> model classes, columns -> attributes) + # exact_table_names: authors => Authors instead of Author + def initialize(proto_catalog, rename: {}, exact_table_names: false) + @rename = rename + @exact_table_names = exact_table_names @default_schema = proto_catalog&.default_schema.to_s schemas = (proto_catalog&.schemas || []).reject { |s| INTERNAL_SCHEMAS.include?(s.name) } @tables = schemas.flat_map do |schema| schema.tables.map do |t| - Table.new(schema.name, t.rel.name, t.columns, nil) + Table.new(schema.name, t.rel.name, t.columns, nil, t.comment) end end assign_model_names # qualified enum name => values; default-schema enums are reachable both bare and qualified @enums = {} + @enum_comments = {} + @composites = schemas.flat_map do |schema| + schema.composite_types.flat_map do |c| + default_schema?(schema.name) ? [c.name, "#{schema.name}.#{c.name}"] : ["#{schema.name}.#{c.name}"] + end + end schemas.each do |schema| schema.enums.each do |e| @enums["#{schema.name}.#{e.name}"] = e.vals.to_a + @enum_comments[e.name] = @enum_comments["#{schema.name}.#{e.name}"] = e.comment @enums[e.name] = e.vals.to_a if default_schema?(schema.name) end end @@ -32,14 +43,20 @@ def initialize(proto_catalog) def enum?(type) = @enums.key?(qualified_name(type)) + # CREATE TYPE ... AS (...): the pg gem returns the record text, e.g. "(1,abc)" + def composite?(type) = @composites.include?(qualified_name(type)) + def enum_values(type) = @enums[qualified_name(type)] # Enums with their Ruby module names, one entry per distinct enum. def enum_modules @enums.reject { |name, _| name.include?('.') && default_schema?(name.split('.').first) } - .map { |name, vals| [Code.camelize(name), vals] } + .map { |name, vals| [Code.camelize(name), vals, @enum_comments[name]] } end + # Column name after `rename:` (rows and models). + def column_name(name) = @rename.fetch(name, name) + # embed_table: Identifier(schema, name) as sqlc sends it for sqlc.embed def table_for(identifier) @tables.find do |t| @@ -62,10 +79,11 @@ def qualified_name(type) # authors => Author; when two tables would share a model (author and authors) the plural one keeps its plural name. def assign_model_names - @tables.each { |t| t.model_name = model_name(t, Code.singularize(t.name)) } + @tables.each { |t| t.model_name = model_name(t, @exact_table_names ? t.name : Code.singularize(t.name)) } @tables.group_by(&:model_name).each_value do |same| same.each { |t| t.model_name = model_name(t, t.name) } if same.size > 1 end + @tables.each { |t| t.model_name = renamed_model(t) if renamed_model(t) } @tables.group_by(&:model_name).each do |name, same| next if same.size == 1 @@ -74,6 +92,15 @@ def assign_model_names end end + # rename: { "authors": "Writer" } or { "extended.bios": "Bio" } + def renamed_model(table) + name = @rename["#{table.schema}.#{table.name}"] || (default_schema?(table.schema) && @rename[table.name]) + return nil unless name + raise ArgumentError, "rename #{table.name}: #{name.inspect} is not a Ruby class name" unless name.match?(/\A[A-Z]\w*\z/) + + name + end + def model_name(table, base) name = Code.camelize(base) default_schema?(table.schema) ? name : Code.camelize(table.schema) + name diff --git a/lib/sqlc_gen_ruby/code.rb b/lib/sqlc_gen_ruby/code.rb index 57f2ddc..2b24e67 100644 --- a/lib/sqlc_gen_ruby/code.rb +++ b/lib/sqlc_gen_ruby/code.rb @@ -108,6 +108,13 @@ def constant_part(str) name.match?(/\A[0-9_]/) ? "Q#{name}" : name end + # "Line 1\nLine 2" => "# Line 1\n# Line 2" ('' for no comment) + def comment(text) + return '' if text.to_s.strip.empty? + + text.to_s.strip.lines.map { |l| "# #{l.rstrip}".rstrip }.join("\n") + end + # GetAuthor => GET_AUTHOR_SQL def sql_constant(query_name) = "#{snake_case(query_name).upcase}_SQL" diff --git a/lib/sqlc_gen_ruby/drivers/base.rb b/lib/sqlc_gen_ruby/drivers/base.rb index f3e7b65..1170d5a 100644 --- a/lib/sqlc_gen_ruby/drivers/base.rb +++ b/lib/sqlc_gen_ruby/drivers/base.rb @@ -75,13 +75,16 @@ def quote_identifier(name) = %("#{name.gsub('"', '""')}") # Ruby expression that turns a parameter value into what the driver can bind (e.g. bytea escaping). def encode_param(expr, _column) = expr + # :execresult -> [*statements, rows_affected expression, last_insert_id expression] + def exec_result_lines(_exec) = raise(NotImplementedError) + # Private helper methods the class needs for these queries: [[code, rbs], ...] def helper_methods(_queries) = [] # Ruby/RBS type of a column's DB type (no nullability / array wrapper). Enums are Strings. # With fallback: nil unknown types return it instead of raising (used for models, which cover every table). def column_type(column, fallback: :raise) - return 'String' if catalog&.enum?(column.type) + return 'String' if catalog&.enum?(column.type) || catalog&.composite?(column.type) db_type = normalize_type(column.type.name) return 'untyped' if db_type == 'any' # sqlc could not infer the type of the expression @@ -94,6 +97,8 @@ def column_type(column, fallback: :raise) end return mapping.ruby_type if matched end + affinity = affinity_type(db_type) + return affinity if affinity return fallback unless fallback == :raise raise ArgumentError, "Unsupported column type: #{column.type.name}" @@ -149,6 +154,9 @@ class NotFound < StandardError; end # from sqlc - char(5), int4, pg_catalog.int8, BLOB, VARCHAR(255), ... => char, int4, int8, blob, varchar def normalize_type(type_name) = type_name.downcase.delete_prefix('pg_catalog.').sub(/\s*\(.*\)\z/, '') + # Type of a declared type no mapping matched (SQLite's type affinity), nil for none. + def affinity_type(_db_type) = nil + def driver_gems = raise(NotImplementedError) def column_mappings = raise(NotImplementedError) diff --git a/lib/sqlc_gen_ruby/drivers/mysql2.rb b/lib/sqlc_gen_ruby/drivers/mysql2.rb index c795583..5270402 100644 --- a/lib/sqlc_gen_ruby/drivers/mysql2.rb +++ b/lib/sqlc_gen_ruby/drivers/mysql2.rb @@ -50,6 +50,8 @@ def exec_rows_lines(exec, _query) = [exec, 'stmt.affected_rows'] def exec_last_id_lines(exec, _query) = [exec, 'client.last_id'] + def exec_result_lines(exec) = [exec, 'stmt.affected_rows', 'client.last_id'] + # LOAD DATA LOCAL INFILE from a temporary CSV (the client needs `local_infile: true`, the server # `local_infile=1`). def copy_from_lines(rows_var, table, columns, values, label) diff --git a/lib/sqlc_gen_ruby/drivers/pg.rb b/lib/sqlc_gen_ruby/drivers/pg.rb index fb7f9f4..6b380b2 100644 --- a/lib/sqlc_gen_ruby/drivers/pg.rb +++ b/lib/sqlc_gen_ruby/drivers/pg.rb @@ -60,6 +60,9 @@ def execute_stmt(stmt_name, const, has_params, array_rows: false) def exec_rows_lines(exec, _query) = ["result = #{exec}", 'result.cmd_tuples'] + # PostgreSQL has no last insert id (use RETURNING). + def exec_result_lines(exec) = ["result = #{exec}", 'result.cmd_tuples', 'nil'] + # Relies on a user-written `RETURNING `; nil when no row is returned (e.g. ON CONFLICT DO NOTHING). def exec_last_id_lines(exec, query) if query.columns.empty? @@ -245,14 +248,18 @@ def column_mappings mapping('bool', %w[bool boolean]), mapping('String', %w[char bpchar varchar text time uuid money macaddr macaddr8 interval xml name citext bit varbit bytea tinytext mediumtext longtext tinyblob blob mediumblob longblob - binary varbinary]), - mapping('Integer', %w[int2 int4 int8 serial bigserial integer smallint bigint int smallserial]), + binary varbinary timetz tsvector tsquery hstore ltree lquery ltxtquery + point line lseg box path polygon circle int4range int8range numrange tsrange + tstzrange daterange int4multirange int8multirange nummultirange tsmultirange + tstzmultirange datemultirange regclass regproc regtype pg_lsn int2vector oidvector + tid xid cid]), + mapping('Integer', %w[int2 int4 int8 serial bigserial integer smallint bigint int smallserial oid]), mapping('Float', %w[float4 float8 real double]), mapping('BigDecimal', %w[numeric decimal]), mapping('Date', %w[date]), mapping('Time', %w[timestamp timestamptz datetime]), mapping('IPAddr', %w[inet cidr]), - mapping('untyped', %w[json jsonb]) + mapping('untyped', %w[json jsonb any anyelement anyarray record void unknown]) ] end end diff --git a/lib/sqlc_gen_ruby/drivers/sqlite3.rb b/lib/sqlc_gen_ruby/drivers/sqlite3.rb index b712497..95b5652 100644 --- a/lib/sqlc_gen_ruby/drivers/sqlite3.rb +++ b/lib/sqlc_gen_ruby/drivers/sqlite3.rb @@ -45,6 +45,8 @@ def exec_lines(exec, query) = [query.columns.empty? ? exec : "#{exec}.to_a", 'ni def exec_rows_lines(exec, query) = [query.columns.empty? ? exec : "#{exec}.to_a", 'client.changes'] + def exec_result_lines(exec) = [exec, 'client.changes', 'client.last_insert_row_id'] + # Relies on a user-written `RETURNING `. def exec_last_id_lines(exec, query) if query.columns.empty? @@ -103,6 +105,19 @@ def transaction(mode: nil) private + # VARCHAR(255) -> varchar, UNSIGNED BIG INT -> unsignedbigint + def normalize_type(type_name) = super.sub(/\(.*\z/m, '').delete(" \t\n") + + # https://www.sqlite.org/datatype3.html#determination_of_column_affinity + def affinity_type(db_type) + case db_type + when /int/ then 'Integer' + when /char|clob|text/, /blob/, '' then 'String' + when /real|floa|doub/ then 'Float' + else 'Float' # NUMERIC affinity, see decimal / numeric above + end + end + def driver_gems = %w[sqlite3] def column_mappings diff --git a/lib/sqlc_gen_ruby/generator.rb b/lib/sqlc_gen_ruby/generator.rb index 3d932da..24f130c 100644 --- a/lib/sqlc_gen_ruby/generator.rb +++ b/lib/sqlc_gen_ruby/generator.rb @@ -20,9 +20,10 @@ class Generator CONNECTION_FILE = 'connection' def generate(request) - @options = Options.new(request.plugin_options) + @options = Options.new(request.plugin_options, request.global_options) + check_engine(request.settings&.engine.to_s) @driver = instantiate_driver - @catalog = Catalog.new(request.catalog) + @catalog = Catalog.new(request.catalog, rename: @options.rename, exact_table_names: @options.exact_table_names) @driver.catalog = @catalog @types = TypeResolver.new(@driver, @options) @layout = RowLayout.new(@driver, @catalog, @types) @@ -52,6 +53,15 @@ def name_anonymous_params(queries) end end + ENGINES = { 'pg' => 'postgresql', 'mysql2' => 'mysql', 'sqlite3' => 'sqlite' }.freeze + + def check_engine(engine) + expected = ENGINES.fetch(@options.driver_name) + return if engine.empty? || engine == expected + + raise ArgumentError, "driver #{@options.driver_name} needs engine: #{expected} (this sql block uses #{engine})" + end + # Everything is defined in one module: models, enums, query classes, rows, ... must not share a name. def check_constants(queries) names = @catalog.enum_modules.map { |name, _| [name, 'enum'] } @@ -137,7 +147,7 @@ def query_members(query, method_gen) members << [@driver.query_text_constant(query), "#{Code.sql_constant(query.name)}: String"] end members << dataclass("#{Code.upcase_first(query.name)}Row", row_pairs(query)) unless query.columns.empty? - if query.cmd == ':copyfrom' + if MethodGen::ROWS_COMMANDS.include?(query.cmd) pairs = method_gen.params(query).reject(&:reused).map { |p| [p.name, p.type] } members << dataclass("#{Code.upcase_first(query.name)}Args", pairs) end @@ -149,6 +159,10 @@ def row_pairs(query) = @layout.entries(query).map { |e| [e.attr, e.type] } # constructor / connect / transaction, one method per query, then the private helpers def class_members(queries, method_gen) members = @driver.class_methods.dup + if queries.any? { |q| q.cmd == ':execresult' } + # last_insert_id is nil on PostgreSQL (use RETURNING) + members << dataclass('ExecResult', [%w[rows_affected Integer], ['last_insert_id', 'Integer?']]) + end helpers = @driver.private_methods + method_gen.helper_methods(queries) check_method_names(queries, (members + helpers).flat_map { |code, _| code.scan(/^def (?:self\.)?([\w?!]+)/) }.flatten + Code::OBJECT_METHODS) @@ -175,27 +189,38 @@ def check_method_names(queries, generated) # --- data classes (Data.define / Struct.new) --- - def dataclass(name, pairs) + # doc: comment of the class (table comment), attr_docs: { attr => comment } (column comments, RBS only) + def dataclass(name, pairs, doc: nil, attr_docs: {}) dupes = pairs.map(&:first).tally.select { |_, n| n > 1 }.keys raise ArgumentError, "duplicate column names in #{name}: #{dupes.join(', ')}" unless dupes.empty? - @options.ruby_version.immutable_data_supported? ? data_define(name, pairs) : new_struct(name, pairs) + code, rbs = @options.ruby_version.immutable_data_supported? ? data_define(name, pairs, attr_docs) : new_struct(name, pairs, attr_docs) + comment = Code.comment(doc) + comment.empty? ? [code, rbs] : ["#{comment}\n#{code}", "#{comment}\n#{rbs}"] + end + + def attribute_lines(pairs, attr_docs, keyword) + pairs.map do |n, t| + comment = Code.comment(attr_docs[n]) + line = "#{keyword} #{n}: #{t}" + comment.empty? ? line : "#{comment}\n#{line}" + end.join("\n") end - def data_define(name, pairs) + def data_define(name, pairs, attr_docs = {}) attrs = Code.comma_join(pairs.map { |n, _| ":#{n}" }) code = "class #{name} < Data.define(#{attrs})\nend" - readers = Code.indent(pairs.map { |n, t| "attr_reader #{n}: #{t}" }.join("\n")) + readers = Code.indent(attribute_lines(pairs, attr_docs, 'attr_reader')) init_args = Code.indent(Code.comma_join(pairs.map { |n, t| "#{t} #{n}" })) init = Code.indent("def initialize: (#{init_args}) -> void") [code, "class #{name}\n#{readers}\n#{init}\nend"] end # Ruby < 3.2: keyword_init so that rows are built like Data rows (`Row.new(id: 1, name: 'x')`). - def new_struct(name, pairs) + def new_struct(name, pairs, attr_docs = {}) attrs = Code.comma_join(pairs.map { |n, _| ":#{n}" } + ['keyword_init: true']) code = "#{name} = Struct.new(#{attrs})" - accessors = Code.indent(pairs.map { |n, t| "attr_accessor #{n}: #{t}" }.join("\n")) + accessors = Code.indent(attribute_lines(pairs, attr_docs, 'attr_accessor')) init_args = Code.indent(Code.comma_join(pairs.map { |n, t| "#{n}: #{t}" })) init = Code.indent("def initialize: (#{init_args}) -> void") new = Code.indent("def self.new: (#{init_args}) -> #{name}") @@ -205,12 +230,14 @@ def new_struct(name, pairs) # --- models.rb: one class per table + one module per enum --- def models_files - members = @catalog.enum_modules.map { |name, vals| enum_module(name, vals) } + members = @catalog.enum_modules.map { |name, vals, comment| enum_module(name, vals, comment) } @catalog.tables.each do |table| next if table.columns.empty? - pairs = table.columns.map { |c| [@types.member_name(c.name), @types.type(c, fallback: 'untyped')] } - members << dataclass(table.model_name, pairs) + ident = Plugin::Identifier.new(schema: table.schema, name: table.name) + pairs = table.columns.map { |c| [@types.member_name(@catalog.column_name(c.name)), @types.type(c, table: ident, fallback: 'untyped')] } + docs = table.columns.to_h { |c| [@types.member_name(@catalog.column_name(c.name)), c.comment.to_s] } + members << dataclass(table.model_name, pairs, doc: table.comment, attr_docs: docs) end return [] if members.empty? @@ -238,13 +265,15 @@ def connection_files end # module BioType; AUTOBIOGRAPHY = 'Autobiography'; ...; ALL = [...].freeze; end - def enum_module(name, values) + def enum_module(name, values, comment = nil) consts = unique_constants(values) code = consts.map { |const, value| "#{const} = #{quote(value)}" } code << "ALL = [#{consts.map(&:first).join(', ')}].freeze" rbs = consts.map { |const, _| "#{const}: String" } rbs << 'ALL: Array[String]' - [Code.block("module #{name}", code.join("\n")), Code.block("module #{name}", rbs.join("\n"))] + doc = Code.comment(comment) + doc = doc.empty? ? '' : "#{doc}\n" + ["#{doc}#{Code.block("module #{name}", code.join("\n"))}", "#{doc}#{Code.block("module #{name}", rbs.join("\n"))}"] end # 'Memoir' (single quotes unless the value needs escaping) diff --git a/lib/sqlc_gen_ruby/method_gen.rb b/lib/sqlc_gen_ruby/method_gen.rb index 4c5ba4f..b6f14f2 100644 --- a/lib/sqlc_gen_ruby/method_gen.rb +++ b/lib/sqlc_gen_ruby/method_gen.rb @@ -12,6 +12,11 @@ class MethodGen 'def slice_placeholders: (Array[untyped] values) -> String' ].freeze + # :batch* run the query once per element of an Array of Args, on one connection + BATCH_COMMANDS = { ':batchexec' => ':exec', ':batchone' => ':one', ':batchmany' => ':many' }.freeze + # commands whose method takes `rows` (an Array of Args) instead of keyword arguments + ROWS_COMMANDS = [':copyfrom', *BATCH_COMMANDS.keys].freeze + # optional: nullable and named by the user (sqlc.arg / sqlc.narg) -> keyword argument defaulting to nil # reused: a later occurrence of a named parameter (MySQL sends one parameter per `?`): bound, not a keyword Param = Struct.new(:name, :column, :slice, :optional, :type, :reused) @@ -29,7 +34,7 @@ def initialize(driver, layout, types, class_name) # `created_at > $1 AND created_at < $2` -> created_at:, created_at_2: def params(query) - copy = query.cmd == ':copyfrom' + copy = ROWS_COMMANDS.include?(query.cmd) by_name = {} query.params.map do |p| column = p.column @@ -54,6 +59,9 @@ def code(query) if query.cmd == ':copyfrom' statements = copy_statements(query, params) signature = '(rows)' + elsif BATCH_COMMANDS.key?(query.cmd) + statements = batch_statements(query, func, params) + signature = '(rows)' else statements = query_statements(query, func, params) keywords = params.reject(&:reused).map { |p| p.optional ? "#{p.name}: nil" : "#{p.name}:" } @@ -73,7 +81,7 @@ def bang_method(query, func) def type(query) func = Code.snake_case(query.name) - if query.cmd == ':copyfrom' + if ROWS_COMMANDS.include?(query.cmd) signature = "(Array[#{query.name}Args] rows)" else list = params(query).reject(&:reused).map { |p| "#{p.optional ? '?' : ''}#{p.name}: #{p.type}" } @@ -123,6 +131,9 @@ def return_type(query) when ':many' then "Array[#{Code.upcase_first(query.name)}Row]" when ':execrows' then 'Integer' when ':execlastid' then last_id_type(query) + when ':execresult' then 'ExecResult' + when ':batchone' then "Array[#{query.name}Row?]" + when ':batchmany' then "Array[Array[#{query.name}Row]]" else 'void' end end @@ -169,6 +180,29 @@ def query_statements(query, func, params) statements end + # Prepared once, executed for every element of rows. + def batch_statements(query, func, params) + raise ArgumentError, "#{query.name}: sqlc.slice cannot be used in #{query.cmd}" if slice?(query) + + const = Code.sql_constant(query.name) + stmt_name = "#{@statement_prefix}_#{func}" + statements = [@driver.prepare_stmt(stmt_name, const)].compact + cmd = BATCH_COMMANDS.fetch(query.cmd) + body = [] + body << "query_params = [#{Code.comma_join(params.map { |p| "args.#{p.name}" })}]" unless params.empty? + body.concat(command_statements(query, stmt_name, const, !params.empty?, cmd: cmd)) + body.pop if cmd == ':exec' # the trailing nil + block_arg = params.empty? ? '_args' : 'args' + loop = Code.block("rows.#{cmd == ':exec' ? 'each' : 'map'} do |#{block_arg}|", body.join("\n")) + commands = cmd == ':exec' ? [loop, 'nil'] : [loop] + if @driver.closes_statements? + statements << Code.block('begin', commands.join("\n")).sub(/\nend\z/, "\nensure\n stmt.close\nend") + else + statements.concat(commands) + end + statements + end + def query_params(params) list = params.map { |p| p.slice ? "*#{p.name}" : @driver.encode_param(p.name, p.column) } "query_params = [#{Code.comma_join(list)}]" @@ -182,9 +216,9 @@ def slice_sql(const, params) end # The value of the last expression is the return value of the method (no explicit `return`). - def command_statements(query, stmt_name, const, has_params) + def command_statements(query, stmt_name, const, has_params, cmd: query.cmd) exec = @driver.execute_stmt(stmt_name, const, has_params, array_rows: !query.columns.empty? && @layout.array_rows?(query)) - case query.cmd + case cmd when ':one' ["result = #{exec}", 'row = result.first', "row && #{new_row(query)}"] when ':many' @@ -192,6 +226,7 @@ def command_statements(query, stmt_name, const, has_params) when ':exec' then @driver.exec_lines(exec, query) when ':execrows' then @driver.exec_rows_lines(exec, query) when ':execlastid' then @driver.exec_last_id_lines(exec, query) + when ':execresult' then exec_result_lines(exec) else raise ArgumentError, "unsupported query command: #{query.cmd}" end end @@ -223,6 +258,11 @@ def new_row(query) end # Struct rows (Ruby < 3.2) are built with keywords (keyword_init: true), Data rows positionally. + def exec_result_lines(exec) + *lines, rows_affected, last_insert_id = @driver.exec_result_lines(exec) + lines << "ExecResult.new(#{constructor_args(%w[rows_affected last_insert_id], [rows_affected, last_insert_id]).join(', ')})" + end + def constructor_args(names, values) @types.keyword_rows? ? names.zip(values).map { |n, v| "#{n}: #{v}" } : values end diff --git a/lib/sqlc_gen_ruby/options.rb b/lib/sqlc_gen_ruby/options.rb index aedce6a..d190278 100644 --- a/lib/sqlc_gen_ruby/options.rb +++ b/lib/sqlc_gen_ruby/options.rb @@ -7,21 +7,25 @@ module SqlcGenRuby # Port of PluginOptions/Options.cs + RawOptions.cs: parses the JSON `options:` block of the sqlc config. class Options DRIVERS = %w[mysql2 pg sqlite3].freeze - KEYS = %w[driver rubyVersion generateGemfile generateTypes overrides moduleName preparedStatements].freeze + KNOWN = %w[driver generateGemfile generateTypes rubyVersion overrides moduleName preparedStatements rename + exactTableNames].freeze - # `overrides:` entry - column is ":" or "*:"; type/not_null come from `ruby_type:` - Override = Struct.new(:query, :column, :type, :not_null) + # `overrides:` entry, matched by `column:` (":", "*:", "
.", + # ".
.") or `db_type:`; type/not_null come from `ruby_type:` + Override = Struct.new(:query, :column, :type, :not_null, :table, :schema, :db_type, keyword_init: true) attr_reader :driver_name, :ruby_version, :generate_gemfile, :generate_types, :overrides, :module_name, - :prepared_statements - - def initialize(plugin_options_bytes) - text = plugin_options_bytes.to_s.dup.force_encoding(Encoding::UTF_8) - raw = JSON.parse(text) - raise ArgumentError, 'plugin options must be a JSON object' unless raw.is_a?(Hash) - - unknown = raw.keys - KEYS - raise ArgumentError, "unknown option(s): #{unknown.join(', ')} (supported: #{KEYS.join(', ')})" unless unknown.empty? + :prepared_statements, :rename, :exact_table_names + + # global_options_bytes: the plugin's entry of the top-level `options:` of sqlc.yaml; per-codegen options win, + # overrides of both apply (per-codegen first). + def initialize(plugin_options_bytes, global_options_bytes = nil) + raw = parse_object(plugin_options_bytes, 'plugin options') + global = parse_object(global_options_bytes, 'global options') + reject_unknown(global, 'global options') + reject_unknown(raw, 'plugin options') + raw = global.merge(raw).merge('overrides' => raw.fetch('overrides', []) + global.fetch('overrides', []), + 'rename' => global.fetch('rename', {}).merge(raw.fetch('rename', {}))) @driver_name = fetch_string(raw, 'driver', required: true).downcase raise ArgumentError, "unknown driver: #{raw['driver']}" unless DRIVERS.include?(@driver_name) @@ -32,38 +36,96 @@ def initialize(plugin_options_bytes) @overrides = parse_overrides(raw.fetch('overrides', [])) @module_name = raw.key?('moduleName') ? fetch_module_name(raw) : nil # default: Codegen @prepared_statements = fetch_bool(raw, 'preparedStatements', true) + @rename = parse_rename(raw['rename']) + @exact_table_names = fetch_bool(raw, 'exactTableNames', false) if raw.key?('preparedStatements') && @driver_name != 'pg' raise ArgumentError, 'option preparedStatements is only supported by the pg driver' end end - # The override matching a result column / parameter of a query: "Query:col" wins over "*:col". - def override_for(query_name, column_name) - overrides.find { |o| o.query == query_name && o.column == column_name } || - overrides.find { |o| o.query == '*' && o.column == column_name } + # The override for a result column / parameter of a query (query_name nil for model columns); the first match of + # "Query:col", "table.col", "*:col", db_type wins. table: Identifier of the column's table, if known. + def override_for(query_name, column, table: column.table) + name = column.name + overrides.find { |o| o.query && o.query == query_name && o.column == name } || + overrides.find { |o| o.table && table && o.table == table.name && o.column == name && schema_match?(o, table) } || + overrides.find { |o| o.query == '*' && o.column == name } || + overrides.find { |o| o.db_type && o.db_type == normalize_db_type(column.type&.name) } end private + def schema_match?(override, table) = override.schema.nil? || override.schema == table.schema.to_s + + def normalize_db_type(name) = name.to_s.downcase.delete_prefix('pg_catalog.') + + def parse_object(bytes, label) + text = bytes.to_s.dup.force_encoding(Encoding::UTF_8) + return {} if text.strip.empty? && label != 'plugin options' + + raw = JSON.parse(text) + raise ArgumentError, "#{label} must be a JSON object" unless raw.is_a?(Hash) + + raw + end + + def reject_unknown(raw, label) + unknown = raw.keys - KNOWN + return if unknown.empty? + + hints = unknown.map do |key| + near = KNOWN.find { |k| k.casecmp?(key) || k.casecmp?(key.delete('_')) } + near ? "#{key} (did you mean #{near}?)" : key + end + raise ArgumentError, "unknown #{label}: #{hints.join(', ')}; supported: #{KNOWN.join(', ')}" + end + + # rename: { "authors": "Writer", "bio": "biography" } - table name -> model class, column name -> attribute + def parse_rename(value) + raise ArgumentError, 'option rename must be an object' unless value.is_a?(Hash) + + value.each do |key, name| + raise ArgumentError, "rename #{key} must be a string" unless name.is_a?(String) && !name.empty? + end + value + end + def parse_overrides(list) raise ArgumentError, 'option overrides must be a list' unless list.is_a?(Array) list.map do |entry| raise ArgumentError, 'each override must be an object' unless entry.is_a?(Hash) - query, column = entry['column'].to_s.split(':', 2) - raise ArgumentError, "override column must look like ':' or '*:'" if column.to_s.empty? - + label = entry['column'] || entry['db_type'] type = entry['ruby_type'] - raise ArgumentError, "override #{entry['column']} needs ruby_type.type" unless type.is_a?(Hash) && type['type'].is_a?(String) + raise ArgumentError, "override #{label} needs ruby_type.type" unless type.is_a?(Hash) && type['type'].is_a?(String) not_null = type['notNull'] - unless [nil, true, false].include?(not_null) - raise ArgumentError, "override #{entry['column']}: notNull must be a boolean" + raise ArgumentError, "override #{label}: notNull must be a boolean" unless [nil, true, false].include?(not_null) + + override_target(entry).tap do |o| + o.type = type['type'] + o.not_null = not_null end + end + end + + def override_target(entry) + if entry.key?('db_type') + return Override.new(db_type: normalize_db_type(entry['db_type'])) if entry['db_type'].is_a?(String) - Override.new(query, column, type['type'], not_null) + raise ArgumentError, 'override db_type must be a string' + end + spec = entry['column'].to_s + if spec.include?(':') + query, column = spec.split(':', 2) + return Override.new(query: query, column: column) unless query.empty? || column.empty? + else + *schema, table, column = spec.split('.') + return Override.new(table: table, column: column, schema: schema.first) if column && schema.size <= 1 end + raise ArgumentError, "override column must look like ':', '*:' or '
.', " \ + "got #{spec.inspect}" end def fetch_string(raw, key, default: nil, required: false) diff --git a/lib/sqlc_gen_ruby/plugin_pb.rb b/lib/sqlc_gen_ruby/plugin_pb.rb index 1745d1a..454a63a 100644 --- a/lib/sqlc_gen_ruby/plugin_pb.rb +++ b/lib/sqlc_gen_ruby/plugin_pb.rb @@ -66,12 +66,17 @@ class Table < P field :comment, 3, :string end + class CompositeType < P + field :name, 1, :string + field :comment, 2, :string + end + class Schema < P - # composite_types(5) is skipped field :comment, 1, :string field :name, 2, :string field :tables, 3, :message, repeated: true, klass: Table field :enums, 4, :message, repeated: true, klass: Enum + field :composite_types, 5, :message, repeated: true, klass: CompositeType end class Catalog < P @@ -81,8 +86,13 @@ class Catalog < P field :schemas, 4, :message, repeated: true, klass: Schema end + class Settings < P + field :version, 1, :string + field :engine, 2, :string + end + class GenerateRequest < P - # settings(1) is skipped: the generator does not need it. + field :settings, 1, :message, klass: Settings field :catalog, 2, :message, klass: Catalog field :queries, 3, :message, repeated: true, klass: Query field :sqlc_version, 4, :string diff --git a/lib/sqlc_gen_ruby/row_layout.rb b/lib/sqlc_gen_ruby/row_layout.rb index f2d59dc..1048b77 100644 --- a/lib/sqlc_gen_ruby/row_layout.rb +++ b/lib/sqlc_gen_ruby/row_layout.rb @@ -32,7 +32,7 @@ def array_rows?(query) def embed?(column) = !column.embed_table.nil? && !column.embed_table.name.empty? def build(query) - taken = query.columns.reject { |c| embed?(c) }.map { |c| @types.member_name(c.name) } + taken = query.columns.reject { |c| embed?(c) }.map { |c| @types.member_name(@catalog.column_name(c.name)) } # sqlc does not say which side of a join an embedded table is on: in an outer join any of them may be missing outer = query.text.match?(OUTER_JOIN) query.columns.map do |column| @@ -42,7 +42,7 @@ def build(query) name = unique_name(taken, @types.member_name(Code.singularize(column.name))) Entry.new(name, outer ? "#{table.model_name}?" : table.model_name, column, table, outer) else - Entry.new(@types.member_name(column.name), @types.type(column, query_name: query.name), column, nil, false) + Entry.new(@types.member_name(@catalog.column_name(column.name)), @types.type(column, query_name: query.name), column, nil, false) end end.tap { |list| reject_duplicate_columns(query, list) } end diff --git a/lib/sqlc_gen_ruby/type_resolver.rb b/lib/sqlc_gen_ruby/type_resolver.rb index e0e6851..8436afa 100644 --- a/lib/sqlc_gen_ruby/type_resolver.rb +++ b/lib/sqlc_gen_ruby/type_resolver.rb @@ -10,23 +10,23 @@ def initialize(driver, options) @options = options end - # query_name: used to look up `overrides:` entries; nil for model (table) columns. + # query_name: used to look up `overrides:` entries; nil for model (table) columns, which pass their table. # fallback: see Drivers::Base#column_type. - def type(column, query_name: nil, fallback: :raise) - override = query_name && @options.override_for(query_name, column.name) + def type(column, query_name: nil, table: column.table, fallback: 'untyped') + override = @options.override_for(query_name, column, table: table) base = override ? override.type : driver_type(column, query_name, fallback) base = array_of(base, column) if column.is_array && !override not_null?(column, override) || base == 'untyped' ? base : "#{base}?" end def nullable?(column, query_name) - !not_null?(column, @options.override_for(query_name, column.name)) && !column.is_sqlc_slice + !not_null?(column, @options.override_for(query_name, column)) && !column.is_sqlc_slice end # A sqlc.slice parameter is passed as an Array of the element type. def slice_type(column, query_name) - override = @options.override_for(query_name, column.name) - "Array[#{override ? override.type : driver_type(column, query_name, :raise)}]" + override = @options.override_for(query_name, column) + "Array[#{override ? override.type : driver_type(column, query_name, 'untyped')}]" end # Rows are Struct (keyword_init) before Ruby 3.2, Data from 3.2. diff --git a/steep_expectations.yml b/steep_expectations.yml index 9c30482..0bb84f3 100644 --- a/steep_expectations.yml +++ b/steep_expectations.yml @@ -3,30 +3,30 @@ diagnostics: - range: start: - line: 218 + line: 249 character: 10 end: - line: 218 + line: 249 character: 24 severity: WARNING message: 'Cannot find the declaration of constant: `ConnectionPool`' code: Ruby::UnknownConstant - range: start: - line: 218 + line: 249 character: 47 end: - line: 218 + line: 249 character: 53 severity: WARNING message: 'Cannot find the declaration of constant: `Mysql2`' code: Ruby::UnknownConstant - range: start: - line: 218 + line: 249 character: 66 end: - line: 218 + line: 249 character: 81 severity: WARNING message: Empty kwargs doesn't have type annotation @@ -35,150 +35,150 @@ diagnostics: - range: start: - line: 291 + line: 346 character: 10 end: - line: 291 + line: 346 character: 24 severity: WARNING message: 'Cannot find the declaration of constant: `ConnectionPool`' code: Ruby::UnknownConstant - range: start: - line: 291 + line: 346 character: 47 end: - line: 291 + line: 346 character: 49 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 291 + line: 346 character: 58 end: - line: 291 + line: 346 character: 69 severity: WARNING message: Empty kwargs doesn't have type annotation code: Ruby::UnannotatedEmptyCollection - range: start: - line: 306 + line: 361 character: 49 end: - line: 306 + line: 361 character: 51 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 513 + line: 571 character: 18 end: - line: 513 + line: 571 character: 20 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 806 + line: 977 character: 18 end: - line: 806 + line: 977 character: 20 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 873 + line: 1044 character: 17 end: - line: 873 + line: 1044 character: 19 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 876 + line: 1047 character: 36 end: - line: 876 + line: 1047 character: 38 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 882 + line: 1053 character: 16 end: - line: 882 + line: 1053 character: 18 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 883 + line: 1054 character: 15 end: - line: 883 + line: 1054 character: 17 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 884 + line: 1055 character: 79 end: - line: 884 + line: 1055 character: 81 severity: WARNING message: Empty array doesn't have type annotation code: Ruby::UnannotatedEmptyCollection - range: start: - line: 885 + line: 1056 character: 25 end: - line: 885 + line: 1056 character: 27 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 886 + line: 1057 character: 67 end: - line: 886 + line: 1057 character: 69 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 894 + line: 1065 character: 15 end: - line: 894 + line: 1065 character: 17 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' code: Ruby::UnknownConstant - range: start: - line: 899 + line: 1070 character: 27 end: - line: 899 + line: 1070 character: 29 severity: WARNING message: 'Cannot find the declaration of constant: `PG`' diff --git a/test/features_test.rb b/test/features_test.rb index 2a0a0b6..e427d11 100644 --- a/test/features_test.rb +++ b/test/features_test.rb @@ -55,10 +55,52 @@ def test_models_fall_back_to_untyped_for_unknown_types assert_includes generate(request).fetch('models.rbs'), 'attr_reader id: untyped' end - def test_unknown_types_in_queries_fail + def test_unknown_types_in_queries_fall_back_to_untyped request = load_request('pg') query(request, 'GetAuthor').columns.first.type.name = 'geometry' - assert_raises(RuntimeError) { generate(request) } + query(request, 'GetAuthor').params.first.column.type.name = 'geometry' + rbs = generate(request).fetch('query_sql.rbs') + assert_includes rbs, "class GetAuthorRow\n attr_reader id: untyped\n" + assert_includes rbs, 'def get_author: (id: untyped) -> GetAuthorRow?' + end + + def test_more_postgres_types_composites_and_domains + rbs = generate(load_request('pg')).fetch('query_sql.rbs') + %w[c_timetz c_tsvector c_point c_int4range c_dimensions].each { |c| assert_includes rbs, "attr_reader #{c}: String?" } + assert_includes rbs, 'attr_reader c_oid: Integer?' + assert_includes rbs, 'attr_reader c_positive: untyped' # sqlc does not send the base type of a domain + end + + def test_sqlite_declared_types_follow_type_affinity + rbs = generate(load_request('sqlite3')).fetch('query_sql.rbs') + assert_match(/attr_reader c_varchar: String\?\n\s+attr_reader c_decimal: Float\?\n\s+attr_reader c_unsigned_big_int: Integer\?/, rbs) + request = load_request('sqlite3') + query(request, 'GetAuthor').columns.first.type.name = 'MEDIUMINT(4)' + query(request, 'GetAuthor').columns[1].type.name = 'STRING' + assert_match(/class GetAuthorRow\n\s+attr_reader id: Integer\n\s+attr_reader name: Float\n/, generate(request).fetch('query_sql.rbs')) + end + + def test_execresult_returns_rows_affected_and_last_insert_id + { 'pg' => 'ExecResult.new(result.cmd_tuples, nil)', 'mysql2' => 'ExecResult.new(stmt.affected_rows, client.last_id)', + 'sqlite3' => 'ExecResult.new(client.changes, client.last_insert_row_id)' }.each do |driver, line| + files = generate(load_request(driver)) + assert_includes files.fetch('query_sql.rb'), line + assert_includes files.fetch('query_sql.rb'), 'class ExecResult < Data.define(:rows_affected, :last_insert_id)' + assert_includes files.fetch('query_sql.rbs'), 'def create_author_result: (name: String, bio: String?) -> ExecResult' + end + end + + def test_batch_commands_take_an_array_of_args + %w[pg mysql2 sqlite3].each do |driver| + files = generate(load_request(driver)) + rbs = files.fetch('query_sql.rbs') + assert_includes rbs, 'def create_authors_batch: (Array[CreateAuthorsBatchArgs] rows) -> void' + assert_includes rbs, 'def get_author_batch: (Array[GetAuthorBatchArgs] rows) -> Array[GetAuthorBatchRow?]' + assert_includes rbs, 'def list_authors_by_name_batch: (Array[ListAuthorsByNameBatchArgs] rows) -> ' \ + 'Array[Array[ListAuthorsByNameBatchRow]]' + assert_includes files.fetch('query_sql.rb'), 'class CreateAuthorsBatchArgs < Data.define(:name, :bio)' + assert_includes files.fetch('query_sql.rb'), "rows.map do |args|\n" + end end def test_embedded_tables_become_model_attributes @@ -297,6 +339,54 @@ def test_multi_dimensional_arrays 'attr_reader c_integer_matrix: Array[Array[Integer]]?' end + def test_table_column_and_db_type_overrides_apply_to_models_and_rows + overrides = [{ column: 'authors.bio', ruby_type: { type: 'Symbol' } }, + { column: '*:c_uuid', ruby_type: { type: 'Pathname', notNull: true } }, + { db_type: 'jsonb', ruby_type: { type: 'Hash[String, untyped]' } }] + files = generate(load_request('pg', overrides: overrides)) + assert_includes files.fetch('models.rbs'), "attr_reader bio: Symbol?\n" + assert_includes files.fetch('models.rbs'), 'attr_reader c_uuid: Pathname' + assert_includes files.fetch('models.rbs'), 'attr_reader c_jsonb: Hash[String, untyped]?' + assert_match(/class GetAuthorRow\n.*\n.*\n\s+attr_reader bio: Symbol\?/, files.fetch('query_sql.rbs')) + assert_includes files.fetch('query_sql.rbs'), 'attr_reader c_uuid: Pathname' + end + + def test_rename_and_exact_table_names + files = generate(load_request('pg', rename: { 'authors' => 'Writer', 'bio' => 'biography' })) + assert_includes files.fetch('models.rb'), 'class Writer < Data.define(:id, :name, :biography)' + assert_includes files.fetch('query_sql.rb'), 'class GetAuthorRow < Data.define(:id, :name, :biography)' + assert_includes files.fetch('query_sql.rb'), "GetAuthorRow.new(row['id'], row['name'], row['bio'])" + assert_includes files.fetch('query_sql.rb'), 'Writer.new(row[0], row[1], row[2])' + assert_includes generate(load_request('pg', exactTableNames: true)).fetch('models.rb'), 'class Authors < Data.define' + assert_raises(RuntimeError) { generate(load_request('pg', rename: { 'authors' => 'writer' })) } + end + + def test_global_options_are_merged_under_plugin_options + request = load_request('pg') + request.global_options = JSON.generate(moduleName: 'Global::Db', generateTypes: false, + overrides: [{ column: '*:bio', ruby_type: { type: 'Symbol' } }]) + files = generate(request) + assert_includes files.fetch('query_sql.rb'), "module Global\n module Db\n" + assert_includes files.keys, 'query_sql.rbs' # generateTypes: true in the plugin options wins + assert_match(/class GetAuthorRow\n.*\n.*\n\s+attr_reader bio: Symbol\?/, files.fetch('query_sql.rbs')) + end + + def test_engine_must_match_the_driver + request = load_request('pg') + request.settings = Plugin::Settings.new(engine: 'mysql') + error = assert_raises(RuntimeError) { generate(request) } + assert_includes error.message, 'driver pg needs engine: postgresql (this sql block uses mysql)' + request.settings.engine = 'postgresql' + generate(request) + end + + def test_table_column_and_enum_comments_become_doc_comments + files = generate(load_request('pg')) + assert_includes files.fetch('models.rb'), " # People who wrote at least one book.\n class Author < Data.define" + assert_includes files.fetch('models.rbs'), " # Short biography, NULL when unknown.\n attr_reader bio: String?" + assert_includes files.fetch('models.rb'), " # Kinds of biographies.\n module ExtendedBioType" + end + # ---- regressions (examples/authors/*: the `regressions` table) ----------------------------------------------- def test_reused_and_repeated_parameters_get_one_keyword_each diff --git a/test/fixtures/mysql2.bin b/test/fixtures/mysql2.bin index e676a68..0a8560c 100644 --- a/test/fixtures/mysql2.bin +++ b/test/fixtures/mysql2.bin @@ -1,4 +1,7 @@ -public"public + + + +2mysqlpublic"public authors& id0R authorsbbigint& name0R authorsbtext# @@ -236,7 +239,25 @@ XINSERT INTO extended_bios (author_name, name, bio_type, author_type) VALUES (?, bio_type0 R extended_biosbextended_bios_bio_typezbio_type"J author_type0R extended_biosbextended_bios_author_typez author_type*EA bio_type0 R extended_biosbextended_bios_bio_typezbio_type: query.sqlC -DELETE FROM extended_biosTruncateExtendedBios:exec: query.sql +DELETE FROM extended_biosTruncateExtendedBios:exec: query.sql +-INSERT INTO authors (name, bio) VALUES (?, ?)CreateAuthorResult :execresult*84 +name0Rpublicauthorsbtextzname*40 +bio0Rpublicauthorsbtextzbio: query.sqlB authors +-INSERT INTO authors (name, bio) VALUES (?, ?)CreateAuthorsBatch +:batchexec*84 +name0Rpublicauthorsbtextzname*40 +bio0Rpublicauthorsbtextzbio: query.sqlB authors +.SELECT id, name, bio FROM authors WHERE id = ?GetAuthorBatch :batchone"* +id0R authorsbbigintzid", +name0R authorsbtextzname"( +bio0R authorsbtextzbio*.* +id0R authorsbbigintzid: query.sql +