[3.0][Testing] Check that an upgraded database matches a fresh install

[albertlast]

Description

The installer builds the schema from Sources/Db/Schema/v3_0/ in one go. The upgrader arrives at the same place through a hundred-odd migrations applied to whatever 2.1 left behind. They are meant to converge, and nothing we have checks that they do — so a column left at the wrong type, an index that was never created, or a primary key quietly dropped goes unnoticed until it turns up as a bug report from someone whose forum upgraded two years ago.

Two files:

• .docker/schema-tool.php reads the shape of a database — tables, columns, indexes, sequences, the variable names in settings — and compares two of those readings. It talks to the engine directly rather than through SMF, since the database worth looking at is frequently one SMF would refuse to run on.
• .docker/compare-upgrade.sh drives it: empty the database, load a 2.1 dump, upgrade it, read the schema, reinstall from scratch, read that too, report the difference.

.docker/compare-upgrade.sh --engine mysql --baseline path/to/a-2.1-dump.sql

--baseline takes any SQL dump of a 2.1 database, so this works against a real forum as well as against the synthetic baseline from #9330.

Two kinds of difference are reported but do not decide the exit code, because a real forum always has some: what is in settings, and the order the columns of a table sit in. And if the upgrade does not reach the end, the script stops there and says so rather than comparing anyway — a half-upgraded database differs from a fresh install in hundreds of places, every one of them the honest consequence of the migrations that never ran, and none of them worth reading. That check is worth more than it sounds: the first run of this produced 315 differences, all of them because the upgrade had stopped a third of the way through.

The tool underneath is usable on its own against any two SMF databases on the same engine — two forums you have, or one forum before and after something you are testing.

What it says today

On MySQL, against the 2.1 baseline from #9330, with the DropTimeOffset defects from #9521 patched so the upgrade can finish: 55 differences in the schema, in four groups.

Text columns are one size larger than they should be. Around 35 of the 55. ALTER TABLE … CONVERT TO CHARACTER SET utf8mb4 promotes a text column to mediumtext to preserve its byte capacity, so an upgraded forum ends up with messages.body as mediumtext where a fresh install has text, and mail_queue.body as longtext against mediumtext. Wider rather than narrower, so nothing breaks, but it is not the schema the code was written against.

Index prefix lengths disagree, in both directions.

admin_info_files.idx_filename   fresh: filename(30)        upgraded: filename(191)
boards.idx_member_groups        fresh: member_groups(48)   upgraded: member_groups(191)
members.idx_email_address       fresh: email_address(191)  upgraded: email_address
members.idx_real_name           fresh: real_name(191)      upgraded: real_name
qanda.idx_lngfile               fresh: lngfile(191)        upgraded: lngfile

Indexes 2.1 had that 3.0 does not define are never dropped, so an upgraded forum carries log_spider_hits.idx_id_spider, log_spider_hits.idx_log_time and a unique log_subscribed.id_subscribe that a fresh install has not got. log_packages ends up with the same column indexed twice, as filename(191) and idx_filename(15).

mentions keeps defaults the fresh schema drops — id_member, id_mentioned and time are DEFAULT 0 after an upgrade and have no default at all in a fresh install.

Outside the schema, four settings a fresh install writes are missing after an upgrade — cpu_count, forum_uuid, mostOnlineUpdated, robots_txt_search — and the 2.1-era ones the upgrade means to delete are all still there, which is #9530.

On PostgreSQL the script stops before it can compare anything:

[smf-dev] postgresql: the upgrade stopped at SMF 2.1.7, expected 3.0 Alpha 4

AlertsObsolete uses UPDATE … JOIN, which is MySQL-only. That is #9519, and #9524 is the fix.

None of the above is addressed here. This adds the thing that finds it.

Notes for review

Depends on #9344 for .docker/install-forum.sh, and reads better after #9330, which is where the baseline it was written against comes from. Nothing outside .docker/ and one .gitignore entry changes, so it cannot affect a running forum.

Issues References (Fixes|Related|Closes)

Related to #9330, #9344, #9519, #9521, #9530

[albertlast]

Checked the reading against pg_dump --schema-only of the same database, which was a good idea and found two gaps. Both are pushed.

Three indexes were missing entirely. An index on an expression stores 0 in indkey and has no pg_attribute row, so the join to that table dropped those keys — and an index every one of whose keys is an expression disappeared with them:

members_idx_birthdate2         indexable_month_day(birthdate)
members_idx_member_name_low    lower(member_name::text)
members_idx_real_name_low      lower(real_name::text)

Three out of 113 on a stock install, and silently: the tool reported no difference where an upgrade had failed to create any of them. pg_get_indexdef(indexrelid, ord, true) renders a plain column and an expression alike, and needs no join at all.

The compatibility functions were not read. find_in_set(), instr(), from_unixtime(), the group_concat aggregate and the rest of the MySQL shims are created at install time. A query naming one of them fails outright where it is absent, which makes a missing one a worse problem than a missing index rather than a lesser one, so they are now a schema difference too.

The two readings agree object for object now:

	pg_dump	schema-tool
tables	72	72
indexes	113	113
primary keys	69 (all 69 constraints)	69
sequences	41	41
functions	18 + 1 aggregate	19

All pg_dump still reports that this does not are the public schema itself and the comment on it.

Verified by breaking a database on purpose — dropping instr(text, text) and idx_real_name_low, both of which used to go unnoticed:

Tables
------
  members
    index members_idx_real_name_low is missing from damaged (index (lower(real_name::text)))

Functions
---------
  instr(text, text)
    missing from damaged

2 difference(s) in the schema.

I checked MySQL the same way, against mysqldump --no-data --routines --triggers --events, in case it had a matching blind spot. It does not: 72 tables, 538 columns and 179 keys on both sides, and SMF creates no routines, triggers or events there, so there is nothing equivalent to miss.

Worth saying why the tool reads the catalogue rather than diffing two dumps, since that is the obvious alternative. A dump is ordered and formatted for restoring, not for comparing — text diffs against mediumtext in the middle of a CREATE TABLE, and a column appended by a migration shifts every line after it — so a plain diff of two dumps reports the whole table where one column changed, and cannot tell "this index is missing" from "this index moved". The catalogue gives the same facts already keyed by name.

[albertlast]

Did the same against mysqldump --no-data --routines --triggers --events, name by name rather than by count. The names all agree — 538 columns and 179 keys, matching table for table — and SMF creates no routines, triggers or events on MySQL, so there is nothing there like the missing PostgreSQL functions.

Two things were being read too shallowly, though.

A generated column was compared without its expression. EXTRA says that a column is generated and never what from. smf_messages has three STORED columns read out of the edit_history JSON:

`modified_time`   bigint unsigned GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][0]')),0)) STORED
`modified_name`   varchar(255)    GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][6]')),'')) STORED
`modified_reason` varchar(255)    GENERATED ALWAYS AS (coalesce(json_unquote(json_extract(`edit_history`,'$[0][7]')),'')) STORED

A wrong path in one of those produces a column of exactly the right type holding quietly the wrong value, which is about the worst thing this could fail to notice. Comparing GENERATION_EXPRESSION is what tells $[0][7] from $[0][9].

ROW_FORMAT was not read at all, and it is not decoration here: COMPACT caps an index key at 767 bytes where DYNAMIC allows 3072, so a table left behind in the older format is one where half of SMF's indexes cannot be created at their full width. Index width is already the second largest group of differences an upgrade produces, so a table quietly in the wrong row format is worth knowing about.

AUTO_INCREMENT is still deliberately not read. It measures how much a database has been used rather than what shape it is, and would differ between any two databases.

Checked by breaking things on purpose again — moving one JSON path and putting one table back to COMPACT:

Tables
------
  messages
    column modified_reason
      fresh:    varchar(255) NULL STORED GENERATED AS (… json_extract(`edit_history`,'$[0][7]') …)
      damaged:  varchar(255) NULL STORED GENERATED AS (… json_extract(`edit_history`,'$[0][9]') …)
  qanda
    row_format: Dynamic in fresh, Compact in damaged

2 difference(s) in the schema.

So both engines have now been checked against their own dump tool, and both found something. That is the more useful outcome than agreement would have been.