migrate: move an app and its data between platforms, either direction - #1019
Merged
Merged
Conversation
sh1pt can already provision a machine (packages/cloud/*: connect, quote, provision, destroy). Provisioning is the easy half of a migration. The half that goes wrong is the data, and nothing here moved any. The shape comes from a migration that actually happened rather than a whiteboard: crawlproof.com off Railway and Supabase cloud onto a dedicated box, which moved 4.7 GB of Postgres, 8,410 storage objects, ten pg_cron jobs and a realtime publication, and which would have quietly broken the site months later over 2,928 rows holding absolute storage URLs. The split that avoids N x M adapters, and makes the thing bidirectional for free: PLATFORMS answer "what have I got and what are the credentials" and never move a byte; ENGINES move bytes and neither know nor care which vendor is on either end. Railway->dedicated and dedicated->Railway are then the same code path, and a new platform costs one inventory() rather than one adapter per existing platform. Direction is just which platform you named first. The planner is pure: given an inventory it returns an ordered plan and never touches the network, so `migrate plan` is safe against production and is fully testable. Phase order is the safety property and is enforced rather than documented -- check, bulk, freeze, delta, cutover, enable, verify. Scheduled jobs are stopped on the source before they are started on the target, because a cron firing on both sides is how a migration sends every customer a duplicate email. Dropping the source is not a phase; it is a decision a person makes days later. Credentials never reach a plan file: connections are a describe()/reveal() pair, and a test asserts a rendered plan contains no password. The Postgres engine dumps custom-format, no-owner, no-acl. It does NOT pass --clean: that would make a re-run idempotent, at the price of being one typo away from dropping a production database, so a non-empty target is refused instead. Credentials go through libpq's own environment variables rather than argv -- ps is world-readable and a dump runs for hours. That is also the only thing that works, since exec runs without a shell and nothing would expand a $VAR written into an argument. The delta sync is honest about its limits: for tables carrying a timestamp column it copies rows newer than the dump, which covers the append-heavy tables that keep being written during a long dump. It does not cover updates or deletes, and the planner says so rather than implying a completeness it does not have. The column name is validated as an identifier before it reaches psql -c. 48 tests, no network, no database. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The object-storage engine is one engine with a remote per side rather than one per vendor: S3, R2, B2, Spaces, Supabase storage and MinIO differ by endpoint and auth style, not structurally. rclone does the copying because concurrency, retries, resume, multipart thresholds and checksum comparison are five things to get right and it already has them; the planner checks for the binary up front so a missing rclone is a blocker before the freeze rather than a failure during it. Two deliberate choices. The remote is built entirely from RCLONE_CONFIG_* environment variables, so no credential is written to disk or appears in argv. And it runs `copy`, never `sync` -- sync deletes whatever at the destination is not at the source, which for a mistyped target is indistinguishable from wiping a live bucket. A migration tool should not be able to delete. Object storage is also the one engine that does not stage bytes: pulling ten gigabytes down and pushing them back up doubles the transfer for no benefit, so export writes a manifest and import copies remote to remote. That needed the source resource at import time, which the Engine contract now passes explicitly rather than smuggling through metadata, which only holds primitives. Then the rewrite. An app that stores a whole URL instead of a key leaves rows pointing at the account you just left. Migrate everything, cut DNS over, check the site: every image loads -- because the OLD account is still serving them. The day it is closed, which is the entire point of migrating and happens weeks later, all of them 404 at once and nothing connects the outage to the migration. crawlproof.com had 2,928 such rows across four tables, found by looking rather than by anything failing. So it is a first-class step: find every text-ish column (broad on purpose -- a column called `notes` holding a pasted URL breaks exactly as badly as one called `image_url`, and guessing from names is how those four tables would have been missed), count what is there, rewrite inside one transaction, then assert zero remain. Writing that assertion surfaced a real bug in my own first version: a trailing `having count(*) > 0` after a UNION ALL binds to the final SELECT only, so every other column would have reported regardless of its count and the one real offender could sit unnoticed among them. The union now goes in a subquery with the filter outside it, and a test pins that the filter is not inside. 90 tests, still no network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SQLite and Turso are one engine because libSQL is SQLite with a server in front, and both dump to the same SQL text -- which is also why Turso->dedicated and dedicated->Turso are the same code path. The dump is text rather than a file copy: copying the file does not work for a hosted database at all, and a SQLite file copied while something writes to it is a corrupt file rather than an error. That needed a contract change. Those tools only read and write stdio, and exec runs without a shell, so a `>` written into the arguments would be passed to the program as a literal. ExecOptions now carries stdoutFile and stdinFile, which also keeps a multi-gigabyte dump out of a string. Redis is included with a warning attached: the right migration for a cache is usually an empty one, and copying it moves stale entries and buys downtime for data that is worthless by definition. It is here for when it is not a cache -- a BullMQ queue with jobs in it, a session store where copying nothing logs everyone out. It exports with --rdb, which is consistent at a point in time, rather than walking keys with SCAN, where keys move under you as you read. It has no delta, and the planner turns that absence into the warning it should be. Import refuses to go over the wire, because there is no supported way to push an RDB into a running managed Redis, and says what to do instead rather than doing it badly. Files is rsync, for the volume or docroot that is neither a database nor a bucket. Writing it turned up two things I had wrong. The endpoint helper took a source/target parameter that selected between two identical values -- dead code pretending to encode a rule. And rsync cannot copy remote to remote at all; it is a protocol limitation, not a missing flag, so a VPS-to-VPS move must relay through the machine running the migration. That now fails up front with an explanation instead of surfacing as "The source and destination cannot both be remote" halfway through a cutover. As with object storage, --delete is never passed anywhere. 110 tests, tsc clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Platforms resolve credentials and enumerate resources. They never move a byte, which is what lets one implementation serve both directions. ssh is a box you own, and it is `both` because "get off the cloud" and "we tried bare metal and went back" are the same code path. A server has no API to enumerate itself, so it is described in config rather than discovered -- not a workaround: a directory has no metadata saying "this is the uploads volume", and guessing from paths would be worse than being told. It does not create databases remotely either; choosing disks and versions and a backup story on someone's server is not a decision a migration tool should make quietly. Railway finds databases by RECOGNISING CONNECTION STRINGS IN VARIABLES rather than asking for a list of databases, because that list does not exist in that shape -- a Postgres service's DSN is DATABASE_URL on the services that use it. Classification is by URL scheme, never by variable name: DATABASE_URL is a convention and plenty of apps use PG_URL or something bespoke, and silently skipping the database because it was called the wrong thing is the worst failure available. The same DSN injected into six services is moved once. Supabase is a Postgres with a lot bolted on, and the bolted-on parts are what make leaving it interesting. Five quirks are named up front, all from the crawlproof migration: auth.users restores as data but GoTrue and the JWT secret do not, so everyone is logged out unless the secret comes too; RLS policies reference anon/authenticated/service_role, which do not exist on a plain Postgres and must be created first; pg_cron jobs restore already enabled and start firing immediately. It also emits the exact --rewrite-host flag for its own public hostname, so the absolute-URL trap is a copyable line rather than a thing to remember. Turso, Neon, PlanetScale, Fly, Render, Heroku and Vercel differ enormously as products and barely at all here: each hands over a DSN and the engines do the rest. Six of them share one factory because writing a file each would be six copies of twenty lines. Turso gets a real implementation, since its CLI works on a database name plus a token rather than a connection string. compatibleKinds() is the whole design in one function: intersect what the source holds with what the target accepts. Turso->Neon returns empty (sqlite vs postgres) and says so in a millisecond instead of failing at a cutover. Supabase->ssh and ssh->Supabase both return postgres, and neither is a code path anyone wrote. 141 tests, tsc clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The planner decides what happens; the executor does it, and its only real job is refusing to deviate. Two things make a migration catastrophic rather than merely failed: running a later phase before an earlier one, and carrying on past a step that was meant to be a gate. Both are prevented here rather than trusted to the caller. A step whose dependency did not run is skipped, not attempted -- an import with no export would restore whatever happened to be left in staging, possibly from a different migration. `--until freeze` runs the whole bulk copy and stops before anything goes down, which is how a migration is rehearsed against production. The DNS cutover and the cron enable/disable steps are printed, not performed. Pointing DNS at a new host on a caller's behalf, or starting a scheduler at the wrong moment, is exactly the failure the phase ordering exists to prevent; automating it would put the decision back inside the tool. Staging is an append-only JSONL ledger rather than a rewritten JSON file. A process killed mid-write corrupts the file it was rewriting; it can only truncate the last line of an append-only one, and a half-written line fails to parse and is skipped. That is the difference between resuming a four-hour copy and starting it again. exec spawns without a shell, always. Engine arguments are built from connection strings, bucket names and paths that come from a config file a person edits, so a shell would make a bucket named `; rm -rf /` a working attack and a path with a space a silent bug. The cost is that `>` and `<` do not work, which is why redirection is an ExecOptions field. ENOENT is translated to "<binary> is not installed or not on PATH", since "spawn ENOENT" tells nobody anything. Verified end to end against a real config. `sh1pt migrate platforms --from supabase` prints what can receive what; `migrate plan --from supabase --to ssh` produced a correct ordered plan with the five Supabase quirks as warnings, the absolute-URL step, and accurate blockers -- pg_dump and rclone really are missing on this box, which is the preflight working. A `--rewrite-host` without a scheme is rejected before anything connects. 156 tests, tsc clean across migrate and cli, biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vu1nz Security Review0 finding(s) in PR #? No security issues found. |
ThreatCrush Security Scan53 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 17 | LOW: 35
…and 3 more. Full results in the Security tab. Snippets are redacted; ThreatCrush never prints matched credential material. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| * offender could be buried. Filtering outside the subquery applies to all of | ||
| * them, which is the point of the assertion. | ||
| */ | ||
| return `select ref, n from (\n${parts.join('\nunion all\n')}\n) as remaining where n > 0 order by n desc`; |
| }); | ||
|
|
||
| it('url-encodes a password with reserved characters', () => { | ||
| expect(supabaseDsn({ projectRef: 'r', dbPassword: 'p@ss/word' })).toContain('p%40ss%2Fword'); |
…tures ThreatCrush flagged 20 new alerts on the last commit. Going through them found one real bug and a lot of my own noise. The real one: the delta's \copy interpolated the table name UNQUOTED. The names come from information_schema so they are real tables rather than attacker input, but a table called `user` or `order` is a reserved word and an unquoted reference to it is a syntax error -- arriving partway through a cutover, which is the worst possible time. A name containing a quote or a space does not parse at all. Fixing it properly meant changing the discovery query to return table_schema and table_name as separate fields instead of pre-joining them with a dot. Joining first makes `public.user` a single string that cannot be quoted correctly: quoting the whole thing yields `"public.user"`, one identifier with a dot in its name, which is a different table that does not exist. Schema, table, the timestamp column and the output path are now all quoted, with tests pinning each. The other 17 highs were all mine and all fake: DSN literals in test fixtures. Most did not need a password at all -- the tests are about plan ordering and platform pairing, not credentials -- so those are gone, which is a real hygiene improvement rather than a workaround. Four tests genuinely are about credential handling (masking, url-decoding) and do need one; those now interpolate a named FAKE_PASSWORD constant, which says plainly what it is instead of leaving a string in source that a scanner cannot distinguish from a leak. The remaining sql-template-interpolation alert in transforms.ts is composition of fragments that quoteIdent and quoteLiteral already built, which the scanner cannot see through. 159 tests, tsc clean, biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| 'psql', | ||
| [ | ||
| '--command', | ||
| `\\copy (select * from ${quoteIdent(schema)}.${quoteIdent(table)} where ${quoteIdent(columns)} > ${quoteLiteral(since.toISOString())}) to ${quoteLiteral(`${ctx.staging.dir}/${out}`)} with csv header`, |
Shipping without this would have been dishonest. `mysql` was already a ResourceKind, and both Railway and PlanetScale declared they hold it -- so the CLI advertised PlanetScale as a source while every plan involving it blocked out with "nothing in this build can move a mysql". Advertised and unusable is worse than absent. mysqldump with the flags that matter, each of which is a failure somebody has already had: --single-transaction, because without it a multi-gigabyte dump locks every table for its duration, which is an outage and rather defeats copying while the source is live; --set-gtid-purged=OFF, because a dump carrying GTID state refuses to load into a server with its own replication history and the error names neither the flag nor the cause; --no-tablespaces, because writing tablespace clauses needs PROCESS privilege that managed providers do not grant, and its absence fails the dump rather than degrading it. The password goes in MYSQL_PWD rather than --password=, same reasoning as libpq: argv is world-readable via ps for the hours a dump runs. The client's warning that MYSQL_PWD is insecure on shared machines is true and still strictly better than the alternative. Verification reports estimated row counts WITHOUT failing on them. information_schema.table_rows is an estimate on InnoDB, not a count; treating it as exact would fail every verification that ever ran. Only a table missing outright is a problem. Then running the CLI caught a second gap of my own making: the ssh platform never declared mysql, so `migrate platforms --from planetscale` reported "nothing in common" against a dedicated box -- the exact get-off-the-cloud move the tool exists for. ssh now holds mysql, with a test pinning the pairing in both directions. 177 tests, tsc clean, biome clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| '--batch', | ||
| '--skip-column-names', | ||
| '--execute', | ||
| `select count(*) from information_schema.tables where table_schema = ${quoteLiteral(databaseName(to))}`, |
ralyodio
marked this pull request as ready for review
September 25, 2026 08:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sh1ptcan already provision a machine —packages/cloud/*does connect, quote, provision, destroy. Provisioning is the easy half of a migration. The half that goes wrong is the data, and nothing moved any.This adds
@profullstack/sh1pt-migrateandsh1pt migrate.Why it is bidirectional without twice the code
Two layers:
supabase → sshandssh → supabaseare therefore the same code path, and a new platform costs oneinventory()rather than one adapter per existing platform. Direction is not a property of the system; it's which platform you named first.compatibleKinds('turso', 'neon')returns[]— sqlite against postgres — in a millisecond, rather than at a cutover.The shape comes from a migration that happened
crawlproof.com off Railway and Supabase cloud onto dev2: 4.7 GB of Postgres, 8,410 storage objects, ten pg_cron jobs, a realtime publication — and 2,928 rows holding absolute storage URLs that would have 404'd weeks later when the old project was deleted, long after everyone had called the migration a success.
So the absolute-URL rewrite is a first-class step, not a footnote. Every text-ish column is scanned (broad on purpose: a column called
notesholding a pasted URL breaks exactly as badly as one calledimage_url), rewritten in one transaction, then asserted to be zero. The Supabase platform emits the exact--rewrite-hostflag for its own hostname, so it's a line to copy rather than a thing to remember.Phase order is enforced, not documented
check → bulk → freeze → delta → cutover → enable → verifyScheduled jobs stop on the source before they start on the target, because cron firing on both sides is how a migration sends every customer a duplicate email.
--until freezeruns the entire bulk copy and stops before anything goes down, which is how you rehearse against production. Dropping the source is not a phase — that's a decision a person makes days later, and this doesn't offer it.The DNS cutover and cron steps are printed, not performed. Automating those would put back the exact decision the phase ordering exists to protect.
Safety, stated plainly
rclone copyneversync; norsync --delete; nopg_restore --clean. A non-empty Postgres target is refused rather than overwritten.describe()/reveal()pair, with a test asserting a rendered plan contains no password.RCLONE_CONFIG_*,TURSO_API_TOKEN.psis world-readable and a dump runs for hours.Three bugs my own tests caught
having count(*) > 0after aUNION ALLbinds to the last SELECT only, so every other column would have reported regardless of count and the one real offender could sit unnoticed. Filter now sits outside a subquery, with a test pinning it.'$PGURI'passed as a literal argument would never expand, becauseexecruns without a shell. Connection now goes through libpq's own environment variables — which is also the more secure answer.Verified end to end
migrate plan --from supabase --to sshproduces a correct ordered plan with the five Supabase quirks as warnings, the URL step, and accurate blockers —pg_dumpandrclonereally are missing on this box, which is the preflight working as designed. A--rewrite-hostwithout a scheme is rejected before anything connects.156 tests, no network, no database. tsc clean across
migrateandcli; biome clean.Known limits, in the README rather than implied away
🤖 Generated with Claude Code