From c577dcad02012eb0118ed691f41cfc2eb68cf9c4 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:38:09 +0800 Subject: [PATCH 01/26] docs: record approved v18 compatibility preparation design --- ...-11-08-mx-core-v18-compatibility-design.md | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md new file mode 100644 index 00000000000..6a478f27bac --- /dev/null +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -0,0 +1,164 @@ +# mx-core v18:兼容性收口与文档迁移 + +状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。当前执行 **v17 准备阶段**,不发布 v18,不操作真实站点数据。 + +基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 + +## 意图 + +让凭据存储、读取与生成、配置写入和已废弃接口形成明确契约。现有实例先在可以继续服务的准备版中发现和处理升级问题,再切换到 v18。不能在滚动切换时删除旧副本仍使用的列,也不能用新密钥代替丢失的原秘密。 + +## 已确认范围 + +| 项目 | 决定 | +| ----------- | ------------------------------------------------------------------------------------------- | +| 加密必填 | v18 要求显式 64 位十六进制字段密钥,结束关闭加密与机器标识回退 | +| V2 写入 | v18 统一配置/Snippet 的 GCM、Webhook 加密、API Key 哈希与 OAuth 加密,退役旧格式写入开关 | +| 历史读取 | 保留历史明文/ECB 的读取和迁移、旧 ZIP/v1 加密备份恢复、旧密码登录后升级 | +| 升级基线 | 只有完成准备的 v17.x 可直接升 v18;更老版本先完成历史迁移 | +| 已废弃入口 | 移除评论列表的 `state` 查询别名和速记表态 GET;先迁 SDK、CLI、官方主题 | +| 旧配置 | AI、第三方服务、会员配置先持久化迁移,再移除运行时读取兼容;清理无效参数和 MongoDB 日志残留 | +| 文档节奏 | 先纠正 v17 和补 v15–v17 指南;v18 发布时切默认示例,并保留 v17 关键入口 | +| AI 普通读取 | v18 摘要/精读 GET 只读;缺结果返回 `{ data: null }`,生成显式触发 | +| 执行类 GET | 非流式备份/清缓存/测试邮件等改 POST,保留现有 GET SSE 和帧协议 | +| 配置验证 | v18 拒绝未知配置字段,保留大小写归一化、部分更新和自由 JSON | +| AI 指标 | 先补齐 generationMetrics 并停止旧列引用,v18 再删除旧字段和物理列 | +| 排序 | 服务端统一 `asc`/`desc`;兼容 SDK 先转换 `1`/`-1` 的调用 | +| SDK 与别名 | 协调 SDK 6.x 清理确认废弃项;保留 SDK legacy、CLI V2 适配和其他有效路由别名、长期 URL | + +## 不纳入 + +API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全部备份、密码重置、历史 SQL 迁移删除、长期 `/objects` 和 `/fn` URL 改名。内部重构不自动成为用户可见的 Breaking Changes。 + +## 当前准备阶段的交付 + +1. 增加后续需要的 POST 入口,并迁移 Admin 和兼容 SDK;保留旧 GET。 +2. 将旧配置转换抽为可复用模块,支持持久化准备与旧接口写入转换,保持 v17 的已承诺形状。 +3. 补齐 AI 指标;让服务端运行时不再读写旧元数据列,旧 v17 响应从新指标派生,暂不删除物理列。 +4. 提供只读 `upgrade:check`、显式 `upgrade:prepare`、安全的 JWT 轮换工具。命令支持构建后的交付与本地源码执行。 +5. 完成 v17 事实文档纠错、v15–v17 迁移入口及 v18 预告,提供可执行准备说明。 +6. 行为测试、真实临时 PostgreSQL 升级演练、构建和独立整体验证;保留原有工作。 + +准备阶段不切换普通 AI GET 的既有默认行为,不拒绝现有合法旧参数,不删除 SDK 5.x 的公开签名,不删除数据库列,不强制关闭 v17 逃生口,不修改线上环境或发布标签。 + +## 安全配置与迁移契约 + +### v18 的启动条件 + +- 显式 `ENCRYPT_KEY`/`MX_ENCRYPT_KEY` 为 64 位十六进制;`JWT_SECRET` 至少 32 字符。保留既有别名优先级。 +- 新装生成每实例随机密钥并持久保存;现有密钥原样保留。生产模板不能内置公共有效密钥。 +- 工程选择:旧 `ENCRYPT_ENABLE=true`、`MX_SECRET_STORAGE_V2=true` 暂接受并提示可删除;显式 false 拒绝启动。新写入固定 V2,`ENCRYPT_ALGORITHM` 只服务历史读取。 +- 新装空库由发布前迁移初始化当前状态,不要求 v17 历史;既有实例需要准备完成。 +- 普通启动只检查配置、schema 和准备状态,不做业务数据迁移。 + +### 检查与显式准备 + +`upgrade:check` 不启动 HTTP/业务模块,不因普通配置初始化提前退出而丢失诊断。输出人可读摘要或 JSON:新装识别、配置缺失、密文类型计数、旧配置计数、指标覆盖和准备状态。退出状态区分 ready、needs-action 和检查失败;不得输出秘密值。 + +`upgrade:prepare` 使用 slim runner、数据库 advisory lock 和迁移账本,支持 dry-run、事务失败保护和幂等重试。运行需要显式确认所有副本已经停止旧格式写入;进程不能凭自己猜测集群状态。只在转换成功、残留检查通过后写入新的 v18 准备标记,旧凭据迁移账本行不能充当完成证明。 + +顺序:识别旧记录 → 用原路径解密 → 转换旧 AI/第三方/会员配置 → 按新路径加密并持久化 → 转换 Snippet/Webhook/API Key/OAuth → 验证残留及指标覆盖 → 记录准备完成。API Key 原始字符串仍可用,不重复哈希,不把哈希当作客户端凭据。 + +准备版对仍需兼容的旧配置写请求在落库边界转换为新形状;旧读取可由新形状派生,不能让旧客户端把旧结构写回。转换和常规 PATCH 串行保护同一配置。旧配置转换留在迁移/导入/恢复共用模块,v18 常规读取不再归一化旧结构。 + +坏密钥、坏密文或未知格式必须失败,不回落为明文成功。报告只带字段路径/记录 ID/计数,不带值。原实例若曾用 machine-id 加密,准备期先取得等效密钥并显式保存;不能生成新随机密钥替代。 + +### 短 JWT 的维护窗口处理 + +工程选择:不自动补齐/截断/覆盖 JWT;暂停全部业务副本和后台写入,以只读秘密文件向独立进程提供旧、新 JWT。事务内验证并用旧 JWT 解封装 OAuth,再用新 JWT 重封装;旧明文按已知格式转换。命令行只出现文件路径,日志不含秘密。 + +成功后由操作者将全部副本配置为同一新 JWT,再启动。明确要求重新登录;保留 provider token,不主动联系服务商,不改变原字段密钥和 API Key 字符串。错误旧 JWT 整批回滚,不清空 token。缺少原 JWT 且存在密文时阻塞并报告记录,由操作者恢复原秘密或完成账户重新授权。没有现有密文时也要说明轮换与会话影响。 + +## AI 指标的 expand-contract + +旧列:`ai_translations.ai_model`、`ai_translations.ai_provider`、`ai_insights.model_info`。 + +准备版先补齐新指标,并停止运行时 SELECT/INSERT/UPDATE/RETURNING 对旧列的引用,包括隐式全列展开。旧 v17 API 的 `aiModel`/`aiProvider`/`modelInfo` 从新指标派生,物理列保留。 + +所有副本运行准备版且覆盖验证通过后,v18 发布前迁移才删旧列;v18 views、Admin 和 SDK 6.x 同步去除旧字段/fallback。相关响应缓存版本化或定向失效,不清空所有 Redis 状态。验收须将准备版代码运行在已删旧列的临时 schema 上。 + +## API 与消费者 + +| 入口 | v18 契约 | 准备版工作 | +| ---------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------- | +| 摘要/精读 GET `/article/:id` | 只读,缺结果 `data: null`;资源存在性、隐藏/会员等权限不放宽 | 保持 v17 默认;SDK 6.x 读取准备版时显式 `onlyDb=true` | +| 显式 AI 生成 | 保留既有任务或生成入口;GET SSE 与 token/done/error 帧保留 | 不改公共 SSE 字节协议 | +| `/comments?state=` | 明确错误并引导 `tab`,不静默忽略;更新 body 的 state 仍有效 | 消费者用 tab,旧书签可在客户端转换 | +| `/recently/attitude/:id` | POST,保留 attitude 查询参数与业务结果 | SDK 提前用已有 POST | +| `/backups/new` | POST,保持二进制、文件名、权限与内容类型 | 新增 POST,同时保留 GET;不能占用上传恢复的 POST `/backups` | +| `/clean_catch`、`/clean_redis`、`/health/email/test` | 原路径改 POST,保持响应/权限;GET 不再执行 | 增加 POST,迁 Admin/脚本,旧 GET 留到 v18 | +| sortOrder / sort_order | 仅 asc/desc,数字及数字字符串明确验证错误 | SDK 输出规范化,CLI/主题迁字符串 | + +业务 JSON/envelope、wire snake_case、请求大小写归一化保持。v18 移除的执行 GET 返回无副作用 404/405,不用会执行旧动作的重定向。生成缺结果的主题显示“尚未生成”,不自动换一种请求继续隐式生成。测试邮件只用替身验收,不发送真实邮件。 + +严格配置仅作用于配置 PATCH/初始化配置写入。在归一化后,未知的声明字段路径返回 HTTP 400 `CONFIG_VALIDATION_FAILED`,列出路径而不含值;嵌套声明对象也校验,record/freeform/meta/Snippet JSON 动态键保持。部分更新、空秘密保留旧值保持,Admin/CLI 用实际 PATCH 验证而非静态表测试。 + +## SDK 与版本矩阵 + +- 先发 SDK 5.x 兼容版:速记 POST、排序输出规范化、新执行入口、必要认证修正。保留已承诺签名,不在 patch 删除方法。 +- 协调 SDK 6.x:删除确认失效的 `getDeepReading`、`getMasterInfo` 别名、误导 query token 参数、通用 `ResponseMeta` 等;仍应工作的认证能力有替代调用,不能只是删除。 +- SDK 6.x 的普通 AI 读取在准备版和 v18 上都只读,类型可空,元数据使用 generationMetrics。 +- 保留 `api-client/legacy` 与 CLI V2 响应适配。CLI 访问 Core v18 使用 API v3 profile,不把产品版本当 API 版本。 +- 保留其他有效路由别名和长期 `/objects`、`/fn` URL,文档只标规范入口。 + +准备版 Core 必须兼容原客户端、SDK 5.x 兼容版和新 SDK 6.x;v18 正式支持匹配的内置 Admin、SDK 6.x 及实测 CLI/主题版本,不宣称未适配客户端兼容。准确最低版本在发布验证后写入矩阵。 + +## 文档修改地图 + +正式文档仓库是 `mx-space/docs`,内容目录保持 `content/docs`,使用已有 Astro/Fumadocs 组件。内部设计不能直接充当用户操作指南。 + +第一批按当前 v17 修正事实: + +- `configure/encryption.mdx`:显式 hex 校验、逃生口边界、已有密钥保留,删除错误的 machine-id/短密钥建议;区别字段密钥、JWT 与恢复私钥。 +- `configure/environment.mdx`:V2/backfill 前置条件,Redis 真实优先级(CONNECTION_STRING → CONNECTION → URL),版本化必填要求。 +- `deploy/docker.mdx` 和 `src/components/env-variable-config.tsx`:生成 YAML 映射而非 `- KEY=value` 列表,字符串正确引用;app/migrate 共用配置;新装密钥持久保存。 +- `deploy/source.mdx`:Node 26 已是 v15 要求,补发布前迁移步骤,避免重复等价构建。 +- `use/backup-restore.mdx`:补 `.mxbackup`、公私钥分工、旧备份读取、原凭据密钥仍需保留;字段必填不代表备份加密必填。 +- 补 `migrate/v14-to-v15.mdx`、`v15-to-v16.mdx`、`v16-to-v17.mdx` 与导航:Node 26、BCP 47、Agent 协议和历史对话清空是历史变更,不算 v18 新破坏。 +- 增加明确标“计划/准备中”的 v18 预告与准备说明,不能提前把默认文档写成 v18 已上线。 + +第二批随 v18 正式发布: + +- 新增 `migrate/v17-to-v18.mdx`;接入 `migrate/index.mdx`、`meta.json`、`use/update.mdx`。 +- 快速体验、Docker/源码示例切换必填配置,持久保存随机实例密钥,不维持公共密钥的“零配置”。 +- AI、评论、账号安全页面按真实 UI 与语言代码核对;补方法/路径、排序、未知字段、元数据对照表。 +- 保留固定到 v17 的关键部署/加密说明入口,旧链接明确版本与升级方向。 +- 同步 Agent 模板和静态 skills;验证生成的 llms.txt/llms-full.txt,不手工维护另一套副本。 + +每项 Breaking Changes 说明受影响对象、旧新行为、迁移步骤、验证和回退边界。部署说明固定已发布镜像/编排版本,不仅让用户 pull latest。 + +## 发布与恢复顺序 + +```mermaid +flowchart TD + A[历史实例] --> B[准备版 v17.x] + B --> C[保存秘密与快照 必要时维护轮换 JWT] + C --> D[所有副本启用 V2 停止旧写入] + D --> E[持久化配置迁移 凭据和指标 backfill] + E --> F[检查通过 迁移消费者] + F --> G[v18 RC 升级与恢复演练] + G --> H[正式发布 v18 与匹配客户端 切文档] +``` + +回退最低版本是实测准备版(能读 V2/新配置且不再引用旧列),不是任意 v17。回退代码不回退 schema;更老版本需要完整旧快照与当时的秘密。区分代码回退、数据库恢复、文件恢复。 + +历史备份恢复先验证归档与当时的字段密钥/JWT/恢复私钥,隔离验证转换后再服务。环境不能自动隔离预检时提供离线“恢复到准备版 → prepare → v18”路径,不能报告成功后把未准备数据交给业务。保留恢复前快照;不得持久保存新的公开秘密副本。 + +## 验收 + +- 新装、历史明文、ECB、V2;缺密钥/坏密钥/短 JWT/显式禁用的独立诊断。 +- 错误原密钥/JWT、损坏密文、未知形状失败不写完成标记;dry-run 无持久写入,重跑不重复哈希或重置 Token。 +- AI 普通 GET 缺结果不调用模型/任务/写仓库;权限、SSE 字节和显式生成保持。 +- 新 POST 与旧 GET 在准备版并存;鉴权、下载、Admin 操作、SDK 调用真实通过。 +- 严格配置与规范排序在 v18 生效;准备版保证旧契约,SDK 转换后的实际排序正确。 +- 删除旧列的临时 schema 上运行准备版仓库/服务,不再引用旧列;旧字段从指标派生且数据覆盖完整。 +- 历史归档恢复和原秘密要求有演练;失败不报告成功,不暴露未准备数据。 +- docs:build、Astro check、产物死链、pangu、lint、真实预览并解析复制的 YAML,核对 llms 与版本入口。 + +既有基线问题:Core 全量测试的 `admin-agent.e2e-spec.ts` 缺 `AiAgentConversationRepository`,初始化和清理失败。验证时明确报告,不伪称全绿,也不把无关修理混入本次范围。 + +## 实现者裁量与记录 + +可自行决定内部模块、命令参数布局、事务批次和测试 fixture,但不得改变行为和失败边界。准备版最低版本、迁移 id、消费者最低 tag 以实施后的证据固定。测试验证行为,不对静态常量、源码行或内部对象做快照。 + +讨论文档:`document/mx-core-v18-compatibility`,路径 `.ryugi/artifacts/document/mx-core-v18-compatibility/doc.xml`,链接 https://ryugi.innei.dev/a/ZTsqm6DOJbAGibGoCjgD1。两轮 13 项选择及处理说明已归档,独立设计审阅结论为 “nothing important left”;用户随后明确同意开始准备版。未制作 UI mockup,保持现有视觉风格。 From ca5a29777d762600d8c85e507ae441d57647cbfc Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:41:59 +0800 Subject: [PATCH 02/26] feat(core): add compatible POST maintenance actions --- apps/core/src/app.controller.ts | 14 ++++ .../src/modules/backup/backup.controller.ts | 7 ++ .../src/modules/health/health.controller.ts | 9 ++- .../upgrade/compatibility-http.e2e-spec.ts | 76 +++++++++++++++++++ 4 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 apps/core/test/src/modules/upgrade/compatibility-http.e2e-spec.ts diff --git a/apps/core/src/app.controller.ts b/apps/core/src/app.controller.ts index cb75496bfdb..4eace5ac51c 100644 --- a/apps/core/src/app.controller.ts +++ b/apps/core/src/app.controller.ts @@ -88,10 +88,24 @@ export class AppController { await this.redisService.cleanCatch() } + @Post('/clean_catch') + @HttpCache.disable + @Auth() + async cleanCatchPost() { + return this.cleanCatch() + } + @Get('/clean_redis') @HttpCache.disable @Auth() async cleanAllRedisKey() { await this.redisService.cleanAllRedisKey() } + + @Post('/clean_redis') + @HttpCache.disable + @Auth() + async cleanAllRedisKeyPost() { + return this.cleanAllRedisKey() + } } diff --git a/apps/core/src/modules/backup/backup.controller.ts b/apps/core/src/modules/backup/backup.controller.ts index 8450e9317f7..93fb877ef19 100644 --- a/apps/core/src/modules/backup/backup.controller.ts +++ b/apps/core/src/modules/backup/backup.controller.ts @@ -31,6 +31,13 @@ import { BackupUploadInterceptor, RestoreKeySchema } from './backup-restore' export class BackupController { constructor(private readonly backupService: BackupService) {} + @Post('/new') + @HttpCode(200) + @HTTPDecorators.RawResponse + async createNewBackupPost(@Res({ passthrough: true }) reply: FastifyReply) { + return this.createNewBackup(reply) + } + @Get('/new') @HTTPDecorators.RawResponse async createNewBackup(@Res({ passthrough: true }) reply: FastifyReply) { diff --git a/apps/core/src/modules/health/health.controller.ts b/apps/core/src/modules/health/health.controller.ts index 538dec294a9..22c25120031 100644 --- a/apps/core/src/modules/health/health.controller.ts +++ b/apps/core/src/modules/health/health.controller.ts @@ -1,4 +1,4 @@ -import { Get } from '@nestjs/common' +import { Get, HttpCode, Post } from '@nestjs/common' import { ApiController } from '~/common/decorators/api-controller.decorator' import { Auth } from '~/common/decorators/auth.decorator' @@ -26,4 +26,11 @@ export class HealthController { } }) } + + @Post('/email/test') + @HttpCode(200) + @Auth() + async testEmailPost() { + return this.testEmail() + } } diff --git a/apps/core/test/src/modules/upgrade/compatibility-http.e2e-spec.ts b/apps/core/test/src/modules/upgrade/compatibility-http.e2e-spec.ts new file mode 100644 index 00000000000..04d28f8531b --- /dev/null +++ b/apps/core/test/src/modules/upgrade/compatibility-http.e2e-spec.ts @@ -0,0 +1,76 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +import { AppController } from '~/app.controller' +import { BackupController } from '~/modules/backup/backup.controller' +import { BackupService } from '~/modules/backup/backup.service' +import { ConfigsService } from '~/modules/configs/configs.service' +import { HealthController } from '~/modules/health/health.controller' +import { EmailService } from '~/processors/helper/helper.email.service' +import { RedisService } from '~/processors/redis/redis.service' + +import { createE2EApp } from '../../../helper/create-e2e-app' + +describe('v18 preparation HTTP compatibility', () => { + const cleanCatch = vi.fn() + const cleanAllRedisKey = vi.fn() + const sendTestEmail = vi.fn().mockResolvedValue(undefined) + const backup = vi.fn() + const proxy = createE2EApp({ + controllers: [AppController, HealthController, BackupController], + providers: [ + { provide: RedisService, useValue: { cleanCatch, cleanAllRedisKey } }, + { provide: ConfigsService, useValue: {} }, + { provide: EmailService, useValue: { sendTestEmail } }, + { provide: BackupService, useValue: { backup } }, + ], + }) + + beforeEach(() => vi.clearAllMocks()) + + for (const method of ['GET', 'POST'] as const) { + for (const [url, action] of [ + ['/clean_catch', cleanCatch], + ['/clean_redis', cleanAllRedisKey], + ['/health/email/test', sendTestEmail], + ] as const) { + it(`${method} ${url} requires authentication and executes once`, async () => { + const denied = await proxy.app.inject({ method, url }) + expect(denied.statusCode).toBe(401) + expect(action).not.toHaveBeenCalled() + const response = await proxy.app.inject({ + method, + url, + headers: { 'test-token': 1 }, + }) + expect(response.statusCode).toBe(204) + expect(action).toHaveBeenCalledTimes(1) + }) + } + + for (const extension of ['zip', 'mxbackup']) { + it(`${method} backup preserves ${extension} bytes, filename and type`, async () => { + const bytes = Buffer.from([0, 255, 80, 75, 1, 2]) + backup.mockResolvedValue({ + path: `/tmp/site.${extension}`, + buffer: bytes, + }) + const denied = await proxy.app.inject({ method, url: '/backups/new' }) + expect(denied.statusCode).toBe(401) + expect(backup).not.toHaveBeenCalled() + const response = await proxy.app.inject({ + method, + url: '/backups/new', + headers: { 'test-token': 1 }, + }) + expect(response.statusCode).toBe(200) + expect(response.rawPayload).toEqual(bytes) + expect(response.headers['content-disposition']).toBe( + `attachment; filename="site.${extension}"`, + ) + expect(response.headers['content-type']).toContain( + extension === 'zip' ? 'application/zip' : 'application/octet-stream', + ) + }) + } + } +}) From 41d877fe43a3ac776bfbf4701bd5e739ae09568f Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:44:34 +0800 Subject: [PATCH 03/26] feat(clients): prepare transport for v18 contracts --- apps/admin/src/api/aggregate.ts | 6 +- apps/admin/src/api/backups.ts | 2 +- apps/admin/src/api/health.ts | 7 +- .../src/api/upgrade-compatibility.test.ts | 40 ++++++++++ .../__tests__/controllers/owner.test.ts | 2 +- .../__tests__/controllers/recently.test.ts | 12 ++- .../__tests__/upgrade-compatibility.test.ts | 75 +++++++++++++++++++ packages/api-client/controllers/ai.ts | 6 +- packages/api-client/controllers/base.ts | 11 ++- packages/api-client/controllers/note.ts | 13 +++- packages/api-client/controllers/owner.ts | 5 +- packages/api-client/controllers/page.ts | 5 +- packages/api-client/controllers/post.ts | 5 +- packages/api-client/controllers/recently.ts | 2 +- 14 files changed, 167 insertions(+), 24 deletions(-) create mode 100644 apps/admin/src/api/upgrade-compatibility.test.ts create mode 100644 packages/api-client/__tests__/upgrade-compatibility.test.ts diff --git a/apps/admin/src/api/aggregate.ts b/apps/admin/src/api/aggregate.ts index 48a642b5cc5..54d10d2fd6e 100644 --- a/apps/admin/src/api/aggregate.ts +++ b/apps/admin/src/api/aggregate.ts @@ -1,7 +1,7 @@ import type { ContentDocument, DraftModel } from '~/models/draft' import type { RecentActivities } from './activity' -import { getJson } from './http' +import { getJson, requestJson } from './http' export interface StatCount { allComments?: number @@ -180,9 +180,9 @@ export function getSiteLikeCount() { } export function cleanCache() { - return getJson('/clean_catch') + return requestJson('/clean_catch', { method: 'POST' }) } export function cleanRedis() { - return getJson('/clean_redis') + return requestJson('/clean_redis', { method: 'POST' }) } diff --git a/apps/admin/src/api/backups.ts b/apps/admin/src/api/backups.ts index 2ce311d36ef..fba3a1b3ac9 100644 --- a/apps/admin/src/api/backups.ts +++ b/apps/admin/src/api/backups.ts @@ -18,7 +18,7 @@ export async function getBackups() { } export function createBackup() { - return requestBlob('/backups/new') + return requestBlob('/backups/new', { method: 'POST' }) } export function downloadBackup(filename: string) { diff --git a/apps/admin/src/api/health.ts b/apps/admin/src/api/health.ts index 6c2f4946b54..4847254903e 100644 --- a/apps/admin/src/api/health.ts +++ b/apps/admin/src/api/health.ts @@ -1,5 +1,8 @@ -import { getJson } from './http' +import { requestJson } from './http' export function sendTestEmail() { - return getJson<{ message?: string; trace?: string }>('/health/email/test') + return requestJson<{ message?: string; trace?: string }>( + '/health/email/test', + { method: 'POST' }, + ) } diff --git a/apps/admin/src/api/upgrade-compatibility.test.ts b/apps/admin/src/api/upgrade-compatibility.test.ts new file mode 100644 index 00000000000..23995c18e66 --- /dev/null +++ b/apps/admin/src/api/upgrade-compatibility.test.ts @@ -0,0 +1,40 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' + +import { cleanCache, cleanRedis } from './aggregate' +import { createBackup } from './backups' +import { sendTestEmail } from './health' + +describe('maintenance transport', () => { + afterEach(() => vi.restoreAllMocks()) + + it.each([ + ['/clean_catch', cleanCache], + ['/clean_redis', cleanRedis], + ['/health/email/test', sendTestEmail], + ] as const)('posts authenticated action %s', async (path, action) => { + const request = vi + .spyOn(globalThis, 'fetch') + .mockResolvedValue(new Response(null, { status: 204 })) + await action() + expect(String(request.mock.calls[0][0])).toMatch(new RegExp(`${path}$`)) + expect(request.mock.calls[0][1]).toMatchObject({ + method: 'POST', + credentials: 'include', + }) + }) + + it('posts backup creation and retains the downloaded bytes', async () => { + const bytes = new Uint8Array([0, 255, 80, 75]) + const request = vi.spyOn(globalThis, 'fetch').mockResolvedValue( + new Response(bytes, { + headers: { 'content-type': 'application/octet-stream' }, + }), + ) + const result = await createBackup() + expect(new Uint8Array(await result.arrayBuffer())).toEqual(bytes) + expect(request.mock.calls[0][1]).toMatchObject({ + method: 'POST', + credentials: 'include', + }) + }) +}) diff --git a/packages/api-client/__tests__/controllers/owner.test.ts b/packages/api-client/__tests__/controllers/owner.test.ts index eeaad5a1345..56511854d60 100644 --- a/packages/api-client/__tests__/controllers/owner.test.ts +++ b/packages/api-client/__tests__/controllers/owner.test.ts @@ -52,7 +52,7 @@ describe('test owner client', () => { }) test('GET /owner/check_logged', async () => { - const mocked = mockResponse('/owner/check_logged?token=token', { + const mocked = mockResponse('/owner/check_logged', { isGuest: true, ok: 0, }) diff --git a/packages/api-client/__tests__/controllers/recently.test.ts b/packages/api-client/__tests__/controllers/recently.test.ts index fe2dd3a2f5c..e957c3f1d56 100644 --- a/packages/api-client/__tests__/controllers/recently.test.ts +++ b/packages/api-client/__tests__/controllers/recently.test.ts @@ -55,11 +55,15 @@ describe('test recently client, /recently', () => { expect(data).toEqual(mocked) }) - test('GET /attitude', async () => { + test('POST /attitude', async () => { const id = `1212121` - const mocked = mockResponse(`/recently/attitude/${id}?attitude=1`, { - code: 1, - }) + const mocked = mockResponse( + `/recently/attitude/${id}`, + { + code: 1, + }, + 'post', + ) const data = await client.recently.attitude(id, 1) expect(data).toEqual(mocked) }) diff --git a/packages/api-client/__tests__/upgrade-compatibility.test.ts b/packages/api-client/__tests__/upgrade-compatibility.test.ts new file mode 100644 index 00000000000..1618b8ef635 --- /dev/null +++ b/packages/api-client/__tests__/upgrade-compatibility.test.ts @@ -0,0 +1,75 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' + +import { axiosAdaptor } from '~/adaptors/axios' +import { allControllers } from '~/controllers' +import { createClient } from '~/core' + +describe('preparation transport', () => { + const client = createClient(axiosAdaptor)('https://example.com/api/v3') + client.injectControllers(allControllers) + afterEach(() => vi.restoreAllMocks()) + + it('uses POST for attitudes with the existing query and response', async () => { + vi.spyOn(axiosAdaptor, 'get').mockRejectedValue( + new Error('GET must not execute an attitude'), + ) + const request = vi + .spyOn(axiosAdaptor, 'post') + .mockResolvedValue({ data: { data: { code: 1 } } }) + expect(await client.recently.attitude('123', 1)).toEqual({ code: 1 }) + expect(request.mock.calls[0][0]).toBe( + 'https://example.com/api/v3/recently/attitude/123', + ) + expect(request.mock.calls[0][1]?.params).toEqual({ attitude: 1 }) + }) + + it.each([ + [1, 'asc'], + [-1, 'desc'], + ] as const)( + 'normalizes %s sorting without changing caller input', + async (input, expected) => { + const request = vi + .spyOn(axiosAdaptor, 'get') + .mockResolvedValue({ data: { data: [] } }) + const sort = Object.freeze({ sortBy: 'created', sortOrder: input }) + await client.link.getAllPaginated(1, 10, sort) + const url = new URL(request.mock.calls[0][0]) + expect(url.searchParams.get('sortOrder')).toBe(expected) + expect(sort.sortOrder).toBe(input) + }, + ) + + it('authenticates the retained token check signature using a header', async () => { + const request = vi + .spyOn(axiosAdaptor, 'get') + .mockResolvedValue({ data: { data: { ok: 1 } } }) + await client.owner.checkTokenValid(' Bearer secret-token ') + expect(request.mock.calls[0][0]).not.toContain('token') + expect(request.mock.calls[0][1]?.params).toBeUndefined() + expect(request.mock.calls[0][1]?.headers).toMatchObject({ + Authorization: 'Bearer secret-token', + }) + }) + + it('normalizes article list sorting at the adapter boundary', async () => { + const request = vi + .spyOn(axiosAdaptor, 'get') + .mockResolvedValue({ data: { data: [] } }) + await client.post.getList(1, 10, { sortOrder: -1 }) + expect( + new URL(request.mock.calls[0][0]).searchParams.get('sortOrder'), + ).toBe('desc') + }) + + it('keeps the legacy summary signature without leaking its token in URLs', async () => { + const request = vi + .spyOn(axiosAdaptor, 'post') + .mockResolvedValue({ data: { data: {} } }) + await client.ai.generateSummary('123', 'zh-CN', 'secret-token') + expect(request.mock.calls[0][0]).not.toContain('token') + expect(request.mock.calls[0][1]?.headers).toMatchObject({ + Authorization: 'Bearer secret-token', + }) + }) +}) diff --git a/packages/api-client/controllers/ai.ts b/packages/api-client/controllers/ai.ts index 12f98393910..9c54bdb276b 100644 --- a/packages/api-client/controllers/ai.ts +++ b/packages/api-client/controllers/ai.ts @@ -55,9 +55,9 @@ export class AIController implements IController { async generateSummary(articleId: string, lang = 'zh-CN', token = '') { return this.proxy.summaries.generate.post({ - params: { - token, - }, + headers: token.trim() + ? { Authorization: `Bearer ${token.trim().replace(/^bearer\s+/i, '')}` } + : undefined, data: { lang, refId: articleId, diff --git a/packages/api-client/controllers/base.ts b/packages/api-client/controllers/base.ts index 55726a975cc..586169ce171 100644 --- a/packages/api-client/controllers/base.ts +++ b/packages/api-client/controllers/base.ts @@ -8,6 +8,10 @@ export type SortOptions = { sortBy?: string sortOrder?: 1 | -1 | 'asc' | 'desc' } +export function normalizeSortOrder(order: SortOptions['sortOrder']) { + return order === 1 ? 'asc' : order === -1 ? 'desc' : order +} + export abstract class BaseCrudController { base!: string constructor(protected client: HTTPClient) { @@ -31,7 +35,12 @@ export abstract class BaseCrudController { */ getAllPaginated(page?: number, perPage?: number, sortOption?: SortOptions) { return this.proxy.get>({ - params: { page, size: perPage, ...sortOption }, + params: { + page, + size: perPage, + ...sortOption, + sortOrder: normalizeSortOrder(sortOption?.sortOrder), + }, }) } } diff --git a/packages/api-client/controllers/note.ts b/packages/api-client/controllers/note.ts index b15b4c80a72..1e28b42b135 100644 --- a/packages/api-client/controllers/note.ts +++ b/packages/api-client/controllers/note.ts @@ -7,6 +7,7 @@ import { autoBind } from '~/utils/auto-bind' import type { HTTPClient } from '../core/client' import type { SortOptions } from './base' +import { normalizeSortOrder } from './base' declare module '@mx-space/api-client' { interface HTTPClient< @@ -20,7 +21,7 @@ declare module '@mx-space/api-client' { export type NoteListOptions = { year?: number sortBy?: 'weather' | 'mood' | 'title' | 'createdAt' | 'modifiedAt' - sortOrder?: 1 | -1 + sortOrder?: 1 | -1 | 'asc' | 'desc' lang?: string withSummary?: boolean } @@ -149,7 +150,7 @@ export class NoteController implements IController { page, size: perPage, sortBy, - sortOrder, + sortOrder: normalizeSortOrder(sortOrder), year, lang, withSummary: withSummary ? '1' : undefined, @@ -188,7 +189,13 @@ export class NoteController implements IController { ) { const { lang, ...sortOptions } = options return this.proxy.topics(topicId).get>({ - params: { page, size, lang, ...sortOptions }, + params: { + page, + size, + lang, + ...sortOptions, + sortOrder: normalizeSortOrder(sortOptions.sortOrder), + }, }) } diff --git a/packages/api-client/controllers/owner.ts b/packages/api-client/controllers/owner.ts index 2595c0caac7..4a884db478b 100644 --- a/packages/api-client/controllers/owner.ts +++ b/packages/api-client/controllers/owner.ts @@ -11,6 +11,7 @@ import type { UserModel, } from '~/models/user' import { autoBind } from '~/utils/auto-bind' + import type { HTTPClient } from '../core' declare module '@mx-space/api-client' { @@ -130,7 +131,9 @@ export class UserController implements IController { const normalized = this.normalizeToken(token) return this.proxy.check_logged.get({ - params: normalized ? { token: normalized } : undefined, + headers: normalized + ? { Authorization: `Bearer ${normalized}` } + : undefined, }) } } diff --git a/packages/api-client/controllers/page.ts b/packages/api-client/controllers/page.ts index 22f9383802a..3f6a16e8c85 100644 --- a/packages/api-client/controllers/page.ts +++ b/packages/api-client/controllers/page.ts @@ -6,6 +6,7 @@ import type { PageModel } from '~/models/page' import { autoBind } from '~/utils/auto-bind' import type { HTTPClient } from '../core' +import { normalizeSortOrder } from './base' declare module '@mx-space/api-client' { interface HTTPClient< @@ -18,7 +19,7 @@ declare module '@mx-space/api-client' { export type PageListOptions = { sortBy?: 'order' | 'subtitle' | 'title' | 'createdAt' | 'modifiedAt' - sortOrder?: 1 | -1 + sortOrder?: 1 | -1 | 'asc' | 'desc' } export class PageController implements IController { @@ -40,7 +41,7 @@ export class PageController implements IController { page, size: perPage, sortBy, - sortOrder, + sortOrder: normalizeSortOrder(sortOrder), }, }) } diff --git a/packages/api-client/controllers/post.ts b/packages/api-client/controllers/post.ts index 96da02f87fe..85a8f47bd0a 100644 --- a/packages/api-client/controllers/post.ts +++ b/packages/api-client/controllers/post.ts @@ -9,6 +9,7 @@ import type { PostModel } from '~/models/post' import { autoBind } from '~/utils/auto-bind' import type { HTTPClient } from '../core/client' +import { normalizeSortOrder } from './base' declare module '@mx-space/api-client' { interface HTTPClient< @@ -22,7 +23,7 @@ declare module '@mx-space/api-client' { export type PostListOptions = { year?: number sortBy?: 'categoryId' | 'title' | 'createdAt' | 'modifiedAt' | 'pinAt' - sortOrder?: 1 | -1 + sortOrder?: 1 | -1 | 'asc' | 'desc' truncate?: number /** 语言代码,用于获取翻译版本 */ lang?: string @@ -48,7 +49,7 @@ export class PostController implements IController { page, size: perPage, sortBy, - sortOrder, + sortOrder: normalizeSortOrder(sortOrder), year, truncate, lang, diff --git a/packages/api-client/controllers/recently.ts b/packages/api-client/controllers/recently.ts index 1b8252289e1..3a9cb7ba5f8 100644 --- a/packages/api-client/controllers/recently.ts +++ b/packages/api-client/controllers/recently.ts @@ -72,7 +72,7 @@ export class RecentlyController implements IController { /** 表态:点赞,点踩 */ attitude(id: string, attitude: RecentlyAttitudeEnum) { - return this.proxy.attitude(id).get<{ code: RecentlyAttitudeResultEnum }>({ + return this.proxy.attitude(id).post<{ code: RecentlyAttitudeResultEnum }>({ params: { attitude, }, From 5d5dc53ca21aa96a83f9e9accb6cd277deefb3c4 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:48:02 +0800 Subject: [PATCH 04/26] refactor(core): canonicalize legacy configuration at storage boundary --- .../modules/configs/configs.encrypt.util.ts | 6 +- .../src/modules/configs/configs.service.ts | 309 +++-------------- .../src/modules/configs/legacy-config.util.ts | 322 ++++++++++++++++++ .../src/modules/configs/options.repository.ts | 13 + .../modules/configs/configs.service.spec.ts | 67 +++- .../src/modules/configs/legacy-config.spec.ts | 77 +++++ 6 files changed, 525 insertions(+), 269 deletions(-) create mode 100644 apps/core/src/modules/configs/legacy-config.util.ts create mode 100644 apps/core/test/src/modules/configs/legacy-config.spec.ts diff --git a/apps/core/src/modules/configs/configs.encrypt.util.ts b/apps/core/src/modules/configs/configs.encrypt.util.ts index b987804bf39..63cf47de578 100644 --- a/apps/core/src/modules/configs/configs.encrypt.util.ts +++ b/apps/core/src/modules/configs/configs.encrypt.util.ts @@ -17,7 +17,11 @@ function getEncryptedPaths(): Set { return cachedEncryptedPaths } - cachedEncryptedPaths = new Set() + cachedEncryptedPaths = new Set([ + 'thirdPartyServiceIntegration.githubToken', + 'membership.dodoApiKey', + 'membership.dodoWebhookKey', + ]) for (const [sectionKey, schema] of Object.entries(configSchemaMapping)) { const paths = collectEncryptedPathsFromSchema(schema, sectionKey) diff --git a/apps/core/src/modules/configs/configs.service.ts b/apps/core/src/modules/configs/configs.service.ts index 4ec878ec62d..8a431695a4d 100644 --- a/apps/core/src/modules/configs/configs.service.ts +++ b/apps/core/src/modules/configs/configs.service.ts @@ -11,7 +11,6 @@ import { type AIModelAssignment, type AIProviderCapability, type AIProviderConfig, - AIProviderType, } from '~/modules/ai/ai.types' import { EventManagerService } from '~/processors/helper/helper.event.service' import { @@ -32,9 +31,13 @@ import { } from './configs.encrypt.util' import { configDtoMapping, IConfig } from './configs.interface' import type { OAuthConfig } from './configs.schema' +import { + normalizeLegacyConfig, + normalizeLegacyPatch, + updateLegacyMedia, +} from './legacy-config.util' import { OptionsRepository } from './options.repository' -const configsKeySet = new Set(Object.keys(configDtoMapping)) const aggregateConfigKeys = new Set([ 'ai', 'commentOptions', @@ -60,27 +63,6 @@ function normalizeS3StorageOptionNulls(value: T): T { return normalized as T } -function isGoogleVertexEndpoint(endpoint: string): boolean { - try { - const url = new URL(endpoint) - return ( - url.hostname === 'aiplatform.googleapis.com' && - url.pathname.endsWith('/endpoints/openapi') - ) - } catch { - return false - } -} - -function extractGoogleVertexProjectId(endpoint: string): string | undefined { - try { - const match = new URL(endpoint).pathname.match(/\/projects\/([^/]+)/) - return match?.[1] ? decodeURIComponent(match[1]) : undefined - } catch { - return undefined - } -} - /* * NOTE: * 1. Configs live in Redis. `getConfig` is the single entry point; all reads @@ -172,247 +154,11 @@ export class ConfigsService implements OnModuleInit { return } - const configs = await this.optionsRepository.findAll() - const storedAiConfig = configs.find((field) => field.name === 'ai') - ?.value as Record | undefined - const mergedConfig = generateDefaultConfig() - const mergeStoredConfig = ( - name: T, - value: unknown, - ) => { - const storedValue = - value && typeof value === 'object' ? (value as Partial) : {} - mergedConfig[name] = { - ...mergedConfig[name], - ...storedValue, - } - } - configs.forEach((field) => { - const name = field.name as keyof IConfig - - if (!configsKeySet.has(name)) { - return - } - // skip url patch in dev mode - if (isDev && name === 'url') { - return - } - - // Backward-compat: migrate old flat thirdPartyServiceIntegration - if (name === 'thirdPartyServiceIntegration') { - const normalized = this.normalizeThirdPartyConfig( - field.value as Record, - ) - mergeStoredConfig(name, normalized) - } else if (name === 'membership') { - const normalized = this.normalizeMembershipConfig( - field.value as Record, - ) - mergeStoredConfig(name, normalized) - } else { - mergeStoredConfig(name, field.value) - } - }) - - mergedConfig.ai = this.normalizeAiConfig( - storedAiConfig, - mergedConfig.ai, - mergedConfig.imageGenerationOptions, - mergedConfig.ttsOptions, - ) - - await this.setConfig(mergedConfig) - } - - private normalizeAiConfig( - storedAiConfig: Record | undefined, - aiConfig: IConfig['ai'], - legacyImageConfig: IConfig['imageGenerationOptions'], - legacyTtsConfig: IConfig['ttsOptions'], - ): IConfig['ai'] { - const providers = (aiConfig.providers ?? []).map((provider) => { - const endpoint = provider.endpoint?.trim() ?? '' - const isLegacyVertex = - provider.type === AIProviderType.OpenAICompatible && - isGoogleVertexEndpoint(endpoint) - return { - ...provider, - ...(isLegacyVertex - ? { - type: AIProviderType.GoogleVertex, - modelListUrl: undefined, - } - : {}), - projectId: provider.projectId ?? extractGoogleVertexProjectId(endpoint), - capabilities: isLegacyVertex - ? { text: true, image: true, speech: true } - : { - decision: provider.capabilities?.decision ?? false, - text: provider.capabilities?.text ?? true, - image: provider.capabilities?.image ?? false, - speech: provider.capabilities?.speech ?? false, - }, - } - }) - - if (storedAiConfig?.version === 2) { - return { ...aiConfig, version: 2, providers } - } - - const imageGeneration = { - enable: legacyImageConfig.enable, - model: undefined as { providerId: string; model?: string } | undefined, - defaultAspectRatio: legacyImageConfig.defaultAspectRatio, - defaultQuality: legacyImageConfig.defaultQuality, - defaultFormat: legacyImageConfig.defaultFormat, - } - const tts = { - enable: legacyTtsConfig.enable, - model: undefined as { providerId: string; model?: string } | undefined, - voice: legacyTtsConfig.voice, - speed: legacyTtsConfig.speed, - maxCharsPerChunk: legacyTtsConfig.maxCharsPerChunk, - concurrency: legacyTtsConfig.concurrency, - maxCharsPerRun: legacyTtsConfig.maxCharsPerRun, - } - - const addLegacyProvider = (input: { - apiKey?: string | null - capability: 'image' | 'speech' - defaultModel?: string | null - endpoint?: string | null - id: string - name: string - }): string | undefined => { - if (!input.apiKey) return undefined - - const normalizedEndpoint = input.endpoint?.trim().replace(/\/+$/, '') - const existing = providers.find( - (provider) => - provider.apiKey === input.apiKey && - provider.endpoint?.trim().replace(/\/+$/, '') === normalizedEndpoint, - ) - if (existing) { - existing.capabilities = { - text: existing.capabilities?.text ?? true, - image: existing.capabilities?.image || input.capability === 'image', - speech: - existing.capabilities?.speech || input.capability === 'speech', - } - return existing.id - } - - let id = input.id - let suffix = 2 - while (providers.some((provider) => provider.id === id)) { - id = `${input.id}-${suffix}` - suffix += 1 - } - - providers.push({ - id, - name: input.name, - type: AIProviderType.OpenAICompatible, - apiKey: input.apiKey, - endpoint: normalizedEndpoint, - projectId: undefined, - defaultModel: input.defaultModel ?? '', - enabled: true, - capabilities: { - text: false, - image: input.capability === 'image', - speech: input.capability === 'speech', - }, - }) - return id - } - - const imageProviderId = addLegacyProvider({ - apiKey: legacyImageConfig.apiKey, - capability: 'image', - defaultModel: legacyImageConfig.model, - endpoint: - legacyImageConfig.endpoint || - (legacyImageConfig.provider === 'openrouter' - ? 'https://openrouter.ai/api/v1' - : undefined), - id: '__legacy_image_generation__', - name: 'Imported image generation provider', - }) - if (imageProviderId && legacyImageConfig.model) { - imageGeneration.model = { - providerId: imageProviderId, - model: legacyImageConfig.model, - } - } - - const ttsProviderId = addLegacyProvider({ - apiKey: legacyTtsConfig.apiKey, - capability: 'speech', - defaultModel: legacyTtsConfig.model, - endpoint: - legacyTtsConfig.endpoint || - (legacyTtsConfig.provider === 'openrouter' - ? 'https://openrouter.ai/api/v1' - : legacyTtsConfig.provider === 'openai' - ? 'https://api.openai.com/v1' - : undefined), - id: '__legacy_tts__', - name: 'Imported speech provider', - }) - if (ttsProviderId && legacyTtsConfig.model) { - tts.model = { - providerId: ttsProviderId, - model: legacyTtsConfig.model, - } - } - - return { - ...aiConfig, - version: 2, - providers, - imageGeneration, - tts, - } - } - - /** - * Backward-compat: migrate old flat { githubToken } to new nested - * { github: { enabled, token } } shape at read time. - */ - private normalizeThirdPartyConfig( - raw: Record, - ): IConfig['thirdPartyServiceIntegration'] { - // Already new shape - if (raw?.github && typeof raw.github === 'object') { - return raw as IConfig['thirdPartyServiceIntegration'] - } - // Old flat shape — convert - return { - github: { enabled: true, token: raw.githubToken || '' }, - tmdb: { enabled: false, apiKey: '' }, - bangumi: { enabled: true, accessToken: '' }, - neodb: { enabled: true }, - arxiv: { enabled: true }, - leetcode: { enabled: true }, - neteaseMusic: { enabled: true }, - qqMusic: { enabled: true }, - } as IConfig['thirdPartyServiceIntegration'] - } - - private normalizeMembershipConfig( - raw: Record, - ): IConfig['membership'] { - const { dodoApiKey, dodoEnvironment, dodoWebhookKey, ...providerConfig } = - raw - - return { - ...providerConfig, - apiKey: providerConfig.apiKey ?? dodoApiKey ?? '', - webhookSigningKey: - providerConfig.webhookSigningKey ?? dodoWebhookKey ?? '', - environment: providerConfig.environment ?? dodoEnvironment ?? 'live_mode', - } as IConfig['membership'] + const rows = await this.optionsRepository.findAll() + const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) + const plaintext = decryptObject(cloneDeep(raw)) + if (isDev) delete plaintext.url + await this.setConfig(normalizeLegacyConfig(plaintext)) } public async get( @@ -475,6 +221,20 @@ export class ConfigsService implements OnModuleInit { data: Partial, ): Promise { const config = await this.getConfig() + if (key === 'imageGenerationOptions' || key === 'ttsOptions') { + const canonical = updateLegacyMedia(config as IConfig, key, data) + await this.optionsRepository.upsert( + 'ai', + encryptObject(cloneDeep(canonical.ai), 'ai'), + ) + await this.optionsRepository.deleteByName?.('imageGenerationOptions') + await this.optionsRepository.deleteByName?.('ttsOptions') + await this.setConfig(canonical) + await this.eventManager.emit(EventBusEvents.ConfigChanged, canonical.ai, { + scope: EventScope.TO_SYSTEM, + }) + return encryptObject(cloneDeep(canonical[key]), key) as IConfig[T] + } const updatedConfigRow = await this.optionsRepository.upsert( key as string, encryptObject( @@ -540,7 +300,26 @@ export class ConfigsService implements OnModuleInit { key: T, value: Partial, ) { - value = camelcaseKeys(value) as any + if (this.optionsRepository.withConfigLock) { + return this.optionsRepository.withConfigLock(async (repository) => { + const scoped = new ConfigsService( + repository, + this.redisService, + this.configVersionService, + this.eventManager, + ) + await scoped.ensureConfigInitialized() + return scoped.patchAndValidUnlocked(key, value) + }) + } + return this.patchAndValidUnlocked(key, value) + } + + private async patchAndValidUnlocked( + key: T, + value: Partial, + ) { + value = normalizeLegacyPatch(key, camelcaseKeys(value)) as any value = removeEmptyEncryptedFields(value as object, key) as Partial< IConfig[T] > diff --git a/apps/core/src/modules/configs/legacy-config.util.ts b/apps/core/src/modules/configs/legacy-config.util.ts new file mode 100644 index 00000000000..7e433bc7252 --- /dev/null +++ b/apps/core/src/modules/configs/legacy-config.util.ts @@ -0,0 +1,322 @@ +import { cloneDeep } from 'es-toolkit/compat' + +import { AIProviderType } from '~/modules/ai/ai.types' + +import { generateDefaultConfig } from './configs.default' +import type { IConfig } from './configs.interface' + +function isGoogleVertexEndpoint(endpoint: string): boolean { + try { + const url = new URL(endpoint) + return ( + url.hostname === 'aiplatform.googleapis.com' && + url.pathname.endsWith('/endpoints/openapi') + ) + } catch { + return false + } +} + +function extractGoogleVertexProjectId(endpoint: string): string | undefined { + try { + const match = new URL(endpoint).pathname.match(/\/projects\/([^/]+)/) + return match?.[1] ? decodeURIComponent(match[1]) : undefined + } catch { + return undefined + } +} + +export function normalizeAiConfig( + storedAiConfig: Record | undefined, + aiConfig: IConfig['ai'], + legacyImageConfig: IConfig['imageGenerationOptions'], + legacyTtsConfig: IConfig['ttsOptions'], +): IConfig['ai'] { + const providers = (aiConfig.providers ?? []).map((provider) => { + const endpoint = provider.endpoint?.trim() ?? '' + const isLegacyVertex = + provider.type === AIProviderType.OpenAICompatible && + isGoogleVertexEndpoint(endpoint) + return { + ...provider, + ...(isLegacyVertex + ? { + type: AIProviderType.GoogleVertex, + modelListUrl: undefined, + } + : {}), + projectId: provider.projectId ?? extractGoogleVertexProjectId(endpoint), + capabilities: isLegacyVertex + ? { text: true, image: true, speech: true } + : { + decision: provider.capabilities?.decision ?? false, + text: provider.capabilities?.text ?? true, + image: provider.capabilities?.image ?? false, + speech: provider.capabilities?.speech ?? false, + }, + } + }) + + if (storedAiConfig?.version === 2) { + return { ...aiConfig, version: 2, providers } + } + + const imageGeneration = { + enable: legacyImageConfig.enable, + model: undefined as { providerId: string; model?: string } | undefined, + defaultAspectRatio: legacyImageConfig.defaultAspectRatio, + defaultQuality: legacyImageConfig.defaultQuality, + defaultFormat: legacyImageConfig.defaultFormat, + } + const tts = { + enable: legacyTtsConfig.enable, + model: undefined as { providerId: string; model?: string } | undefined, + voice: legacyTtsConfig.voice, + speed: legacyTtsConfig.speed, + maxCharsPerChunk: legacyTtsConfig.maxCharsPerChunk, + concurrency: legacyTtsConfig.concurrency, + maxCharsPerRun: legacyTtsConfig.maxCharsPerRun, + } + + const addLegacyProvider = (input: { + apiKey?: string | null + capability: 'image' | 'speech' + defaultModel?: string | null + endpoint?: string | null + id: string + name: string + }): string | undefined => { + if (!input.apiKey) return undefined + + const normalizedEndpoint = input.endpoint?.trim().replace(/\/+$/, '') + const existing = providers.find( + (provider) => + provider.apiKey === input.apiKey && + provider.endpoint?.trim().replace(/\/+$/, '') === normalizedEndpoint, + ) + if (existing) { + existing.capabilities = { + text: existing.capabilities?.text ?? true, + image: existing.capabilities?.image || input.capability === 'image', + speech: existing.capabilities?.speech || input.capability === 'speech', + } + return existing.id + } + + let id = input.id + let suffix = 2 + while (providers.some((provider) => provider.id === id)) { + id = `${input.id}-${suffix}` + suffix += 1 + } + + providers.push({ + id, + name: input.name, + type: AIProviderType.OpenAICompatible, + apiKey: input.apiKey, + endpoint: normalizedEndpoint, + projectId: undefined, + defaultModel: input.defaultModel ?? '', + enabled: true, + capabilities: { + decision: false, + text: false, + image: input.capability === 'image', + speech: input.capability === 'speech', + }, + }) + return id + } + + const imageProviderId = addLegacyProvider({ + apiKey: legacyImageConfig.apiKey, + capability: 'image', + defaultModel: legacyImageConfig.model, + endpoint: + legacyImageConfig.endpoint || + (legacyImageConfig.provider === 'openrouter' + ? 'https://openrouter.ai/api/v1' + : undefined), + id: '__legacy_image_generation__', + name: 'Imported image generation provider', + }) + if (imageProviderId && legacyImageConfig.model) { + imageGeneration.model = { + providerId: imageProviderId, + model: legacyImageConfig.model, + } + } + + const ttsProviderId = addLegacyProvider({ + apiKey: legacyTtsConfig.apiKey, + capability: 'speech', + defaultModel: legacyTtsConfig.model, + endpoint: + legacyTtsConfig.endpoint || + (legacyTtsConfig.provider === 'openrouter' + ? 'https://openrouter.ai/api/v1' + : legacyTtsConfig.provider === 'openai' + ? 'https://api.openai.com/v1' + : undefined), + id: '__legacy_tts__', + name: 'Imported speech provider', + }) + if (ttsProviderId && legacyTtsConfig.model) { + tts.model = { + providerId: ttsProviderId, + model: legacyTtsConfig.model, + } + } + + return { + ...aiConfig, + version: 2, + providers, + imageGeneration, + tts, + } +} + +/** + * Backward-compat: migrate old flat { githubToken } to new nested + * { github: { enabled, token } } shape at read time. + */ +export function normalizeThirdPartyConfig( + raw: Record, +): IConfig['thirdPartyServiceIntegration'] { + // Already new shape + if (raw?.github && typeof raw.github === 'object') { + return raw as IConfig['thirdPartyServiceIntegration'] + } + // Old flat shape — convert + return { + github: { enabled: true, token: raw.githubToken || '' }, + tmdb: { enabled: false, apiKey: '' }, + bangumi: { enabled: true, accessToken: '' }, + neodb: { enabled: true }, + arxiv: { enabled: true }, + leetcode: { enabled: true }, + neteaseMusic: { enabled: true }, + qqMusic: { enabled: true }, + } as IConfig['thirdPartyServiceIntegration'] +} + +export function normalizeMembershipConfig( + raw: Record, +): IConfig['membership'] { + const { dodoApiKey, dodoEnvironment, dodoWebhookKey, ...providerConfig } = raw + + return { + ...providerConfig, + apiKey: providerConfig.apiKey ?? dodoApiKey ?? '', + webhookSigningKey: providerConfig.webhookSigningKey ?? dodoWebhookKey ?? '', + environment: providerConfig.environment ?? dodoEnvironment ?? 'live_mode', + } as IConfig['membership'] +} + +/** Input must be decrypted at its original paths before calling this converter. */ +export function normalizeLegacyConfig(raw: Record): IConfig { + const config = generateDefaultConfig() + for (const key of Object.keys(config) as Array) { + if (raw[key] && typeof raw[key] === 'object' && !Array.isArray(raw[key])) { + ;(config as any)[key] = { ...config[key], ...cloneDeep(raw[key]) } + } + } + if (raw.thirdPartyServiceIntegration) + config.thirdPartyServiceIntegration = { + ...config.thirdPartyServiceIntegration, + ...normalizeThirdPartyConfig(raw.thirdPartyServiceIntegration), + } + if (raw.membership) + config.membership = { + ...config.membership, + ...normalizeMembershipConfig(raw.membership), + } + delete (config.thirdPartyServiceIntegration as any).githubToken + for (const key of ['dodoApiKey', 'dodoEnvironment', 'dodoWebhookKey']) + delete (config.membership as any)[key] + config.ai = normalizeAiConfig( + raw.ai, + config.ai, + config.imageGenerationOptions, + config.ttsOptions, + ) + { + for (const [section, feature, legacyId] of [ + [ + 'imageGenerationOptions', + 'imageGeneration', + '__legacy_image_generation__', + ], + ['ttsOptions', 'tts', '__legacy_tts__'], + ] as const) { + const route = config.ai[feature] + const provider = + (config.ai.providers ?? []).find( + (p) => p.id === route?.model?.providerId, + ) ?? (config.ai.providers ?? []).find((p) => p.id === legacyId) + const { model: assignment, ...settings } = route ?? {} + ;(config as any)[section] = { + ...config[section], + ...settings, + ...(provider + ? { + apiKey: provider.apiKey, + endpoint: provider.endpoint, + model: assignment?.model ?? provider.defaultModel, + provider: provider.endpoint?.includes('openrouter.ai') + ? 'openrouter' + : 'custom', + } + : {}), + } + } + } + return config +} + +/** Canonical storage has one source for media credentials and assignments. */ +export function canonicalConfigRows(config: IConfig): Record { + const { + imageGenerationOptions: _image, + ttsOptions: _speech, + ...canonical + } = config + return canonical +} + +export function normalizeLegacyPatch(key: keyof IConfig, input: object) { + const value = cloneDeep(input) as Record + if (key === 'thirdPartyServiceIntegration' && 'githubToken' in value) { + value.github = { token: value.githubToken, ...value.github } + delete value.githubToken + } + if (key === 'membership') { + for (const [old, current] of [ + ['dodoApiKey', 'apiKey'], + ['dodoWebhookKey', 'webhookSigningKey'], + ['dodoEnvironment', 'environment'], + ]) { + if (old in value && !(current in value)) value[current] = value[old] + delete value[old] + } + } + return value +} + +export function updateLegacyMedia( + config: IConfig, + section: 'imageGenerationOptions' | 'ttsOptions', + value: object, +): IConfig { + const updated = { ...config, [section]: { ...config[section], ...value } } + // Reuse the import converter, then expose the v17 view from the canonical result. + updated.ai = normalizeAiConfig( + undefined, + updated.ai, + updated.imageGenerationOptions, + updated.ttsOptions, + ) + return normalizeLegacyConfig(updated) +} diff --git a/apps/core/src/modules/configs/options.repository.ts b/apps/core/src/modules/configs/options.repository.ts index f95b500d5f6..34de59ac8a6 100644 --- a/apps/core/src/modules/configs/options.repository.ts +++ b/apps/core/src/modules/configs/options.repository.ts @@ -28,6 +28,19 @@ export class OptionsRepository extends BaseRepository { super(db) } + async withConfigLock( + work: (repository: OptionsRepository) => Promise, + ): Promise { + return this.db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(5183248167463294042::bigint)`, + ) + return work( + new OptionsRepository(tx as unknown as AppDatabase, this.snowflake), + ) + }) + } + async findAll(): Promise { const rows = await this.db.select().from(options).orderBy(options.name) return rows.map(mapRow) diff --git a/apps/core/test/src/modules/configs/configs.service.spec.ts b/apps/core/test/src/modules/configs/configs.service.spec.ts index 361135d360c..8e9b683ba4a 100644 --- a/apps/core/test/src/modules/configs/configs.service.spec.ts +++ b/apps/core/test/src/modules/configs/configs.service.spec.ts @@ -21,6 +21,55 @@ function createDeferred() { } describe('ConfigsService', () => { + it('persists legacy media writes through canonical AI and retains blank secrets', async () => { + let cache: string | null = null + const rows = new Map() + const repository = { + findAll: async () => [...rows].map(([name, value]) => ({ name, value })), + upsert: vi.fn(async (name, value) => { + rows.set(name, value) + return { name, value } + }), + deleteByName: vi.fn(async (name) => rows.delete(name)), + } + const service = new ConfigsService( + repository as any, + { + waitForReady: async () => {}, + getClient: () => ({ + get: async () => cache, + set: async (_key, value) => { + cache = value + }, + }), + } as any, + {} as any, + { emit: vi.fn() } as any, + ) + await service.patchAndValid('imageGenerationOptions', { + enable: true, + apiKey: 'image-secret', + model: 'image-v1', + }) + expect(rows.has('imageGenerationOptions')).toBe(false) + expect(JSON.stringify(rows.get('ai'))).not.toContain('image-secret') + await service.patchAndValid('imageGenerationOptions', { + apiKey: '', + model: 'image-v2', + }) + expect(await service.get('imageGenerationOptions')).toMatchObject({ + apiKey: 'image-secret', + model: 'image-v2', + }) + await service.patchAndValid('membership', { + dodoApiKey: 'billing-secret', + } as any) + expect(decryptObject(rows.get('membership'), 'membership').apiKey).toBe( + 'billing-secret', + ) + expect(rows.get('membership')).not.toHaveProperty('dodoApiKey') + }) + it('requires a public key to enable backup encryption and never accepts private keys in settings', async () => { const pair = generateKeyPairSync('rsa', { modulusLength: 2048, @@ -235,7 +284,11 @@ describe('ConfigsService', () => { expect(cachedConfig.ai.version).toBe(2) expect(imageProvider).toMatchObject({ apiKey: 'legacy-image-key', - capabilities: { text: false, image: true, speech: false }, + capabilities: expect.objectContaining({ + text: false, + image: true, + speech: false, + }), endpoint: 'https://openrouter.ai/api/v1', }) expect(speechProvider).toMatchObject({ @@ -345,7 +398,11 @@ describe('ConfigsService', () => { apiKey: 'credential-without-model', defaultModel: '', endpoint: 'https://images.example.com/v1', - capabilities: { text: false, image: true, speech: false }, + capabilities: expect.objectContaining({ + text: false, + image: true, + speech: false, + }), }), ]) expect(cachedConfig.ai.imageGeneration.model).toBeUndefined() @@ -908,7 +965,11 @@ describe('ConfigsService', () => { apiKey: 'secret', defaultModel: '', enabled: true, - capabilities: { text: false, image: true, speech: false }, + capabilities: expect.objectContaining({ + text: false, + image: true, + speech: false, + }), }, ] const { service } = createService(currentConfig) diff --git a/apps/core/test/src/modules/configs/legacy-config.spec.ts b/apps/core/test/src/modules/configs/legacy-config.spec.ts new file mode 100644 index 00000000000..9a95a9e605f --- /dev/null +++ b/apps/core/test/src/modules/configs/legacy-config.spec.ts @@ -0,0 +1,77 @@ +import { describe, expect, it } from 'vitest' + +import { + decryptObject, + encryptObject, +} from '~/modules/configs/configs.encrypt.util' +import { normalizeLegacyConfig } from '~/modules/configs/legacy-config.util' +import { EncryptUtil } from '~/utils/encrypt.util' + +describe('canonical legacy config conversion', () => { + it('moves decrypted legacy credentials and binds ciphertext to the new path', () => { + const original = { + thirdPartyServiceIntegration: { + githubToken: EncryptUtil.encrypt( + 'github-secret', + 'thirdPartyServiceIntegration.githubToken', + ), + }, + membership: { + dodoApiKey: EncryptUtil.encrypt( + 'billing-secret', + 'membership.dodoApiKey', + ), + }, + imageGenerationOptions: { + apiKey: EncryptUtil.encrypt( + 'image-secret', + 'imageGenerationOptions.apiKey', + ), + model: 'image-model', + enable: true, + }, + } + const converted = normalizeLegacyConfig( + decryptObject(structuredClone(original)), + ) + expect(converted.thirdPartyServiceIntegration.github.token).toBe( + 'github-secret', + ) + expect(converted.membership.apiKey).toBe('billing-secret') + expect(converted.ai.providers[0].apiKey).toBe('image-secret') + const persisted = encryptObject(structuredClone(converted)) + expect( + EncryptUtil.decrypt( + persisted.ai.providers[0].apiKey, + 'ai.providers.0.apiKey', + ), + ).toBe('image-secret') + expect(persisted.membership).not.toHaveProperty('dodoApiKey') + expect(persisted.thirdPartyServiceIntegration).not.toHaveProperty( + 'githubToken', + ) + expect(normalizeLegacyConfig(converted)).toEqual(converted) + }) + + it('retains legacy media views from canonical AI configuration', () => { + const config = normalizeLegacyConfig({ + imageGenerationOptions: { + enable: true, + apiKey: 'legacy-image', + model: 'image-model', + endpoint: 'https://api.example.com/v1', + }, + }) + const { + imageGenerationOptions: _image, + ttsOptions: _speech, + ...stored + } = config + expect(normalizeLegacyConfig(stored).imageGenerationOptions).toMatchObject({ + enable: true, + apiKey: 'legacy-image', + model: 'image-model', + endpoint: 'https://api.example.com/v1', + }) + }) +}) From b88a95015a9d7f307d5e15859615b9e90233496d Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:49:11 +0800 Subject: [PATCH 05/26] test(core): preserve concrete image-only provider fixture --- apps/core/test/src/modules/configs/configs.service.spec.ts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/apps/core/test/src/modules/configs/configs.service.spec.ts b/apps/core/test/src/modules/configs/configs.service.spec.ts index 8e9b683ba4a..2a79a323dea 100644 --- a/apps/core/test/src/modules/configs/configs.service.spec.ts +++ b/apps/core/test/src/modules/configs/configs.service.spec.ts @@ -965,11 +965,11 @@ describe('ConfigsService', () => { apiKey: 'secret', defaultModel: '', enabled: true, - capabilities: expect.objectContaining({ + capabilities: { text: false, image: true, speech: false, - }), + }, }, ] const { service } = createService(currentConfig) From 4510d08a0493e756d4ded50138ac1f4667144615 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:52:03 +0800 Subject: [PATCH 06/26] refactor(core): derive AI metadata from generation metrics --- .../20261108-ai-metrics-coverage.ts | 49 +++++++++++ .../src/database/app-migrations/registry.ts | 2 + apps/core/src/database/schema/index.ts | 3 + .../ai-generation-metrics/legacy-metadata.ts | 12 +++ .../ai/ai-insights/ai-insights.repository.ts | 34 +++++--- .../ai/ai-overview/ai-overview.repository.ts | 9 +- .../ai-translation.repository.ts | 53 ++++++++---- .../modules/ai/ai-metrics-backfill.spec.ts | 35 ++++++++ .../ai/ai-metrics-compatibility.spec.ts | 86 +++++++++++++++++++ packages/db-schema/src/schema/ai.ts | 82 +++++++++++------- 10 files changed, 303 insertions(+), 62 deletions(-) create mode 100644 apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts create mode 100644 apps/core/src/modules/ai/ai-generation-metrics/legacy-metadata.ts create mode 100644 apps/core/test/src/modules/ai/ai-metrics-backfill.spec.ts create mode 100644 apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts diff --git a/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts b/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts new file mode 100644 index 00000000000..b75f5a47b95 --- /dev/null +++ b/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts @@ -0,0 +1,49 @@ +import type { Pool, PoolClient } from 'pg' + +import type { AppMigration } from './types' + +/** Reusable for explicit preparation; never overwrites a recorded generation. */ +export async function backfillAiMetrics( + client: Pick, +): Promise { + let changed = 0 + for (const [table, type, model, provider, timestamp] of [ + [ + 'ai_translations', + 'translation', + "to_jsonb(r)->>'ai_model'", + "to_jsonb(r)->>'ai_provider'", + 'coalesce(r.updated_at, r.created_at)', + ], + [ + 'ai_insights', + 'insights', + "to_jsonb(r)->'model_info'->>'model'", + "to_jsonb(r)->'model_info'->>'provider'", + 'r.created_at', + ], + ]) { + // All SQL identifiers/expressions above are fixed application constants. + const result = await client.query( + ` + INSERT INTO ai_generation_metrics (id, created_at, resource_type, resource_id, ref_id, lang, model, provider_id) + SELECT r.id || ':v18-preparation', ${timestamp}, $1, r.id, r.ref_id, r.lang, ${model}, ${provider} + FROM ${table} r WHERE NOT EXISTS ( + SELECT 1 FROM ai_generation_metrics m WHERE m.resource_type = $1 AND m.resource_id = r.id + ) ON CONFLICT (id) DO NOTHING`, + [type], + ) + changed += result.rowCount ?? 0 + } + return changed +} + +export const migration: AppMigration = { + id: '20261108-ai-metrics-coverage', + description: + 'Backfill generation metadata before retiring runtime column references', + async up({ pool, logger }) { + const changed = await backfillAiMetrics(pool) + logger.log(`Backfilled ${changed} AI metric rows`) + }, +} diff --git a/apps/core/src/database/app-migrations/registry.ts b/apps/core/src/database/app-migrations/registry.ts index 26c4c7baa0b..c32061c58e3 100644 --- a/apps/core/src/database/app-migrations/registry.ts +++ b/apps/core/src/database/app-migrations/registry.ts @@ -2,6 +2,7 @@ import { migration as recentlyDropEnrichmentColumns } from './20260515-recently- import { migration as treeContentRevisions } from './20260831-tree-content-revisions' import { migration as protectStoredSecrets } from './20261018-protect-stored-secrets' import { migration as mintLoroBlockIds } from './20261026-mint-loro-block-ids' +import { migration as aiMetricsCoverage } from './20261108-ai-metrics-coverage' import type { AppMigration } from './types' /** @@ -17,4 +18,5 @@ export const migrations: AppMigration[] = [ treeContentRevisions, ...(process.env.MX_MIGRATE_SECRETS === '1' ? [protectStoredSecrets] : []), mintLoroBlockIds, + aiMetricsCoverage, ] diff --git a/apps/core/src/database/schema/index.ts b/apps/core/src/database/schema/index.ts index 62d8cdda1bb..5088c74b5ed 100644 --- a/apps/core/src/database/schema/index.ts +++ b/apps/core/src/database/schema/index.ts @@ -1 +1,4 @@ +import { createAiRuntimeTables } from '@mx-space/db-schema/schema' + export * from '@mx-space/db-schema/schema' +export const { aiTranslations, aiInsights } = createAiRuntimeTables() diff --git a/apps/core/src/modules/ai/ai-generation-metrics/legacy-metadata.ts b/apps/core/src/modules/ai/ai-generation-metrics/legacy-metadata.ts new file mode 100644 index 00000000000..1cc922fd19b --- /dev/null +++ b/apps/core/src/modules/ai/ai-generation-metrics/legacy-metadata.ts @@ -0,0 +1,12 @@ +import { sql, type SQLWrapper } from 'drizzle-orm' + +/** v17 response fields derived from the durable ledger, without retired columns. */ +export function latestMetricField( + type: 'translation' | 'insights', + id: SQLWrapper, + field: 'model' | 'provider_id', +) { + return sql< + string | null + >`(select ${sql.identifier(field)} from ai_generation_metrics m where m.resource_type = ${type} and m.resource_id = ${id} order by m.created_at desc, m.id desc limit 1)` +} diff --git a/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts b/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts index cb86c58faba..8ac644162e7 100644 --- a/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts +++ b/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts @@ -1,5 +1,5 @@ import { Inject, Injectable } from '@nestjs/common' -import { and, desc, eq, inArray, sql } from 'drizzle-orm' +import { and, desc, eq, getTableColumns, inArray, sql } from 'drizzle-orm' import { PG_DB_TOKEN } from '~/constants/system.constant' import { aiInsights } from '~/database/schema' @@ -14,7 +14,19 @@ import { SnowflakeService } from '~/shared/id/snowflake.service' import type { AiInsightsRow } from './ai-insights.types' -const mapRow = (row: typeof aiInsights.$inferSelect): AiInsightsRow => ({ +const insightsProjection = { + ...getTableColumns(aiInsights), + modelInfo: sql | null>`(select jsonb_build_object('model', m.model, 'provider', m.provider_id) from ai_generation_metrics m where m.resource_type = 'insights' and m.resource_id = ${sql.raw('"ai_insights"."id"')} order by m.created_at desc, m.id desc limit 1)`, +} + +const mapRow = ( + row: typeof aiInsights.$inferSelect & { + modelInfo?: Record | null + }, +): AiInsightsRow => ({ id: toEntityId(row.id) as EntityId, refId: toEntityId(row.refId) as EntityId, lang: row.lang, @@ -25,7 +37,7 @@ const mapRow = (row: typeof aiInsights.$inferSelect): AiInsightsRow => ({ ? (toEntityId(row.sourceInsightsId) as EntityId) : null, sourceLang: row.sourceLang, - modelInfo: row.modelInfo, + modelInfo: row.modelInfo ?? null, createdAt: row.createdAt, }) @@ -44,7 +56,7 @@ export class AiInsightsRepository extends BaseRepository { ): Promise { const refBig = parseEntityId(refId) const [row] = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where(and(eq(aiInsights.refId, refBig), eq(aiInsights.lang, lang))!) .limit(1) @@ -53,7 +65,7 @@ export class AiInsightsRepository extends BaseRepository { async findById(id: EntityId | string): Promise { const [row] = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where(eq(aiInsights.id, parseEntityId(id))) .limit(1) @@ -63,7 +75,7 @@ export class AiInsightsRepository extends BaseRepository { async listForRef(refId: EntityId | string): Promise { const refBig = parseEntityId(refId) const rows = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where(eq(aiInsights.refId, refBig)) return rows.map(mapRow) @@ -74,7 +86,7 @@ export class AiInsightsRepository extends BaseRepository { ): Promise { if (!refIds.length) return [] const rows = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where( inArray( @@ -96,7 +108,7 @@ export class AiInsightsRepository extends BaseRepository { ] if (lang) conds.push(eq(aiInsights.lang, lang)) const [row] = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where(and(...conds)!) .orderBy(desc(aiInsights.createdAt)) @@ -110,7 +122,7 @@ export class AiInsightsRepository extends BaseRepository { const offset = (page - 1) * size const [rows, [{ count }]] = await Promise.all([ this.db - .select() + .select(insightsProjection) .from(aiInsights) .orderBy(desc(aiInsights.createdAt)) .limit(size) @@ -197,7 +209,7 @@ export class AiInsightsRepository extends BaseRepository { }): Promise { const refBig = parseEntityId(input.refId) const [existing] = await this.db - .select() + .select(insightsProjection) .from(aiInsights) .where( and(eq(aiInsights.refId, refBig), eq(aiInsights.lang, input.lang))!, @@ -214,7 +226,6 @@ export class AiInsightsRepository extends BaseRepository { ? parseEntityId(input.sourceInsightsId) : existing.sourceInsightsId, sourceLang: input.sourceLang ?? existing.sourceLang, - modelInfo: input.modelInfo ?? existing.modelInfo, }) .where(eq(aiInsights.id, existing.id)) .returning() @@ -234,7 +245,6 @@ export class AiInsightsRepository extends BaseRepository { ? parseEntityId(input.sourceInsightsId) : null, sourceLang: input.sourceLang ?? null, - modelInfo: input.modelInfo ?? null, }) .returning() return mapRow(row) diff --git a/apps/core/src/modules/ai/ai-overview/ai-overview.repository.ts b/apps/core/src/modules/ai/ai-overview/ai-overview.repository.ts index c6064b6ac14..4a41f1529ad 100644 --- a/apps/core/src/modules/ai/ai-overview/ai-overview.repository.ts +++ b/apps/core/src/modules/ai/ai-overview/ai-overview.repository.ts @@ -1,5 +1,5 @@ import { Inject, Injectable } from '@nestjs/common' -import { desc, eq, inArray } from 'drizzle-orm' +import { desc, eq, inArray, sql } from 'drizzle-orm' import { PG_DB_TOKEN } from '~/constants/system.constant' import { @@ -16,6 +16,7 @@ import type { AppDatabase } from '~/processors/database/postgres.provider' import { parseEntityId } from '~/shared/id/entity-id' import { DEFAULT_SUMMARY_LANG } from '../ai.constants' +import { latestMetricField } from '../ai-generation-metrics/legacy-metadata' import type { InsightsAsset, SummaryAsset, @@ -174,7 +175,11 @@ export class AiOverviewRepository extends BaseRepository { hash: aiTranslations.hash, lang: aiTranslations.lang, sourceLang: aiTranslations.sourceLang, - aiModel: aiTranslations.aiModel, + aiModel: latestMetricField( + 'translation', + sql.raw('"ai_translations"."id"'), + 'model', + ), createdAt: aiTranslations.createdAt, updatedAt: aiTranslations.updatedAt, }) diff --git a/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts b/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts index 260d1b25d72..a36f36bf63f 100644 --- a/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts +++ b/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts @@ -1,5 +1,14 @@ import { ConflictException, Inject, Injectable } from '@nestjs/common' -import { and, desc, eq, inArray, or, type SQL, sql } from 'drizzle-orm' +import { + and, + desc, + eq, + getTableColumns, + inArray, + or, + type SQL, + sql, +} from 'drizzle-orm' import { PG_DB_TOKEN } from '~/constants/system.constant' import { @@ -21,6 +30,7 @@ import { type EntityId, parseEntityId } from '~/shared/id/entity-id' import { SnowflakeService } from '~/shared/id/snowflake.service' import { ContentFormat } from '~/shared/types/content-format.type' +import { latestMetricField } from '../ai-generation-metrics/legacy-metadata' import type { AiTranslationRevision, AiTranslationRow, @@ -28,8 +38,25 @@ import type { } from './ai-translation.types' import type { TranslationEntryKeyPath } from './translation-entry.types' +const translationProjection = { + ...getTableColumns(aiTranslations), + aiModel: latestMetricField( + 'translation', + sql.raw('"ai_translations"."id"'), + 'model', + ), + aiProvider: latestMetricField( + 'translation', + sql.raw('"ai_translations"."id"'), + 'provider_id', + ), +} + const mapTranslation = ( - row: typeof aiTranslations.$inferSelect, + row: typeof aiTranslations.$inferSelect & { + aiModel?: string | null + aiProvider?: string | null + }, ): AiTranslationRow => ({ id: toEntityId(row.id) as EntityId, hash: row.hash, @@ -43,8 +70,8 @@ const mapTranslation = ( summary: row.summary, tags: row.tags, sourceModifiedAt: row.sourceModifiedAt, - aiModel: row.aiModel, - aiProvider: row.aiProvider, + aiModel: row.aiModel ?? null, + aiProvider: row.aiProvider ?? null, contentFormat: row.contentFormat, content: row.content, sourceBlockSnapshots: row.sourceBlockSnapshots, @@ -83,7 +110,7 @@ export class AiTranslationRepository extends BaseRepository { ): Promise { const refBig = parseEntityId(refId) const [row] = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where( and( @@ -102,7 +129,7 @@ export class AiTranslationRepository extends BaseRepository { ): Promise { const refBig = parseEntityId(refId) const [row] = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where( and(eq(aiTranslations.refId, refBig), eq(aiTranslations.lang, lang))!, @@ -141,7 +168,7 @@ export class AiTranslationRepository extends BaseRepository { async findById(id: EntityId | string): Promise { const [row] = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where(eq(aiTranslations.id, parseEntityId(id))) .limit(1) @@ -150,7 +177,7 @@ export class AiTranslationRepository extends BaseRepository { async listByRefId(refId: EntityId | string): Promise { const rows = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where(eq(aiTranslations.refId, parseEntityId(refId))) .orderBy(desc(aiTranslations.createdAt)) @@ -162,7 +189,7 @@ export class AiTranslationRepository extends BaseRepository { ): Promise { if (!refIds.length) return [] const rows = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where( inArray( @@ -180,7 +207,7 @@ export class AiTranslationRepository extends BaseRepository { ): Promise { if (!refIds.length) return [] const rows = await this.db - .select() + .select(translationProjection) .from(aiTranslations) .where( and( @@ -269,7 +296,7 @@ export class AiTranslationRepository extends BaseRepository { const offset = (page - 1) * size const [rows, [{ count }]] = await Promise.all([ this.db - .select() + .select(translationProjection) .from(aiTranslations) .orderBy(desc(aiTranslations.createdAt)) .limit(size) @@ -303,8 +330,6 @@ export class AiTranslationRepository extends BaseRepository { summary: input.summary, tags: input.tags ?? [], sourceModifiedAt: input.sourceModifiedAt, - aiModel: input.aiModel, - aiProvider: input.aiProvider, contentFormat: input.contentFormat, content: input.content, sourceBlockSnapshots: input.sourceBlockSnapshots, @@ -389,8 +414,6 @@ export class AiTranslationRepository extends BaseRepository { if (patch.tags !== undefined) update.tags = patch.tags if (patch.sourceModifiedAt !== undefined) update.sourceModifiedAt = patch.sourceModifiedAt - if (patch.aiModel !== undefined) update.aiModel = patch.aiModel - if (patch.aiProvider !== undefined) update.aiProvider = patch.aiProvider if (patch.contentFormat !== undefined) update.contentFormat = patch.contentFormat if (patch.content !== undefined) update.content = patch.content diff --git a/apps/core/test/src/modules/ai/ai-metrics-backfill.spec.ts b/apps/core/test/src/modules/ai/ai-metrics-backfill.spec.ts new file mode 100644 index 00000000000..986d3039652 --- /dev/null +++ b/apps/core/test/src/modules/ai/ai-metrics-backfill.spec.ts @@ -0,0 +1,35 @@ +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' + +import { backfillAiMetrics } from '~/database/app-migrations/20261108-ai-metrics-coverage' + +describe('AI metrics preparation backfill', () => { + let context: Awaited> + beforeAll(async () => { + context = await createPgTestDatabase('mx_metrics_backfill') + }) + afterAll(async () => context?.close()) + it('fills gaps once and preserves newer generation metrics', async () => { + await context.pool.query( + "INSERT INTO ai_translations (id, ref_id, ref_type, hash, lang, source_lang, title, text, ai_model, ai_provider) VALUES ('100', '200', 'post', 'h', 'ja', 'zh', 'T', 'Body', 'legacy-model', 'legacy-provider'), ('101', '200', 'post', 'h', 'en', 'zh', 'T', 'Body', 'stale-model', 'stale-provider')", + ) + await context.pool.query( + "INSERT INTO ai_generation_metrics (id, resource_type, resource_id, ref_id, model) VALUES ('current', 'translation', '101', '200', 'current-model')", + ) + expect(await backfillAiMetrics(context.pool)).toBe(1) + expect(await backfillAiMetrics(context.pool)).toBe(0) + const rows = ( + await context.pool.query( + 'SELECT resource_id, model, provider_id FROM ai_generation_metrics ORDER BY resource_id', + ) + ).rows + expect(rows).toEqual([ + { + resource_id: '100', + model: 'legacy-model', + provider_id: 'legacy-provider', + }, + { resource_id: '101', model: 'current-model', provider_id: null }, + ]) + }) +}) diff --git a/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts b/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts new file mode 100644 index 00000000000..98424f5f1f7 --- /dev/null +++ b/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts @@ -0,0 +1,86 @@ +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' + +import { AiInsightsRepository } from '~/modules/ai/ai-insights/ai-insights.repository' +import { AiTranslationRepository } from '~/modules/ai/ai-translation/ai-translation.repository' +import { SnowflakeService } from '~/shared/id/snowflake.service' + +describe('prepared AI repositories without retired columns', () => { + let context: Awaited> + let translation: AiTranslationRepository + let insights: AiInsightsRepository + const refId = '424242424242424242' + beforeAll(async () => { + context = await createPgTestDatabase('mx_metrics_contract') + await context.pool.query( + 'ALTER TABLE ai_translations DROP COLUMN ai_model, DROP COLUMN ai_provider; ALTER TABLE ai_insights DROP COLUMN model_info', + ) + translation = new AiTranslationRepository( + context.db, + new SnowflakeService(), + ) + insights = new AiInsightsRepository(context.db, new SnowflakeService()) + }) + afterAll(async () => context?.close()) + + it('inserts, updates and reads translations without implicit old-column references', async () => { + const value = { + refId, + refType: 'post', + hash: 'h', + lang: 'ja', + sourceLang: 'zh', + title: 'T', + text: 'Body', + aiModel: 'old-signature', + aiProvider: 'provider', + } + const created = await translation.upsert(value as any) + await translation.upsert({ ...value, title: 'Changed' } as any) + await translation.updateById(created.id, { + text: 'Updated', + aiModel: 'ignored-old-field', + }) + await context.pool.query( + "INSERT INTO ai_generation_metrics (id, resource_type, resource_id, ref_id, model, provider_id) VALUES ('fixture', 'translation', $1, $2, 'new-model', 'new-provider')", + [created.id, refId], + ) + expect(await translation.findById(created.id)).toMatchObject({ + text: 'Updated', + aiModel: 'new-model', + aiProvider: 'new-provider', + }) + expect(await translation.findByRef(refId, 'post', 'ja')).toBeTruthy() + expect(await translation.findByRefAndLang(refId, 'ja')).toBeTruthy() + expect(await translation.listByRefId(refId)).toHaveLength(1) + expect(await translation.listByRefIds([refId])).toHaveLength(1) + expect(await translation.listByRefIdsAndLang([refId], 'ja')).toHaveLength(1) + expect((await translation.list()).data).toHaveLength(1) + }) + + it('inserts, updates and reads insights while deriving the legacy API view', async () => { + const value = { + refId, + hash: 'h', + lang: 'ja', + content: 'Body', + modelInfo: { model: 'old-signature' }, + } + const created = await insights.upsert(value) + await insights.upsert({ ...value, content: 'Changed' }) + await insights.updateContent(created.id, 'Updated') + await context.pool.query( + "INSERT INTO ai_generation_metrics (id, resource_type, resource_id, ref_id, model, provider_id) VALUES ('insight-metric', 'insights', $1, $2, 'new-model', 'new-provider')", + [created.id, refId], + ) + expect(await insights.findById(created.id)).toMatchObject({ + content: 'Updated', + modelInfo: { model: 'new-model', provider: 'new-provider' }, + }) + expect(await insights.findByRefAndLang(refId, 'ja')).toBeTruthy() + expect(await insights.findSourceForRef(refId, 'ja')).toBeTruthy() + expect(await insights.listForRef(refId)).toHaveLength(1) + expect(await insights.listByRefIds([refId])).toHaveLength(1) + expect((await insights.list()).data).toHaveLength(1) + }) +}) diff --git a/packages/db-schema/src/schema/ai.ts b/packages/db-schema/src/schema/ai.ts index bac404fdf93..95dcbe3f5a9 100644 --- a/packages/db-schema/src/schema/ai.ts +++ b/packages/db-schema/src/schema/ai.ts @@ -15,32 +15,36 @@ import { import { createdAt, pkText, refText, tsCol, updatedAt } from './columns' +const translationColumns = () => ({ + id: pkText(), + createdAt: createdAt(), + updatedAt: updatedAt(), + hash: text('hash').notNull(), + refId: refText('ref_id').notNull(), + refType: text('ref_type').notNull(), + lang: text('lang').notNull(), + sourceLang: text('source_lang').notNull(), + title: text('title').notNull(), + text: text('text').notNull(), + subtitle: text('subtitle'), + summary: text('summary'), + tags: text('tags') + .array() + .notNull() + .default(sql`'{}'::text[]`), + sourceModifiedAt: tsCol('source_modified_at'), + contentFormat: text('content_format'), + content: text('content'), + sourceBlockSnapshots: jsonb('source_block_snapshots').$type(), + sourceMetaHashes: jsonb('source_meta_hashes').$type(), +}) + export const aiTranslations = pgTable( 'ai_translations', { - id: pkText(), - createdAt: createdAt(), - updatedAt: updatedAt(), - hash: text('hash').notNull(), - refId: refText('ref_id').notNull(), - refType: text('ref_type').notNull(), - lang: text('lang').notNull(), - sourceLang: text('source_lang').notNull(), - title: text('title').notNull(), - text: text('text').notNull(), - subtitle: text('subtitle'), - summary: text('summary'), - tags: text('tags') - .array() - .notNull() - .default(sql`'{}'::text[]`), - sourceModifiedAt: tsCol('source_modified_at'), + ...translationColumns(), aiModel: text('ai_model'), aiProvider: text('ai_provider'), - contentFormat: text('content_format'), - content: text('content'), - sourceBlockSnapshots: jsonb('source_block_snapshots').$type(), - sourceMetaHashes: jsonb('source_meta_hashes').$type(), }, (table) => [ uniqueIndex('ai_translations_ref_lang_uniq').on( @@ -99,21 +103,25 @@ export const aiSummaries = pgTable( ], ) +const insightsColumns = () => ({ + id: pkText(), + createdAt: createdAt(), + refId: refText('ref_id').notNull(), + lang: text('lang').notNull(), + hash: text('hash').notNull(), + content: text('content').notNull(), + isTranslation: boolean('is_translation').notNull().default(false), + sourceInsightsId: refText('source_insights_id').references( + (): AnyPgColumn => aiInsights.id, + { onDelete: 'set null' }, + ), + sourceLang: text('source_lang'), +}) + export const aiInsights = pgTable( 'ai_insights', { - id: pkText(), - createdAt: createdAt(), - refId: refText('ref_id').notNull(), - lang: text('lang').notNull(), - hash: text('hash').notNull(), - content: text('content').notNull(), - isTranslation: boolean('is_translation').notNull().default(false), - sourceInsightsId: refText('source_insights_id').references( - (): AnyPgColumn => aiInsights.id, - { onDelete: 'set null' }, - ), - sourceLang: text('source_lang'), + ...insightsColumns(), modelInfo: jsonb('model_info').$type | null>(), }, (table) => [ @@ -227,3 +235,11 @@ export const aiGenerationMetrics = pgTable( index('ai_generation_metrics_ref_id_idx').on(table.refId, table.createdAt), ], ) + +/** Runtime mappings exclude retired columns; Drizzle Kit still sees the full tables above. */ +export function createAiRuntimeTables() { + return { + aiTranslations: pgTable('ai_translations', translationColumns()), + aiInsights: pgTable('ai_insights', insightsColumns()), + } +} From d123f3b3b726954e33867b8724848f1df90a73d2 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:55:41 +0800 Subject: [PATCH 07/26] feat(core): add standalone v18 readiness diagnostics --- apps/core/package.json | 3 +- .../modules/configs/configs.encrypt.util.ts | 212 +--------------- .../modules/configs/configs.secret-paths.ts | 231 ++++++++++++++++++ apps/core/src/upgrade.ts | 5 + apps/core/src/upgrade/cli.ts | 68 ++++++ apps/core/src/upgrade/readiness.ts | 190 ++++++++++++++ apps/core/src/utils/encrypt.util.ts | 151 +----------- apps/core/src/utils/secret-codec.ts | 163 ++++++++++++ apps/core/test/src/upgrade/cli.spec.ts | 76 ++++++ apps/core/test/src/upgrade/readiness.spec.ts | 76 ++++++ apps/core/vite.config.ts | 1 + 11 files changed, 824 insertions(+), 352 deletions(-) create mode 100644 apps/core/src/modules/configs/configs.secret-paths.ts create mode 100644 apps/core/src/upgrade.ts create mode 100644 apps/core/src/upgrade/cli.ts create mode 100644 apps/core/src/upgrade/readiness.ts create mode 100644 apps/core/src/utils/secret-codec.ts create mode 100644 apps/core/test/src/upgrade/cli.spec.ts create mode 100644 apps/core/test/src/upgrade/readiness.spec.ts diff --git a/apps/core/package.json b/apps/core/package.json index e13020899e6..449b3f297bc 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -45,7 +45,8 @@ "test:live:mix": "NODE_ENV=development RUN_LIVE_TESTS=1 SMOKE_TRANSLATOR_PROVIDER=openai-compatible SMOKE_REVIEWER_PROVIDER=openrouter SMOKE_REVIEWER_MODEL=openai/gpt-5.5 vitest run translation-review-pipeline.live.e2e-spec.ts", "test:live:deepseek-mix": "NODE_ENV=development RUN_LIVE_TESTS=1 SMOKE_REVIEWER_MODEL=openai/gpt-5.5 vitest run translation-review-pipeline.live.e2e-spec.ts", "docs": "npx @compodoc/compodoc -p tsconfig.json -s -d docs", - "publish": "bump" + "publish": "bump", + "upgrade:check": "vite-node src/upgrade.ts check" }, "bump": { "before": [ diff --git a/apps/core/src/modules/configs/configs.encrypt.util.ts b/apps/core/src/modules/configs/configs.encrypt.util.ts index 63cf47de578..e38b8064f42 100644 --- a/apps/core/src/modules/configs/configs.encrypt.util.ts +++ b/apps/core/src/modules/configs/configs.encrypt.util.ts @@ -1,213 +1,10 @@ import { isArrayLike, isObject } from 'es-toolkit/compat' -import type { z } from 'zod' import { EncryptUtil } from '~/utils/encrypt.util' -import { configSchemaMapping } from './configs.schema' -import { getMeta } from './configs.zod-schema.util' +import { isEncryptedPath } from './configs.secret-paths' -// Cached encrypted paths extracted from schemas -let cachedEncryptedPaths: Set | null = null - -/** - * Extract encrypted field paths from all config schemas - */ -function getEncryptedPaths(): Set { - if (cachedEncryptedPaths) { - return cachedEncryptedPaths - } - - cachedEncryptedPaths = new Set([ - 'thirdPartyServiceIntegration.githubToken', - 'membership.dodoApiKey', - 'membership.dodoWebhookKey', - ]) - - for (const [sectionKey, schema] of Object.entries(configSchemaMapping)) { - const paths = collectEncryptedPathsFromSchema(schema, sectionKey) - paths.forEach((p) => cachedEncryptedPaths!.add(p)) - } - - return cachedEncryptedPaths -} - -/** - * Get the type name from a Zod schema (Zod 4.x compatible) - */ -function getZodTypeName(schema: z.ZodType): string { - // Zod 4.x uses _zod.def.type - const zodDef = (schema as any)._zod?.def - if (zodDef?.type) { - return zodDef.type - } - // Fallback for direct .type property - if ((schema as any).type) { - return (schema as any).type - } - return '' -} - -/** - * Get the inner type from wrapper types (optional, nullable, default, etc.) - */ -function getInnerType(schema: z.ZodType): z.ZodType | null { - const zodDef = (schema as any)._zod?.def - return zodDef?.innerType || null -} - -/** - * Get the shape from an object schema - */ -function getObjectShape(schema: z.ZodType): Record | null { - // Try direct .shape property first (Zod 4.x) - if ((schema as any).shape && typeof (schema as any).shape === 'object') { - return (schema as any).shape - } - // Try _zod.def.shape (might be a getter) - const zodDef = (schema as any)._zod?.def - if (typeof zodDef?.shape === 'function') { - return zodDef.shape() - } - if (zodDef?.shape && typeof zodDef.shape === 'object') { - return zodDef.shape - } - return null -} - -/** - * Get the element type from an array schema - */ -function getArrayElementType(schema: z.ZodType): z.ZodType | null { - const zodDef = (schema as any)._zod?.def - return zodDef?.element || zodDef?.type || null -} - -/** - * Recursively collect encrypted field paths from a Zod schema - * @param schema - The Zod schema to analyze - * @param prefix - Current path prefix - * @param parentEncrypt - Whether a parent is marked as encrypt (for subtree encryption) - */ -function collectEncryptedPathsFromSchema( - schema: z.ZodType, - prefix: string, - parentEncrypt = false, -): Set { - const paths = new Set() - - // Check if this schema has encrypt metadata - const meta = getMeta(schema) - const isEncrypt = meta?.encrypt || parentEncrypt - - const typeName = getZodTypeName(schema) - - // Unwrap wrapper types - if ( - typeName === 'optional' || - typeName === 'nullable' || - typeName === 'default' - ) { - const innerType = getInnerType(schema) - if (innerType) { - const innerPaths = collectEncryptedPathsFromSchema( - innerType, - prefix, - isEncrypt, - ) - innerPaths.forEach((p) => paths.add(p)) - } - return paths - } - - // For leaf types (string, number, etc.), if marked as encrypt, add the path - if ( - isEncrypt && - typeName !== 'object' && - typeName !== 'array' && - typeName !== 'record' - ) { - paths.add(prefix) - return paths - } - - // Handle object types - if (typeName === 'object') { - const shape = getObjectShape(schema) - if (shape) { - for (const [key, value] of Object.entries(shape)) { - const childPaths = collectEncryptedPathsFromSchema( - value, - `${prefix}.${key}`, - isEncrypt, - ) - childPaths.forEach((p) => paths.add(p)) - } - } - } - - // Handle array types - if (typeName === 'array') { - const elementType = getArrayElementType(schema) - if (elementType) { - const elementPaths = collectEncryptedPathsFromSchema( - elementType, - `${prefix}.*`, - isEncrypt, - ) - elementPaths.forEach((p) => paths.add(p)) - } - } - - // Handle record types (for oauth.secrets) - if (typeName === 'record') { - const zodDef = (schema as any)._zod?.def - const valueType = zodDef?.valueType - if (valueType) { - const valuePaths = collectEncryptedPathsFromSchema( - valueType, - `${prefix}.*`, - isEncrypt, - ) - valuePaths.forEach((p) => paths.add(p)) - } - } - - return paths -} - -// Pre-compiled regex patterns cache -let compiledPatterns: Array | null = null - -function getCompiledPatterns(): Array { - if (compiledPatterns) { - return compiledPatterns - } - - const encryptedPaths = getEncryptedPaths() - compiledPatterns = Array.from(encryptedPaths).map((pattern) => { - if (pattern.includes('*')) { - const regexStr = `^${pattern.replaceAll('.', String.raw`\.`).replaceAll('*', '[^.]+')}$` - return new RegExp(regexStr) - } - return pattern - }) - - return compiledPatterns -} - -/** - * Check if a path matches any encrypted path pattern - */ -function isEncryptedPath(path: string): boolean { - for (const pattern of getCompiledPatterns()) { - if (typeof pattern === 'string') { - if (pattern === path) return true - } else if (pattern.test(path)) { - return true - } - } - return false -} +export { getExtractedEncryptedPaths } from './configs.secret-paths' /** * Recursively decrypt all encrypted fields in an object @@ -263,11 +60,6 @@ export const encryptObject = (target: T, prefix = ''): T => { return target } -// For debugging: export function to view extracted paths -export const getExtractedEncryptedPaths = (): string[] => { - return Array.from(getEncryptedPaths()) -} - /** * Remove encrypted fields from an object before sending to frontend * Sets encrypted field values to empty string diff --git a/apps/core/src/modules/configs/configs.secret-paths.ts b/apps/core/src/modules/configs/configs.secret-paths.ts new file mode 100644 index 00000000000..0fdade2193e --- /dev/null +++ b/apps/core/src/modules/configs/configs.secret-paths.ts @@ -0,0 +1,231 @@ +import type { z } from 'zod' + +import { configSchemaMapping } from './configs.schema' +import { getMeta } from './configs.zod-schema.util' + +// Cached encrypted paths extracted from schemas +let cachedEncryptedPaths: Set | null = null + +/** + * Extract encrypted field paths from all config schemas + */ +function getEncryptedPaths(): Set { + if (cachedEncryptedPaths) { + return cachedEncryptedPaths + } + + cachedEncryptedPaths = new Set([ + 'thirdPartyServiceIntegration.githubToken', + 'membership.dodoApiKey', + 'membership.dodoWebhookKey', + ]) + + for (const [sectionKey, schema] of Object.entries(configSchemaMapping)) { + const paths = collectEncryptedPathsFromSchema(schema, sectionKey) + paths.forEach((p) => cachedEncryptedPaths!.add(p)) + } + + return cachedEncryptedPaths +} + +/** + * Get the type name from a Zod schema (Zod 4.x compatible) + */ +function getZodTypeName(schema: z.ZodType): string { + // Zod 4.x uses _zod.def.type + const zodDef = (schema as any)._zod?.def + if (zodDef?.type) { + return zodDef.type + } + // Fallback for direct .type property + if ((schema as any).type) { + return (schema as any).type + } + return '' +} + +/** + * Get the inner type from wrapper types (optional, nullable, default, etc.) + */ +function getInnerType(schema: z.ZodType): z.ZodType | null { + const zodDef = (schema as any)._zod?.def + return zodDef?.innerType || null +} + +/** + * Get the shape from an object schema + */ +function getObjectShape(schema: z.ZodType): Record | null { + // Try direct .shape property first (Zod 4.x) + if ((schema as any).shape && typeof (schema as any).shape === 'object') { + return (schema as any).shape + } + // Try _zod.def.shape (might be a getter) + const zodDef = (schema as any)._zod?.def + if (typeof zodDef?.shape === 'function') { + return zodDef.shape() + } + if (zodDef?.shape && typeof zodDef.shape === 'object') { + return zodDef.shape + } + return null +} + +/** + * Get the element type from an array schema + */ +function getArrayElementType(schema: z.ZodType): z.ZodType | null { + const zodDef = (schema as any)._zod?.def + return zodDef?.element || zodDef?.type || null +} + +/** + * Recursively collect encrypted field paths from a Zod schema + * @param schema - The Zod schema to analyze + * @param prefix - Current path prefix + * @param parentEncrypt - Whether a parent is marked as encrypt (for subtree encryption) + */ +function collectEncryptedPathsFromSchema( + schema: z.ZodType, + prefix: string, + parentEncrypt = false, +): Set { + const paths = new Set() + + // Check if this schema has encrypt metadata + const meta = getMeta(schema) + const isEncrypt = meta?.encrypt || parentEncrypt + + const typeName = getZodTypeName(schema) + + // Unwrap wrapper types + if ( + typeName === 'optional' || + typeName === 'nullable' || + typeName === 'default' + ) { + const innerType = getInnerType(schema) + if (innerType) { + const innerPaths = collectEncryptedPathsFromSchema( + innerType, + prefix, + isEncrypt, + ) + innerPaths.forEach((p) => paths.add(p)) + } + return paths + } + + // For leaf types (string, number, etc.), if marked as encrypt, add the path + if ( + isEncrypt && + typeName !== 'object' && + typeName !== 'array' && + typeName !== 'record' + ) { + paths.add(prefix) + return paths + } + + // Handle object types + if (typeName === 'object') { + const shape = getObjectShape(schema) + if (shape) { + for (const [key, value] of Object.entries(shape)) { + const childPaths = collectEncryptedPathsFromSchema( + value, + `${prefix}.${key}`, + isEncrypt, + ) + childPaths.forEach((p) => paths.add(p)) + } + } + } + + // Handle array types + if (typeName === 'array') { + const elementType = getArrayElementType(schema) + if (elementType) { + const elementPaths = collectEncryptedPathsFromSchema( + elementType, + `${prefix}.*`, + isEncrypt, + ) + elementPaths.forEach((p) => paths.add(p)) + } + } + + // Handle record types (for oauth.secrets) + if (typeName === 'record') { + const zodDef = (schema as any)._zod?.def + const valueType = zodDef?.valueType + if (valueType) { + const valuePaths = collectEncryptedPathsFromSchema( + valueType, + `${prefix}.*`, + isEncrypt, + ) + valuePaths.forEach((p) => paths.add(p)) + } + } + + return paths +} + +// Pre-compiled regex patterns cache +let compiledPatterns: Array | null = null + +function getCompiledPatterns(): Array { + if (compiledPatterns) { + return compiledPatterns + } + + const encryptedPaths = getEncryptedPaths() + compiledPatterns = Array.from(encryptedPaths).map((pattern) => { + if (pattern.includes('*')) { + const regexStr = `^${pattern.replaceAll('.', String.raw`\.`).replaceAll('*', '[^.]+')}$` + return new RegExp(regexStr) + } + return pattern + }) + + return compiledPatterns +} + +/** + * Check if a path matches any encrypted path pattern + */ +export function isEncryptedPath(path: string): boolean { + for (const pattern of getCompiledPatterns()) { + if (typeof pattern === 'string') { + if (pattern === path) return true + } else if (pattern.test(path)) { + return true + } + } + return false +} + +export const getExtractedEncryptedPaths = (): string[] => + Array.from(getEncryptedPaths()) + +export function visitConfigSecrets( + value: T, + prefix: string, + visit: (secret: string, path: string) => string, +): T { + if (typeof value === 'string') + return (isEncryptedPath(prefix) ? visit(value, prefix) : value) as T + if (Array.isArray(value)) + return value.map((item, index) => + visitConfigSecrets(item, `${prefix}.${index}`, visit), + ) as T + if (value && typeof value === 'object') + return Object.fromEntries( + Object.entries(value).map(([key, item]) => [ + key, + visitConfigSecrets(item, prefix ? `${prefix}.${key}` : key, visit), + ]), + ) as T + return value +} diff --git a/apps/core/src/upgrade.ts b/apps/core/src/upgrade.ts new file mode 100644 index 00000000000..15c5ff96a8d --- /dev/null +++ b/apps/core/src/upgrade.ts @@ -0,0 +1,5 @@ +import 'dotenv-expand/config' + +import { runUpgradeCli } from './upgrade/cli' + +process.exitCode = await runUpgradeCli(process.argv.slice(2)) diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts new file mode 100644 index 00000000000..52581223a96 --- /dev/null +++ b/apps/core/src/upgrade/cli.ts @@ -0,0 +1,68 @@ +import { Pool } from 'pg' + +import { readUpgradeState, type UpgradeConfig } from './readiness' + +export function upgradeConfigFromEnv(env: NodeJS.ProcessEnv): UpgradeConfig { + const key = env.MX_ENCRYPT_KEY || env.ENCRYPT_KEY || '' + const enabled = env.MX_ENCRYPT_ENABLE ?? env.ENCRYPT_ENABLE + return { + key, + enable: enabled === undefined ? !!key : /^(?:true|1)$/i.test(enabled), + v2: /^(?:true|1)$/i.test(env.MX_SECRET_STORAGE_V2 ?? ''), + jwtSecret: env.JWT_SECRET || env.JWTSECRET || '', + algorithm: env.ENCRYPT_ALGORITHM || 'aes-256-ecb', + } +} + +export async function runUpgradeCli( + args: string[], + env = process.env, + write = (output: string) => process.stdout.write(`${output}\n`), +): Promise { + const [command, ...flags] = args + if (command !== 'check' || flags.some((flag) => flag !== '--json')) { + write('Usage: upgrade check [--json]') + return 1 + } + const pool = new Pool({ + connectionString: env.PG_URL || env.PG_CONNECTION_STRING, + host: env.PG_HOST || '127.0.0.1', + port: Number(env.PG_PORT || 5432), + user: env.PG_USER || 'mx', + password: env.PG_PASSWORD || 'mx', + database: env.PG_DATABASE || 'mx_core', + ssl: /^(?:true|1)$/i.test(env.PG_SSL ?? '') + ? { rejectUnauthorized: false } + : false, + max: 1, + connectionTimeoutMillis: 10000, + }) + try { + const client = await pool.connect() + try { + await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY') + const report = await readUpgradeState(client, upgradeConfigFromEnv(env)) + await client.query('COMMIT') + write( + flags.includes('--json') + ? JSON.stringify(report) + : `${report.status}: ${report.installation} installation\n${report.issues.map((issue) => `${issue.code}${issue.path ? ` (${issue.path})` : ''}`).join('\n')}`, + ) + return report.status === 'ready' ? 0 : 2 + } finally { + client.release() + } + } catch { + // Driver/decryption messages can contain connection strings or stored values. + write( + JSON.stringify({ + target: 18, + status: 'error', + issues: [{ code: 'UPGRADE_CHECK_FAILED' }], + }), + ) + return 1 + } finally { + await pool.end() + } +} diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts new file mode 100644 index 00000000000..db0cc84b220 --- /dev/null +++ b/apps/core/src/upgrade/readiness.ts @@ -0,0 +1,190 @@ +import { parseEnvelope, symmetricDecrypt } from 'better-auth/crypto' +import type { Pool, PoolClient } from 'pg' + +import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' +import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' +import { SecretCodec } from '../utils/secret-codec' + +export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' +export const CONFIG_UPGRADE_LOCK = '5183248167463294042' +export type UpgradeConfig = { + key: string + jwtSecret: string + enable: boolean + v2: boolean + algorithm?: string +} +export type UpgradeIssue = { code: string; path?: string } +export type UpgradeReport = { + target: 18 + status: 'ready' | 'needs-action' | 'error' + installation: 'empty' | 'existing' + prepared: boolean + counts: { + plaintext: number + ecb: number + v2: number + legacyConfig: number + plainApiKeys: number + hashedApiKeys: number + plainOAuth: number + encryptedOAuth: number + metricsMissing: number + } + issues: UpgradeIssue[] +} +export type UpgradeDatabase = Pick + +export function oauthKey(secret: string) { + return { + currentVersion: 1, + keys: new Map([[1, secret]]), + legacySecret: secret, + } +} + +export function isLegacyConfig(name: string, value: any): boolean { + return ( + name === 'imageGenerationOptions' || + name === 'ttsOptions' || + (name === 'ai' && value?.version !== 2) || + (name === 'thirdPartyServiceIntegration' && + !!value && + 'githubToken' in value) || + (name === 'membership' && + !!value && + ['dodoApiKey', 'dodoEnvironment', 'dodoWebhookKey'].some( + (key) => key in value, + )) + ) +} + +/** Read-only and independent of app.config, Nest bootstrap, Redis and business modules. */ +export async function readUpgradeState( + database: UpgradeDatabase, + config: UpgradeConfig, +): Promise { + const issues: UpgradeIssue[] = [] + const keyValid = /^[\da-f]{64}$/i.test(config.key) + if (!config.key) issues.push({ code: 'FIELD_KEY_MISSING' }) + else if (!keyValid) issues.push({ code: 'FIELD_KEY_INVALID' }) + if (!config.enable) issues.push({ code: 'ENCRYPTION_DISABLED' }) + if (config.jwtSecret.length < 32) issues.push({ code: 'JWT_TOO_SHORT' }) + if (!config.v2) issues.push({ code: 'V2_WRITES_DISABLED' }) + const counts: UpgradeReport['counts'] = { + plaintext: 0, + ecb: 0, + v2: 0, + legacyConfig: 0, + plainApiKeys: 0, + hashedApiKeys: 0, + plainOAuth: 0, + encryptedOAuth: 0, + metricsMissing: 0, + } + const codec = new SecretCodec({ ...config, enable: true, v2: true }) + const inspect = (secret: string, path: string) => { + if (!secret) return secret + if (secret.startsWith('$${mx}$$v2.')) counts.v2++ + else if (codec.isEncryptedString(secret)) counts.ecb++ + else counts.plaintext++ + if (keyValid && codec.isEncryptedString(secret)) { + try { + codec.decrypt(secret, path) + } catch { + issues.push({ code: 'SECRET_UNREADABLE', path }) + } + } + return secret + } + const options = ( + await database.query('SELECT id, name, value FROM options ORDER BY name') + ).rows + for (const row of options) { + if (isLegacyConfig(row.name, row.value)) counts.legacyConfig++ + if (row.value && typeof row.value === 'object') + visitConfigSecrets(row.value, row.name, inspect) + } + for (const table of ['snippets', 'webhooks']) { + for (const row of (await database.query(`SELECT id, secret FROM ${table}`)) + .rows) { + if (row.secret) inspect(row.secret, `${table}.${row.id}.secret`) + } + } + for (const row of (await database.query('SELECT config_id FROM api_keys')) + .rows) { + if (row.config_id === HASHED_API_KEY_CONFIG_ID) counts.hashedApiKeys++ + else counts.plainApiKeys++ + } + const accounts = ( + await database.query('SELECT id, access_token, refresh_token FROM accounts') + ).rows + for (const row of accounts) { + for (const field of ['access_token', 'refresh_token']) { + const token = row[field] + if (!token) continue + if (!token.startsWith('$ba$')) { + counts.plainOAuth++ + continue + } + counts.encryptedOAuth++ + if (!parseEnvelope(token)) + issues.push({ + code: 'OAUTH_UNREADABLE', + path: `accounts.${row.id}.${field}`, + }) + else if (config.jwtSecret) { + try { + await symmetricDecrypt({ + key: oauthKey(config.jwtSecret), + data: token, + }) + } catch { + issues.push({ + code: 'OAUTH_UNREADABLE', + path: `accounts.${row.id}.${field}`, + }) + } + } + } + } + for (const [table, type] of [ + ['ai_translations', 'translation'], + ['ai_insights', 'insights'], + ]) { + const result = await database.query( + `SELECT count(*)::int AS count FROM ${table} r WHERE NOT EXISTS (SELECT 1 FROM ai_generation_metrics m WHERE m.resource_type = $1 AND m.resource_id = r.id)`, + [type], + ) + counts.metricsMissing += result.rows[0].count + } + const prepared = + ( + await database.query('SELECT 1 FROM _app_migrations WHERE id = $1', [ + PREPARED_MIGRATION_ID, + ]) + ).rowCount !== 0 + const existing = await database.query(`SELECT EXISTS ( + SELECT 1 FROM options UNION ALL SELECT 1 FROM readers UNION ALL SELECT 1 FROM posts UNION ALL SELECT 1 FROM notes UNION ALL SELECT 1 FROM pages UNION ALL SELECT 1 FROM snippets UNION ALL SELECT 1 FROM webhooks UNION ALL SELECT 1 FROM accounts UNION ALL SELECT 1 FROM api_keys UNION ALL SELECT 1 FROM ai_translations UNION ALL SELECT 1 FROM ai_insights + ) AS present`) + const installation = existing.rows[0].present ? 'existing' : 'empty' + if ( + counts.plaintext || + counts.ecb || + counts.plainApiKeys || + counts.plainOAuth + ) + issues.push({ code: 'CREDENTIALS_NEED_PREPARATION' }) + if (counts.legacyConfig) issues.push({ code: 'LEGACY_CONFIG' }) + if (counts.metricsMissing) issues.push({ code: 'METRICS_INCOMPLETE' }) + if (installation === 'existing' && !prepared) + issues.push({ code: 'PREPARATION_MARKER_MISSING' }) + return { + target: 18, + status: issues.length ? 'needs-action' : 'ready', + installation, + prepared, + counts, + issues, + } +} diff --git a/apps/core/src/utils/encrypt.util.ts b/apps/core/src/utils/encrypt.util.ts index cd9105f1a9d..f5f0fac8491 100644 --- a/apps/core/src/utils/encrypt.util.ts +++ b/apps/core/src/utils/encrypt.util.ts @@ -1,155 +1,24 @@ -import crypto from 'node:crypto' - import { ENCRYPT, SECRET_STORAGE_V2 } from '~/app.config' -const PREFIX = '$${mx}$$' -const V2_PREFIX = `${PREFIX}v2.` +import { SecretCodec } from './secret-codec' -export const mapString = (key: string) => { - if (!key || key.length > 64) throw new Error('Invalid encryption key length') - return key.length === 64 - ? key - : crypto.createHash('sha256').update(`${key}mx-encrypt-salt`).digest('hex') -} +export { mapString } from './secret-codec' +/** v17 facade keeps runtime configuration and the plaintext escape hatch. */ export class EncryptUtil { + private static codec() { + return new SecretCodec({ ...ENCRYPT, v2: SECRET_STORAGE_V2 }) + } static assertConfigured() { - if (!ENCRYPT.enable) { - throw new Error( - 'Secret encryption is disabled; set MX_ENCRYPT_ENABLE=true or ENCRYPT_ENABLE=true', - ) - } - if (!/^[\da-f]{64}$/i.test(ENCRYPT.key)) { - throw new Error( - 'Storing secrets requires a 64-character hexadecimal encryption key', - ) - } + return this.codec().assertConfigured() } - static isEncryptedString(value: string) { - return value.startsWith(PREFIX) + return this.codec().isEncryptedString(value) } - static encrypt(data: T, context = ''): T { - if (typeof data === 'string') { - if (!data) return data - // TODO(v18): remove the legacy plaintext escape hatch and require encryption. - if ( - !ENCRYPT.enable && - !SECRET_STORAGE_V2 && - !EncryptUtil.isEncryptedString(data) - ) { - return data - } - EncryptUtil.assertConfigured() - let plain: string = data - if (EncryptUtil.isEncryptedString(data)) { - plain = EncryptUtil.decrypt(data, context) - if (data.startsWith(V2_PREFIX) || !SECRET_STORAGE_V2) return data - } - if (!plain) return '' as T - const key = Buffer.from(ENCRYPT.key, 'hex') - if (!SECRET_STORAGE_V2) { - const cipher = crypto.createCipheriv( - ENCRYPT.algorithm || 'aes-256-ecb', - key, - '', - ) - return (PREFIX + - Buffer.concat([ - cipher.update(plain, 'utf8'), - cipher.final(), - ]).toString('base64')) as T - } - const keyId = crypto - .createHash('sha256') - .update(key) - .digest('hex') - .slice(0, 16) - const nonce = crypto.randomBytes(12) - const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce) - cipher.setAAD(Buffer.from(`mx:v2:${keyId}:${context}`)) - const ciphertext = Buffer.concat([ - cipher.update(plain, 'utf8'), - cipher.final(), - ]) - return `${V2_PREFIX}${keyId}.${nonce.toString('base64url')}.${cipher.getAuthTag().toString('base64url')}.${ciphertext.toString('base64url')}` as T - } - if (Array.isArray(data)) { - return data.map((value, index) => - EncryptUtil.encrypt(value, `${context}.${index}`), - ) as T - } - if (data && typeof data === 'object') { - return Object.fromEntries( - Object.entries(data).map(([key, value]) => [ - key, - EncryptUtil.encrypt(value, `${context}.${key}`), - ]), - ) as T - } - return data + return this.codec().encrypt(data, context) } - static decrypt(data: T, context = ''): T { - if (typeof data === 'string') { - if (!EncryptUtil.isEncryptedString(data)) return data - EncryptUtil.assertConfigured() - const key = Buffer.from(ENCRYPT.key, 'hex') - if (data.startsWith(V2_PREFIX)) { - const parts = data.slice(V2_PREFIX.length).split('.') - const [keyId, iv, tag, ciphertext] = parts - if (parts.length !== 4 || !keyId || !iv || !tag || !ciphertext) { - throw new Error('Malformed encrypted secret') - } - const expectedId = crypto - .createHash('sha256') - .update(key) - .digest('hex') - .slice(0, 16) - const nonce = Buffer.from(iv, 'base64url') - const authTag = Buffer.from(tag, 'base64url') - if ( - keyId !== expectedId || - nonce.length !== 12 || - authTag.length !== 16 - ) { - throw new Error('Invalid secret key or encrypted envelope') - } - const decipher = crypto.createDecipheriv('aes-256-gcm', key, nonce) - decipher.setAAD(Buffer.from(`mx:v2:${keyId}:${context}`)) - decipher.setAuthTag(authTag) - return Buffer.concat([ - decipher.update(Buffer.from(ciphertext, 'base64url')), - decipher.final(), - ]).toString('utf8') as T - } - if (!/^[\d+/a-z]+={0,2}$/i.test(data.slice(PREFIX.length))) { - throw new Error('Unsupported encrypted secret format') - } - const decipher = crypto.createDecipheriv( - ENCRYPT.algorithm || 'aes-256-ecb', - key, - '', - ) - return Buffer.concat([ - decipher.update(Buffer.from(data.slice(PREFIX.length), 'base64')), - decipher.final(), - ]).toString('utf8') as T - } - if (Array.isArray(data)) { - return data.map((value, index) => - EncryptUtil.decrypt(value, `${context}.${index}`), - ) as T - } - if (data && typeof data === 'object') { - return Object.fromEntries( - Object.entries(data).map(([key, value]) => [ - key, - EncryptUtil.decrypt(value, `${context}.${key}`), - ]), - ) as T - } - return data + return this.codec().decrypt(data, context) } } diff --git a/apps/core/src/utils/secret-codec.ts b/apps/core/src/utils/secret-codec.ts new file mode 100644 index 00000000000..a6d74c461e5 --- /dev/null +++ b/apps/core/src/utils/secret-codec.ts @@ -0,0 +1,163 @@ +import crypto from 'node:crypto' + +const PREFIX = '$${mx}$$' +const V2_PREFIX = `${PREFIX}v2.` + +export const mapString = (key: string) => { + if (!key || key.length > 64) throw new Error('Invalid encryption key length') + return key.length === 64 + ? key + : crypto.createHash('sha256').update(`${key}mx-encrypt-salt`).digest('hex') +} + +export interface SecretCodecOptions { + key: string + enable: boolean + v2: boolean + algorithm?: string +} + +/** No runtime config imports: usable by offline upgrade tools. */ +export class SecretCodec { + constructor(private readonly options: SecretCodecOptions) {} + + assertConfigured() { + if (!this.options.enable) { + throw new Error( + 'Secret encryption is disabled; set MX_ENCRYPT_ENABLE=true or ENCRYPT_ENABLE=true', + ) + } + if (!/^[\da-f]{64}$/i.test(this.options.key)) { + throw new Error( + 'Storing secrets requires a 64-character hexadecimal encryption key', + ) + } + } + + isEncryptedString(value: string) { + return value.startsWith(PREFIX) + } + + encrypt(data: T, context = ''): T { + if (typeof data === 'string') { + if (!data) return data + // TODO(v18): remove the legacy plaintext escape hatch and require encryption. + if ( + !this.options.enable && + !this.options.v2 && + !this.isEncryptedString(data) + ) { + return data + } + this.assertConfigured() + let plain: string = data + if (this.isEncryptedString(data)) { + plain = this.decrypt(data, context) + if (data.startsWith(V2_PREFIX) || !this.options.v2) return data + } + if (!plain) return '' as T + const key = Buffer.from(this.options.key, 'hex') + if (!this.options.v2) { + const cipher = crypto.createCipheriv( + this.options.algorithm || 'aes-256-ecb', + key, + '', + ) + return (PREFIX + + Buffer.concat([ + cipher.update(plain, 'utf8'), + cipher.final(), + ]).toString('base64')) as T + } + const keyId = crypto + .createHash('sha256') + .update(key) + .digest('hex') + .slice(0, 16) + const nonce = crypto.randomBytes(12) + const cipher = crypto.createCipheriv('aes-256-gcm', key, nonce) + cipher.setAAD(Buffer.from(`mx:v2:${keyId}:${context}`)) + const ciphertext = Buffer.concat([ + cipher.update(plain, 'utf8'), + cipher.final(), + ]) + return `${V2_PREFIX}${keyId}.${nonce.toString('base64url')}.${cipher.getAuthTag().toString('base64url')}.${ciphertext.toString('base64url')}` as T + } + if (Array.isArray(data)) { + return data.map((value, index) => + this.encrypt(value, `${context}.${index}`), + ) as T + } + if (data && typeof data === 'object') { + return Object.fromEntries( + Object.entries(data).map(([key, value]) => [ + key, + this.encrypt(value, `${context}.${key}`), + ]), + ) as T + } + return data + } + + decrypt(data: T, context = ''): T { + if (typeof data === 'string') { + if (!this.isEncryptedString(data)) return data + this.assertConfigured() + const key = Buffer.from(this.options.key, 'hex') + if (data.startsWith(V2_PREFIX)) { + const parts = data.slice(V2_PREFIX.length).split('.') + const [keyId, iv, tag, ciphertext] = parts + if (parts.length !== 4 || !keyId || !iv || !tag || !ciphertext) { + throw new Error('Malformed encrypted secret') + } + const expectedId = crypto + .createHash('sha256') + .update(key) + .digest('hex') + .slice(0, 16) + const nonce = Buffer.from(iv, 'base64url') + const authTag = Buffer.from(tag, 'base64url') + if ( + keyId !== expectedId || + nonce.length !== 12 || + authTag.length !== 16 + ) { + throw new Error('Invalid secret key or encrypted envelope') + } + const decipher = crypto.createDecipheriv('aes-256-gcm', key, nonce) + decipher.setAAD(Buffer.from(`mx:v2:${keyId}:${context}`)) + decipher.setAuthTag(authTag) + return Buffer.concat([ + decipher.update(Buffer.from(ciphertext, 'base64url')), + decipher.final(), + ]).toString('utf8') as T + } + if (!/^[\d+/a-z]+={0,2}$/i.test(data.slice(PREFIX.length))) { + throw new Error('Unsupported encrypted secret format') + } + const decipher = crypto.createDecipheriv( + this.options.algorithm || 'aes-256-ecb', + key, + '', + ) + return Buffer.concat([ + decipher.update(Buffer.from(data.slice(PREFIX.length), 'base64')), + decipher.final(), + ]).toString('utf8') as T + } + if (Array.isArray(data)) { + return data.map((value, index) => + this.decrypt(value, `${context}.${index}`), + ) as T + } + if (data && typeof data === 'object') { + return Object.fromEntries( + Object.entries(data).map(([key, value]) => [ + key, + this.decrypt(value, `${context}.${key}`), + ]), + ) as T + } + return data + } +} diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts new file mode 100644 index 00000000000..aa340da2aa5 --- /dev/null +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -0,0 +1,76 @@ +import { execFile } from 'node:child_process' +import { promisify } from 'node:util' + +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' + +import { runUpgradeCli } from '~/upgrade/cli' + +describe('upgrade CLI isolated process', () => { + let context: Awaited> + beforeAll(async () => { + context = await createPgTestDatabase('mx_upgrade_cli') + }) + afterAll(async () => context?.close()) + + it('reports bad runtime settings in a fresh process without bootstrap failure', async () => { + const result = await promisify(execFile)( + 'node_modules/.bin/vite-node', + ['src/upgrade.ts', 'check', '--json'], + { + cwd: process.cwd().endsWith('/test') + ? `${process.cwd()}/..` + : process.cwd(), + env: { + ...process.env, + PG_URL: context.connectionString, + MX_ENCRYPT_KEY: 'invalid-key', + MX_SECRET_STORAGE_V2: 'true', + JWT_SECRET: 'bad-jwt-secret', + }, + }, + ).catch((error) => error) + expect(result.code).toBe(2) + const report = JSON.parse(result.stdout.trim()) + expect(report.status).toBe('needs-action') + expect(report.issues.map((issue) => issue.code)).toContain( + 'FIELD_KEY_INVALID', + ) + expect(result.stdout + result.stderr).not.toContain('bad-jwt-secret') + }, 30000) + + it('separates ready, needs-action and connection errors by exit status', async () => { + const output: string[] = [] + const env = { + PG_URL: context.connectionString, + MX_ENCRYPT_KEY: 'ab'.repeat(32), + MX_SECRET_STORAGE_V2: 'true', + JWT_SECRET: 'long-enough-test-jwt-123456789012345', + } + expect( + await runUpgradeCli(['check', '--json'], env, (value) => { + output.push(value) + }), + ).toBe(0) + expect( + await runUpgradeCli( + ['check', '--json'], + { ...env, JWT_SECRET: 'short' }, + (value) => { + output.push(value) + }, + ), + ).toBe(2) + expect( + await runUpgradeCli( + ['check', '--json'], + { ...env, PG_URL: 'postgres://no-secret@127.0.0.1:1/missing' }, + (value) => { + output.push(value) + }, + ), + ).toBe(1) + expect(JSON.parse(output[2]).status).toBe('error') + expect(output.join('')).not.toContain('no-secret') + }) +}) diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts new file mode 100644 index 00000000000..a8c5921c0e2 --- /dev/null +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -0,0 +1,76 @@ +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' + +import { readUpgradeState } from '~/upgrade/readiness' +import { SecretCodec } from '~/utils/secret-codec' + +describe('standalone upgrade readiness', () => { + let context: Awaited> + const config = { + key: 'ab'.repeat(32), + jwtSecret: 'jwt-secret-for-upgrade-123456789012345', + enable: true, + v2: true, + } + beforeAll(async () => { + context = await createPgTestDatabase('mx_upgrade_check') + }) + afterAll(async () => context?.close()) + + it('identifies an empty installation without requiring a historical marker', async () => { + expect(await readUpgradeState(context.pool, config)).toMatchObject({ + installation: 'empty', + status: 'ready', + prepared: false, + }) + }) + + it('reports missing/invalid settings without importing the business application', async () => { + const report = await readUpgradeState(context.pool, { + key: '', + jwtSecret: 'short-secret', + enable: false, + v2: false, + }) + expect(report.status).toBe('needs-action') + expect(report.issues.map((issue) => issue.code)).toEqual( + expect.arrayContaining([ + 'FIELD_KEY_MISSING', + 'JWT_TOO_SHORT', + 'V2_WRITES_DISABLED', + ]), + ) + expect(JSON.stringify(report)).not.toContain('short-secret') + }) + + it('counts old formats, detects wrong keys and remains read-only', async () => { + const codec = new SecretCodec({ ...config, v2: false }) + await context.pool.query( + 'INSERT INTO options (id, name, value) VALUES ($1, $2, $3)', + [ + '100', + 'membership', + { dodoApiKey: codec.encrypt('private-value', 'membership.dodoApiKey') }, + ], + ) + const before = (await context.pool.query('SELECT * FROM options')).rows + const report = await readUpgradeState(context.pool, config) + expect(report).toMatchObject({ + installation: 'existing', + status: 'needs-action', + counts: { ecb: 1, legacyConfig: 1 }, + }) + expect(JSON.stringify(report)).not.toContain('private-value') + expect( + ( + await readUpgradeState(context.pool, { + ...config, + key: 'cd'.repeat(32), + }) + ).issues.map((issue) => issue.code), + ).toContain('SECRET_UNREADABLE') + expect((await context.pool.query('SELECT * FROM options')).rows).toEqual( + before, + ) + }) +}) diff --git a/apps/core/vite.config.ts b/apps/core/vite.config.ts index 16f08ee320b..86a7ecc00bf 100644 --- a/apps/core/vite.config.ts +++ b/apps/core/vite.config.ts @@ -77,6 +77,7 @@ export default defineConfig(({ command }) => { input: { main: resolve(__dirname, 'src/main.ts'), migrate: resolve(__dirname, 'src/migrate.ts'), + upgrade: resolve(__dirname, 'src/upgrade.ts'), 'app-migrate': resolve(__dirname, 'src/app-migrate.ts'), }, output: { From 9a8b96f8d9bf3f9befcdc1e9da9488384bf59f28 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 20:57:17 +0800 Subject: [PATCH 08/26] fix(core): isolate upgrade CLI from application Vite transforms --- apps/core/package.json | 2 +- apps/core/src/modules/configs/configs.schema.ts | 2 +- apps/core/test/src/upgrade/cli.spec.ts | 15 ++++++++++----- apps/core/vite.upgrade.config.ts | 8 ++++++++ 4 files changed, 20 insertions(+), 7 deletions(-) create mode 100644 apps/core/vite.upgrade.config.ts diff --git a/apps/core/package.json b/apps/core/package.json index 449b3f297bc..3436035e77b 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -46,7 +46,7 @@ "test:live:deepseek-mix": "NODE_ENV=development RUN_LIVE_TESTS=1 SMOKE_REVIEWER_MODEL=openai/gpt-5.5 vitest run translation-review-pipeline.live.e2e-spec.ts", "docs": "npx @compodoc/compodoc -p tsconfig.json -s -d docs", "publish": "bump", - "upgrade:check": "vite-node src/upgrade.ts check" + "upgrade:check": "vite-node --config vite.upgrade.config.ts src/upgrade.ts check" }, "bump": { "before": [ diff --git a/apps/core/src/modules/configs/configs.schema.ts b/apps/core/src/modules/configs/configs.schema.ts index eea9a96d40b..24f4fd2e82e 100644 --- a/apps/core/src/modules/configs/configs.schema.ts +++ b/apps/core/src/modules/configs/configs.schema.ts @@ -1,7 +1,7 @@ import { canonicalizeLanguageTag, isKnownLanguageTag } from '@mx-space/ai' import { z } from 'zod' -import { zAllowedUrl } from '~/common/zod' +import { zAllowedUrl } from '~/common/zod/primitives' import { AIProviderType } from '~/modules/ai/ai.types' import { field, section, withMeta } from './configs.zod-schema.util' diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts index aa340da2aa5..91eeda47541 100644 --- a/apps/core/test/src/upgrade/cli.spec.ts +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -1,4 +1,5 @@ import { execFile } from 'node:child_process' +import { fileURLToPath } from 'node:url' import { promisify } from 'node:util' import { createPgTestDatabase } from 'test/helper/pg-verify-url' @@ -16,11 +17,15 @@ describe('upgrade CLI isolated process', () => { it('reports bad runtime settings in a fresh process without bootstrap failure', async () => { const result = await promisify(execFile)( 'node_modules/.bin/vite-node', - ['src/upgrade.ts', 'check', '--json'], + [ + '--config', + 'vite.upgrade.config.ts', + 'src/upgrade.ts', + 'check', + '--json', + ], { - cwd: process.cwd().endsWith('/test') - ? `${process.cwd()}/..` - : process.cwd(), + cwd: fileURLToPath(new URL('../../../', import.meta.url)), env: { ...process.env, PG_URL: context.connectionString, @@ -30,7 +35,7 @@ describe('upgrade CLI isolated process', () => { }, }, ).catch((error) => error) - expect(result.code).toBe(2) + expect(result.code, result.stdout + result.stderr).toBe(2) const report = JSON.parse(result.stdout.trim()) expect(report.status).toBe('needs-action') expect(report.issues.map((issue) => issue.code)).toContain( diff --git a/apps/core/vite.upgrade.config.ts b/apps/core/vite.upgrade.config.ts new file mode 100644 index 00000000000..db890f91648 --- /dev/null +++ b/apps/core/vite.upgrade.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from 'vite' +import tsconfigPaths from 'vite-tsconfig-paths' + +// Offline tooling needs no application transforms, decorators or dependency optimizer. +export default defineConfig({ + plugins: [tsconfigPaths()], + optimizeDeps: { noDiscovery: true }, +}) From c9543421d166b08071046143ae1d4a274172f871 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:00:58 +0800 Subject: [PATCH 09/26] feat(core): prepare existing installations transactionally for v18 --- apps/core/package.json | 3 +- apps/core/src/upgrade/cli.ts | 35 +++- apps/core/src/upgrade/oauth-token.ts | 24 +++ apps/core/src/upgrade/preparation.ts | 196 ++++++++++++++++++ apps/core/src/upgrade/readiness.ts | 67 ++++-- .../core/test/src/upgrade/preparation.spec.ts | 182 ++++++++++++++++ apps/core/test/src/upgrade/readiness.spec.ts | 10 + 7 files changed, 490 insertions(+), 27 deletions(-) create mode 100644 apps/core/src/upgrade/oauth-token.ts create mode 100644 apps/core/src/upgrade/preparation.ts create mode 100644 apps/core/test/src/upgrade/preparation.spec.ts diff --git a/apps/core/package.json b/apps/core/package.json index 3436035e77b..f76bb9107bb 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -46,7 +46,8 @@ "test:live:deepseek-mix": "NODE_ENV=development RUN_LIVE_TESTS=1 SMOKE_REVIEWER_MODEL=openai/gpt-5.5 vitest run translation-review-pipeline.live.e2e-spec.ts", "docs": "npx @compodoc/compodoc -p tsconfig.json -s -d docs", "publish": "bump", - "upgrade:check": "vite-node --config vite.upgrade.config.ts src/upgrade.ts check" + "upgrade:check": "vite-node --config vite.upgrade.config.ts src/upgrade.ts check", + "upgrade:prepare": "vite-node --config vite.upgrade.config.ts src/upgrade.ts prepare" }, "bump": { "before": [ diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts index 52581223a96..30f8472d46a 100644 --- a/apps/core/src/upgrade/cli.ts +++ b/apps/core/src/upgrade/cli.ts @@ -1,5 +1,6 @@ import { Pool } from 'pg' +import { prepareUpgrade, UpgradeError } from './preparation' import { readUpgradeState, type UpgradeConfig } from './readiness' export function upgradeConfigFromEnv(env: NodeJS.ProcessEnv): UpgradeConfig { @@ -20,8 +21,17 @@ export async function runUpgradeCli( write = (output: string) => process.stdout.write(`${output}\n`), ): Promise { const [command, ...flags] = args - if (command !== 'check' || flags.some((flag) => flag !== '--json')) { - write('Usage: upgrade check [--json]') + const allowedFlags = + command === 'prepare' + ? ['--json', '--dry-run', '--all-replicas-v2'] + : ['--json'] + if ( + !['check', 'prepare'].includes(command) || + flags.some((flag) => !allowedFlags.includes(flag)) + ) { + write( + 'Usage: upgrade check [--json] | prepare --all-replicas-v2 [--dry-run] [--json]', + ) return 1 } const pool = new Pool({ @@ -38,6 +48,14 @@ export async function runUpgradeCli( connectionTimeoutMillis: 10000, }) try { + if (command === 'prepare') { + const result = await prepareUpgrade(pool, upgradeConfigFromEnv(env), { + allReplicasV2: flags.includes('--all-replicas-v2'), + dryRun: flags.includes('--dry-run'), + }) + write(JSON.stringify(result)) + return 0 + } const client = await pool.connect() try { await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY') @@ -52,13 +70,22 @@ export async function runUpgradeCli( } finally { client.release() } - } catch { + } catch (error) { // Driver/decryption messages can contain connection strings or stored values. write( JSON.stringify({ target: 18, status: 'error', - issues: [{ code: 'UPGRADE_CHECK_FAILED' }], + issues: [ + error instanceof UpgradeError + ? { code: error.code, path: error.path } + : { + code: + command === 'check' + ? 'UPGRADE_CHECK_FAILED' + : 'UPGRADE_PREPARATION_FAILED', + }, + ], }), ) return 1 diff --git a/apps/core/src/upgrade/oauth-token.ts b/apps/core/src/upgrade/oauth-token.ts new file mode 100644 index 00000000000..11baaeb6de4 --- /dev/null +++ b/apps/core/src/upgrade/oauth-token.ts @@ -0,0 +1,24 @@ +import { parseEnvelope, symmetricDecrypt } from 'better-auth/crypto' + +export function oauthKey(secret: string) { + return { + currentVersion: 1, + keys: new Map([[1, secret]]), + legacySecret: secret, + } +} + +export function isEncryptedOAuth(token: string) { + // Better Auth's legacy XChaCha20 payload: 24-byte nonce + 16-byte tag + text. + return ( + token.startsWith('$ba$') || + (token.length >= 80 && token.length % 2 === 0 && /^[\da-f]+$/i.test(token)) + ) +} + +export async function unwrapOAuthToken(token: string, secret: string) { + if (!isEncryptedOAuth(token)) return token + if (token.startsWith('$ba$') && !parseEnvelope(token)) + throw new Error('Malformed OAuth envelope') + return symmetricDecrypt({ key: oauthKey(secret), data: token }) +} diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts new file mode 100644 index 00000000000..4081d5c28fe --- /dev/null +++ b/apps/core/src/upgrade/preparation.ts @@ -0,0 +1,196 @@ +import { isDeepStrictEqual } from 'node:util' + +import { defaultKeyHasher } from '@better-auth/api-key' +import { SnowflakeGenerator } from '@mx-space/db-schema/id' +import { symmetricEncrypt } from 'better-auth/crypto' +import type { Pool } from 'pg' + +import { backfillAiMetrics } from '../database/app-migrations/20261108-ai-metrics-coverage' +import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' +import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' +import { + canonicalConfigRows, + normalizeLegacyConfig, +} from '../modules/configs/legacy-config.util' +import { SecretCodec } from '../utils/secret-codec' +import { unwrapOAuthToken } from './oauth-token' +import { + CONFIG_UPGRADE_LOCK, + oauthKey, + PREPARED_MIGRATION_ID, + readUpgradeState, + type UpgradeConfig, +} from './readiness' + +export class UpgradeError extends Error { + constructor( + public readonly code: string, + public readonly path?: string, + ) { + super(path ? `${code} (${path})` : code) + } +} + +export async function prepareUpgrade( + pool: Pool, + config: UpgradeConfig, + options: { allReplicasV2: boolean; dryRun?: boolean }, +) { + if (!options.allReplicasV2) + throw new UpgradeError('REPLICA_CONFIRMATION_REQUIRED') + const client = await pool.connect() + const codec = new SecretCodec({ ...config, v2: true }) + const ids = new SnowflakeGenerator({ workerId: 1023 }) + let changed = 0 + try { + await client.query('BEGIN') + await client.query("SET LOCAL lock_timeout = '10s'") + await client.query('SELECT pg_advisory_xact_lock($1::bigint)', [ + CONFIG_UPGRADE_LOCK, + ]) + const before = await readUpgradeState(client, config) + const allowed = new Set([ + 'CREDENTIALS_NEED_PREPARATION', + 'LEGACY_CONFIG', + 'METRICS_INCOMPLETE', + 'PREPARATION_MARKER_MISSING', + ]) + const blocking = before.issues.find((issue) => !allowed.has(issue.code)) + if (blocking) throw new UpgradeError(blocking.code, blocking.path) + const rows = ( + await client.query( + 'SELECT id, name, value FROM options ORDER BY name FOR UPDATE', + ) + ).rows + const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) + for (const key of [ + 'ai', + 'membership', + 'thirdPartyServiceIntegration', + 'imageGenerationOptions', + 'ttsOptions', + ]) { + if ( + key in raw && + (!raw[key] || typeof raw[key] !== 'object' || Array.isArray(raw[key])) + ) + throw new UpgradeError('CONFIG_SHAPE_UNSUPPORTED', key) + } + const plaintext = Object.fromEntries( + rows.map((row) => [ + row.name, + visitConfigSecrets(row.value, row.name, (secret, path) => + codec.decrypt(secret, path), + ), + ]), + ) + const canonical = canonicalConfigRows(normalizeLegacyConfig(plaintext)) + const names = new Set(rows.map((row) => row.name)) + if (names.has('imageGenerationOptions') || names.has('ttsOptions')) + names.add('ai') + names.delete('imageGenerationOptions') + names.delete('ttsOptions') + for (const name of names) { + const value = canonical[name] ?? plaintext[name] + const encrypted = visitConfigSecrets(value, name, (secret, path) => { + const previous = path + .split('.') + .reduce((node, part) => node?.[part], raw) + return typeof previous === 'string' && + codec.isEncryptedString(previous) && + codec.decrypt(previous, path) === secret + ? codec.encrypt(previous, path) + : codec.encrypt(secret, path) + }) + if (isDeepStrictEqual(encrypted, raw[name])) continue + await client.query( + 'INSERT INTO options (id, name, value) VALUES ($1, $2, $3) ON CONFLICT (name) DO UPDATE SET value = excluded.value', + [ids.nextId(), name, JSON.stringify(encrypted)], + ) + changed++ + } + changed += + ( + await client.query( + "DELETE FROM options WHERE name IN ('imageGenerationOptions', 'ttsOptions')", + ) + ).rowCount ?? 0 + for (const table of ['snippets', 'webhooks']) { + for (const row of ( + await client.query(`SELECT id, secret FROM ${table} FOR UPDATE`) + ).rows) { + if (!row.secret) continue + const secret = codec.encrypt(row.secret, `${table}.${row.id}.secret`) + if (secret === row.secret) continue + await client.query(`UPDATE ${table} SET secret = $1 WHERE id = $2`, [ + secret, + row.id, + ]) + changed++ + } + } + for (const row of ( + await client.query( + 'SELECT id, key, config_id, user_id, reference_id, start FROM api_keys FOR UPDATE', + ) + ).rows) { + if (row.config_id === HASHED_API_KEY_CONFIG_ID) continue + await client.query( + 'UPDATE api_keys SET key = $1, config_id = $2, reference_id = $3, start = $4 WHERE id = $5', + [ + await defaultKeyHasher(row.key), + HASHED_API_KEY_CONFIG_ID, + row.reference_id || row.user_id, + row.start || row.key.slice(0, 6), + row.id, + ], + ) + changed++ + } + for (const row of ( + await client.query( + 'SELECT id, access_token, refresh_token FROM accounts FOR UPDATE', + ) + ).rows) { + for (const field of ['access_token', 'refresh_token']) { + const token = row[field] + if (!token || token.startsWith('$ba$')) continue // Authenticated by readUpgradeState. + const encrypted = await symmetricEncrypt({ + key: oauthKey(config.jwtSecret), + data: await unwrapOAuthToken(token, config.jwtSecret), + }) + await client.query(`UPDATE accounts SET ${field} = $1 WHERE id = $2`, [ + encrypted, + row.id, + ]) + changed++ + } + } + changed += await backfillAiMetrics(client) + const after = await readUpgradeState(client, config) + const residual = after.issues.find( + (issue) => issue.code !== 'PREPARATION_MARKER_MISSING', + ) + if (residual) throw new UpgradeError(residual.code, residual.path) + if (!after.prepared) { + await client.query( + 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', + [PREPARED_MIGRATION_ID], + ) + } + await client.query(options.dryRun ? 'ROLLBACK' : 'COMMIT') + return { + status: options.dryRun ? 'would-be-ready' : 'prepared', + dryRun: !!options.dryRun, + changed, + requiresRestart: !options.dryRun, + } + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error instanceof UpgradeError + ? error + : new UpgradeError('UPGRADE_PREPARATION_FAILED') + } finally { + client.release() + } +} diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index db0cc84b220..102386b32d9 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -1,9 +1,13 @@ -import { parseEnvelope, symmetricDecrypt } from 'better-auth/crypto' import type { Pool, PoolClient } from 'pg' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' +import { configSchemaMapping } from '../modules/configs/configs.schema' import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' +import { normalizeLegacyConfig } from '../modules/configs/legacy-config.util' import { SecretCodec } from '../utils/secret-codec' +import { isEncryptedOAuth, unwrapOAuthToken } from './oauth-token' + +export { oauthKey } from './oauth-token' export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' export const CONFIG_UPGRADE_LOCK = '5183248167463294042' @@ -35,24 +39,18 @@ export type UpgradeReport = { } export type UpgradeDatabase = Pick -export function oauthKey(secret: string) { - return { - currentVersion: 1, - keys: new Map([[1, secret]]), - legacySecret: secret, - } -} - export function isLegacyConfig(name: string, value: any): boolean { return ( name === 'imageGenerationOptions' || name === 'ttsOptions' || (name === 'ai' && value?.version !== 2) || (name === 'thirdPartyServiceIntegration' && - !!value && + value && + typeof value === 'object' && 'githubToken' in value) || (name === 'membership' && - !!value && + value && + typeof value === 'object' && ['dodoApiKey', 'dodoEnvironment', 'dodoWebhookKey'].some( (key) => key in value, )) @@ -105,6 +103,39 @@ export async function readUpgradeState( if (row.value && typeof row.value === 'object') visitConfigSecrets(row.value, row.name, inspect) } + const knownRows = options.filter((row) => row.name in configSchemaMapping) + for (const row of knownRows) { + if (!row.value || typeof row.value !== 'object' || Array.isArray(row.value)) + issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: row.name }) + } + if ( + keyValid && + !issues.some( + (issue) => + issue.code === 'SECRET_UNREADABLE' || + issue.code === 'CONFIG_SHAPE_UNSUPPORTED', + ) + ) { + try { + const raw = Object.fromEntries( + knownRows.map((row) => [ + row.name, + visitConfigSecrets(row.value, row.name, (secret, path) => + codec.decrypt(secret, path), + ), + ]), + ) + const normalized = normalizeLegacyConfig(raw) + for (const row of knownRows) { + if ( + !configSchemaMapping[row.name].safeParse(normalized[row.name]).success + ) + issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: row.name }) + } + } catch { + issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED' }) + } + } for (const table of ['snippets', 'webhooks']) { for (const row of (await database.query(`SELECT id, secret FROM ${table}`)) .rows) { @@ -123,22 +154,14 @@ export async function readUpgradeState( for (const field of ['access_token', 'refresh_token']) { const token = row[field] if (!token) continue - if (!token.startsWith('$ba$')) { + if (!isEncryptedOAuth(token)) { counts.plainOAuth++ continue } counts.encryptedOAuth++ - if (!parseEnvelope(token)) - issues.push({ - code: 'OAUTH_UNREADABLE', - path: `accounts.${row.id}.${field}`, - }) - else if (config.jwtSecret) { + if (config.jwtSecret) { try { - await symmetricDecrypt({ - key: oauthKey(config.jwtSecret), - data: token, - }) + await unwrapOAuthToken(token, config.jwtSecret) } catch { issues.push({ code: 'OAUTH_UNREADABLE', diff --git a/apps/core/test/src/upgrade/preparation.spec.ts b/apps/core/test/src/upgrade/preparation.spec.ts new file mode 100644 index 00000000000..f32cac8dc61 --- /dev/null +++ b/apps/core/test/src/upgrade/preparation.spec.ts @@ -0,0 +1,182 @@ +import { defaultKeyHasher } from '@better-auth/api-key' +import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' + +import { prepareUpgrade } from '~/upgrade/preparation' +import { + CONFIG_UPGRADE_LOCK, + oauthKey, + readUpgradeState, +} from '~/upgrade/readiness' +import { SecretCodec } from '~/utils/secret-codec' + +describe('transactional v18 preparation', () => { + let context: Awaited> + const config = { + key: 'ab'.repeat(32), + jwtSecret: 'long-test-jwt-secret-1234567890123456789', + enable: true, + v2: true, + } + const codec = new SecretCodec(config) + beforeAll(async () => { + context = await createPgTestDatabase('mx_upgrade_prepare') + }) + afterAll(async () => context?.close()) + beforeEach(async () => { + await context.pool.query( + 'TRUNCATE options, snippets, webhooks, api_keys, accounts, readers, _app_migrations CASCADE', + ) + await context.pool.query( + "INSERT INTO options (id, name, value) VALUES ('100', 'membership', $1), ('101', 'thirdPartyServiceIntegration', $2), ('102', 'imageGenerationOptions', $3)", + [ + { + dodoApiKey: codec.encrypt('billing-secret', 'membership.dodoApiKey'), + }, + { + githubToken: new SecretCodec({ ...config, v2: false }).encrypt( + 'github-secret', + 'thirdPartyServiceIntegration.githubToken', + ), + }, + { enable: true, apiKey: 'image-secret', model: 'image-v1' }, + ], + ) + await context.pool.query( + "INSERT INTO snippets (id, raw, path, secret) VALUES ('200', '{}', 'sample', 'snippet-secret')", + ) + await context.pool.query( + "INSERT INTO webhooks (id, payload_url, events, secret) VALUES ('201', 'https://example.test', '{}', 'webhook-secret')", + ) + await context.pool.query( + "INSERT INTO readers (id, email, name) VALUES ('owner', 'owner@example.test', 'Owner')", + ) + await context.pool.query( + "INSERT INTO api_keys (id, user_id, key) VALUES ('key', 'owner', 'raw-client-key')", + ) + await context.pool.query( + "INSERT INTO accounts (id, user_id, account_id, provider_id, access_token) VALUES ('account', 'owner', 'remote', 'github', 'provider-token')", + ) + }) + async function snapshot() { + return Promise.all( + [ + 'options', + 'snippets', + 'webhooks', + 'api_keys', + 'accounts', + '_app_migrations', + ].map( + async (table) => + (await context.pool.query(`SELECT * FROM ${table} ORDER BY id`)).rows, + ), + ) + } + it('requires an explicit all-replicas confirmation and leaves data untouched', async () => { + const before = await snapshot() + await expect( + prepareUpgrade(context.pool, config, { allReplicasV2: false }), + ).rejects.toThrow('REPLICA_CONFIRMATION_REQUIRED') + expect(await snapshot()).toEqual(before) + }) + it('dry-run proves conversion and rolls back every write', async () => { + const before = await snapshot() + expect( + await prepareUpgrade(context.pool, config, { + allReplicasV2: true, + dryRun: true, + }), + ).toMatchObject({ dryRun: true, status: 'would-be-ready' }) + expect(await snapshot()).toEqual(before) + }) + it('converts at original paths, hashes once and preserves OAuth/provider secrets', async () => { + await prepareUpgrade(context.pool, config, { allReplicasV2: true }) + expect(await readUpgradeState(context.pool, config)).toMatchObject({ + status: 'ready', + prepared: true, + counts: { + legacyConfig: 0, + ecb: 0, + plaintext: 0, + plainApiKeys: 0, + plainOAuth: 0, + }, + }) + const rows = Object.fromEntries( + (await context.pool.query('SELECT name, value FROM options')).rows.map( + (row) => [row.name, row.value], + ), + ) + expect(codec.decrypt(rows.membership.apiKey, 'membership.apiKey')).toBe( + 'billing-secret', + ) + expect( + codec.decrypt( + rows.thirdPartyServiceIntegration.github.token, + 'thirdPartyServiceIntegration.github.token', + ), + ).toBe('github-secret') + expect( + codec.decrypt(rows.ai.providers[0].apiKey, 'ai.providers.0.apiKey'), + ).toBe('image-secret') + const key = (await context.pool.query('SELECT key FROM api_keys')).rows[0] + .key + expect(key).toBe(await defaultKeyHasher('raw-client-key')) + expect( + await symmetricDecrypt({ + key: oauthKey(config.jwtSecret), + data: (await context.pool.query('SELECT access_token FROM accounts')) + .rows[0].access_token, + }), + ).toBe('provider-token') + const before = await snapshot() + await prepareUpgrade(context.pool, config, { allReplicasV2: true }) + expect(await snapshot()).toEqual(before) + }) + it('rolls back and omits a marker if original decryption fails', async () => { + const before = await snapshot() + await expect( + prepareUpgrade( + context.pool, + { ...config, key: 'cd'.repeat(32) }, + { allReplicasV2: true }, + ), + ).rejects.toThrow('SECRET_UNREADABLE') + expect(await snapshot()).toEqual(before) + }) + it('upgrades a legacy Better Auth bare-hex token without double encryption', async () => { + const bare = await symmetricEncrypt({ + key: config.jwtSecret, + data: 'original-provider-token', + }) + await context.pool.query('UPDATE accounts SET access_token = $1', [bare]) + await prepareUpgrade(context.pool, config, { allReplicasV2: true }) + const stored = ( + await context.pool.query('SELECT access_token FROM accounts') + ).rows[0].access_token + expect( + await symmetricDecrypt({ key: oauthKey(config.jwtSecret), data: stored }), + ).toBe('original-provider-token') + }) + it('serializes preparation with normal configuration writes', async () => { + const writer = await context.pool.connect() + await writer.query('BEGIN') + await writer.query('SELECT pg_advisory_xact_lock($1::bigint)', [ + CONFIG_UPGRADE_LOCK, + ]) + let finished = false + const preparing = prepareUpgrade(context.pool, config, { + allReplicasV2: true, + }).then(() => { + finished = true + }) + await new Promise((resolve) => setTimeout(resolve, 100)) + expect(finished).toBe(false) + await writer.query('COMMIT') + writer.release() + await preparing + expect(finished).toBe(true) + }) +}) diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts index a8c5921c0e2..0b9b495ed55 100644 --- a/apps/core/test/src/upgrade/readiness.spec.ts +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -73,4 +73,14 @@ describe('standalone upgrade readiness', () => { before, ) }) + it('does not trust a marker over malformed canonical configuration', async () => { + await context.pool.query( + "DELETE FROM options; INSERT INTO _app_migrations (id) VALUES ('20261108-v18-prepared'); INSERT INTO options (id, name, value) VALUES ('300', 'membership', '42'::jsonb)", + ) + expect( + (await readUpgradeState(context.pool, config)).issues.map( + (issue) => issue.code, + ), + ).toContain('CONFIG_SHAPE_UNSUPPORTED') + }) }) From e16210ec8a89af1dea1934f041337ff6b23ebdf1 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:03:18 +0800 Subject: [PATCH 10/26] feat(core): rotate JWT secrets during explicit maintenance --- apps/core/package.json | 3 +- apps/core/src/upgrade/cli.ts | 51 ++++++- apps/core/src/upgrade/jwt-rotation.ts | 91 ++++++++++++ .../test/src/upgrade/jwt-rotation.spec.ts | 137 ++++++++++++++++++ 4 files changed, 273 insertions(+), 9 deletions(-) create mode 100644 apps/core/src/upgrade/jwt-rotation.ts create mode 100644 apps/core/test/src/upgrade/jwt-rotation.spec.ts diff --git a/apps/core/package.json b/apps/core/package.json index f76bb9107bb..6b072ba938e 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -47,7 +47,8 @@ "docs": "npx @compodoc/compodoc -p tsconfig.json -s -d docs", "publish": "bump", "upgrade:check": "vite-node --config vite.upgrade.config.ts src/upgrade.ts check", - "upgrade:prepare": "vite-node --config vite.upgrade.config.ts src/upgrade.ts prepare" + "upgrade:prepare": "vite-node --config vite.upgrade.config.ts src/upgrade.ts prepare", + "upgrade:rotate-jwt": "vite-node --config vite.upgrade.config.ts src/upgrade.ts rotate-jwt" }, "bump": { "before": [ diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts index 30f8472d46a..9ec5a1c1303 100644 --- a/apps/core/src/upgrade/cli.ts +++ b/apps/core/src/upgrade/cli.ts @@ -1,5 +1,8 @@ +import { parseArgs } from 'node:util' + import { Pool } from 'pg' +import { rotateJwtSecret } from './jwt-rotation' import { prepareUpgrade, UpgradeError } from './preparation' import { readUpgradeState, type UpgradeConfig } from './readiness' @@ -21,14 +24,32 @@ export async function runUpgradeCli( write = (output: string) => process.stdout.write(`${output}\n`), ): Promise { const [command, ...flags] = args - const allowedFlags = - command === 'prepare' - ? ['--json', '--dry-run', '--all-replicas-v2'] - : ['--json'] - if ( - !['check', 'prepare'].includes(command) || - flags.some((flag) => !allowedFlags.includes(flag)) - ) { + let values: ReturnType['values'] + try { + values = parseArgs({ + args: flags, + options: { + json: { type: 'boolean' }, + ...(command === 'prepare' + ? { + 'dry-run': { type: 'boolean' as const }, + 'all-replicas-v2': { type: 'boolean' as const }, + } + : {}), + ...(command === 'rotate-jwt' + ? { + 'maintenance-confirmed': { type: 'boolean' as const }, + 'old-jwt-file': { type: 'string' as const }, + 'new-jwt-file': { type: 'string' as const }, + } + : {}), + }, + }).values + } catch { + write('Invalid upgrade arguments') + return 1 + } + if (!['check', 'prepare', 'rotate-jwt'].includes(command)) { write( 'Usage: upgrade check [--json] | prepare --all-replicas-v2 [--dry-run] [--json]', ) @@ -48,6 +69,20 @@ export async function runUpgradeCli( connectionTimeoutMillis: 10000, }) try { + if (command === 'rotate-jwt') { + if ( + typeof values['old-jwt-file'] !== 'string' || + typeof values['new-jwt-file'] !== 'string' + ) + throw new UpgradeError('JWT_FILE_PATHS_REQUIRED') + const result = await rotateJwtSecret(pool, { + maintenanceConfirmed: values['maintenance-confirmed'] === true, + oldJwtFile: values['old-jwt-file'], + newJwtFile: values['new-jwt-file'], + }) + write(JSON.stringify(result)) + return 0 + } if (command === 'prepare') { const result = await prepareUpgrade(pool, upgradeConfigFromEnv(env), { allReplicasV2: flags.includes('--all-replicas-v2'), diff --git a/apps/core/src/upgrade/jwt-rotation.ts b/apps/core/src/upgrade/jwt-rotation.ts new file mode 100644 index 00000000000..54b3a325b33 --- /dev/null +++ b/apps/core/src/upgrade/jwt-rotation.ts @@ -0,0 +1,91 @@ +import { readFile, stat } from 'node:fs/promises' + +import { symmetricEncrypt } from 'better-auth/crypto' +import type { Pool } from 'pg' + +import { oauthKey, unwrapOAuthToken } from './oauth-token' +import { UpgradeError } from './preparation' +import { CONFIG_UPGRADE_LOCK } from './readiness' + +async function readSecretFile(file: string) { + try { + if (!(await stat(file)).isFile()) throw new Error('Not a file') + const value = (await readFile(file, 'utf8')).replace(/\r?\n$/, '') + if (!value || /[\r\n]/.test(value)) throw new Error('Invalid secret file') + return value + } catch { + throw new UpgradeError('JWT_FILE_UNREADABLE') + } +} + +/** Offline only: the operator must stop every business/background writer first. */ +export async function rotateJwtSecret( + pool: Pool, + input: { + maintenanceConfirmed: boolean + oldJwtFile: string + newJwtFile: string + }, +) { + if (!input.maintenanceConfirmed) + throw new UpgradeError('MAINTENANCE_CONFIRMATION_REQUIRED') + const [oldSecret, newSecret] = await Promise.all([ + readSecretFile(input.oldJwtFile), + readSecretFile(input.newJwtFile), + ]) + if (newSecret.length < 32) throw new UpgradeError('NEW_JWT_TOO_SHORT') + if (newSecret === oldSecret) throw new UpgradeError('JWT_UNCHANGED') + const client = await pool.connect() + let changed = 0 + try { + await client.query('BEGIN') + await client.query("SET LOCAL lock_timeout = '10s'") + await client.query('SELECT pg_advisory_xact_lock($1::bigint)', [ + CONFIG_UPGRADE_LOCK, + ]) + for (const row of ( + await client.query( + 'SELECT id, access_token, refresh_token FROM accounts FOR UPDATE', + ) + ).rows) { + for (const field of ['access_token', 'refresh_token']) { + if (!row[field]) continue + let plaintext: string + try { + plaintext = await unwrapOAuthToken(row[field], oldSecret) + } catch { + throw new UpgradeError( + 'OAUTH_UNREADABLE', + `accounts.${row.id}.${field}`, + ) + } + const encrypted = await symmetricEncrypt({ + key: oauthKey(newSecret), + data: plaintext, + }) + await client.query(`UPDATE accounts SET ${field} = $1 WHERE id = $2`, [ + encrypted, + row.id, + ]) + changed++ + } + } + const expiredSessions = + (await client.query('DELETE FROM sessions')).rowCount ?? 0 + await client.query('COMMIT') + return { + status: 'rotated', + changed, + expiredSessions, + reLoginRequired: true, + environmentUpdateRequired: true, + } + } catch (error) { + await client.query('ROLLBACK').catch(() => {}) + throw error instanceof UpgradeError + ? error + : new UpgradeError('JWT_ROTATION_FAILED') + } finally { + client.release() + } +} diff --git a/apps/core/test/src/upgrade/jwt-rotation.spec.ts b/apps/core/test/src/upgrade/jwt-rotation.spec.ts new file mode 100644 index 00000000000..17ae449fda5 --- /dev/null +++ b/apps/core/test/src/upgrade/jwt-rotation.spec.ts @@ -0,0 +1,137 @@ +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import path from 'node:path' + +import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' + +import { rotateJwtSecret } from '~/upgrade/jwt-rotation' +import { oauthKey } from '~/upgrade/oauth-token' + +describe('maintenance JWT rotation', () => { + let context: Awaited> + let directory: string + const oldSecret = 'old-short-jwt' + const newSecret = 'new-long-jwt-for-upgrade-123456789012345' + let input: { + oldJwtFile: string + newJwtFile: string + maintenanceConfirmed: boolean + } + beforeAll(async () => { + context = await createPgTestDatabase('mx_jwt_rotation') + directory = await mkdtemp(path.join(tmpdir(), 'mx-jwt-test-')) + input = { + oldJwtFile: path.join(directory, 'old'), + newJwtFile: path.join(directory, 'new'), + maintenanceConfirmed: true, + } + await writeFile(input.oldJwtFile, oldSecret, { mode: 0o400 }) + await writeFile(input.newJwtFile, `${newSecret}\n`, { mode: 0o400 }) + }) + afterAll(async () => { + await context?.close() + if (directory) await rm(directory, { recursive: true }) + }) + beforeEach(async () => { + await context.pool.query( + 'TRUNCATE accounts, sessions, readers, api_keys, options CASCADE', + ) + await context.pool.query( + "INSERT INTO readers (id, email, name) VALUES ('owner', 'owner@example.test', 'Owner')", + ) + await context.pool.query( + "INSERT INTO sessions (id, token, user_id, expires_at) VALUES ('session', 'session-token', 'owner', now() + interval '1 day')", + ) + await context.pool.query( + "INSERT INTO api_keys (id, key) VALUES ('key', 'unchanged-api-key')", + ) + await context.pool.query( + "INSERT INTO options (id, name, value) VALUES ('100', 'non-secret-setting', '{\"untouched\":true}')", + ) + await context.pool.query( + "INSERT INTO accounts (id, user_id, account_id, provider_id, access_token, refresh_token) VALUES ('a', 'owner', 'remote', 'github', $1, 'plaintext-refresh'), ('b', 'owner', 'remote-b', 'google', $2, NULL)", + [ + await symmetricEncrypt({ + key: oauthKey(oldSecret), + data: 'access-token', + }), + await symmetricEncrypt({ key: oldSecret, data: 'legacy-access' }), + ], + ) + }) + async function snapshot() { + return (await context.pool.query('SELECT * FROM accounts ORDER BY id')).rows + } + it('requires maintenance confirmation before any mutation', async () => { + const before = await snapshot() + await expect( + rotateJwtSecret(context.pool, { ...input, maintenanceConfirmed: false }), + ).rejects.toThrow('MAINTENANCE_CONFIRMATION_REQUIRED') + expect(await snapshot()).toEqual(before) + }) + it('rewraps all known OAuth formats, expires sessions, and preserves other secrets/files', async () => { + expect(await rotateJwtSecret(context.pool, input)).toMatchObject({ + reLoginRequired: true, + environmentUpdateRequired: true, + }) + const rows = await snapshot() + expect( + await symmetricDecrypt({ + key: oauthKey(newSecret), + data: rows[0].access_token, + }), + ).toBe('access-token') + expect( + await symmetricDecrypt({ + key: oauthKey(newSecret), + data: rows[0].refresh_token, + }), + ).toBe('plaintext-refresh') + expect( + await symmetricDecrypt({ + key: oauthKey(newSecret), + data: rows[1].access_token, + }), + ).toBe('legacy-access') + expect( + (await context.pool.query('SELECT count(*)::int AS n FROM sessions')) + .rows[0].n, + ).toBe(0) + expect( + (await context.pool.query('SELECT key FROM api_keys')).rows[0].key, + ).toBe('unchanged-api-key') + expect( + (await context.pool.query('SELECT value FROM options')).rows[0].value, + ).toEqual({ untouched: true }) + expect(await readFile(input.oldJwtFile, 'utf8')).toBe(oldSecret) + expect(await readFile(input.newJwtFile, 'utf8')).toBe(`${newSecret}\n`) + }) + it('rolls the whole batch back when an OAuth record cannot be opened', async () => { + await context.pool.query( + "UPDATE accounts SET access_token = '$ba$1$invalid-ciphertext' WHERE id = 'b'", + ) + const before = await snapshot() + const error = await rotateJwtSecret(context.pool, input).catch( + (error) => error, + ) + expect(error.message).toContain('OAUTH_UNREADABLE') + expect(error.message).not.toContain(oldSecret) + expect(error.message).not.toContain('invalid-ciphertext') + expect(await snapshot()).toEqual(before) + expect( + (await context.pool.query('SELECT count(*)::int AS n FROM sessions')) + .rows[0].n, + ).toBe(1) + }) + it('rejects a wrong original JWT without resetting tokens', async () => { + const wrongFile = path.join(directory, 'wrong') + await writeFile(wrongFile, 'incorrect-original', { mode: 0o400 }) + const before = await snapshot() + await expect( + rotateJwtSecret(context.pool, { ...input, oldJwtFile: wrongFile }), + ).rejects.toThrow('OAUTH_UNREADABLE') + expect(await snapshot()).toEqual(before) + }) +}) From 6b9352080d4307f674c53cc33c66c7ba1546b198 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:17:46 +0800 Subject: [PATCH 11/26] docs(core): document and rehearse the v18 preparation phase --- .env.example | 5 +- apps/core/README.md | 13 ++ .../src/modules/configs/configs.service.ts | 9 +- .../src/modules/configs/legacy-config.util.ts | 15 +- apps/core/src/upgrade/cli.ts | 10 +- .../modules/configs/configs.service.spec.ts | 4 +- .../src/modules/configs/legacy-config.spec.ts | 16 ++ .../src/upgrade/archive-rehearsal.spec.ts | 172 ++++++++++++++++++ apps/core/test/src/upgrade/cli.spec.ts | 17 +- docs/security-secret-storage.md | 11 +- docs/v18-preparation.md | 124 +++++++++++++ 11 files changed, 373 insertions(+), 23 deletions(-) create mode 100644 apps/core/test/src/upgrade/archive-rehearsal.spec.ts create mode 100644 docs/v18-preparation.md diff --git a/.env.example b/.env.example index 25980342774..93abac03f76 100644 --- a/.env.example +++ b/.env.example @@ -1,10 +1,11 @@ # THIS ENV FILE EXAMPLE ONLY FOR DOCKER COMPOSE # SEE https://docs.docker.com/compose/environment-variables/#the-env-file -JWT_SECRET=asffasgvxczfqreqw213 +# Set a private instance secret (at least 32 characters for V2). Preserve an existing value. +JWT_SECRET= ALLOWED_ORIGINS=innei.ren,www.innei.ren # Must be 64 hexadecimal characters. Keep the existing key when upgrading. -ENCRYPT_KEY=593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab +ENCRYPT_KEY= # Temporary legacy plaintext escape hatch; v18 will require encryption. ENCRYPT_ENABLE=false diff --git a/apps/core/README.md b/apps/core/README.md index 5dabddfcf98..34d83c3e47e 100644 --- a/apps/core/README.md +++ b/apps/core/README.md @@ -299,3 +299,16 @@ Inspired in part by [nodepress](https://github.com/surmon-china/nodepress). --- Since 2021-08-31. + +## Preparing for v18 (unreleased) + +This branch adds standalone `upgrade:check`, `upgrade:prepare` and +`upgrade:rotate-jwt` tooling. These commands are not in v17.0.4; a minimum +released preparation tag and client matrix will be recorded after release +verification. v17 behavior remains compatible, including legacy GET actions +and SDK 5.x signatures. No columns are dropped in this phase. + +See the [preparation runbook](../../docs/v18-preparation.md) for read-only +diagnostics, dry-run/transactional preparation, original-secret requirements +and offline JWT rotation. Do not change an existing encryption key or JWT +to fix decryption failures. diff --git a/apps/core/src/modules/configs/configs.service.ts b/apps/core/src/modules/configs/configs.service.ts index 8a431695a4d..cae617d2be2 100644 --- a/apps/core/src/modules/configs/configs.service.ts +++ b/apps/core/src/modules/configs/configs.service.ts @@ -32,6 +32,7 @@ import { import { configDtoMapping, IConfig } from './configs.interface' import type { OAuthConfig } from './configs.schema' import { + canonicalConfigRows, normalizeLegacyConfig, normalizeLegacyPatch, updateLegacyMedia, @@ -260,7 +261,13 @@ export class ConfigsService implements OnModuleInit { ), ) const newData = updatedConfigRow.value as IConfig[T] - const mergedFullConfig = Object.assign({}, config, { [key]: newData }) + const mergedFullConfig = normalizeLegacyConfig( + Object.assign( + {}, + key === 'ai' ? canonicalConfigRows(config as IConfig) : config, + { [key]: decryptObject(cloneDeep(newData), key) }, + ), + ) await this.setConfig(mergedFullConfig) this.eventManager.emit( diff --git a/apps/core/src/modules/configs/legacy-config.util.ts b/apps/core/src/modules/configs/legacy-config.util.ts index 7e433bc7252..c54b72cc119 100644 --- a/apps/core/src/modules/configs/legacy-config.util.ts +++ b/apps/core/src/modules/configs/legacy-config.util.ts @@ -57,10 +57,6 @@ export function normalizeAiConfig( } }) - if (storedAiConfig?.version === 2) { - return { ...aiConfig, version: 2, providers } - } - const imageGeneration = { enable: legacyImageConfig.enable, model: undefined as { providerId: string; model?: string } | undefined, @@ -96,6 +92,7 @@ export function normalizeAiConfig( ) if (existing) { existing.capabilities = { + ...existing.capabilities, text: existing.capabilities?.text ?? true, image: existing.capabilities?.image || input.capability === 'image', speech: existing.capabilities?.speech || input.capability === 'speech', @@ -169,13 +166,9 @@ export function normalizeAiConfig( } } - return { - ...aiConfig, - version: 2, - providers, - imageGeneration, - tts, - } + return storedAiConfig?.version === 2 + ? { ...aiConfig, version: 2, providers } + : { ...aiConfig, version: 2, providers, imageGeneration, tts } } /** diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts index 9ec5a1c1303..7e107842dab 100644 --- a/apps/core/src/upgrade/cli.ts +++ b/apps/core/src/upgrade/cli.ts @@ -8,11 +8,11 @@ import { readUpgradeState, type UpgradeConfig } from './readiness' export function upgradeConfigFromEnv(env: NodeJS.ProcessEnv): UpgradeConfig { const key = env.MX_ENCRYPT_KEY || env.ENCRYPT_KEY || '' - const enabled = env.MX_ENCRYPT_ENABLE ?? env.ENCRYPT_ENABLE + const enabled = env.MX_ENCRYPT_ENABLE || env.ENCRYPT_ENABLE return { key, - enable: enabled === undefined ? !!key : /^(?:true|1)$/i.test(enabled), - v2: /^(?:true|1)$/i.test(env.MX_SECRET_STORAGE_V2 ?? ''), + enable: enabled === undefined ? !!key : enabled === 'true', + v2: env.MX_SECRET_STORAGE_V2 === '1' || env.MX_SECRET_STORAGE_V2 === 'true', jwtSecret: env.JWT_SECRET || env.JWTSECRET || '', algorithm: env.ENCRYPT_ALGORITHM || 'aes-256-ecb', } @@ -62,9 +62,7 @@ export async function runUpgradeCli( user: env.PG_USER || 'mx', password: env.PG_PASSWORD || 'mx', database: env.PG_DATABASE || 'mx_core', - ssl: /^(?:true|1)$/i.test(env.PG_SSL ?? '') - ? { rejectUnauthorized: false } - : false, + ssl: env.PG_SSL === 'true' ? { rejectUnauthorized: false } : false, max: 1, connectionTimeoutMillis: 10000, }) diff --git a/apps/core/test/src/modules/configs/configs.service.spec.ts b/apps/core/test/src/modules/configs/configs.service.spec.ts index 2a79a323dea..18277a2a889 100644 --- a/apps/core/test/src/modules/configs/configs.service.spec.ts +++ b/apps/core/test/src/modules/configs/configs.service.spec.ts @@ -61,6 +61,8 @@ describe('ConfigsService', () => { apiKey: 'image-secret', model: 'image-v2', }) + await service.patchAndValid('ai', { imageGeneration: { enable: false } }) + expect((await service.get('imageGenerationOptions')).enable).toBe(false) await service.patchAndValid('membership', { dodoApiKey: 'billing-secret', } as any) @@ -348,7 +350,7 @@ describe('ConfigsService', () => { JSON.parse(redisClient.set.mock.calls[0][1]), ) expect(cachedConfig.ai.providers).toHaveLength(1) - expect(cachedConfig.ai.providers[0].capabilities).toEqual({ + expect(cachedConfig.ai.providers[0].capabilities).toMatchObject({ text: false, image: true, speech: true, diff --git a/apps/core/test/src/modules/configs/legacy-config.spec.ts b/apps/core/test/src/modules/configs/legacy-config.spec.ts index 9a95a9e605f..eb90538d0b0 100644 --- a/apps/core/test/src/modules/configs/legacy-config.spec.ts +++ b/apps/core/test/src/modules/configs/legacy-config.spec.ts @@ -74,4 +74,20 @@ describe('canonical legacy config conversion', () => { endpoint: 'https://api.example.com/v1', }) }) + it('retains leftover media credentials alongside an already-v2 AI row', () => { + const converted = normalizeLegacyConfig({ + ai: { version: 2, providers: [], imageGeneration: { enable: false } }, + imageGenerationOptions: { + apiKey: 'retained-legacy-key', + model: 'legacy-model', + enable: true, + }, + }) + expect( + converted.ai.providers?.some( + (provider) => provider.apiKey === 'retained-legacy-key', + ), + ).toBe(true) + expect(converted.ai.imageGeneration?.enable).toBe(false) + }) }) diff --git a/apps/core/test/src/upgrade/archive-rehearsal.spec.ts b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts new file mode 100644 index 00000000000..b596b19b78b --- /dev/null +++ b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts @@ -0,0 +1,172 @@ +import { execFileSync } from 'node:child_process' +import { createCipheriv, randomBytes } from 'node:crypto' +import { existsSync } from 'node:fs' +import { fileURLToPath } from 'node:url' + +import JSZip from 'jszip' +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { afterAll, beforeAll, describe, expect, it } from 'vitest' + +import { decryptBackup } from '~/modules/backup/backup-crypto' +import { SecretCodec } from '~/utils/secret-codec' + +const entry = fileURLToPath( + new URL('../../../out/upgrade.mjs', import.meta.url), +) + +describe.skipIf(!existsSync(entry))( + 'historical archive → private PG → built upgrade CLI', + () => { + let source: Awaited> + let target: Awaited> + beforeAll(async () => { + source = await createPgTestDatabase('mx_archive_source') + target = await createPgTestDatabase('mx_archive_target') + }) + afterAll(async () => { + await target?.close() + await source?.close() + }) + + it('restores ZIP/v1 contents using original keys, then prepares through the shipped entry', async (test) => { + const url = new URL(source.connectionString) + const container = execFileSync( + 'docker', + ['ps', '--filter', `publish=${url.port}`, '--format', '{{.ID}}'], + { encoding: 'utf8' }, + ).trim() + if (!container || container.includes('\n')) { + test.skip() + return + } + const key = 'ab'.repeat(32) + const codec = new SecretCodec({ key, enable: true, v2: false }) + await source.pool.query( + "INSERT INTO options (id, name, value) VALUES ('100', 'membership', $1)", + [ + { + dodoApiKey: codec.encrypt( + 'retained-billing-token', + 'membership.dodoApiKey', + ), + }, + ], + ) + const pgEnv = (database: string) => ({ + ...process.env, + PGUSER: decodeURIComponent(url.username), + PGPASSWORD: decodeURIComponent(url.password), + PGDATABASE: database, + }) + const dump = execFileSync( + 'docker', + [ + 'exec', + '-e', + 'PGUSER', + '-e', + 'PGPASSWORD', + '-e', + 'PGDATABASE', + container, + 'pg_dump', + '-Fc', + '--table=public.options', + ], + { env: pgEnv(url.pathname.slice(1)), maxBuffer: 10 * 1024 * 1024 }, + ) + const zip = await new JSZip() + .file('mx-space/pg.dump', dump) + .file('backup_data/proof.txt', 'retained-file') + .generateAsync({ type: 'nodebuffer' }) + expect(decryptBackup(zip, undefined)).toEqual(zip) + const recoveryKey = randomBytes(32) + const nonce = randomBytes(12) + const header = Buffer.from('MXBACKUP\x01') + const cipher = createCipheriv('aes-256-gcm', recoveryKey, nonce) + cipher.setAAD(header) + const payload = Buffer.concat([cipher.update(zip), cipher.final()]) + const archive = Buffer.concat([ + header, + nonce, + cipher.getAuthTag(), + payload, + ]) + expect(() => decryptBackup(archive, 'cd'.repeat(32))).toThrow() + const restored = await JSZip.loadAsync( + decryptBackup(archive, recoveryKey.toString('hex')), + ) + expect( + await restored.file('backup_data/proof.txt')!.async('string'), + ).toBe('retained-file') + execFileSync( + 'docker', + [ + 'exec', + '-i', + '-e', + 'PGUSER', + '-e', + 'PGPASSWORD', + '-e', + 'PGDATABASE', + container, + 'pg_restore', + '--single-transaction', + '--exit-on-error', + '--clean', + '--if-exists', + '--no-owner', + '--no-acl', + '-d', + new URL(target.connectionString).pathname.slice(1), + ], + { + env: pgEnv(new URL(target.connectionString).pathname.slice(1)), + input: await restored.file('mx-space/pg.dump')!.async('nodebuffer'), + }, + ) + const env = { + ...process.env, + PG_URL: target.connectionString, + MX_ENCRYPT_KEY: key, + MX_ENCRYPT_ENABLE: 'true', + MX_SECRET_STORAGE_V2: 'true', + JWT_SECRET: 'private-rehearsal-jwt-123456789012345', + } + const run = (args: string[]) => + JSON.parse( + execFileSync(process.execPath, [entry, ...args], { + env, + encoding: 'utf8', + }), + ) + expect(run(['prepare', '--all-replicas-v2', '--dry-run'])).toMatchObject({ + status: 'would-be-ready', + dryRun: true, + }) + expect( + ( + await target.pool.query( + "SELECT value FROM options WHERE name = 'membership'", + ) + ).rows[0].value, + ).toHaveProperty('dodoApiKey') + expect(run(['prepare', '--all-replicas-v2'])).toMatchObject({ + status: 'prepared', + }) + expect(run(['check', '--json'])).toMatchObject({ status: 'ready' }) + const canonical = ( + await target.pool.query( + "SELECT value FROM options WHERE name = 'membership'", + ) + ).rows[0].value + expect( + new SecretCodec({ key, enable: true, v2: true }).decrypt( + canonical.apiKey, + 'membership.apiKey', + ), + ).toBe('retained-billing-token') + }, 30000) + }, +) diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts index 91eeda47541..eeb0028fa82 100644 --- a/apps/core/test/src/upgrade/cli.spec.ts +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -5,9 +5,24 @@ import { promisify } from 'node:util' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { runUpgradeCli } from '~/upgrade/cli' +import { runUpgradeCli, upgradeConfigFromEnv } from '~/upgrade/cli' describe('upgrade CLI isolated process', () => { + it('matches runtime enable-flag precedence and boolean boundaries', () => { + expect( + upgradeConfigFromEnv({ + MX_ENCRYPT_KEY: 'ab'.repeat(32), + MX_ENCRYPT_ENABLE: '', + ENCRYPT_ENABLE: 'true', + }).enable, + ).toBe(true) + expect( + upgradeConfigFromEnv({ + MX_ENCRYPT_KEY: 'ab'.repeat(32), + MX_ENCRYPT_ENABLE: '1', + }).enable, + ).toBe(false) + }) let context: Awaited> beforeAll(async () => { context = await createPgTestDatabase('mx_upgrade_cli') diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index 9b65b31def6..5eecbfe3371 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -10,7 +10,16 @@ This escape hatch does not bypass encryption checks for existing ciphertext, v2 **v18 will require field encryption and an explicit 64-character hexadecimal encryption key.** Before upgrading to v18, set `ENCRYPT_ENABLE=true` (or `MX_ENCRYPT_ENABLE=true`) and configure `ENCRYPT_KEY` (or `MX_ENCRYPT_KEY`). Keep the original key when one already exists; configure a random key only for installations that have never encrypted data. Follow the rollout below separately to activate v2 writes and backfill existing stored credentials. -## Rollout +## v18 preparation tooling (unreleased) + +The current branch adds a separate readiness/preparation ledger and standalone +CLI. v17.0.4 does not contain these commands. Follow the +[v18 preparation runbook](v18-preparation.md) after deploying a compatible +preparation build. The older `migrate:secrets` ledger is not proof that legacy +config and AI metadata are prepared. Never rotate a short JWT by replacing its +environment value alone: use the maintenance file-based rewrap operation. + +## Original credential rollout This is an expand-before-contract rollout. Two replicas may overlap, so the new reader ships before new writes. diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md new file mode 100644 index 00000000000..60c5e9a1e26 --- /dev/null +++ b/docs/v18-preparation.md @@ -0,0 +1,124 @@ +# v18 preparation (unreleased) + +This is the v17 preparation phase. v18 enforcement, SDK 6 cleanup and column +removal remain future work. These commands are not part of released v17.0.4. +No minimum released preparation tag is established yet. Build and deploy this +preparation code to every replica before using it; do not publish or migrate +production as part of reviewing this branch. + +## Preconditions and checks + +Save database/files snapshots, the original field key, original JWT, and any +archive recovery private key. Never replace existing secrets with newly +generated ones to resolve a decryption error. Older instances complete their +historical schema/data migrations before using this runner. + +The independent CLI consumes environment configuration, not the application's +CLI/YAML flags. Supply the same effective `PG_URL` (preferred), +`PG_CONNECTION_STRING` or `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE`, +optional `PG_SSL`, `MX_ENCRYPT_KEY`/`ENCRYPT_KEY`, enable flag, +`JWT_SECRET`/`JWTSECRET`, and `MX_SECRET_STORAGE_V2`. `.env` is loaded relative +to the command's working directory. It does not start HTTP, business services, +background jobs or Redis. Migrate the schema separately first. + +```sh +pnpm -C apps/core upgrade:check --json +``` + +`check` is read-only: exit 0 means ready, 2 means needs-action, and 1 means +inspection failed. Reports contain counts, record paths and stable codes, +never secret values. Missing/bad encryption settings and short JWT values do +not prevent diagnostics. Empty installations do not need a historical marker. +Existing installations require a new marker plus current data verification; +an old secret-migration ledger row alone is insufficient. + +## Short JWT maintenance + +Stop all application replicas, background jobs and other business writers. +Read the old JWT from the existing secret store into a protected secret file; +generate and save a distinct new JWT of at least 32 characters in another. +The command takes file paths only and never rewrites either file: + +```sh +pnpm -C apps/core upgrade:rotate-jwt --maintenance-confirmed --old-jwt-file /secure/old-jwt --new-jwt-file /secure/new-jwt +``` + +OAuth access/refresh tokens (plaintext, legacy Better Auth bare hex or versioned +envelopes) are unwrapped with the old secret and rewrapped with the new secret +in one transaction. A suspicious encrypted format or wrong old secret fails +without plaintext fallback. The batch rolls back on any failure, including +session invalidation. Provider tokens, field keys and API-key strings remain. +No provider calls occur. Successful rotation clears login sessions: everyone +must sign in again. The operator then applies the same new JWT to every +replica before starting them. Missing original JWT plus encrypted tokens +requires recovering the original secret or reauthorizing the account. + +## Explicit preparation + +After every replica has the compatible reader, enable field encryption and +V2 writes everywhere. Confirm no old writer remains. A short JWT must be +rotated before enabling V2. Use the original field key; only an instance that +has never encrypted data may generate its first random key. + +```sh +pnpm -C apps/core upgrade:prepare --all-replicas-v2 --dry-run +pnpm -C apps/core upgrade:prepare --all-replicas-v2 +pnpm -C apps/core upgrade:check --json +``` + +`--all-replicas-v2` is an operator attestation, not automatic cluster discovery. +Preparation holds a PostgreSQL advisory lock shared by configuration PATCH. +It decrypts secrets at their old AAD paths, converts legacy AI/media, GitHub +and membership options to canonical storage, encrypts at new paths, hashes +API keys once, protects OAuth, and fills missing AI metrics. Existing client +API-key strings continue to verify. It writes the new marker only after no +residuals remain. Malformed data or wrong secrets roll the whole transaction +back. Unknown fields retained by v17 are not made strict in this phase. + +Dry-run executes and rolls back the complete transaction. `would-be-ready` +does not mean a marker was persisted. Repeated successful preparation is +idempotent and does not rehash keys or rotate tokens. After a real commit, +restart business replicas to retire old caches, then check original API-key +authentication, OAuth, configuration consumption and AI metadata. Do not flush +all Redis state. + +## Built artifacts and containers + +Run `node upgrade.mjs check --json`, `node upgrade.mjs prepare` or +`node upgrade.mjs rotate-jwt` with the same arguments in the built output. +Docker copies this entry and its shared chunks with the main application. +A future preparation image can run the tool without starting app services: + +```sh +docker compose run --rm --no-deps app node upgrade.mjs check --json +``` + +The database must already be running. For JWT rotation mount both secret files +read-only and pass their in-container paths. Ensure the app replicas are +stopped; `--no-deps` by itself does not stop an existing replica. Keep app and +migration images pinned to the same verified release, not an unreviewed latest. + +## Compatibility and recovery + +The prepared runtime no longer selects/inserts/updates/returns the retired +AI metadata columns; the migration schema still retains them. v17 response +fields derive from metrics. The columns may be removed only in a later +release after every replica runs this code. Fresh AI generation metadata must +continue to use the metrics ledger. + +Old GET maintenance routes coexist with the new POST routes. Compatible SDK +5.x code uses POST for attitudes, string sorting and header authentication; +public signatures remain. Admin uses POST for backup creation, cache clearing +and test mail. SDK/CLI/theme minimum published versions are not yet verified; +do not advertise an unsupported v18 compatibility matrix. + +Rollback code only to an actually tested preparation build. The minimum +published tag remains unset. Code rollback does not reverse schema/data. +Older versions require their old complete snapshot and secrets. + +The restore service does not currently provide automatic v18 isolation or +preflight. Restore historical ZIP/v1 encrypted archives into an isolated +preparation installation using original archive/field/JWT secrets, run schema +migration and check/prepare, verify credentials and files, and only then plan +a v18 switch. Keep a pre-restore snapshot. Database replacement is atomic, +file copying is separate; failures must not be reported as a complete restore. From d455552d8e76aadc1e36bfac6ac7d82aa7487f2c Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:38:11 +0800 Subject: [PATCH 12/26] fix(core): close preparation review safety and compatibility gaps --- .../ai/ai-insights/ai-insights.repository.ts | 6 +- .../ai/ai-multilang/ai-multilang.service.ts | 10 +- .../ai-translation.repository.ts | 4 +- .../ai-translation/ai-translation.service.ts | 16 +-- .../src/modules/configs/configs.service.ts | 94 ++++++++++--- .../src/modules/configs/legacy-config.util.ts | 76 ++++++++++- .../src/modules/configs/options.repository.ts | 46 +++++-- apps/core/src/shared/config-lock.ts | 2 + apps/core/src/upgrade/oauth-token.ts | 2 +- apps/core/src/upgrade/readiness.ts | 15 +- .../ai/ai-metrics-compatibility.spec.ts | 14 ++ .../modules/ai/ai-multilang.service.spec.ts | 22 +++ .../modules/ai/ai-translation.service.spec.ts | 20 ++- .../review-demo-config-persistence.spec.ts | 11 +- .../configs/configs.transaction.spec.ts | 129 ++++++++++++++++++ .../src/modules/configs/legacy-config.spec.ts | 37 +++++ .../test/src/upgrade/jwt-rotation.spec.ts | 15 ++ .../core/test/src/upgrade/preparation.spec.ts | 25 ++++ docs/v18-preparation.md | 7 + 19 files changed, 495 insertions(+), 56 deletions(-) create mode 100644 apps/core/src/shared/config-lock.ts create mode 100644 apps/core/test/src/modules/configs/configs.transaction.spec.ts diff --git a/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts b/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts index 8ac644162e7..0af42a93023 100644 --- a/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts +++ b/apps/core/src/modules/ai/ai-insights/ai-insights.repository.ts @@ -228,7 +228,7 @@ export class AiInsightsRepository extends BaseRepository { sourceLang: input.sourceLang ?? existing.sourceLang, }) .where(eq(aiInsights.id, existing.id)) - .returning() + .returning(insightsProjection) return mapRow(row) } const id = this.snowflake.nextId() @@ -246,7 +246,7 @@ export class AiInsightsRepository extends BaseRepository { : null, sourceLang: input.sourceLang ?? null, }) - .returning() + .returning(insightsProjection) return mapRow(row) } @@ -275,7 +275,7 @@ export class AiInsightsRepository extends BaseRepository { .update(aiInsights) .set({ content }) .where(eq(aiInsights.id, parseEntityId(id))) - .returning() + .returning(insightsProjection) return row ? mapRow(row) : null } diff --git a/apps/core/src/modules/ai/ai-multilang/ai-multilang.service.ts b/apps/core/src/modules/ai/ai-multilang/ai-multilang.service.ts index 92f5f17839c..7d58b3146d2 100644 --- a/apps/core/src/modules/ai/ai-multilang/ai-multilang.service.ts +++ b/apps/core/src/modules/ai/ai-multilang/ai-multilang.service.ts @@ -107,12 +107,13 @@ export class MultilangGenerationService { model: generated.model, usage: generated.usage, }) - adapter.emitGenerated(doc, { + const hydrated = (await adapter.findById(view.id)) ?? doc + adapter.emitGenerated(hydrated, { refId: options.refId, sourceLang: options.lang, sourceHash: contentMd5, }) - return { result: doc, resultId: view.id } + return { result: hydrated, resultId: view.id } }, parseResult: async (resultId) => { const doc = await adapter.findById(resultId) @@ -187,7 +188,10 @@ export class MultilangGenerationService { }) const totalCost = generated.usage.cost?.total ?? 0 if (options.onCost && totalCost > 0) await options.onCost(totalCost) - return { result: doc, resultId: view.id } + return { + result: (await adapter.findById(view.id)) ?? doc, + resultId: view.id, + } }, parseResult: async (resultId) => { const doc = await adapter.findById(resultId) diff --git a/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts b/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts index a36f36bf63f..805824ea18b 100644 --- a/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts +++ b/apps/core/src/modules/ai/ai-translation/ai-translation.repository.ts @@ -392,7 +392,7 @@ export class AiTranslationRepository extends BaseRepository { ], set: updatableColumns, }) - .returning() + .returning(translationProjection) return mapTranslation(row) }) } @@ -426,7 +426,7 @@ export class AiTranslationRepository extends BaseRepository { .update(aiTranslations) .set(update) .where(eq(aiTranslations.id, parseEntityId(id))) - .returning() + .returning(translationProjection) return row ? mapTranslation(row) : null } diff --git a/apps/core/src/modules/ai/ai-translation/ai-translation.service.ts b/apps/core/src/modules/ai/ai-translation/ai-translation.service.ts index e14070a9fd2..c2d9315f1f2 100644 --- a/apps/core/src/modules/ai/ai-translation/ai-translation.service.ts +++ b/apps/core/src/modules/ai/ai-translation/ai-translation.service.ts @@ -902,25 +902,23 @@ export class AiTranslationService usage, }) + const hydrated = + (await this.aiTranslationRepository.findById(persisted.id)) ?? + persisted + if (existing) { this.logger.log( `AI translation updated: article=${articleId} target=${targetLang}`, ) - this.emitTranslationEvent( - BusinessEvents.TRANSLATION_UPDATE, - persisted, - ) + this.emitTranslationEvent(BusinessEvents.TRANSLATION_UPDATE, hydrated) } else { this.logger.log( `AI translation created: article=${articleId} target=${targetLang}`, ) - this.emitTranslationEvent( - BusinessEvents.TRANSLATION_CREATE, - persisted, - ) + this.emitTranslationEvent(BusinessEvents.TRANSLATION_CREATE, hydrated) } - return { result: persisted, resultId: persisted.id } + return { result: hydrated, resultId: persisted.id } }, parseResult: async (resultId) => { const doc = await this.aiTranslationRepository.findById(resultId) diff --git a/apps/core/src/modules/configs/configs.service.ts b/apps/core/src/modules/configs/configs.service.ts index cae617d2be2..aee03120631 100644 --- a/apps/core/src/modules/configs/configs.service.ts +++ b/apps/core/src/modules/configs/configs.service.ts @@ -33,6 +33,8 @@ import { configDtoMapping, IConfig } from './configs.interface' import type { OAuthConfig } from './configs.schema' import { canonicalConfigRows, + LegacyConfigConflictError, + legacySecretConflicts, normalizeLegacyConfig, normalizeLegacyPatch, updateLegacyMedia, @@ -75,6 +77,9 @@ function normalizeS3StorageOptionNulls(value: T): T { @Injectable() export class ConfigsService implements OnModuleInit { private readonly logger = new Logger(ConfigsService.name) + private stageCache = false + private stagedConfig?: IConfig + private sourceConfig: Record = {} private configInitd = false private configInitPromise?: Promise @@ -99,10 +104,15 @@ export class ConfigsService implements OnModuleInit { } private async setConfig(config: IConfig) { + const encrypted = encryptObject(cloneDeep(config)) + if (this.stageCache) { + this.stagedConfig = encrypted + return + } const redis = await this.getRedisClient() await redis.set( getRedisKey(RedisKeys.ConfigCache), - JSON.stringify(encryptObject(cloneDeep(config))), + JSON.stringify(encrypted), ) } @@ -158,8 +168,11 @@ export class ConfigsService implements OnModuleInit { const rows = await this.optionsRepository.findAll() const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) const plaintext = decryptObject(cloneDeep(raw)) + this.sourceConfig = cloneDeep(plaintext) if (isDev) delete plaintext.url - await this.setConfig(normalizeLegacyConfig(plaintext)) + await this.setConfig( + normalizeLegacyConfig(plaintext, { allowConflicts: true }), + ) } public async get( @@ -172,6 +185,8 @@ export class ConfigsService implements OnModuleInit { // Single entry point for config reads private async getStoredConfig(errorRetryCount = 3): Promise { await this.ensureConfigInitialized() + if (this.stageCache && this.stagedConfig) + return cloneDeep(this.stagedConfig) const redis = await this.getRedisClient() const configCache = await redis.get(getRedisKey(RedisKeys.ConfigCache)) @@ -222,18 +237,32 @@ export class ConfigsService implements OnModuleInit { data: Partial, ): Promise { const config = await this.getConfig() + const conflicts = legacySecretConflicts(key, this.sourceConfig[key] ?? {}) + for (const path of conflicts) { + const replacement = path + .split('.') + .reduce((node, part) => node?.[part], data) + if (typeof replacement !== 'string' || !replacement) + throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { + message: `CONFIG_SECRET_CONFLICT (${key}.${path}); explicitly replace the canonical credential to resolve it`, + }) + } if (key === 'imageGenerationOptions' || key === 'ttsOptions') { const canonical = updateLegacyMedia(config as IConfig, key, data) - await this.optionsRepository.upsert( + const storedAi = await this.optionsRepository.upsert( 'ai', encryptObject(cloneDeep(canonical.ai), 'ai'), ) await this.optionsRepository.deleteByName?.('imageGenerationOptions') await this.optionsRepository.deleteByName?.('ttsOptions') await this.setConfig(canonical) - await this.eventManager.emit(EventBusEvents.ConfigChanged, canonical.ai, { - scope: EventScope.TO_SYSTEM, - }) + await this.eventManager.emit( + EventBusEvents.ConfigChanged, + storedAi.value, + { + scope: EventScope.TO_SYSTEM, + }, + ) return encryptObject(cloneDeep(canonical[key]), key) as IConfig[T] } const updatedConfigRow = await this.optionsRepository.upsert( @@ -261,6 +290,7 @@ export class ConfigsService implements OnModuleInit { ), ) const newData = updatedConfigRow.value as IConfig[T] + this.sourceConfig[key] = decryptObject(cloneDeep(newData), key) const mergedFullConfig = normalizeLegacyConfig( Object.assign( {}, @@ -308,16 +338,38 @@ export class ConfigsService implements OnModuleInit { value: Partial, ) { if (this.optionsRepository.withConfigLock) { - return this.optionsRepository.withConfigLock(async (repository) => { - const scoped = new ConfigsService( - repository, - this.redisService, - this.configVersionService, - this.eventManager, - ) - await scoped.ensureConfigInitialized() - return scoped.patchAndValidUnlocked(key, value) - }) + const effects: Array<() => Promise> = [] + const publication = await this.optionsRepository.withConfigLock( + async (repository) => { + const versions = { + bump: (...args: Parameters) => { + effects.push(() => this.configVersionService.bump(...args)) + return Promise.resolve(0) + }, + } as ConfigVersionService + const events = { + emit: (...args: Parameters) => { + effects.push(() => this.eventManager.emit(...args)) + return Promise.resolve() + }, + } as EventManagerService + const scoped = new ConfigsService( + repository, + this.redisService, + versions, + events, + ) + scoped.stageCache = true + await scoped.ensureConfigInitialized() + const result = await scoped.patchAndValidUnlocked(key, value) + return { result, config: scoped.stagedConfig! } + }, + async ({ config }) => { + await this.setConfig(config) + for (const effect of effects) await effect() + }, + ) + return publication.result } return this.patchAndValidUnlocked(key, value) } @@ -326,7 +378,15 @@ export class ConfigsService implements OnModuleInit { key: T, value: Partial, ) { - value = normalizeLegacyPatch(key, camelcaseKeys(value)) as any + try { + value = normalizeLegacyPatch(key, camelcaseKeys(value)) as any + } catch (error) { + if (error instanceof LegacyConfigConflictError) + throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { + message: error.message, + }) + throw error + } value = removeEmptyEncryptedFields(value as object, key) as Partial< IConfig[T] > diff --git a/apps/core/src/modules/configs/legacy-config.util.ts b/apps/core/src/modules/configs/legacy-config.util.ts index c54b72cc119..1e399bdc338 100644 --- a/apps/core/src/modules/configs/legacy-config.util.ts +++ b/apps/core/src/modules/configs/legacy-config.util.ts @@ -5,6 +5,46 @@ import { AIProviderType } from '~/modules/ai/ai.types' import { generateDefaultConfig } from './configs.default' import type { IConfig } from './configs.interface' +export class LegacyConfigConflictError extends Error { + readonly code = 'CONFIG_SECRET_CONFLICT' + constructor(public readonly path: string) { + super(`CONFIG_SECRET_CONFLICT (${path})`) + } +} + +export function legacySecretConflicts( + key: string, + raw: Record, +): string[] { + const pairs = + key === 'membership' + ? [ + ['apiKey', 'dodoApiKey'], + ['webhookSigningKey', 'dodoWebhookKey'], + ] + : key === 'thirdPartyServiceIntegration' + ? [['github.token', 'githubToken']] + : [] + return pairs + .filter(([current, old]) => { + const value = current + .split('.') + .reduce((node, part) => node?.[part], raw) + return ( + typeof value === 'string' && + value !== '' && + typeof raw[old] === 'string' && + raw[old] !== '' && + value !== raw[old] + ) + }) + .map(([current]) => current) +} + +function legacySecret(current: any, previous: any) { + return current === '' || current == null ? (previous ?? '') : current +} + function isGoogleVertexEndpoint(endpoint: string): boolean { try { const url = new URL(endpoint) @@ -180,7 +220,13 @@ export function normalizeThirdPartyConfig( ): IConfig['thirdPartyServiceIntegration'] { // Already new shape if (raw?.github && typeof raw.github === 'object') { - return raw as IConfig['thirdPartyServiceIntegration'] + return { + ...raw, + github: { + ...raw.github, + token: legacySecret(raw.github.token, raw.githubToken), + }, + } as IConfig['thirdPartyServiceIntegration'] } // Old flat shape — convert return { @@ -202,14 +248,26 @@ export function normalizeMembershipConfig( return { ...providerConfig, - apiKey: providerConfig.apiKey ?? dodoApiKey ?? '', - webhookSigningKey: providerConfig.webhookSigningKey ?? dodoWebhookKey ?? '', + apiKey: legacySecret(providerConfig.apiKey, dodoApiKey), + webhookSigningKey: legacySecret( + providerConfig.webhookSigningKey, + dodoWebhookKey, + ), environment: providerConfig.environment ?? dodoEnvironment ?? 'live_mode', } as IConfig['membership'] } /** Input must be decrypted at its original paths before calling this converter. */ -export function normalizeLegacyConfig(raw: Record): IConfig { +export function normalizeLegacyConfig( + raw: Record, + options: { allowConflicts?: boolean } = {}, +): IConfig { + if (!options.allowConflicts) { + for (const key of ['membership', 'thirdPartyServiceIntegration']) { + const conflict = legacySecretConflicts(key, raw[key] ?? {})[0] + if (conflict) throw new LegacyConfigConflictError(`${key}.${conflict}`) + } + } const config = generateDefaultConfig() for (const key of Object.keys(config) as Array) { if (raw[key] && typeof raw[key] === 'object' && !Array.isArray(raw[key])) { @@ -281,8 +339,13 @@ export function canonicalConfigRows(config: IConfig): Record { export function normalizeLegacyPatch(key: keyof IConfig, input: object) { const value = cloneDeep(input) as Record + const conflict = legacySecretConflicts(key, value)[0] + if (conflict) throw new LegacyConfigConflictError(`${key}.${conflict}`) if (key === 'thirdPartyServiceIntegration' && 'githubToken' in value) { - value.github = { token: value.githubToken, ...value.github } + value.github = { + ...value.github, + token: legacySecret(value.github?.token, value.githubToken), + } delete value.githubToken } if (key === 'membership') { @@ -291,7 +354,8 @@ export function normalizeLegacyPatch(key: keyof IConfig, input: object) { ['dodoWebhookKey', 'webhookSigningKey'], ['dodoEnvironment', 'environment'], ]) { - if (old in value && !(current in value)) value[current] = value[old] + if (old in value) + value[current] = legacySecret(value[current], value[old]) delete value[old] } } diff --git a/apps/core/src/modules/configs/options.repository.ts b/apps/core/src/modules/configs/options.repository.ts index 34de59ac8a6..2eba5c0416b 100644 --- a/apps/core/src/modules/configs/options.repository.ts +++ b/apps/core/src/modules/configs/options.repository.ts @@ -1,5 +1,7 @@ import { Inject, Injectable } from '@nestjs/common' import { eq, sql } from 'drizzle-orm' +import { drizzle } from 'drizzle-orm/node-postgres' +import type { Pool } from 'pg' import { PG_DB_TOKEN } from '~/constants/system.constant' import { options } from '~/database/schema' @@ -8,6 +10,7 @@ import { toEntityId, } from '~/processors/database/base.repository' import type { AppDatabase } from '~/processors/database/postgres.provider' +import { CONFIG_UPGRADE_LOCK } from '~/shared/config-lock' import { type EntityId } from '~/shared/id/entity-id' import { SnowflakeService } from '~/shared/id/snowflake.service' @@ -30,15 +33,42 @@ export class OptionsRepository extends BaseRepository { async withConfigLock( work: (repository: OptionsRepository) => Promise, + afterCommit?: (value: T) => Promise, ): Promise { - return this.db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(5183248167463294042::bigint)`, - ) - return work( - new OptionsRepository(tx as unknown as AppDatabase, this.snowflake), - ) - }) + const client = await ( + this.db as AppDatabase & { $client: Pool } + ).$client.connect() + let committed = false + let locked = false + let destroy = false + try { + await client.query('BEGIN') + await client.query("SET LOCAL lock_timeout = '10s'") + await client.query('SELECT pg_advisory_lock($1::bigint)', [ + CONFIG_UPGRADE_LOCK, + ]) + locked = true + const scopedDb = drizzle(client, { casing: 'snake_case' }) as AppDatabase + const value = await work(new OptionsRepository(scopedDb, this.snowflake)) + await client.query('COMMIT') + committed = true + await afterCommit?.(value) + return value + } catch (error) { + if (!committed) + await client.query('ROLLBACK').catch(() => { + destroy = true + }) + throw error + } finally { + if (locked) + await client + .query('SELECT pg_advisory_unlock($1::bigint)', [CONFIG_UPGRADE_LOCK]) + .catch(() => { + destroy = true + }) + client.release(destroy) + } } async findAll(): Promise { diff --git a/apps/core/src/shared/config-lock.ts b/apps/core/src/shared/config-lock.ts new file mode 100644 index 00000000000..d0ddc19d6c6 --- /dev/null +++ b/apps/core/src/shared/config-lock.ts @@ -0,0 +1,2 @@ +/** Shared by configuration writes and offline preparation/rotation. */ +export const CONFIG_UPGRADE_LOCK = '5183248167463294042' diff --git a/apps/core/src/upgrade/oauth-token.ts b/apps/core/src/upgrade/oauth-token.ts index 11baaeb6de4..19a0d4d7b34 100644 --- a/apps/core/src/upgrade/oauth-token.ts +++ b/apps/core/src/upgrade/oauth-token.ts @@ -12,7 +12,7 @@ export function isEncryptedOAuth(token: string) { // Better Auth's legacy XChaCha20 payload: 24-byte nonce + 16-byte tag + text. return ( token.startsWith('$ba$') || - (token.length >= 80 && token.length % 2 === 0 && /^[\da-f]+$/i.test(token)) + (token.length >= 80 && /^[\da-f]+$/i.test(token)) ) } diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index 102386b32d9..4f74c3c3993 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -3,14 +3,17 @@ import type { Pool, PoolClient } from 'pg' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' import { configSchemaMapping } from '../modules/configs/configs.schema' import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' -import { normalizeLegacyConfig } from '../modules/configs/legacy-config.util' +import { + LegacyConfigConflictError, + normalizeLegacyConfig, +} from '../modules/configs/legacy-config.util' import { SecretCodec } from '../utils/secret-codec' import { isEncryptedOAuth, unwrapOAuthToken } from './oauth-token' export { oauthKey } from './oauth-token' export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' -export const CONFIG_UPGRADE_LOCK = '5183248167463294042' +export { CONFIG_UPGRADE_LOCK } from '../shared/config-lock' export type UpgradeConfig = { key: string jwtSecret: string @@ -132,8 +135,12 @@ export async function readUpgradeState( ) issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: row.name }) } - } catch { - issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED' }) + } catch (error) { + issues.push( + error instanceof LegacyConfigConflictError + ? { code: error.code, path: error.path } + : { code: 'CONFIG_SHAPE_UNSUPPORTED' }, + ) } } for (const table of ['snippets', 'webhooks']) { diff --git a/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts b/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts index 98424f5f1f7..34e122b5c5c 100644 --- a/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts +++ b/apps/core/test/src/modules/ai/ai-metrics-compatibility.spec.ts @@ -50,6 +50,12 @@ describe('prepared AI repositories without retired columns', () => { aiModel: 'new-model', aiProvider: 'new-provider', }) + expect( + await translation.updateById(created.id, { title: 'Edited' }), + ).toMatchObject({ aiModel: 'new-model', aiProvider: 'new-provider' }) + expect( + await translation.upsert({ ...value, title: 'Edited again' } as any), + ).toMatchObject({ aiModel: 'new-model', aiProvider: 'new-provider' }) expect(await translation.findByRef(refId, 'post', 'ja')).toBeTruthy() expect(await translation.findByRefAndLang(refId, 'ja')).toBeTruthy() expect(await translation.listByRefId(refId)).toHaveLength(1) @@ -77,6 +83,14 @@ describe('prepared AI repositories without retired columns', () => { content: 'Updated', modelInfo: { model: 'new-model', provider: 'new-provider' }, }) + expect(await insights.updateContent(created.id, 'Edited')).toMatchObject({ + modelInfo: { model: 'new-model', provider: 'new-provider' }, + }) + expect( + await insights.upsert({ ...value, content: 'Edited again' }), + ).toMatchObject({ + modelInfo: { model: 'new-model', provider: 'new-provider' }, + }) expect(await insights.findByRefAndLang(refId, 'ja')).toBeTruthy() expect(await insights.findSourceForRef(refId, 'ja')).toBeTruthy() expect(await insights.listForRef(refId)).toHaveLength(1) diff --git a/apps/core/test/src/modules/ai/ai-multilang.service.spec.ts b/apps/core/test/src/modules/ai/ai-multilang.service.spec.ts index 69b6b60d8d0..9725b2d6e72 100644 --- a/apps/core/test/src/modules/ai/ai-multilang.service.spec.ts +++ b/apps/core/test/src/modules/ai/ai-multilang.service.spec.ts @@ -10,6 +10,7 @@ interface FakeDoc { content: string isTranslation: boolean sourceLang: string | null + modelInfo?: { model: string; provider: string } } const createHarness = (overrides?: { @@ -101,6 +102,7 @@ const createHarness = (overrides?: { return { adapter, + docs, aiInFlightService, configService, context, @@ -111,6 +113,26 @@ const createHarness = (overrides?: { } describe('MultilangGenerationService.executeMultilangTask', () => { + it('hydrates insights after recording current generation metrics before emitting', async () => { + const { adapter, context, docs, generationMetrics, service } = + createHarness({ adapter: { feature: 'insights' } }) + generationMetrics.record.mockImplementation(async (input: any) => { + const doc = docs.get(input.resourceId)! + docs.set(doc.id, { + ...doc, + modelInfo: { model: input.model, provider: input.providerId }, + }) + }) + await service.executeMultilangTask( + adapter, + { refId: 'post-1', targetLanguages: [] }, + context as any, + ) + expect(adapter.emitGenerated).toHaveBeenCalledWith( + expect.objectContaining({ modelInfo: { model: 'm', provider: 'p' } }), + expect.anything(), + ) + }) it('generates the base in the source language, then translates the remaining targets', async () => { const { adapter, context, service } = createHarness() diff --git a/apps/core/test/src/modules/ai/ai-translation.service.spec.ts b/apps/core/test/src/modules/ai/ai-translation.service.spec.ts index 50938ac205f..5f19f48213b 100644 --- a/apps/core/test/src/modules/ai/ai-translation.service.spec.ts +++ b/apps/core/test/src/modules/ai/ai-translation.service.spec.ts @@ -3,6 +3,7 @@ import { describe, expect, it, vi } from 'vitest' import { createAiGenerationMetricsMock } from '@/helper/ai-generation-metrics-mock' import { createPgRepositoryMock, now } from '@/helper/pg-repository-mock' import { AppException } from '~/common/errors/exception.types' +import { BusinessEvents } from '~/constants/business-event.constant' import { CollectionRefTypes } from '~/constants/db.constant' import type { AiService } from '~/modules/ai/ai.service' import type { AiInFlightService } from '~/modules/ai/ai-inflight/ai-inflight.service' @@ -114,6 +115,7 @@ const createService = () => { aiTaskService, configService, databaseService, + eventManager, generationMetrics, lexicalService, lexicalStrategy, @@ -639,6 +641,7 @@ describe('AiTranslationService — force regeneration', () => { aiService, configService, databaseService, + eventManager, generationMetrics, lexicalStrategy, repository, @@ -654,6 +657,9 @@ describe('AiTranslationService — force regeneration', () => { }) repository.findByRef.mockResolvedValue(null) repository.upsert.mockResolvedValue(row()) + repository.findById.mockResolvedValue( + row({ aiModel: 'model-x', aiProvider: 'provider-x' }), + ) let writerUsage: any let reviewerUsage: any aiService.getTranslationModelWithInfo.mockImplementation( @@ -692,7 +698,19 @@ describe('AiTranslationService — force regeneration', () => { }, ) stubLeaderRun(aiInFlightService) - await service.generateTranslation('post-1', 'en') + const result = await service.generateTranslation('post-1', 'en') + expect(result).toMatchObject({ + aiModel: 'model-x', + aiProvider: 'provider-x', + }) + expect(eventManager.emit).toHaveBeenCalledWith( + BusinessEvents.TRANSLATION_CREATE, + expect.objectContaining({ + aiModel: 'model-x', + aiProvider: 'provider-x', + }), + expect.anything(), + ) expect(generationMetrics.record).toHaveBeenCalledWith( expect.objectContaining({ usage: expect.objectContaining({ diff --git a/apps/core/test/src/modules/auth/review-demo-config-persistence.spec.ts b/apps/core/test/src/modules/auth/review-demo-config-persistence.spec.ts index d6a461c6525..6998a2517de 100644 --- a/apps/core/test/src/modules/auth/review-demo-config-persistence.spec.ts +++ b/apps/core/test/src/modules/auth/review-demo-config-persistence.spec.ts @@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest' import { RedisKeys } from '~/constants/cache.constant' import { ConfigsService } from '~/modules/configs/configs.service' +import { EncryptUtil } from '~/utils/encrypt.util' import { getRedisKey } from '~/utils/redis.util' const REVIEW_PASSWORD = 'persistent-review-password' @@ -62,9 +63,15 @@ describe('review demo OAuth secret persistence', () => { secrets: { apple: { reviewDemoPassword: REVIEW_PASSWORD } }, }) const cachedOauth = JSON.parse(cachedConfig!).oauth - expect(cachedOauth.secrets.apple.reviewDemoPassword).toBe( - rawOauth.secrets.apple.reviewDemoPassword, + expect(cachedOauth.secrets.apple.reviewDemoPassword).not.toBe( + REVIEW_PASSWORD, ) + expect( + EncryptUtil.decrypt( + cachedOauth.secrets.apple.reviewDemoPassword, + 'oauth.secrets.apple.reviewDemoPassword', + ), + ).toBe(REVIEW_PASSWORD) } expect(redisClient.set).toHaveBeenCalledWith( diff --git a/apps/core/test/src/modules/configs/configs.transaction.spec.ts b/apps/core/test/src/modules/configs/configs.transaction.spec.ts new file mode 100644 index 00000000000..f46d01de0af --- /dev/null +++ b/apps/core/test/src/modules/configs/configs.transaction.spec.ts @@ -0,0 +1,129 @@ +import { createPgTestDatabase } from 'test/helper/pg-verify-url' +import { + afterAll, + beforeAll, + beforeEach, + describe, + expect, + it, + vi, +} from 'vitest' + +import { ConfigsService } from '~/modules/configs/configs.service' +import { OptionsRepository } from '~/modules/configs/options.repository' +import { SnowflakeService } from '~/shared/id/snowflake.service' + +describe('configuration commit boundary', () => { + let context: Awaited> + let service: ConfigsService + let repository: OptionsRepository + let cache: string + let setCache: ReturnType + const emit = vi.fn() + const bump = vi.fn() + beforeAll(async () => { + context = await createPgTestDatabase('mx_config_commit') + repository = new OptionsRepository(context.db, new SnowflakeService()) + await context.pool + .query(`CREATE FUNCTION reject_config_commit() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.value->>'title' = 'reject-commit' OR NEW.value->>'webUrl' = 'https://reject-commit.example' THEN RAISE EXCEPTION 'test commit rejection'; END IF; RETURN NEW; END $$; + CREATE CONSTRAINT TRIGGER reject_commit AFTER INSERT OR UPDATE ON options DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE FUNCTION reject_config_commit()`) + }) + afterAll(async () => context?.close()) + beforeEach(async () => { + await context.pool.query('TRUNCATE options') + await repository.upsert('seo', { title: 'committed' }) + setCache = vi.fn(async (_key, value) => { + cache = value + }) + emit.mockReset().mockResolvedValue(undefined) + bump.mockReset().mockResolvedValue(1) + service = new ConfigsService( + repository, + { + waitForReady: async () => {}, + getClient: () => ({ get: async () => cache, set: setCache }), + } as any, + { bump } as any, + { emit } as any, + ) + await service.onModuleInit() + setCache.mockClear() + emit.mockClear() + bump.mockClear() + }) + it.each([ + ['seo', { title: 'reject-commit' }], + ['url', { webUrl: 'https://reject-commit.example' }], + ] as const)( + 'does not publish cache/events/versions when %s commit fails', + async (key, value) => { + const beforeCache = cache + const beforeRows = await repository.findAll() + await expect(service.patchAndValid(key, value)).rejects.toThrow() + expect(await repository.findAll()).toEqual(beforeRows) + expect(cache).toBe(beforeCache) + expect(setCache).not.toHaveBeenCalled() + expect(emit).not.toHaveBeenCalled() + expect(bump).not.toHaveBeenCalled() + }, + ) + it('keeps committed data when a subsequent event delivery fails', async () => { + emit.mockRejectedValue(new Error('event delivery failed')) + await expect( + service.patchAndValid('seo', { title: 'after-commit' }), + ).rejects.toThrow('event delivery failed') + expect(await repository.get('seo')).toMatchObject({ title: 'after-commit' }) + expect(JSON.parse(cache).seo.title).toBe('after-commit') + }) + it('keeps conflicting stored secrets readable and requires an explicit replacement to save', async () => { + await repository.upsert('membership', { + apiKey: 'canonical-key', + dodoApiKey: 'legacy-key', + }) + await expect( + service.patchAndValid('membership', { enabled: true }), + ).rejects.toMatchObject({ code: 'CONFIG_VALIDATION_FAILED' }) + expect(await repository.get('membership')).toMatchObject({ + apiKey: 'canonical-key', + dodoApiKey: 'legacy-key', + }) + await service.patchAndValid('membership', { apiKey: 'chosen-key' }) + const stored = await repository.get>('membership') + expect(stored).not.toHaveProperty('dodoApiKey') + expect((await service.get('membership')).apiKey).toBe('chosen-key') + }) + it('orders cache publication before allowing the next configuration writer', async () => { + let entered!: () => void + let release!: () => void + const publishing = new Promise((resolve) => { + entered = resolve + }) + const mayPublish = new Promise((resolve) => { + release = resolve + }) + setCache.mockImplementation(async (_key, value) => { + if (JSON.parse(value).seo.title === 'first') { + entered() + await mayPublish + } + cache = value + }) + const first = service.patchAndValid('seo', { title: 'first' }) + await publishing + let secondDone = false + let second: Promise = Promise.resolve() + try { + expect(await repository.get('seo')).toMatchObject({ title: 'first' }) + second = service.patchAndValid('seo', { title: 'second' }).then(() => { + secondDone = true + }) + await new Promise((resolve) => setTimeout(resolve, 100)) + expect(secondDone).toBe(false) + } finally { + release() + await Promise.allSettled([first, second]) + } + expect(JSON.parse(cache).seo.title).toBe('second') + expect(await repository.get('seo')).toMatchObject({ title: 'second' }) + }) +}) diff --git a/apps/core/test/src/modules/configs/legacy-config.spec.ts b/apps/core/test/src/modules/configs/legacy-config.spec.ts index eb90538d0b0..ff37bd234be 100644 --- a/apps/core/test/src/modules/configs/legacy-config.spec.ts +++ b/apps/core/test/src/modules/configs/legacy-config.spec.ts @@ -8,6 +8,43 @@ import { normalizeLegacyConfig } from '~/modules/configs/legacy-config.util' import { EncryptUtil } from '~/utils/encrypt.util' describe('canonical legacy config conversion', () => { + it('preserves populated legacy secrets when canonical fields are blank', () => { + const value = normalizeLegacyConfig({ + membership: { + apiKey: '', + webhookSigningKey: '', + dodoApiKey: 'billing-key', + dodoWebhookKey: 'webhook-key', + }, + thirdPartyServiceIntegration: { + github: { enabled: true, token: '' }, + githubToken: 'github-key', + }, + }) + expect(value.membership).toMatchObject({ + apiKey: 'billing-key', + webhookSigningKey: 'webhook-key', + }) + expect(value.thirdPartyServiceIntegration.github.token).toBe('github-key') + }) + it('blocks conflicting populated secrets instead of choosing and deleting one', () => { + expect(() => + normalizeLegacyConfig({ + membership: { + apiKey: 'canonical-private', + dodoApiKey: 'legacy-private', + }, + }), + ).toThrow('CONFIG_SECRET_CONFLICT') + expect(() => + normalizeLegacyConfig({ + thirdPartyServiceIntegration: { + github: { token: 'canonical-private' }, + githubToken: 'legacy-private', + }, + }), + ).toThrow('CONFIG_SECRET_CONFLICT') + }) it('moves decrypted legacy credentials and binds ciphertext to the new path', () => { const original = { thirdPartyServiceIntegration: { diff --git a/apps/core/test/src/upgrade/jwt-rotation.spec.ts b/apps/core/test/src/upgrade/jwt-rotation.spec.ts index 17ae449fda5..309fd20ddf3 100644 --- a/apps/core/test/src/upgrade/jwt-rotation.spec.ts +++ b/apps/core/test/src/upgrade/jwt-rotation.spec.ts @@ -134,4 +134,19 @@ describe('maintenance JWT rotation', () => { ).rejects.toThrow('OAUTH_UNREADABLE') expect(await snapshot()).toEqual(before) }) + it('rejects truncated legacy ciphertext and rolls back earlier token changes', async () => { + const bare = await symmetricEncrypt({ + key: oldSecret, + data: 'legacy-provider-token', + }) + await context.pool.query( + "UPDATE accounts SET access_token = $1 WHERE id = 'b'", + [bare.slice(0, -1)], + ) + const before = await snapshot() + await expect(rotateJwtSecret(context.pool, input)).rejects.toThrow( + 'OAUTH_UNREADABLE', + ) + expect(await snapshot()).toEqual(before) + }) }) diff --git a/apps/core/test/src/upgrade/preparation.spec.ts b/apps/core/test/src/upgrade/preparation.spec.ts index f32cac8dc61..d780bcd7f76 100644 --- a/apps/core/test/src/upgrade/preparation.spec.ts +++ b/apps/core/test/src/upgrade/preparation.spec.ts @@ -146,6 +146,31 @@ describe('transactional v18 preparation', () => { ).rejects.toThrow('SECRET_UNREADABLE') expect(await snapshot()).toEqual(before) }) + it('rolls back when mixed config contains conflicting secrets', async () => { + await context.pool.query( + "UPDATE options SET value = $1 WHERE name = 'membership'", + [{ apiKey: 'canonical-private', dodoApiKey: 'legacy-private' }], + ) + const before = await snapshot() + await expect( + prepareUpgrade(context.pool, config, { allReplicasV2: true }), + ).rejects.toThrow('CONFIG_SECRET_CONFLICT') + expect(await snapshot()).toEqual(before) + }) + it('rolls back on truncated legacy OAuth ciphertext rather than wrapping it as plaintext', async () => { + const bare = await symmetricEncrypt({ + key: config.jwtSecret, + data: 'original-provider-token', + }) + await context.pool.query('UPDATE accounts SET access_token = $1', [ + bare.slice(0, -1), + ]) + const before = await snapshot() + await expect( + prepareUpgrade(context.pool, config, { allReplicasV2: true }), + ).rejects.toThrow('OAUTH_UNREADABLE') + expect(await snapshot()).toEqual(before) + }) it('upgrades a legacy Better Auth bare-hex token without double encryption', async () => { const bare = await symmetricEncrypt({ key: config.jwtSecret, diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index 60c5e9a1e26..4a09754fb93 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -82,6 +82,13 @@ restart business replicas to retire old caches, then check original API-key authentication, OAuth, configuration consumption and AI metadata. Do not flush all Redis state. +A blank canonical credential falls back to a populated legacy credential. +Different nonempty old/new values report `CONFIG_SECRET_CONFLICT` with the +field path and stop preparation. Reads stay compatible; to resolve the +conflict, explicitly save the intended canonical credential in settings, +then run check again. An unrelated partial save of that section is rejected +so it cannot silently delete the other secret. + ## Built artifacts and containers Run `node upgrade.mjs check --json`, `node upgrade.mjs prepare` or From c62cf18e89bab0e9b191386a0b4ecca3c01287de Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:42:58 +0800 Subject: [PATCH 13/26] fix(core): require truly empty application data for new-install readiness --- apps/core/src/upgrade/readiness.ts | 29 ++++++++++++++++++-- apps/core/test/src/upgrade/readiness.spec.ts | 10 +++++++ 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index 4f74c3c3993..1c29dfc9384 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -194,9 +194,32 @@ export async function readUpgradeState( PREPARED_MIGRATION_ID, ]) ).rowCount !== 0 - const existing = await database.query(`SELECT EXISTS ( - SELECT 1 FROM options UNION ALL SELECT 1 FROM readers UNION ALL SELECT 1 FROM posts UNION ALL SELECT 1 FROM notes UNION ALL SELECT 1 FROM pages UNION ALL SELECT 1 FROM snippets UNION ALL SELECT 1 FROM webhooks UNION ALL SELECT 1 FROM accounts UNION ALL SELECT 1 FROM api_keys UNION ALL SELECT 1 FROM ai_translations UNION ALL SELECT 1 FROM ai_insights - ) AS present`) + // A new database may contain migration ledgers, but no application data. + const tables = ( + await database.query( + `SELECT tablename FROM pg_catalog.pg_tables + WHERE schemaname = 'public' AND tablename <> ALL($1::text[])`, + [ + [ + '_app_migrations', + '__drizzle_migrations', + 'schema_migrations', + 'data_migration_runs', + 'mongo_id_map', + 'auth_id_map', + ], + ], + ) + ).rows + const probes = tables.map( + ({ tablename }) => + `SELECT 1 FROM public."${String(tablename).replaceAll('"', '""')}"`, + ) + const existing = await database.query( + probes.length + ? `SELECT EXISTS (${probes.join(' UNION ALL ')}) AS present` + : 'SELECT false AS present', + ) const installation = existing.rows[0].present ? 'existing' : 'empty' if ( counts.plaintext || diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts index 0b9b495ed55..248631d1aaa 100644 --- a/apps/core/test/src/upgrade/readiness.spec.ts +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -83,4 +83,14 @@ describe('standalone upgrade readiness', () => { ), ).toContain('CONFIG_SHAPE_UNSUPPORTED') }) + it('does not classify retained business data without an owner as a new empty installation', async () => { + await context.pool.query( + "DELETE FROM options; DELETE FROM _app_migrations; INSERT INTO categories (id, name, slug) VALUES ('400', 'Retained category', 'retained')", + ) + expect(await readUpgradeState(context.pool, config)).toMatchObject({ + installation: 'existing', + status: 'needs-action', + prepared: false, + }) + }) }) From dde22891ed8154ffd11e975de48d2e990d75a92c Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 21:46:15 +0800 Subject: [PATCH 14/26] docs: record verified v18 preparation outcome --- .../2026-11-08-mx-core-v18-compatibility-design.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 6a478f27bac..f2ddb864dd6 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -1,6 +1,6 @@ # mx-core v18:兼容性收口与文档迁移 -状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。当前执行 **v17 准备阶段**,不发布 v18,不操作真实站点数据。 +状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。本次完成 **v17 准备阶段** 的本地实现与验证,尚未发布;不发布 v18,不操作真实站点数据。 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 @@ -162,3 +162,14 @@ flowchart TD 可自行决定内部模块、命令参数布局、事务批次和测试 fixture,但不得改变行为和失败边界。准备版最低版本、迁移 id、消费者最低 tag 以实施后的证据固定。测试验证行为,不对静态常量、源码行或内部对象做快照。 讨论文档:`document/mx-core-v18-compatibility`,路径 `.ryugi/artifacts/document/mx-core-v18-compatibility/doc.xml`,链接 https://ryugi.innei.dev/a/ZTsqm6DOJbAGibGoCjgD1。两轮 13 项选择及处理说明已归档,独立设计审阅结论为 “nothing important left”;用户随后明确同意开始准备版。未制作 UI mockup,保持现有视觉风格。 + +## 准备阶段验证记录(2026-11-08) + +- 准备功能代码提交至 `c62cf18e8`;Core 保留 v17 行为,未发版、未推送、未执行生产迁移。最低公开准备版 tag 与 SDK/CLI/主题版本矩阵仍待发布验证。 +- Core 全量回归显式排除原有 `admin-agent.e2e-spec.ts` 的缺失 DI 夹具后,430 个文件、3569 项测试通过,11 项跳过。该夹具仍未修复,不能声称包含它的原始全量测试通过。 +- SDK 207 项测试、Admin 8 项请求行为测试及各包类型检查通过;Core/Admin 生产构建、API envelope/验证配对检查及 50 个迁移文件的 lint 通过。 +- 真实临时 PostgreSQL 验证了旧列删除后的仓库读写/返回值、原路径解密和新路径重加密、dry-run、幂等、错误密钥、损坏 OAuth、冲突凭据、事务提交失败与并发缓存发布顺序。历史 ZIP/v1 归档经真实 pg_restore 后,由构建产物完成 dry-run、prepare、check 并保留原凭据。全部使用隔离测试数据库。 +- 独立整体验证发现的 2 项 Critical、2 项 Important 已通过先失败后通过的行为测试修复;未再次分派审阅。另补「仅有分类等业务数据时不得判为空库」回归。缓存加密验证比较解密后的原值,不把随机 GCM 密文字节相同当作契约。 +- 官方文档工作区 `/Users/innei/git/innei-repo/mx-docs` 的 `545e5c8`、`0719e79` 已本地提交,build/check/links/pangu/lint 与 3 项配置生成测试通过。保留原有 1 条 Astro hint、1 条既有死链;真实预览浏览器和临时 Compose 容器验证了复制映射、字符串类型与美元符号保留,生成的 llms 内容已核对。 + +裁量保持:先抽纯加密/字段路径模块以支持独立诊断,仅改变实现顺序;v18 的启动强制、严格配置、只读 AI 默认行为、SDK 6 清理和物理列删除仍属后续阶段。当前没有自动隔离恢复入口,继续使用离线恢复到准备版再准备的路径;回退最低公开版本与消费者矩阵仍须发布后实测确定。原有 Agent 测试夹具另行处理。 From c42276ea40726f9f553cf0781342d93e2ea188da Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 23:18:16 +0800 Subject: [PATCH 15/26] fix(security): require encrypted storage and retire legacy writers --- apps/core/src/app.config.test.ts | 2 - apps/core/src/app.config.ts | 51 +++----- .../20261018-protect-stored-secrets.ts | 6 +- apps/core/src/main.ts | 17 +-- apps/core/src/modules/auth/auth.implement.ts | 40 +------ apps/core/src/modules/auth/auth.service.ts | 9 +- .../src/modules/push/push-secret.vault.ts | 12 +- .../src/modules/webhook/webhook.repository.ts | 9 +- apps/core/src/upgrade/cli.ts | 11 +- apps/core/src/upgrade/preparation.ts | 2 +- apps/core/src/upgrade/readiness.ts | 21 ++-- apps/core/src/utils/encrypt.util.ts | 8 +- apps/core/src/utils/secret-codec.ts | 36 +----- apps/core/src/utils/secret-storage-config.ts | 66 +++++++++++ .../test/fixtures/read-security-config.ts | 16 +++ apps/core/test/helper/legacy-secret.ts | 12 ++ apps/core/test/setup-global.ts | 1 - .../src/upgrade/archive-rehearsal.spec.ts | 25 ++-- apps/core/test/src/upgrade/cli.spec.ts | 35 +++--- .../core/test/src/upgrade/preparation.spec.ts | 8 +- apps/core/test/src/upgrade/readiness.spec.ts | 12 +- apps/core/test/src/utils/encrypt.util.spec.ts | 19 +-- .../src/utils/required-secret-config.spec.ts | 77 ++++++++++++ .../secret-storage-compatibility.spec.ts | 110 ++++++------------ ...-11-08-mx-core-v18-compatibility-design.md | 2 + packages/e2e/src/helpers/core-app-config.ts | 3 - 26 files changed, 296 insertions(+), 314 deletions(-) create mode 100644 apps/core/src/utils/secret-storage-config.ts create mode 100644 apps/core/test/fixtures/read-security-config.ts create mode 100644 apps/core/test/helper/legacy-secret.ts create mode 100644 apps/core/test/src/utils/required-secret-config.spec.ts diff --git a/apps/core/src/app.config.test.ts b/apps/core/src/app.config.test.ts index 055ed630481..1177d5467ac 100644 --- a/apps/core/src/app.config.test.ts +++ b/apps/core/src/app.config.test.ts @@ -43,10 +43,8 @@ export const DEBUG_MODE = { httpRequestVerbose: false, } -export const SECRET_STORAGE_V2 = true export const ENCRYPT = { key: '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab', - enable: true, } export const THROTTLE_OPTIONS = { diff --git a/apps/core/src/app.config.ts b/apps/core/src/app.config.ts index cc74071c721..f878600192f 100644 --- a/apps/core/src/app.config.ts +++ b/apps/core/src/app.config.ts @@ -4,24 +4,23 @@ import path from 'node:path' import { seconds } from '@nestjs/throttler' import { Option, program } from 'commander' import { load as yamlLoad } from 'js-yaml' -import nodeMachineId from 'node-machine-id' import { isDebugMode, isDev } from './global/env.global' import { parseRedisConnectionString, resolveRedisConnectionStringEnv, } from './utils/redis-config.util' +import { + assertSecretStorageConfig, + secretStorageConfigFromEnv, +} from './utils/secret-storage-config' import { parseBooleanishValue } from './utils/tool.util' -const { machineIdSync } = nodeMachineId - const { PORT: ENV_PORT, ALLOWED_ORIGINS, MX_ENCRYPT_KEY, - MX_ENCRYPT_ENABLE, ENCRYPT_KEY: ENV_ENCRYPT_KEY, - ENCRYPT_ENABLE: ENV_ENCRYPT_ENABLE, CDN_CACHE_HEADER, FORCE_CACHE_HEADER, THROTTLE_TTL, @@ -33,7 +32,6 @@ const { const ENV_JWT_SECRET = JWT_SECRET || JWTSECRET const ENCRYPT_KEY_FROM_ENV = MX_ENCRYPT_KEY || ENV_ENCRYPT_KEY -const ENCRYPT_ENABLE_FROM_ENV = MX_ENCRYPT_ENABLE || ENV_ENCRYPT_ENABLE const REDIS_CONNECTION_STRING_FROM_ENV = resolveRedisConnectionStringEnv() function applyArgvEnvFallback(argv: Record) { @@ -127,13 +125,10 @@ const commander = program '64-character hexadecimal encryption key', ).default(ENCRYPT_KEY_FROM_ENV, '[configured]'), ) - .option( - '--encrypt_enable', - 'enable encrypt security field, please remember encrypt key.', - ) + .option('--encrypt_enable', 'deprecated no-op; encryption is always required') .option( '--encrypt_algorithm ', - 'custom encrypt algorithm, default is aes-256-ecb', + 'historical ciphertext read algorithm, default is aes-256-ecb', ) // throttle .option('--throttle_ttl ', 'throttle ttl') @@ -276,33 +271,19 @@ export const THROTTLE_OPTIONS = { limit: Number(argv.throttle_limit ?? THROTTLE_LIMIT ?? 100), } -const ENCRYPT_KEY = argv.encrypt_key -// shortcut: enable v2 writes only after all replicas have the compatible reader; remove after the compatibility release. -export const SECRET_STORAGE_V2 = - process.env.MX_SECRET_STORAGE_V2 === '1' || - (parseBooleanishValue(process.env.MX_SECRET_STORAGE_V2) ?? false) - +const secretStorageConfig = secretStorageConfigFromEnv(process.env) export const ENCRYPT = { - key: ENCRYPT_KEY || machineIdSync(), - enable: - parseBooleanishValue(argv.encrypt_enable ?? ENCRYPT_ENABLE_FROM_ENV) ?? - !!ENCRYPT_KEY, + key: argv.encrypt_key || '', algorithm: argv.encrypt_algorithm || 'aes-256-ecb', } - -if (ENCRYPT.enable && (!ENCRYPT_KEY || !/^[\da-f]{64}$/i.test(ENCRYPT_KEY))) - throw new Error( - 'Encryption requires an explicit 64-character hexadecimal MX_ENCRYPT_KEY or ENCRYPT_KEY', - ) - -if ( - SECRET_STORAGE_V2 && - (!ENCRYPT.enable || !SECURITY.jwtSecret || SECURITY.jwtSecret.length < 32) -) { - throw new Error( - 'MX_SECRET_STORAGE_V2 requires encryption and a JWT_SECRET of at least 32 characters', - ) -} +assertSecretStorageConfig({ + ...ENCRYPT, + jwtSecret: SECURITY.jwtSecret || '', + deprecatedFlags: { + ...secretStorageConfig.deprecatedFlags, + encrypt_enable: argv.encrypt_enable, + }, +}) export const TELEMETRY = { enable: !parseBooleanishValue(argv.disable_telemetry ?? MX_DISABLE_TELEMETRY), diff --git a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts index 16f38a503c0..b0009d66b95 100644 --- a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts +++ b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts @@ -6,7 +6,7 @@ import { } from 'better-auth/crypto' import { eq } from 'drizzle-orm' -import { SECRET_STORAGE_V2, SECURITY } from '~/app.config' +import { SECURITY } from '~/app.config' import { accounts, apiKeys, @@ -26,9 +26,9 @@ export const migration: AppMigration = { 'Protect existing credentials after every replica enables v2 writes', async up({ db, logger }) { EncryptUtil.assertConfigured() - if (!SECRET_STORAGE_V2 || !SECURITY.jwtSecret) { + if (!SECURITY.jwtSecret || SECURITY.jwtSecret.length < 32) { throw new Error( - 'Secret migration requires MX_SECRET_STORAGE_V2=1 and JWT_SECRET', + 'Secret migration requires JWT_SECRET of at least 32 characters', ) } const oauthKey = { diff --git a/apps/core/src/main.ts b/apps/core/src/main.ts index d34c516bb1f..904097652a7 100644 --- a/apps/core/src/main.ts +++ b/apps/core/src/main.ts @@ -21,7 +21,7 @@ export async function startMain() { initializeApp() const importStartedAt = performance.now() - const [{ bootstrap }, { CLUSTER, ENCRYPT }, { Cluster }] = await Promise.all([ + const [{ bootstrap }, { CLUSTER }, { Cluster }] = await Promise.all([ import('./bootstrap'), import('./app.config.js'), import('./cluster'), @@ -34,21 +34,6 @@ export async function startMain() { logger.log('cwd: ', cwd) } - if (ENCRYPT.enable && ENCRYPT.key) { - const isValidKey = ENCRYPT.key.length === 64 - - if (!isValidKey) { - logger.error('encrypt key must be 64 length') - process.exit(1) - } - - logger.log('Secret encryption is enabled') - } else { - logger.warn( - 'Secret encryption is disabled (legacy compatibility mode). v18 will require ENCRYPT_ENABLE=true and a 64-character hexadecimal ENCRYPT_KEY; keep your existing key when upgrading.', - ) - } - DEBUG_MODE.memoryDump && registerForMemoryDump() if (CLUSTER.enable) { Cluster.register( diff --git a/apps/core/src/modules/auth/auth.implement.ts b/apps/core/src/modules/auth/auth.implement.ts index b845d24e091..9ab939a75e7 100644 --- a/apps/core/src/modules/auth/auth.implement.ts +++ b/apps/core/src/modules/auth/auth.implement.ts @@ -12,11 +12,7 @@ import type { BetterAuthOptions } from 'better-auth' import { betterAuth } from 'better-auth' import { drizzleAdapter } from 'better-auth/adapters/drizzle' import { APIError, createAuthMiddleware } from 'better-auth/api' -import { - hashPassword, - symmetricDecrypt, - verifyPassword, -} from 'better-auth/crypto' +import { hashPassword, verifyPassword } from 'better-auth/crypto' import { toNodeHandler } from 'better-auth/node' import { bearer, @@ -28,12 +24,7 @@ import { and, eq } from 'drizzle-orm' import { customAlphabet } from 'nanoid' import wcmatch from 'wildcard-match' -import { - API_VERSION, - CROSS_DOMAIN, - SECRET_STORAGE_V2, - SECURITY, -} from '~/app.config' +import { API_VERSION, CROSS_DOMAIN, SECURITY } from '~/app.config' import { db } from '~/processors/database/postgres.provider' import { APPLE_ORIGIN } from './apple-client-secret' @@ -63,32 +54,12 @@ const deviceUserCodeAlphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789' const generateDeviceUserCode = customAlphabet(deviceUserCodeAlphabet, 8) const reviewDemoMutableProfileFields = new Set(['image', 'name']) -// Compatibility release: all replicas can read encrypted OAuth tokens before v2 writes are enabled. -async function preserveLegacyOAuthWrites( - account: Record, - context: any, -) { - if (SECRET_STORAGE_V2) return - const data = { ...account } - for (const field of ['accessToken', 'refreshToken']) { - const token = data[field] - if (typeof token === 'string' && token) { - data[field] = await symmetricDecrypt({ - key: context.context.secretConfig, - data: token, - }) - } - } - return { data } -} - export const reviewDemoDatabaseHooks: NonNullable< BetterAuthOptions['databaseHooks'] > = { account: { - create: { before: preserveLegacyOAuthWrites }, update: { - before: async (account, context) => { + before: async (_account, context) => { if ( context?.path === '/change-password' && isReviewDemoEmail(context.context.session?.user ?? {}) @@ -97,7 +68,6 @@ export const reviewDemoDatabaseHooks: NonNullable< message: 'app review demo credentials cannot be changed', }) } - return preserveLegacyOAuthWrites(account, context) }, }, }, @@ -150,9 +120,7 @@ export const createCredentialSignInHook = ( ) => createAuthMiddleware(async (ctx) => { if (ctx.path === '/api-key/create') { - ctx.body.configId = SECRET_STORAGE_V2 - ? HASHED_API_KEY_CONFIG_ID - : 'default' + ctx.body.configId = HASHED_API_KEY_CONFIG_ID } if (ctx.body?.role !== undefined) { throw new APIError('FORBIDDEN', { diff --git a/apps/core/src/modules/auth/auth.service.ts b/apps/core/src/modules/auth/auth.service.ts index 4d070f79216..ceb8d55892c 100644 --- a/apps/core/src/modules/auth/auth.service.ts +++ b/apps/core/src/modules/auth/auth.service.ts @@ -11,7 +11,6 @@ import { hashPassword } from 'better-auth/crypto' import { omit } from 'es-toolkit/compat' import { customAlphabet } from 'nanoid' -import { SECRET_STORAGE_V2 } from '~/app.config' import { RequestContext } from '~/common/contexts/request.context' import { AppErrorCode, createAppException } from '~/common/errors' import { EventBusEvents } from '~/constants/event-bus.constant' @@ -174,7 +173,7 @@ export class AuthService { body: { name: body.name, userId: ownerId, - configId: SECRET_STORAGE_V2 ? HASHED_API_KEY_CONFIG_ID : 'default', + configId: HASHED_API_KEY_CONFIG_ID, // Owner automation keys must not inherit the provider's 10-request daily quota. rateLimitEnabled: false, ...(expiresIn ? { expiresIn } : {}), @@ -201,10 +200,8 @@ export class AuthService { name: model.name, start, prefix, - key: SECRET_STORAGE_V2 - ? await defaultKeyHasher(model.token) - : model.token, - configId: SECRET_STORAGE_V2 ? HASHED_API_KEY_CONFIG_ID : 'default', + key: await defaultKeyHasher(model.token), + configId: HASHED_API_KEY_CONFIG_ID, userId: ownerId, referenceId: ownerId, enabled: true, diff --git a/apps/core/src/modules/push/push-secret.vault.ts b/apps/core/src/modules/push/push-secret.vault.ts index 9b08868cc85..e50d1fc14c9 100644 --- a/apps/core/src/modules/push/push-secret.vault.ts +++ b/apps/core/src/modules/push/push-secret.vault.ts @@ -1,17 +1,13 @@ import crypto from 'node:crypto' import { ENCRYPT } from '~/app.config' -import { mapString } from '~/utils/encrypt.util' +import { EncryptUtil } from '~/utils/encrypt.util' const PREFIX = '$${push}$$v1.' export class PushSecretVault { static assertConfigured() { - if (!ENCRYPT.enable || !ENCRYPT.key) { - throw new Error( - 'Push notifications require MX_ENCRYPT_KEY so source credentials are never stored in plaintext', - ) - } + EncryptUtil.assertConfigured() } static encrypt(value: string) { @@ -19,7 +15,7 @@ export class PushSecretVault { const nonce = crypto.randomBytes(12) const cipher = crypto.createCipheriv( 'aes-256-gcm', - Buffer.from(mapString(ENCRYPT.key), 'hex'), + Buffer.from(ENCRYPT.key, 'hex'), nonce, ) const ciphertext = Buffer.concat([ @@ -46,7 +42,7 @@ export class PushSecretVault { } const decipher = crypto.createDecipheriv( 'aes-256-gcm', - Buffer.from(mapString(ENCRYPT.key), 'hex'), + Buffer.from(ENCRYPT.key, 'hex'), Buffer.from(nonceValue, 'base64url'), ) decipher.setAuthTag(Buffer.from(tagValue, 'base64url')) diff --git a/apps/core/src/modules/webhook/webhook.repository.ts b/apps/core/src/modules/webhook/webhook.repository.ts index b8df7b56430..be263a899be 100644 --- a/apps/core/src/modules/webhook/webhook.repository.ts +++ b/apps/core/src/modules/webhook/webhook.repository.ts @@ -1,7 +1,6 @@ import { Inject, Injectable } from '@nestjs/common' import { desc, eq, sql } from 'drizzle-orm' -import { SECRET_STORAGE_V2 } from '~/app.config' import { PG_DB_TOKEN } from '~/constants/system.constant' import { webhookEvents, webhooks } from '~/database/schema' import { @@ -95,9 +94,7 @@ export class WebhookRepository extends BaseRepository { id, payloadUrl: input.payloadUrl, events: input.events, - secret: SECRET_STORAGE_V2 - ? EncryptUtil.encrypt(input.secret, `webhooks.${id}.secret`) - : input.secret, + secret: EncryptUtil.encrypt(input.secret, `webhooks.${id}.secret`), enabled: input.enabled ?? true, scope: input.scope ?? null, timestamp: new Date(), @@ -121,9 +118,7 @@ export class WebhookRepository extends BaseRepository { if (patch.payloadUrl !== undefined) update.payloadUrl = patch.payloadUrl if (patch.events !== undefined) update.events = patch.events if (patch.secret !== undefined) - update.secret = SECRET_STORAGE_V2 - ? EncryptUtil.encrypt(patch.secret, `webhooks.${id}.secret`) - : patch.secret + update.secret = EncryptUtil.encrypt(patch.secret, `webhooks.${id}.secret`) if (patch.enabled !== undefined) update.enabled = patch.enabled if (patch.scope !== undefined) update.scope = patch.scope const [row] = await this.db diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts index 7e107842dab..3a7fdc1beee 100644 --- a/apps/core/src/upgrade/cli.ts +++ b/apps/core/src/upgrade/cli.ts @@ -2,20 +2,13 @@ import { parseArgs } from 'node:util' import { Pool } from 'pg' +import { secretStorageConfigFromEnv } from '../utils/secret-storage-config' import { rotateJwtSecret } from './jwt-rotation' import { prepareUpgrade, UpgradeError } from './preparation' import { readUpgradeState, type UpgradeConfig } from './readiness' export function upgradeConfigFromEnv(env: NodeJS.ProcessEnv): UpgradeConfig { - const key = env.MX_ENCRYPT_KEY || env.ENCRYPT_KEY || '' - const enabled = env.MX_ENCRYPT_ENABLE || env.ENCRYPT_ENABLE - return { - key, - enable: enabled === undefined ? !!key : enabled === 'true', - v2: env.MX_SECRET_STORAGE_V2 === '1' || env.MX_SECRET_STORAGE_V2 === 'true', - jwtSecret: env.JWT_SECRET || env.JWTSECRET || '', - algorithm: env.ENCRYPT_ALGORITHM || 'aes-256-ecb', - } + return secretStorageConfigFromEnv(env) } export async function runUpgradeCli( diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts index 4081d5c28fe..c059364e2ea 100644 --- a/apps/core/src/upgrade/preparation.ts +++ b/apps/core/src/upgrade/preparation.ts @@ -39,7 +39,7 @@ export async function prepareUpgrade( if (!options.allReplicasV2) throw new UpgradeError('REPLICA_CONFIRMATION_REQUIRED') const client = await pool.connect() - const codec = new SecretCodec({ ...config, v2: true }) + const codec = new SecretCodec(config) const ids = new SnowflakeGenerator({ workerId: 1023 }) let changed = 0 try { diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index 1c29dfc9384..e06f5fb158e 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -8,19 +8,17 @@ import { normalizeLegacyConfig, } from '../modules/configs/legacy-config.util' import { SecretCodec } from '../utils/secret-codec' +import { + inspectSecretStorageConfig, + type SecretStorageConfig, +} from '../utils/secret-storage-config' import { isEncryptedOAuth, unwrapOAuthToken } from './oauth-token' export { oauthKey } from './oauth-token' export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' export { CONFIG_UPGRADE_LOCK } from '../shared/config-lock' -export type UpgradeConfig = { - key: string - jwtSecret: string - enable: boolean - v2: boolean - algorithm?: string -} +export type UpgradeConfig = SecretStorageConfig export type UpgradeIssue = { code: string; path?: string } export type UpgradeReport = { target: 18 @@ -65,13 +63,8 @@ export async function readUpgradeState( database: UpgradeDatabase, config: UpgradeConfig, ): Promise { - const issues: UpgradeIssue[] = [] + const issues: UpgradeIssue[] = inspectSecretStorageConfig(config) const keyValid = /^[\da-f]{64}$/i.test(config.key) - if (!config.key) issues.push({ code: 'FIELD_KEY_MISSING' }) - else if (!keyValid) issues.push({ code: 'FIELD_KEY_INVALID' }) - if (!config.enable) issues.push({ code: 'ENCRYPTION_DISABLED' }) - if (config.jwtSecret.length < 32) issues.push({ code: 'JWT_TOO_SHORT' }) - if (!config.v2) issues.push({ code: 'V2_WRITES_DISABLED' }) const counts: UpgradeReport['counts'] = { plaintext: 0, ecb: 0, @@ -83,7 +76,7 @@ export async function readUpgradeState( encryptedOAuth: 0, metricsMissing: 0, } - const codec = new SecretCodec({ ...config, enable: true, v2: true }) + const codec = new SecretCodec(config) const inspect = (secret: string, path: string) => { if (!secret) return secret if (secret.startsWith('$${mx}$$v2.')) counts.v2++ diff --git a/apps/core/src/utils/encrypt.util.ts b/apps/core/src/utils/encrypt.util.ts index f5f0fac8491..de5b464d060 100644 --- a/apps/core/src/utils/encrypt.util.ts +++ b/apps/core/src/utils/encrypt.util.ts @@ -1,13 +1,11 @@ -import { ENCRYPT, SECRET_STORAGE_V2 } from '~/app.config' +import { ENCRYPT } from '~/app.config' import { SecretCodec } from './secret-codec' -export { mapString } from './secret-codec' - -/** v17 facade keeps runtime configuration and the plaintext escape hatch. */ +/** Runtime facade: authenticated V2 writes, historical plaintext/ECB reads. */ export class EncryptUtil { private static codec() { - return new SecretCodec({ ...ENCRYPT, v2: SECRET_STORAGE_V2 }) + return new SecretCodec(ENCRYPT) } static assertConfigured() { return this.codec().assertConfigured() diff --git a/apps/core/src/utils/secret-codec.ts b/apps/core/src/utils/secret-codec.ts index a6d74c461e5..482c8466d0d 100644 --- a/apps/core/src/utils/secret-codec.ts +++ b/apps/core/src/utils/secret-codec.ts @@ -3,17 +3,8 @@ import crypto from 'node:crypto' const PREFIX = '$${mx}$$' const V2_PREFIX = `${PREFIX}v2.` -export const mapString = (key: string) => { - if (!key || key.length > 64) throw new Error('Invalid encryption key length') - return key.length === 64 - ? key - : crypto.createHash('sha256').update(`${key}mx-encrypt-salt`).digest('hex') -} - export interface SecretCodecOptions { key: string - enable: boolean - v2: boolean algorithm?: string } @@ -22,11 +13,6 @@ export class SecretCodec { constructor(private readonly options: SecretCodecOptions) {} assertConfigured() { - if (!this.options.enable) { - throw new Error( - 'Secret encryption is disabled; set MX_ENCRYPT_ENABLE=true or ENCRYPT_ENABLE=true', - ) - } if (!/^[\da-f]{64}$/i.test(this.options.key)) { throw new Error( 'Storing secrets requires a 64-character hexadecimal encryption key', @@ -41,34 +27,14 @@ export class SecretCodec { encrypt(data: T, context = ''): T { if (typeof data === 'string') { if (!data) return data - // TODO(v18): remove the legacy plaintext escape hatch and require encryption. - if ( - !this.options.enable && - !this.options.v2 && - !this.isEncryptedString(data) - ) { - return data - } this.assertConfigured() let plain: string = data if (this.isEncryptedString(data)) { plain = this.decrypt(data, context) - if (data.startsWith(V2_PREFIX) || !this.options.v2) return data + if (data.startsWith(V2_PREFIX)) return data } if (!plain) return '' as T const key = Buffer.from(this.options.key, 'hex') - if (!this.options.v2) { - const cipher = crypto.createCipheriv( - this.options.algorithm || 'aes-256-ecb', - key, - '', - ) - return (PREFIX + - Buffer.concat([ - cipher.update(plain, 'utf8'), - cipher.final(), - ]).toString('base64')) as T - } const keyId = crypto .createHash('sha256') .update(key) diff --git a/apps/core/src/utils/secret-storage-config.ts b/apps/core/src/utils/secret-storage-config.ts new file mode 100644 index 00000000000..f27c8ced19e --- /dev/null +++ b/apps/core/src/utils/secret-storage-config.ts @@ -0,0 +1,66 @@ +export interface SecretStorageConfig { + key: string + jwtSecret: string + /** Used only when reading historical ECB ciphertext. */ + algorithm?: string + deprecatedFlags?: Record +} + +export function secretStorageConfigFromEnv( + env: NodeJS.ProcessEnv, +): SecretStorageConfig { + return { + key: env.MX_ENCRYPT_KEY || env.ENCRYPT_KEY || '', + jwtSecret: env.JWT_SECRET || env.JWTSECRET || '', + algorithm: env.ENCRYPT_ALGORITHM || 'aes-256-ecb', + deprecatedFlags: { + MX_ENCRYPT_ENABLE: env.MX_ENCRYPT_ENABLE, + ENCRYPT_ENABLE: env.ENCRYPT_ENABLE, + MX_SECRET_STORAGE_V2: env.MX_SECRET_STORAGE_V2, + }, + } +} + +export function inspectSecretStorageConfig(config: SecretStorageConfig) { + const issues: { code: string; path?: string }[] = [] + if (!config.key) issues.push({ code: 'FIELD_KEY_MISSING' }) + else if (typeof config.key !== 'string' || !/^[\da-f]{64}$/i.test(config.key)) + issues.push({ code: 'FIELD_KEY_INVALID' }) + if (typeof config.jwtSecret !== 'string' || config.jwtSecret.length < 32) + issues.push({ code: 'JWT_TOO_SHORT' }) + for (const [path, value] of Object.entries(config.deprecatedFlags ?? {})) { + if ( + value !== undefined && + value !== true && + value !== 'true' && + value !== '1' + ) { + issues.push({ code: 'DEPRECATED_ENCRYPTION_FLAG', path }) + } + } + return issues +} + +export function assertSecretStorageConfig(config: SecretStorageConfig) { + const issues = inspectSecretStorageConfig(config) + if (issues.length) { + throw new Error( + issues + .map(({ code, path }) => { + if (code === 'JWT_TOO_SHORT') + return 'JWT_SECRET must be at least 32 characters; preserve existing secrets or use upgrade:rotate-jwt during maintenance' + if (code === 'DEPRECATED_ENCRYPTION_FLAG') + return `${path} is no longer supported: encryption and V2 writes are required. Remove this deprecated flag` + return 'Encryption requires an explicit 64-character hexadecimal MX_ENCRYPT_KEY or ENCRYPT_KEY; preserve your existing key' + }) + .join('\n'), + ) + } + const flags = Object.entries(config.deprecatedFlags ?? {}) + .filter(([, value]) => value !== undefined) + .map(([name]) => name) + if (flags.length) + console.warn( + `${flags.join(', ')}: deprecated encryption flags are ignored; remove them. Encryption and V2 writes are always required.`, + ) +} diff --git a/apps/core/test/fixtures/read-security-config.ts b/apps/core/test/fixtures/read-security-config.ts new file mode 100644 index 00000000000..d8cf2caa06d --- /dev/null +++ b/apps/core/test/fixtures/read-security-config.ts @@ -0,0 +1,16 @@ +// Vite loads local .env files before running the entry. Isolate each boot case +// from developer credentials and the test runner's fixture environment. +for (const name of [ + 'MX_ENCRYPT_KEY', + 'ENCRYPT_KEY', + 'MX_ENCRYPT_ENABLE', + 'ENCRYPT_ENABLE', + 'MX_SECRET_STORAGE_V2', + 'JWT_SECRET', + 'JWTSECRET', +]) + delete process.env[name] +Object.assign(process.env, JSON.parse(process.env.SECURITY_CONFIG_TEST_ENV!)) +await import('../../src/app.config') + +process.stdout.write('SECURITY_CONFIG_READY') diff --git a/apps/core/test/helper/legacy-secret.ts b/apps/core/test/helper/legacy-secret.ts new file mode 100644 index 00000000000..fa6cb93419c --- /dev/null +++ b/apps/core/test/helper/legacy-secret.ts @@ -0,0 +1,12 @@ +import { createCipheriv } from 'node:crypto' + +/** Historical fixture writer. Production only writes authenticated V2. */ +export function legacySecret(value: string, key: string) { + const cipher = createCipheriv('aes-256-ecb', Buffer.from(key, 'hex'), '') + return ( + '$${mx}$$' + + Buffer.concat([cipher.update(value, 'utf8'), cipher.final()]).toString( + 'base64', + ) + ) +} diff --git a/apps/core/test/setup-global.ts b/apps/core/test/setup-global.ts index a28534f3f10..a417b0902dd 100644 --- a/apps/core/test/setup-global.ts +++ b/apps/core/test/setup-global.ts @@ -3,7 +3,6 @@ import { vi } from 'vitest' import { registerAiEnvironment } from '~/global/ai.global' -process.env.MX_SECRET_STORAGE_V2 ??= 'true' process.env.JWT_SECRET ??= 'mx-test-only-better-auth-secret-at-least-32-characters' process.env.TEST ??= '1' diff --git a/apps/core/test/src/upgrade/archive-rehearsal.spec.ts b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts index b596b19b78b..bc0d1ef6ce6 100644 --- a/apps/core/test/src/upgrade/archive-rehearsal.spec.ts +++ b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts @@ -1,9 +1,12 @@ import { execFileSync } from 'node:child_process' import { createCipheriv, randomBytes } from 'node:crypto' -import { existsSync } from 'node:fs' +import { existsSync, mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' import { fileURLToPath } from 'node:url' import JSZip from 'jszip' +import { legacySecret } from 'test/helper/legacy-secret' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { afterAll, beforeAll, describe, expect, it } from 'vitest' @@ -19,13 +22,16 @@ describe.skipIf(!existsSync(entry))( () => { let source: Awaited> let target: Awaited> + let cliCwd: string beforeAll(async () => { + cliCwd = mkdtempSync(join(tmpdir(), 'mx-upgrade-rehearsal-')) source = await createPgTestDatabase('mx_archive_source') target = await createPgTestDatabase('mx_archive_target') }) afterAll(async () => { await target?.close() await source?.close() + if (cliCwd) rmSync(cliCwd, { recursive: true }) }) it('restores ZIP/v1 contents using original keys, then prepares through the shipped entry', async (test) => { @@ -40,15 +46,11 @@ describe.skipIf(!existsSync(entry))( return } const key = 'ab'.repeat(32) - const codec = new SecretCodec({ key, enable: true, v2: false }) await source.pool.query( "INSERT INTO options (id, name, value) VALUES ('100', 'membership', $1)", [ { - dodoApiKey: codec.encrypt( - 'retained-billing-token', - 'membership.dodoApiKey', - ), + dodoApiKey: legacySecret('retained-billing-token', key), }, ], ) @@ -128,15 +130,17 @@ describe.skipIf(!existsSync(entry))( ) const env = { ...process.env, + ENCRYPT_ENABLE: undefined, + MX_ENCRYPT_ENABLE: undefined, + MX_SECRET_STORAGE_V2: undefined, PG_URL: target.connectionString, MX_ENCRYPT_KEY: key, - MX_ENCRYPT_ENABLE: 'true', - MX_SECRET_STORAGE_V2: 'true', JWT_SECRET: 'private-rehearsal-jwt-123456789012345', } const run = (args: string[]) => JSON.parse( execFileSync(process.execPath, [entry, ...args], { + cwd: cliCwd, env, encoding: 'utf8', }), @@ -162,10 +166,7 @@ describe.skipIf(!existsSync(entry))( ) ).rows[0].value expect( - new SecretCodec({ key, enable: true, v2: true }).decrypt( - canonical.apiKey, - 'membership.apiKey', - ), + new SecretCodec({ key }).decrypt(canonical.apiKey, 'membership.apiKey'), ).toBe('retained-billing-token') }, 30000) }, diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts index eeb0028fa82..0f3ea241f86 100644 --- a/apps/core/test/src/upgrade/cli.spec.ts +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -8,20 +8,25 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { runUpgradeCli, upgradeConfigFromEnv } from '~/upgrade/cli' describe('upgrade CLI isolated process', () => { - it('matches runtime enable-flag precedence and boolean boundaries', () => { - expect( - upgradeConfigFromEnv({ - MX_ENCRYPT_KEY: 'ab'.repeat(32), - MX_ENCRYPT_ENABLE: '', - ENCRYPT_ENABLE: 'true', - }).enable, - ).toBe(true) - expect( - upgradeConfigFromEnv({ - MX_ENCRYPT_KEY: 'ab'.repeat(32), - MX_ENCRYPT_ENABLE: '1', - }).enable, - ).toBe(false) + it('uses the same required-key/JWT validation as startup without an activation flag', async () => { + const { inspectSecretStorageConfig } = + await import('~/utils/secret-storage-config') + const env = { + MX_ENCRYPT_KEY: 'ab'.repeat(32), + JWT_SECRET: 'test-jwt-secret-with-at-least-32-characters', + } + expect(inspectSecretStorageConfig(upgradeConfigFromEnv(env))).toEqual([]) + for (const flag of [ + 'MX_ENCRYPT_ENABLE', + 'ENCRYPT_ENABLE', + 'MX_SECRET_STORAGE_V2', + ]) { + expect( + inspectSecretStorageConfig( + upgradeConfigFromEnv({ ...env, [flag]: 'false' }), + ), + ).toContainEqual({ code: 'DEPRECATED_ENCRYPTION_FLAG', path: flag }) + } }) let context: Awaited> beforeAll(async () => { @@ -45,7 +50,6 @@ describe('upgrade CLI isolated process', () => { ...process.env, PG_URL: context.connectionString, MX_ENCRYPT_KEY: 'invalid-key', - MX_SECRET_STORAGE_V2: 'true', JWT_SECRET: 'bad-jwt-secret', }, }, @@ -64,7 +68,6 @@ describe('upgrade CLI isolated process', () => { const env = { PG_URL: context.connectionString, MX_ENCRYPT_KEY: 'ab'.repeat(32), - MX_SECRET_STORAGE_V2: 'true', JWT_SECRET: 'long-enough-test-jwt-123456789012345', } expect( diff --git a/apps/core/test/src/upgrade/preparation.spec.ts b/apps/core/test/src/upgrade/preparation.spec.ts index d780bcd7f76..d4f9c7abf5a 100644 --- a/apps/core/test/src/upgrade/preparation.spec.ts +++ b/apps/core/test/src/upgrade/preparation.spec.ts @@ -1,5 +1,6 @@ import { defaultKeyHasher } from '@better-auth/api-key' import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' +import { legacySecret } from 'test/helper/legacy-secret' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' @@ -16,8 +17,6 @@ describe('transactional v18 preparation', () => { const config = { key: 'ab'.repeat(32), jwtSecret: 'long-test-jwt-secret-1234567890123456789', - enable: true, - v2: true, } const codec = new SecretCodec(config) beforeAll(async () => { @@ -35,10 +34,7 @@ describe('transactional v18 preparation', () => { dodoApiKey: codec.encrypt('billing-secret', 'membership.dodoApiKey'), }, { - githubToken: new SecretCodec({ ...config, v2: false }).encrypt( - 'github-secret', - 'thirdPartyServiceIntegration.githubToken', - ), + githubToken: legacySecret('github-secret', config.key), }, { enable: true, apiKey: 'image-secret', model: 'image-v1' }, ], diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts index 248631d1aaa..a1399fc7f82 100644 --- a/apps/core/test/src/upgrade/readiness.spec.ts +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -1,16 +1,14 @@ +import { legacySecret } from 'test/helper/legacy-secret' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { afterAll, beforeAll, describe, expect, it } from 'vitest' import { readUpgradeState } from '~/upgrade/readiness' -import { SecretCodec } from '~/utils/secret-codec' describe('standalone upgrade readiness', () => { let context: Awaited> const config = { key: 'ab'.repeat(32), jwtSecret: 'jwt-secret-for-upgrade-123456789012345', - enable: true, - v2: true, } beforeAll(async () => { context = await createPgTestDatabase('mx_upgrade_check') @@ -29,28 +27,26 @@ describe('standalone upgrade readiness', () => { const report = await readUpgradeState(context.pool, { key: '', jwtSecret: 'short-secret', - enable: false, - v2: false, + deprecatedFlags: { MX_SECRET_STORAGE_V2: 'false' }, }) expect(report.status).toBe('needs-action') expect(report.issues.map((issue) => issue.code)).toEqual( expect.arrayContaining([ 'FIELD_KEY_MISSING', 'JWT_TOO_SHORT', - 'V2_WRITES_DISABLED', + 'DEPRECATED_ENCRYPTION_FLAG', ]), ) expect(JSON.stringify(report)).not.toContain('short-secret') }) it('counts old formats, detects wrong keys and remains read-only', async () => { - const codec = new SecretCodec({ ...config, v2: false }) await context.pool.query( 'INSERT INTO options (id, name, value) VALUES ($1, $2, $3)', [ '100', 'membership', - { dodoApiKey: codec.encrypt('private-value', 'membership.dodoApiKey') }, + { dodoApiKey: legacySecret('private-value', config.key) }, ], ) const before = (await context.pool.query('SELECT * FROM options')).rows diff --git a/apps/core/test/src/utils/encrypt.util.spec.ts b/apps/core/test/src/utils/encrypt.util.spec.ts index ddacfcaa001..56bc3e46157 100644 --- a/apps/core/test/src/utils/encrypt.util.spec.ts +++ b/apps/core/test/src/utils/encrypt.util.spec.ts @@ -1,7 +1,7 @@ import crypto from 'node:crypto' import { ENCRYPT } from '~/app.config' -import { EncryptUtil, mapString } from '~/utils/encrypt.util' +import { EncryptUtil } from '~/utils/encrypt.util' const legacyEncrypt = (value: string) => { const cipher = crypto.createCipheriv( @@ -60,26 +60,17 @@ describe('EncryptUtil', () => { } }) - it('never silently writes plaintext or returns ciphertext when encryption is disabled', () => { + it('rejects invalid keys for writes while preserving plaintext historical reads', () => { const encrypted = EncryptUtil.encrypt('secret') - const enabled = ENCRYPT.enable - ENCRYPT.enable = false + const key = ENCRYPT.key + ENCRYPT.key = 'invalid-key' try { expect(() => EncryptUtil.encrypt('secret')).toThrow() expect(() => EncryptUtil.decrypt(encrypted)).toThrow() expect(EncryptUtil.encrypt('')).toBe('') expect(EncryptUtil.decrypt('legacy-plaintext')).toBe('legacy-plaintext') } finally { - ENCRYPT.enable = enabled + ENCRYPT.key = key } }) - - it('keeps the legacy key mapping readable', () => { - expect(mapString('1234567890')).toBe( - '3b5aa8094f9d741d2a5cb7e8120f92aad4916df6beafe1be0008a2add070f523', - ) - expect(mapString('1'.repeat(64))).toBe('1'.repeat(64)) - expect(() => mapString('')).toThrow() - expect(() => mapString('1'.repeat(65))).toThrow() - }) }) diff --git a/apps/core/test/src/utils/required-secret-config.spec.ts b/apps/core/test/src/utils/required-secret-config.spec.ts new file mode 100644 index 00000000000..d7c001ae51f --- /dev/null +++ b/apps/core/test/src/utils/required-secret-config.spec.ts @@ -0,0 +1,77 @@ +import { execFile } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' + +const validEnv = { + MX_ENCRYPT_KEY: 'ab'.repeat(32), + JWT_SECRET: 'runtime-jwt-secret-for-testing-1234567890', +} +async function boot(overrides: NodeJS.ProcessEnv = {}) { + const env = { ...process.env } + for (const name of [ + 'MX_ENCRYPT_KEY', + 'ENCRYPT_KEY', + 'MX_ENCRYPT_ENABLE', + 'ENCRYPT_ENABLE', + 'MX_SECRET_STORAGE_V2', + 'JWT_SECRET', + 'JWTSECRET', + ]) + delete env[name] + try { + return await promisify(execFile)( + 'node_modules/.bin/vite-node', + [ + '--config', + 'vite.upgrade.config.ts', + 'test/fixtures/read-security-config.ts', + ], + { + cwd: fileURLToPath(new URL('../../../', import.meta.url)), + env: { + ...env, + NODE_ENV: 'development', + SECURITY_CONFIG_TEST_ENV: JSON.stringify(overrides), + }, + }, + ) + } catch (error) { + return error as { stdout: string; stderr: string; code: number } + } +} + +it.each([ + { JWT_SECRET: validEnv.JWT_SECRET }, + { ...validEnv, MX_ENCRYPT_KEY: 'old-short-key' }, + { ...validEnv, MX_ENCRYPT_KEY: 'z'.repeat(64) }, + { ...validEnv, JWT_SECRET: 'short-jwt' }, + ...['MX_ENCRYPT_ENABLE', 'ENCRYPT_ENABLE', 'MX_SECRET_STORAGE_V2'].map( + (flag) => ({ ...validEnv, [flag]: 'false' }), + ), +])( + 'rejects unsafe runtime settings before application startup %#', + async (env) => { + const result = await boot(env) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') + expect(result.stderr).not.toContain('old-short-key') + expect(result.stderr).not.toContain('short-jwt') + }, + 30000, +) + +it('starts with only explicit secrets and no encryption mode flags', async () => { + const result = await boot(validEnv) + expect(result.stdout, result.stderr).toContain('SECURITY_CONFIG_READY') +}) + +it('keeps key/JWT aliases and accepts old true flags as deprecated no-ops', async () => { + const result = await boot({ + ENCRYPT_KEY: validEnv.MX_ENCRYPT_KEY, + JWTSECRET: validEnv.JWT_SECRET, + ENCRYPT_ENABLE: 'true', + MX_SECRET_STORAGE_V2: 'true', + }) + expect(result.stdout, result.stderr).toContain('SECURITY_CONFIG_READY') + expect(result.stderr).toContain('deprecated') +}) diff --git a/apps/core/test/src/utils/secret-storage-compatibility.spec.ts b/apps/core/test/src/utils/secret-storage-compatibility.spec.ts index e601c62e7b9..eb70c373391 100644 --- a/apps/core/test/src/utils/secret-storage-compatibility.spec.ts +++ b/apps/core/test/src/utils/secret-storage-compatibility.spec.ts @@ -5,56 +5,19 @@ import { reviewDemoDatabaseHooks } from '~/modules/auth/auth.implement' import { ConfigsService } from '~/modules/configs/configs.service' import { EncryptUtil } from '~/utils/encrypt.util' -const gate = vi.hoisted(() => ({ v2: false })) -vi.mock('../../../src/app.config', async (importOriginal) => ({ - ...(await importOriginal()), - get SECRET_STORAGE_V2() { - return gate.v2 - }, -})) - -const encryptionConfig = { ...ENCRYPT } -afterEach(() => { - Object.assign(ENCRYPT, encryptionConfig) - gate.v2 = false -}) - -it('temporarily preserves legacy plaintext writes when encryption is disabled', () => { - ENCRYPT.enable = false - ENCRYPT.key = 'legacy-machine-id' - const data = { secret: 'legacy-secret', nested: ['another-secret', null, 42] } - - expect(EncryptUtil.encrypt(data)).toEqual(data) - expect(EncryptUtil.decrypt(data)).toEqual(data) -}) - -it.each([false, true])( - 'refuses to pass through existing ciphertext with encryption disabled (v2=%s)', - (v2) => { - gate.v2 = v2 - const encrypted = EncryptUtil.encrypt('stored-secret', 'field') - gate.v2 = false - ENCRYPT.enable = false - - expect(() => EncryptUtil.encrypt(encrypted, 'field')).toThrow( - 'encryption is disabled', - ) - expect(() => EncryptUtil.decrypt(encrypted, 'field')).toThrow( - 'encryption is disabled', - ) - expect(() => EncryptUtil.assertConfigured()).toThrow() - }, -) - -it('rejects invalid keys when legacy encryption is enabled', () => { - ENCRYPT.key = 'not-a-hex-key' - - expect(() => EncryptUtil.encrypt('secret')).toThrow() +it('always writes authenticated V2 envelopes, even when obsolete codec options are supplied', async () => { + const { SecretCodec } = await import('~/utils/secret-codec') + const codec = new SecretCodec({ + key: ENCRYPT.key, + enable: false, + v2: false, + } as any) + const encrypted = codec.encrypt('stored-secret', 'field') + expect(encrypted).toMatch(/^\$\$\{mx\}\$\$v2\./) + expect(codec.decrypt(encrypted, 'field')).toBe('stored-secret') }) -it('initializes and updates legacy plaintext configuration without exposing secrets in responses', async () => { - ENCRYPT.enable = false - ENCRYPT.key = 'legacy-machine-id' +it('reads legacy plaintext configuration and encrypts updated storage and cache', async () => { let cache: string | null = null let stored: unknown const service = new ConfigsService( @@ -81,50 +44,43 @@ it('initializes and updates legacy plaintext configuration without exposing secr ) await service.onModuleInit() - expect(JSON.parse(cache!).backupOptions.secretKey).toBe('existing-secret') + expect( + EncryptUtil.decrypt( + JSON.parse(cache!).backupOptions.secretKey, + 'backupOptions.secretKey', + ), + ).toBe('existing-secret') await expect(service.get('backupOptions')).resolves.toMatchObject({ secretKey: 'existing-secret', }) await service.patchAndValid('backupOptions', { secretKey: 'updated-secret' }) - expect(stored).toMatchObject({ secretKey: 'updated-secret' }) - expect(JSON.parse(cache!).backupOptions.secretKey).toBe('updated-secret') + expect((stored as any).secretKey).toMatch(/^\$\$\{mx\}\$\$v2\./) + expect( + EncryptUtil.decrypt((stored as any).secretKey, 'backupOptions.secretKey'), + ).toBe('updated-secret') + expect( + EncryptUtil.decrypt( + JSON.parse(cache!).backupOptions.secretKey, + 'backupOptions.secretKey', + ), + ).toBe('updated-secret') await expect(service.getForResponse('backupOptions')).resolves.toMatchObject({ secretKey: '', }) }) -it('keeps legacy writes while reading new ciphertext during a rolling upgrade', () => { - gate.v2 = false - const legacy = EncryptUtil.encrypt('legacy-compatible', 'field') - expect(legacy).not.toContain('v2.') - gate.v2 = true - const upgraded = EncryptUtil.encrypt(legacy, 'field') - expect(upgraded).toContain('v2.') - gate.v2 = false - expect(EncryptUtil.decrypt(upgraded, 'field')).toBe('legacy-compatible') - expect(EncryptUtil.encrypt(upgraded, 'field')).toBe(upgraded) -}) - -it('keeps native OAuth encryption readable while preserving legacy write format until activation', async () => { +it('keeps encrypted OAuth tokens encrypted at the database hook boundary', async () => { const key = { currentVersion: 1, keys: new Map([[1, SECURITY.jwtSecret!]]), legacySecret: SECURITY.jwtSecret!, } const encrypted = await symmetricEncrypt({ key, data: 'provider-token' }) - const before = reviewDemoDatabaseHooks.account!.create!.before! - gate.v2 = false - const legacy = await before( - { accessToken: encrypted } as any, + const before = reviewDemoDatabaseHooks.account!.update!.before! + const account = { accessToken: encrypted } + const result = await before( + account as any, { context: { secretConfig: key } } as any, ) - expect(legacy).toMatchObject({ data: { accessToken: 'provider-token' } }) - gate.v2 = true - expect( - await before( - { accessToken: encrypted } as any, - { context: { secretConfig: key } } as any, - ), - ).toBeUndefined() - expect(ENCRYPT.enable).toBe(true) + expect((result ? result.data : account).accessToken).toBe(encrypted) }) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index f2ddb864dd6..41a2a5202bd 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -2,6 +2,8 @@ 状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。本次完成 **v17 准备阶段** 的本地实现与验证,尚未发布;不发布 v18,不操作真实站点数据。 +修订(2026-11-08):用户明确要求本 PR 立即移除非加密运行和旧格式写入,覆盖下文“准备阶段保留逃生口”的安排。显式字段密钥、至少 32 字符 JWT 和固定 V2 写入在本分支立即生效;旧 true 开关仅作为弃用配置接受,false 拒绝启动。历史读取与显式迁移保留。其余 API、SDK 6 和数据库列删除仍按后续阶段执行,当前分支尚未发布。 + 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 ## 意图 diff --git a/packages/e2e/src/helpers/core-app-config.ts b/packages/e2e/src/helpers/core-app-config.ts index f746cf0afed..3b3c8da0aa9 100644 --- a/packages/e2e/src/helpers/core-app-config.ts +++ b/packages/e2e/src/helpers/core-app-config.ts @@ -36,8 +36,6 @@ const currentRedis = () => parseRedisUrl(process.env.REDIS_CONNECTION_STRING) export const PORT = process.env.PORT || 2333 export const API_VERSION = 3 export const DEMO_MODE = false -export const SECRET_STORAGE_V2 = - parseBoolean(process.env.MX_SECRET_STORAGE_V2) ?? false export const CROSS_DOMAIN = { allowedOrigins: process.env.ALLOWED_ORIGINS @@ -104,7 +102,6 @@ export const ENCRYPT = { key: process.env.MX_ENCRYPT_KEY || '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab', - enable: true, algorithm: 'aes-256-ecb', } From 565ebb9064309674e2bd882348e1f4a954ec7705 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 23:22:20 +0800 Subject: [PATCH 16/26] docs(security): align deployments and upgrade guides with required encryption --- .env.example | 10 +-- .github/workflows/ci.yml | 6 +- .github/workflows/release.yml | 3 +- apps/core/README.md | 9 ++- apps/core/ecosystem.config.cjs | 2 +- apps/core/ecosystem.dev.config.js | 3 +- apps/core/package.json | 4 +- .../processors/helper/helper.jwt.service.ts | 12 +-- docker-compose.server.yml | 9 +-- docker-compose.yml | 4 +- ...-11-08-mx-core-v18-compatibility-design.md | 16 +++- docs/security-secret-storage.md | 75 +++++++++---------- docs/v18-preparation.md | 21 ++++-- pnpm-lock.yaml | 8 -- 14 files changed, 91 insertions(+), 91 deletions(-) diff --git a/.env.example b/.env.example index 93abac03f76..39229f5a6c9 100644 --- a/.env.example +++ b/.env.example @@ -1,13 +1,12 @@ # THIS ENV FILE EXAMPLE ONLY FOR DOCKER COMPOSE # SEE https://docs.docker.com/compose/environment-variables/#the-env-file -# Set a private instance secret (at least 32 characters for V2). Preserve an existing value. +# Required: a private JWT secret of at least 32 characters. Preserve an existing value. JWT_SECRET= ALLOWED_ORIGINS=innei.ren,www.innei.ren -# Must be 64 hexadecimal characters. Keep the existing key when upgrading. +# Required: 64 hexadecimal characters. Keep the existing key when upgrading. +# New instances only: openssl rand -hex 32; persist this value before starting. ENCRYPT_KEY= -# Temporary legacy plaintext escape hatch; v18 will require encryption. -ENCRYPT_ENABLE=false # Comma-separated HTTPS origins that mx-core may contact for Push Relay. MX_PUSH_RELAY_ORIGINS= @@ -22,6 +21,3 @@ PG_URL= # Throttle THROTTLE_TTL=10 THROTTLE_LIMIT=20 - -# Enable only after every replica has the compatible secret reader; see docs/security-secret-storage.md -MX_SECRET_STORAGE_V2=false diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2e38f9a5916..c8d160e17b6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,6 +73,9 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Test + env: + JWT_SECRET: test-docker-jwt-secret-at-least-32-characters + MX_ENCRYPT_KEY: '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab' run: | bash ./scripts/workflow/test-docker.sh sudo rm -rf ./data @@ -121,7 +124,8 @@ jobs: PG_DATABASE: mx_core REDIS_HOST: 127.0.0.1 REDIS_PORT: 6379 - JWT_SECRET: test-bundle-server-jwt-secret + JWT_SECRET: test-bundle-server-jwt-secret-at-least-32-characters + MX_ENCRYPT_KEY: '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab' MIGRATIONS_DIR: ${{ github.workspace }}/apps/core/src/database/migrations e2e: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 5a8514055cb..3915f07914c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -83,7 +83,8 @@ jobs: PG_DATABASE: mx_core REDIS_HOST: 127.0.0.1 REDIS_PORT: 6379 - JWT_SECRET: test-bundle-server-jwt-secret + JWT_SECRET: test-bundle-server-jwt-secret-at-least-32-characters + MX_ENCRYPT_KEY: '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab' MIGRATIONS_DIR: ${{ github.workspace }}/apps/core/src/database/migrations run: | bash scripts/workflow/test-server.sh diff --git a/apps/core/README.md b/apps/core/README.md index 34d83c3e47e..78d4cbaa18f 100644 --- a/apps/core/README.md +++ b/apps/core/README.md @@ -223,7 +223,7 @@ pnpm test:watch # watch mode | Variable | Description | Default | |----------|-------------|---------| -| `JWT_SECRET` | Secret for JWT signing | Required | +| `JWT_SECRET` | Explicit secret for JWT signing and OAuth encryption, at least 32 characters | Required | | `ALLOWED_ORIGINS` | CORS allowed origins (comma-separated) | — | | `PG_URL` | Full PostgreSQL connection string | — | | `PG_HOST` | PostgreSQL host | `127.0.0.1` | @@ -238,15 +238,16 @@ pnpm test:watch # watch mode | `REDIS_PORT` | Redis port | `6379` | | `REDIS_PASSWORD` | Redis password | — | | `SNOWFLAKE_WORKER_ID` | Snowflake ID worker ID (0–1023) | Required | -| `ENCRYPT_ENABLE` | Enable field encryption; `false` is a temporary v17 plaintext escape hatch. Encryption is required starting in v18 | `false` | -| `ENCRYPT_KEY` | 64-char hex encryption key | — | -| `MX_SECRET_STORAGE_V2` | Enable new secret formats after all replicas are upgraded; see [rollout guide](../../docs/security-secret-storage.md) | `false` | +| `ENCRYPT_KEY` / `MX_ENCRYPT_KEY` | Required explicit 64-char hex field key; preserve the existing key | — | +| `ENCRYPT_ENABLE` / `MX_ENCRYPT_ENABLE` / `MX_SECRET_STORAGE_V2` | Deprecated: remove these flags. Old true values are ignored; false values refuse startup | — | | `THROTTLE_TTL` | Rate limit window (seconds) | `10` | | `THROTTLE_LIMIT` | Max requests per window | `100` | | `PORT` | Server port | `2333` | | `TZ` | Timezone | `Asia/Shanghai` | | `DISABLE_CACHE` | Disable Redis caching | `false` | +On this unreleased branch, field encryption and V2 writes are mandatory; `JWT_SECRET` must have at least 32 characters. See the [upgrade guide](../../docs/v18-preparation.md) before deploying to an existing instance. + For managed Redis providers such as Upstash, pass the provider URI as `REDIS_CONNECTION_STRING=rediss://user:password@host:port`. The `rediss://` scheme enables TLS. If the provider exposes the same URI as `REDIS_URL`, it is diff --git a/apps/core/ecosystem.config.cjs b/apps/core/ecosystem.config.cjs index 20de19f33de..b074d6a7ce5 100644 --- a/apps/core/ecosystem.config.cjs +++ b/apps/core/ecosystem.config.cjs @@ -15,7 +15,7 @@ module.exports = { watch: false, instances: cpuLen, max_memory_restart: '520M', - args: '--color --encrypt_enable', + args: '--color', env: { NODE_ENV: 'production', NODE_PATH: nodePath, diff --git a/apps/core/ecosystem.dev.config.js b/apps/core/ecosystem.dev.config.js index c9463593c84..fa1dafde6b8 100644 --- a/apps/core/ecosystem.dev.config.js +++ b/apps/core/ecosystem.dev.config.js @@ -1,4 +1,5 @@ const { execSync } = require('node:child_process') + const nodePath = execSync(`npm root --quiet -g`, { encoding: 'utf-8' }).split( '\n', )[0] @@ -13,7 +14,7 @@ module.exports = { watch: false, instances: 2, max_memory_restart: '220M', - args: '--color --encrypt_enable', + args: '--color', env: { NODE_ENV: 'development', NODE_PATH: nodePath, diff --git a/apps/core/package.json b/apps/core/package.json index 6b072ba938e..e5ae771862b 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -18,9 +18,8 @@ "prebuild": "rimraf dist", "build": "npm run bundle", "dev": "cross-env NODE_ENV=development nodemon --signal SIGKILL --watch src --ext ts,json --exec \"vite-node src/dev.ts\"", - "dev:encrypt": "npm run dev -- --encrypt_enable --encrypt_key=33b9405ac49f63ef4ccbf6b338aeeff8152e844d942ef22278abbec2b1f93b5e", "migrate": "vite-node src/migrate.ts", - "migrate:secrets": "cross-env MX_SECRET_STORAGE_V2=1 MX_MIGRATE_SECRETS=1 vite-node src/app-migrate.ts", + "migrate:secrets": "cross-env MX_MIGRATE_SECRETS=1 vite-node src/app-migrate.ts", "migrate:app": "cross-env NODE_ENV=development vite-node src/app-migrate.ts", "lint:migrations": "tsx --tsconfig tsconfig.json scripts/lint-migrations.ts", "openapi:export": "vite-node scripts/export-openapi.ts", @@ -135,7 +134,6 @@ "mime-types": "^3.0.2", "mkdirp": "^3.0.1", "nanoid": "6.0.2", - "node-machine-id": "1.1.12", "nodemailer": "10.2.1", "object-scan": "^20.0.4", "octokit": "^5.0.5", diff --git a/apps/core/src/processors/helper/helper.jwt.service.ts b/apps/core/src/processors/helper/helper.jwt.service.ts index 486c532cc77..5d32aa2373b 100644 --- a/apps/core/src/processors/helper/helper.jwt.service.ts +++ b/apps/core/src/processors/helper/helper.jwt.service.ts @@ -1,7 +1,7 @@ import { Injectable } from '@nestjs/common' import jwt from 'jsonwebtoken' -import { ENCRYPT, SECURITY } from '~/app.config' +import { SECURITY } from '~/app.config' import { RedisKeys } from '~/constants/cache.constant' import { getRedisKey } from '~/utils/redis.util' import { md5 } from '~/utils/tool.util' @@ -23,14 +23,8 @@ export class JWTService { return } - const ENCRYPT_KEY = ENCRYPT.key - - const secret = - SECURITY.jwtSecret || - Buffer.from(ENCRYPT_KEY).toString('base64').slice(0, 15) || - 'asjhczxiucipoiopiqm2376' - - this.secret = secret + // app.config validates the explicit JWT secret before any service starts. + this.secret = SECURITY.jwtSecret! } async verify(token: string) { diff --git a/docker-compose.server.yml b/docker-compose.server.yml index 7509c674974..976c4b55d48 100644 --- a/docker-compose.server.yml +++ b/docker-compose.server.yml @@ -20,12 +20,9 @@ x-mx-env: &mx-env REDIS_PORT: '6379' # REDIS_PASSWORD: ${REDIS_PASSWORD:-} - # Security (optional) - # ENCRYPT_KEY must be length 64 if ENCRYPT_ENABLE=true - ENCRYPT_ENABLE: ${ENCRYPT_ENABLE:-true} - ENCRYPT_KEY: ${ENCRYPT_KEY:-} - MX_SECRET_STORAGE_V2: ${MX_SECRET_STORAGE_V2:-false} - JWT_SECRET: ${JWT_SECRET:-} + # Required persisted secrets, shared by the application and migration runner. + MX_ENCRYPT_KEY: ${MX_ENCRYPT_KEY:-${ENCRYPT_KEY:?Set MX_ENCRYPT_KEY or ENCRYPT_KEY to your persisted 64-character hexadecimal key}} + JWT_SECRET: ${JWT_SECRET:?Set JWT_SECRET to your persisted secret of at least 32 characters} JWT_EXPIRE: ${JWT_EXPIRE:-} # Comma-separated HTTPS origins mx-core may contact for Push Relay. diff --git a/docker-compose.yml b/docker-compose.yml index c52b9ea1343..381d008a464 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,8 +12,8 @@ x-mx-env: &mx-env PG_DATABASE: mx_core SNOWFLAKE_WORKER_ID: '1' ALLOWED_ORIGINS: localhost - JWT_SECRET: YOUR_SUPER_SECURED_JWT_SECRET_STRING - MX_SECRET_STORAGE_V2: ${MX_SECRET_STORAGE_V2:-false} + JWT_SECRET: ${JWT_SECRET:?Set JWT_SECRET to your persisted secret of at least 32 characters} + MX_ENCRYPT_KEY: ${MX_ENCRYPT_KEY:-${ENCRYPT_KEY:?Set MX_ENCRYPT_KEY or ENCRYPT_KEY to your persisted 64-character hexadecimal key}} # Bound crash artifacts and stdout/stderr even during dependency outages. x-runtime-safety: &runtime-safety diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 41a2a5202bd..6b86824f7c1 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -41,11 +41,11 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 5. 完成 v17 事实文档纠错、v15–v17 迁移入口及 v18 预告,提供可执行准备说明。 6. 行为测试、真实临时 PostgreSQL 升级演练、构建和独立整体验证;保留原有工作。 -准备阶段不切换普通 AI GET 的既有默认行为,不拒绝现有合法旧参数,不删除 SDK 5.x 的公开签名,不删除数据库列,不强制关闭 v17 逃生口,不修改线上环境或发布标签。 +准备阶段不切换普通 AI GET 的既有默认行为,不拒绝现有合法旧参数,不删除 SDK 5.x 的公开签名,不删除数据库列,不修改线上环境或发布标签。 ## 安全配置与迁移契约 -### v18 的启动条件 +### 当前分支已生效的启动条件 - 显式 `ENCRYPT_KEY`/`MX_ENCRYPT_KEY` 为 64 位十六进制;`JWT_SECRET` 至少 32 字符。保留既有别名优先级。 - 新装生成每实例随机密钥并持久保存;现有密钥原样保留。生产模板不能内置公共有效密钥。 @@ -174,4 +174,14 @@ flowchart TD - 独立整体验证发现的 2 项 Critical、2 项 Important 已通过先失败后通过的行为测试修复;未再次分派审阅。另补「仅有分类等业务数据时不得判为空库」回归。缓存加密验证比较解密后的原值,不把随机 GCM 密文字节相同当作契约。 - 官方文档工作区 `/Users/innei/git/innei-repo/mx-docs` 的 `545e5c8`、`0719e79` 已本地提交,build/check/links/pangu/lint 与 3 项配置生成测试通过。保留原有 1 条 Astro hint、1 条既有死链;真实预览浏览器和临时 Compose 容器验证了复制映射、字符串类型与美元符号保留,生成的 llms 内容已核对。 -裁量保持:先抽纯加密/字段路径模块以支持独立诊断,仅改变实现顺序;v18 的启动强制、严格配置、只读 AI 默认行为、SDK 6 清理和物理列删除仍属后续阶段。当前没有自动隔离恢复入口,继续使用离线恢复到准备版再准备的路径;回退最低公开版本与消费者矩阵仍须发布后实测确定。原有 Agent 测试夹具另行处理。 +裁量保持:先抽纯加密/字段路径模块以支持独立诊断,仅改变实现顺序;启动强制已按用户修订在本分支实施;严格配置、只读 AI 默认行为、SDK 6 清理和物理列删除仍属后续阶段。当前没有自动隔离恢复入口,继续使用离线恢复到准备版再准备的路径;回退最低公开版本与消费者矩阵仍须发布后实测确定。原有 Agent 测试夹具另行处理。 + +## 强制加密修订验证(2026-11-08) + +用户明确要求本 PR 删除非加密运行,已移除 enable/V2 模式分支、明文/ECB 写入、机器 ID 与短密钥派生;配置、Snippet、Webhook 固定 GCM,API Key 固定哈希,OAuth 不再在数据库 hook 中解密后写入。旧数据读取、离线检查和维护迁移保留。旧 true 标记仅提示弃用,显式 false/无效标记拒绝启动;字段密钥和 JWT 同时必填。 + +- 定向 49 项测试通过,包括独立进程配置拒绝、无开关启动、历史 ECB、事务迁移与构建后 CLI 恢复演练。旧实现下新增启动测试有 6 项失败,固定 V2 写入测试亦先失败。 +- 完整 Core 回归:431 个文件、3573 项通过,11 项跳过;仍显式排除原有 Agent DI 夹具。JWT helper 另 4 项通过。 +- Core 生产构建、Core/E2E 类型检查、API envelope/验证配对检查、50 个迁移 lint、冻结 lockfile 校验通过;构建产物已不含关闭加密报错或 machine-id 回退。 +- 两份 Compose 均验证缺秘密拒绝、别名和主键优先级、app/migrate 同值;示例不再带关闭开关或公共生产秘密。PM2 与 CI/release 测试环境已同步。 +- mx-docs 的 3 项测试、build/check/links/pangu/lint 通过,生成 llms 内容包含新要求;v17.0.4 页面明确固定历史行为,当前开发分支说明明确强制加密已实现。保留既有 hint 和死链。 diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index 5eecbfe3371..2b83331359e 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -2,53 +2,52 @@ Core protects recoverable credentials with authenticated encryption and stores API keys as hashes. This does not isolate the encryption key from a compromised application host; external KMS integration is a separate step. -## Temporary legacy escape hatch (v17 only) - -Existing installations can temporarily keep `ENCRYPT_ENABLE=false` (or `MX_ENCRYPT_ENABLE=false`) with `MX_SECRET_STORAGE_V2=false`. In this mode, legacy configuration and Snippet secrets continue to be read and written as plaintext in PostgreSQL and Redis. API responses still mask sensitive configuration fields. Startup logs a deprecation warning. - -This escape hatch does not bypass encryption checks for existing ciphertext, v2 writes, the secret backfill, or features that explicitly require encrypted storage (such as LobeHub authorization). To read existing ciphertext, enable encryption with its original key. Enabling encryption with a missing or invalid key still fails rather than falling back to plaintext. - -**v18 will require field encryption and an explicit 64-character hexadecimal encryption key.** Before upgrading to v18, set `ENCRYPT_ENABLE=true` (or `MX_ENCRYPT_ENABLE=true`) and configure `ENCRYPT_KEY` (or `MX_ENCRYPT_KEY`). Keep the original key when one already exists; configure a random key only for installations that have never encrypted data. Follow the rollout below separately to activate v2 writes and backfill existing stored credentials. - -## v18 preparation tooling (unreleased) - -The current branch adds a separate readiness/preparation ledger and standalone -CLI. v17.0.4 does not contain these commands. Follow the -[v18 preparation runbook](v18-preparation.md) after deploying a compatible -preparation build. The older `migrate:secrets` ledger is not proof that legacy -config and AI metadata are prepared. Never rotate a short JWT by replacing its -environment value alone: use the maintenance file-based rewrap operation. - -## Original credential rollout - -This is an expand-before-contract rollout. Two replicas may overlap, so the new reader ships before new writes. - -1. Configure an explicit random 32-byte encryption key as 64 hexadecimal characters (`MX_ENCRYPT_KEY` or `ENCRYPT_KEY`), enable encryption, and configure a stable `JWT_SECRET`. Keep the existing keys when upgrading an installed instance. Machine identity is not an encryption secret. -2. Deploy this version to **every replica** with `MX_SECRET_STORAGE_V2=false` (the default). This immediately fixes configuration plaintext writes, removes secret logging, and masks token details. Config and Snippet writes retain the legacy cipher format during this phase. Webhook and OAuth writes retain their old format so the preceding release can still read them. -3. After every replica has the compatible reader, deploy `MX_SECRET_STORAGE_V2=true` to every replica. New configuration/Snippet secrets use AES-256-GCM; Webhook credentials are encrypted; API keys use Better Auth hashes; OAuth access/refresh tokens use Better Auth authenticated encryption. All replicas must finish this step before the backfill, so no legacy writer can reintroduce plaintext afterwards. -4. Run the one-shot data migration using the same database, encryption key and JWT secret as the application: - - ```sh - pnpm -C apps/core run migrate:secrets - ``` - - In the release container, the equivalent is to set `MX_SECRET_STORAGE_V2=true` and `MX_MIGRATE_SECRETS=1` for a one-shot `node migrate.mjs` invocation. Do not put `MX_MIGRATE_SECRETS` in the regular app environment. The migration is explicitly opt-in, transactional and repeatable; the standard app-migration ledger records completion. It does not change schema or contact credential providers. - -5. Refresh running instances after the backfill so old cached representations retire. Verify configuration consumption, API authentication and token revocation. Existing client API key strings remain valid after hashing. - -After activation, rollback only to a version that understands the new formats. Do not roll back to the preceding release or disable v2 writes again. Do not change encryption/JWT keys as part of this rollout. Key IDs identify the required encryption key; they do not provide automatic key rotation or recovery. +## Required configuration (unreleased branch) + +Field encryption is the only runtime mode on this branch. Supply an explicit +64-character hexadecimal `MX_ENCRYPT_KEY` or `ENCRYPT_KEY`, and a stable +`JWT_SECRET` (alias `JWTSECRET`) of at least 32 characters. There is no +machine-ID fallback or short-key derivation. Keep existing effective secrets; +only an instance that has never encrypted data may generate its first random +field key with `openssl rand -hex 32`. Persist secrets before starting and +share the same values between app and migration services. + +Remove `ENCRYPT_ENABLE`, `MX_ENCRYPT_ENABLE` and `MX_SECRET_STORAGE_V2`. +Historical true values are accepted with a deprecation warning; false or +other unsupported values fail before business startup. These flags no longer +select a storage mode. Config/Snippet/Webhook writes always use authenticated +V2 encryption, API-key writes always hash, and OAuth writes stay encrypted. +`ENCRYPT_ALGORITHM` is used only to read historical ECB ciphertext. + +The published v17.0.4 escape hatch is historical behavior, removed here. +This branch is not released yet. The [preparation runbook](v18-preparation.md) +explains standalone check/prepare and maintenance JWT rotation. The tools can +inspect invalid settings without booting the application. Short JWT values +must be rotated with the original secret before starting this runtime; do not +replace their environment value alone. + +Before persistent backfill, stop old-format writers on every replica and +background worker. `upgrade:prepare --all-replicas-v2` requires an explicit +operator attestation and performs transactional conversion/verification. +`migrate:secrets` remains an opt-in credential-only historical migration; its +ledger is not proof that configuration and AI metadata are prepared. It no +longer needs a V2 activation flag. + +After conversion, roll back only to a tested reader of the new formats. +Neither disabling encryption nor switching back to old-format writes exists. +Preserve original field/JWT secrets and snapshots for recovery. ## Storage and API behavior -- With encryption enabled, sensitive configuration is encrypted **after merging**, before PostgreSQL and Redis writes. Blank secret fields retain the existing value. Responses sanitize stored objects without decrypting them, including in legacy plaintext mode. -- The v2 envelope carries a format version, key fingerprint, random 96-bit nonce, authentication tag and ciphertext. AAD binds configuration secrets to their field paths and Snippet/Webhook secrets to their record and field. A wrong key, invalid context, corrupt envelope or encryption failure is an error; it is never a plaintext fallback. Legacy plaintext/ECB reads remain for the staged upgrade. +- Sensitive configuration is encrypted **after merging**, before PostgreSQL and Redis writes. Blank secret fields retain the existing value. Responses sanitize stored objects without decrypting them, including when reading historical plaintext. +- The v2 envelope carries a format version, key fingerprint, random 96-bit nonce, authentication tag and ciphertext. AAD binds configuration secrets to their field paths and Snippet/Webhook secrets to their record and field. A wrong key, invalid context, corrupt envelope or encryption failure is an error; it is never a plaintext fallback. Legacy plaintext/ECB reads remain for explicit upgrades and historical recovery. - API keys are displayed only in the creation response. List/detail responses contain only the visible prefix. `mx-hashed-v1` identifies hashed records, so a hash copied from the database cannot authenticate through the legacy verifier. The legacy reader is retained for upgrade compatibility; migrated records always use the hashed configuration. Revocation uses the key ID and closes associated WebSocket connections on every replica through the existing Redis bus. - Password hashes, OAuth tokens, ID tokens and raw provider payloads are excluded from session profile responses. - OAuth access/refresh token encryption uses `JWT_SECRET` as Better Auth key version 1. Encryption and authentication secrets must remain available for recovery of their respective data. ## Encrypted backups -Enable **Settings → Storage → Backup → Backup encryption**. The browser generates a recovery key and displays it once, with copy and download actions. Confirm that you have saved it, then save the settings. Canceling the dialog leaves encryption disabled. No backup environment variable is required; this setting is independent of `MX_SECRET_STORAGE_V2` and field encryption. The browser must use HTTPS or localhost for Web Crypto. +Enable **Settings → Storage → Backup → Backup encryption**. The browser generates a recovery key and displays it once, with copy and download actions. Confirm that you have saved it, then save the settings. Canceling the dialog leaves encryption disabled. No backup environment variable is required; this setting is independent of mandatory field encryption. The browser must use HTTPS or localhost for Web Crypto. Only the public key is sent to and stored by the server. The RSA-3072 private recovery key stays in the browser's temporary modal state until the user closes it; it is not put in application settings, browser storage or query caches. Every backup gets a fresh random AES-256-GCM data key, wrapped with RSA-OAEP-SHA256 and the configured public key. The versioned `.mxbackup` envelope includes the wrapped key, random 96-bit nonce and authentication tag; its header is authenticated. See the [Node crypto API](https://nodejs.org/api/crypto.html#cryptopublicencryptkey-buffer) for the platform primitives. diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index 4a09754fb93..5920b152b4f 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -1,6 +1,8 @@ # v18 preparation (unreleased) -This is the v17 preparation phase. v18 enforcement, SDK 6 cleanup and column +This unreleased branch now requires field encryption, an explicit hex64 key +and a JWT of at least 32 characters. Plaintext/ECB writes, encryption-disable +flags and machine-ID fallback have been removed. SDK 6 cleanup and column removal remain future work. These commands are not part of released v17.0.4. No minimum released preparation tag is established yet. Build and deploy this preparation code to every replica before using it; do not publish or migrate @@ -16,8 +18,9 @@ historical schema/data migrations before using this runner. The independent CLI consumes environment configuration, not the application's CLI/YAML flags. Supply the same effective `PG_URL` (preferred), `PG_CONNECTION_STRING` or `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE`, -optional `PG_SSL`, `MX_ENCRYPT_KEY`/`ENCRYPT_KEY`, enable flag, -`JWT_SECRET`/`JWTSECRET`, and `MX_SECRET_STORAGE_V2`. `.env` is loaded relative +optional `PG_SSL`, `MX_ENCRYPT_KEY`/`ENCRYPT_KEY`, and +`JWT_SECRET`/`JWTSECRET`. Remove the old encryption/V2 flags; explicit false +values are diagnosed as `DEPRECATED_ENCRYPTION_FLAG`. `.env` is loaded relative to the command's working directory. It does not start HTTP, business services, background jobs or Redis. Migrate the schema separately first. @@ -55,9 +58,10 @@ requires recovering the original secret or reauthorizing the account. ## Explicit preparation -After every replica has the compatible reader, enable field encryption and -V2 writes everywhere. Confirm no old writer remains. A short JWT must be -rotated before enabling V2. Use the original field key; only an instance that +Obtain this build without starting business replicas. Run the standalone +check and rotate a short JWT during maintenance first. Remove old disable +flags, then deploy the required secrets and this runtime to all replicas. +Every new write is V2 automatically. Confirm no old writer remains. Use the original field key; only an instance that has never encrypted data may generate its first random key. ```sh @@ -100,7 +104,10 @@ A future preparation image can run the tool without starting app services: docker compose run --rm --no-deps app node upgrade.mjs check --json ``` -The database must already be running. For JWT rotation mount both secret files +The database must already be running. Compose requires nonempty persisted +field/JWT values before launching any service. Use the source CLI or a direct +`docker run` invocation to diagnose missing settings without Compose validation. +For JWT rotation mount both secret files read-only and pass their in-container paths. Ensure the app replicas are stopped; `--no-deps` by itself does not stop an existing replica. Keep app and migration images pinned to the same verified release, not an unreviewed latest. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6c6f158efa7..6efb4cfc3be 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -914,9 +914,6 @@ importers: nanoid: specifier: 6.0.2 version: 6.0.2 - node-machine-id: - specifier: 1.1.12 - version: 1.1.12 nodemailer: specifier: 10.2.1 version: 10.2.1 @@ -9922,9 +9919,6 @@ packages: engines: {node: '>=16.0.0'} hasBin: true - node-machine-id@1.1.12: - resolution: {integrity: sha512-QNABxbrPa3qEIfrE6GOJ7BYIuignnJw7iQ2YPbc3Nla1HzRJjXzZOiikfF8m7eAMfichLt3M4VgLOetqgDmgGQ==} - node-pty@1.1.0: resolution: {integrity: sha512-20JqtutY6JPXTUnL0ij1uad7Qe1baT46lyolh2sSENDd4sTzKZ4nmAFkeAARDKwmlLjPx6XKRlwRUxwjOy+lUg==} @@ -21917,8 +21911,6 @@ snapshots: node-gyp-build: 4.8.4 optional: true - node-machine-id@1.1.12: {} - node-pty@1.1.0: dependencies: node-addon-api: 7.1.1 From b5eb3d004304576784133e86be1c733b0b2cc4e7 Mon Sep 17 00:00:00 2001 From: Innei Date: Sun, 8 Nov 2026 23:29:24 +0800 Subject: [PATCH 17/26] fix(security): unwrap historical OAuth ciphertext before migration --- .../20261018-protect-stored-secrets.ts | 16 ++-- .../database/protect-stored-secrets.spec.ts | 86 ++++++++++++++++++- ...-11-08-mx-core-v18-compatibility-design.md | 4 +- 3 files changed, 93 insertions(+), 13 deletions(-) diff --git a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts index b0009d66b95..140fc72bd23 100644 --- a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts +++ b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts @@ -1,9 +1,5 @@ import { defaultKeyHasher } from '@better-auth/api-key' -import { - parseEnvelope, - symmetricDecrypt, - symmetricEncrypt, -} from 'better-auth/crypto' +import { symmetricEncrypt } from 'better-auth/crypto' import { eq } from 'drizzle-orm' import { SECURITY } from '~/app.config' @@ -16,6 +12,7 @@ import { } from '~/database/schema' import { HASHED_API_KEY_CONFIG_ID } from '~/modules/auth/auth.constant' import { encryptObject } from '~/modules/configs/configs.encrypt.util' +import { unwrapOAuthToken } from '~/upgrade/oauth-token' import { EncryptUtil } from '~/utils/encrypt.util' import type { AppMigration } from './types' @@ -96,14 +93,11 @@ export const migration: AppMigration = { for (const field of ['accessToken', 'refreshToken'] as const) { const token = row[field] if (!token) continue - if (token.startsWith('$ba$')) { - if (!parseEnvelope(token)) - throw new Error('Malformed OAuth token envelope') - await symmetricDecrypt({ key: oauthKey, data: token }) - } else { + const plain = await unwrapOAuthToken(token, SECURITY.jwtSecret) + if (!token.startsWith('$ba$')) { patch[field] = await symmetricEncrypt({ key: oauthKey, - data: token, + data: plain, }) } } diff --git a/apps/core/test/src/database/protect-stored-secrets.spec.ts b/apps/core/test/src/database/protect-stored-secrets.spec.ts index 63a6c3ba73b..f05301caf24 100644 --- a/apps/core/test/src/database/protect-stored-secrets.spec.ts +++ b/apps/core/test/src/database/protect-stored-secrets.spec.ts @@ -1,7 +1,7 @@ import crypto from 'node:crypto' import { defaultKeyHasher } from '@better-auth/api-key' -import { symmetricDecrypt } from 'better-auth/crypto' +import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { ENCRYPT, SECURITY } from '~/app.config' @@ -17,6 +17,7 @@ import { SnippetRepository } from '~/modules/snippet/snippet.repository' import { WebhookRepository } from '~/modules/webhook/webhook.repository' import { __setTestPostgresInstance } from '~/processors/database/postgres.provider' import { SnowflakeService } from '~/shared/id/snowflake.service' +import { oauthKey } from '~/upgrade/oauth-token' import { EncryptUtil } from '~/utils/encrypt.util' it('upgrades real credential rows without changing their usable values, and can run twice', async () => { @@ -193,3 +194,86 @@ it('upgrades real credential rows without changing their usable values, and can await context.close() } }, 120_000) + +it('unwraps historical OAuth ciphertext before rewrapping and remains idempotent', async () => { + const context = await createPgTestDatabase('mx_secret_legacy_oauth') + try { + const legacy = await symmetricEncrypt({ + key: SECURITY.jwtSecret!, + data: 'retained-provider-token', + }) + expect(legacy).toMatch(/^[\da-f]+$/i) + await context.pool.query( + "INSERT INTO readers (id, name, email) VALUES ('owner', 'Owner', 'owner@example.test')", + ) + await context.pool.query( + "INSERT INTO accounts (id,user_id,provider_id,access_token) VALUES ('oauth','owner','github',$1)", + [legacy], + ) + const args = { + db: context.db, + pool: context.pool, + logger: { log: vi.fn() } as any, + } + await migration.up(args) + const token = ( + await context.pool.query('SELECT access_token FROM accounts') + ).rows[0].access_token + expect( + await symmetricDecrypt({ + key: oauthKey(SECURITY.jwtSecret!), + data: token, + }), + ).toBe('retained-provider-token') + await migration.up(args) + expect( + (await context.pool.query('SELECT access_token FROM accounts')).rows[0] + .access_token, + ).toBe(token) + } finally { + await context.close() + } +}) + +it.each(['wrong-key', 'truncated'])( + 'rolls back the credential batch for %s historical OAuth ciphertext', + async (invalid) => { + const context = await createPgTestDatabase('mx_secret_bad_oauth') + try { + const original = await symmetricEncrypt({ + key: + invalid === 'wrong-key' + ? 'another-original-jwt-at-least-32-characters' + : SECURITY.jwtSecret!, + data: 'retained-provider-token', + }) + const token = invalid === 'truncated' ? original.slice(0, -1) : original + await context.pool.query( + "INSERT INTO readers (id, name, email) VALUES ('owner', 'Owner', 'owner@example.test')", + ) + await context.pool.query( + "INSERT INTO api_keys (id,user_id,key) VALUES ('key','owner','original-client-key')", + ) + await context.pool.query( + "INSERT INTO accounts (id,user_id,provider_id,access_token) VALUES ('oauth','owner','github',$1)", + [token], + ) + await expect( + migration.up({ + db: context.db, + pool: context.pool, + logger: { log: vi.fn() } as any, + }), + ).rejects.toThrow() + expect( + (await context.pool.query('SELECT access_token FROM accounts')).rows[0] + .access_token, + ).toBe(token) + expect( + (await context.pool.query('SELECT key FROM api_keys')).rows[0].key, + ).toBe('original-client-key') + } finally { + await context.close() + } + }, +) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 6b86824f7c1..02b6df3810f 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -181,7 +181,9 @@ flowchart TD 用户明确要求本 PR 删除非加密运行,已移除 enable/V2 模式分支、明文/ECB 写入、机器 ID 与短密钥派生;配置、Snippet、Webhook 固定 GCM,API Key 固定哈希,OAuth 不再在数据库 hook 中解密后写入。旧数据读取、离线检查和维护迁移保留。旧 true 标记仅提示弃用,显式 false/无效标记拒绝启动;字段密钥和 JWT 同时必填。 - 定向 49 项测试通过,包括独立进程配置拒绝、无开关启动、历史 ECB、事务迁移与构建后 CLI 恢复演练。旧实现下新增启动测试有 6 项失败,固定 V2 写入测试亦先失败。 -- 完整 Core 回归:431 个文件、3573 项通过,11 项跳过;仍显式排除原有 Agent DI 夹具。JWT helper 另 4 项通过。 +- 完整 Core 回归:431 个文件、3576 项通过,11 项跳过;仍显式排除原有 Agent DI 夹具。JWT helper 另 4 项通过。 - Core 生产构建、Core/E2E 类型检查、API envelope/验证配对检查、50 个迁移 lint、冻结 lockfile 校验通过;构建产物已不含关闭加密报错或 machine-id 回退。 - 两份 Compose 均验证缺秘密拒绝、别名和主键优先级、app/migrate 同值;示例不再带关闭开关或公共生产秘密。PM2 与 CI/release 测试环境已同步。 - mx-docs 的 3 项测试、build/check/links/pangu/lint 通过,生成 llms 内容包含新要求;v17.0.4 页面明确固定历史行为,当前开发分支说明明确强制加密已实现。保留既有 hint 和死链。 + +最终独立审阅:无 Critical、1 项 Important、无 Minor。保留的旧凭据迁移会将历史 Better Auth hex 密文当作明文二次加密;已复用严格解封装逻辑,新增历史 token 保留/幂等、错误原 JWT、截断密文整批回滚 3 项行为回归,均先失败后通过。修复后完整回归为 431 个文件、3576 项通过、11 项跳过,排除项不变;重新构建和类型检查通过。未再次派发审阅,未操作生产。 From afb4d94fb53a02574e9a8fb6efa35ec5f8ca840f Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 00:04:34 +0800 Subject: [PATCH 18/26] db: migrate legacy configuration in the release phase --- .../20261109-canonical-config.ts | 34 +++++ .../src/database/app-migrations/registry.ts | 2 + apps/core/src/shared/config-format.ts | 2 + apps/core/src/upgrade/config-migration.ts | 96 +++++++++++++ apps/core/src/upgrade/preparation.ts | 74 +--------- .../canonical-config.migration.spec.ts | 135 ++++++++++++++++++ ...-11-08-mx-core-v18-compatibility-design.md | 2 + 7 files changed, 278 insertions(+), 67 deletions(-) create mode 100644 apps/core/src/database/app-migrations/20261109-canonical-config.ts create mode 100644 apps/core/src/shared/config-format.ts create mode 100644 apps/core/src/upgrade/config-migration.ts create mode 100644 apps/core/test/src/database/canonical-config.migration.spec.ts diff --git a/apps/core/src/database/app-migrations/20261109-canonical-config.ts b/apps/core/src/database/app-migrations/20261109-canonical-config.ts new file mode 100644 index 00000000000..c6c3ed49d06 --- /dev/null +++ b/apps/core/src/database/app-migrations/20261109-canonical-config.ts @@ -0,0 +1,34 @@ +import { ENCRYPT } from '~/app.config' +import { CONFIG_FORMAT_MIGRATION_ID } from '~/shared/config-format' +import { CONFIG_UPGRADE_LOCK } from '~/shared/config-lock' +import { migrateConfiguration } from '~/upgrade/config-migration' +import { SecretCodec } from '~/utils/secret-codec' + +import type { AppMigration } from './types' + +export const migration: AppMigration = { + id: CONFIG_FORMAT_MIGRATION_ID, + description: + 'Persist canonical configuration before removing runtime legacy conversion', + async up({ pool, logger }) { + const client = await pool.connect() + try { + await client.query('BEGIN') + await client.query("SET LOCAL lock_timeout = '10s'") + await client.query('SELECT pg_advisory_xact_lock($1::bigint)', [ + CONFIG_UPGRADE_LOCK, + ]) + const changed = await migrateConfiguration( + client, + new SecretCodec(ENCRYPT), + ) + await client.query('COMMIT') + logger.log(`Migrated ${changed} configuration records`) + } catch (error) { + await client.query('ROLLBACK') + throw error + } finally { + client.release() + } + }, +} diff --git a/apps/core/src/database/app-migrations/registry.ts b/apps/core/src/database/app-migrations/registry.ts index c32061c58e3..35dcd52fc5f 100644 --- a/apps/core/src/database/app-migrations/registry.ts +++ b/apps/core/src/database/app-migrations/registry.ts @@ -3,6 +3,7 @@ import { migration as treeContentRevisions } from './20260831-tree-content-revis import { migration as protectStoredSecrets } from './20261018-protect-stored-secrets' import { migration as mintLoroBlockIds } from './20261026-mint-loro-block-ids' import { migration as aiMetricsCoverage } from './20261108-ai-metrics-coverage' +import { migration as canonicalConfig } from './20261109-canonical-config' import type { AppMigration } from './types' /** @@ -19,4 +20,5 @@ export const migrations: AppMigration[] = [ ...(process.env.MX_MIGRATE_SECRETS === '1' ? [protectStoredSecrets] : []), mintLoroBlockIds, aiMetricsCoverage, + canonicalConfig, ] diff --git a/apps/core/src/shared/config-format.ts b/apps/core/src/shared/config-format.ts new file mode 100644 index 00000000000..e09c1131d2e --- /dev/null +++ b/apps/core/src/shared/config-format.ts @@ -0,0 +1,2 @@ +/** Required data migration before the current-only configuration runtime. */ +export const CONFIG_FORMAT_MIGRATION_ID = '20261109-canonical-config' diff --git a/apps/core/src/upgrade/config-migration.ts b/apps/core/src/upgrade/config-migration.ts new file mode 100644 index 00000000000..fd0325fcaeb --- /dev/null +++ b/apps/core/src/upgrade/config-migration.ts @@ -0,0 +1,96 @@ +import { isDeepStrictEqual } from 'node:util' + +import { SnowflakeGenerator } from '@mx-space/db-schema/id' +import type { PoolClient } from 'pg' + +import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' +import { + canonicalConfigRows, + normalizeLegacyConfig, +} from '../modules/configs/legacy-config.util' +import type { SecretCodec } from '../utils/secret-codec' + +export class ConfigMigrationError extends Error { + constructor( + public readonly code: string, + public readonly path?: string, + ) { + super(path ? `${code} (${path})` : code) + } +} + +/** Caller owns the transaction and CONFIG_UPGRADE_LOCK. No business-service imports. */ +export async function migrateConfiguration( + client: Pick, + codec: SecretCodec, +) { + codec.assertConfigured() + const ids = new SnowflakeGenerator({ workerId: 1023 }) + let changed = 0 + const decrypt = (secret: string, path: string) => { + try { + return codec.decrypt(secret, path) + } catch { + throw new ConfigMigrationError('SECRET_UNREADABLE', path) + } + } + const rows = ( + await client.query( + 'SELECT id, name, value FROM options ORDER BY name FOR UPDATE', + ) + ).rows + const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) + for (const key of [ + 'ai', + 'membership', + 'thirdPartyServiceIntegration', + 'imageGenerationOptions', + 'ttsOptions', + ]) { + if ( + key in raw && + (!raw[key] || typeof raw[key] !== 'object' || Array.isArray(raw[key])) + ) + throw new ConfigMigrationError('CONFIG_SHAPE_UNSUPPORTED', key) + } + const plaintext = Object.fromEntries( + rows.map((row) => [ + row.name, + visitConfigSecrets(row.value, row.name, (secret, path) => + decrypt(secret, path), + ), + ]), + ) + const canonical = canonicalConfigRows(normalizeLegacyConfig(plaintext)) + const names = new Set(rows.map((row) => row.name)) + if (names.has('imageGenerationOptions') || names.has('ttsOptions')) + names.add('ai') + names.delete('imageGenerationOptions') + names.delete('ttsOptions') + for (const name of names) { + const value = canonical[name] ?? plaintext[name] + const encrypted = visitConfigSecrets(value, name, (secret, path) => { + const previous = path + .split('.') + .reduce((node, part) => node?.[part], raw) + return typeof previous === 'string' && + codec.isEncryptedString(previous) && + decrypt(previous, path) === secret + ? codec.encrypt(previous, path) + : codec.encrypt(secret, path) + }) + if (isDeepStrictEqual(encrypted, raw[name])) continue + await client.query( + 'INSERT INTO options (id, name, value) VALUES ($1, $2, $3) ON CONFLICT (name) DO UPDATE SET value = excluded.value', + [ids.nextId(), name, JSON.stringify(encrypted)], + ) + changed++ + } + changed += + ( + await client.query( + "DELETE FROM options WHERE name IN ('imageGenerationOptions', 'ttsOptions')", + ) + ).rowCount ?? 0 + return changed +} diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts index c059364e2ea..15a9f17e92c 100644 --- a/apps/core/src/upgrade/preparation.ts +++ b/apps/core/src/upgrade/preparation.ts @@ -1,18 +1,12 @@ -import { isDeepStrictEqual } from 'node:util' - import { defaultKeyHasher } from '@better-auth/api-key' -import { SnowflakeGenerator } from '@mx-space/db-schema/id' import { symmetricEncrypt } from 'better-auth/crypto' import type { Pool } from 'pg' import { backfillAiMetrics } from '../database/app-migrations/20261108-ai-metrics-coverage' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' -import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' -import { - canonicalConfigRows, - normalizeLegacyConfig, -} from '../modules/configs/legacy-config.util' +import { CONFIG_FORMAT_MIGRATION_ID } from '../shared/config-format' import { SecretCodec } from '../utils/secret-codec' +import { migrateConfiguration } from './config-migration' import { unwrapOAuthToken } from './oauth-token' import { CONFIG_UPGRADE_LOCK, @@ -40,7 +34,6 @@ export async function prepareUpgrade( throw new UpgradeError('REPLICA_CONFIRMATION_REQUIRED') const client = await pool.connect() const codec = new SecretCodec(config) - const ids = new SnowflakeGenerator({ workerId: 1023 }) let changed = 0 try { await client.query('BEGIN') @@ -57,64 +50,7 @@ export async function prepareUpgrade( ]) const blocking = before.issues.find((issue) => !allowed.has(issue.code)) if (blocking) throw new UpgradeError(blocking.code, blocking.path) - const rows = ( - await client.query( - 'SELECT id, name, value FROM options ORDER BY name FOR UPDATE', - ) - ).rows - const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) - for (const key of [ - 'ai', - 'membership', - 'thirdPartyServiceIntegration', - 'imageGenerationOptions', - 'ttsOptions', - ]) { - if ( - key in raw && - (!raw[key] || typeof raw[key] !== 'object' || Array.isArray(raw[key])) - ) - throw new UpgradeError('CONFIG_SHAPE_UNSUPPORTED', key) - } - const plaintext = Object.fromEntries( - rows.map((row) => [ - row.name, - visitConfigSecrets(row.value, row.name, (secret, path) => - codec.decrypt(secret, path), - ), - ]), - ) - const canonical = canonicalConfigRows(normalizeLegacyConfig(plaintext)) - const names = new Set(rows.map((row) => row.name)) - if (names.has('imageGenerationOptions') || names.has('ttsOptions')) - names.add('ai') - names.delete('imageGenerationOptions') - names.delete('ttsOptions') - for (const name of names) { - const value = canonical[name] ?? plaintext[name] - const encrypted = visitConfigSecrets(value, name, (secret, path) => { - const previous = path - .split('.') - .reduce((node, part) => node?.[part], raw) - return typeof previous === 'string' && - codec.isEncryptedString(previous) && - codec.decrypt(previous, path) === secret - ? codec.encrypt(previous, path) - : codec.encrypt(secret, path) - }) - if (isDeepStrictEqual(encrypted, raw[name])) continue - await client.query( - 'INSERT INTO options (id, name, value) VALUES ($1, $2, $3) ON CONFLICT (name) DO UPDATE SET value = excluded.value', - [ids.nextId(), name, JSON.stringify(encrypted)], - ) - changed++ - } - changed += - ( - await client.query( - "DELETE FROM options WHERE name IN ('imageGenerationOptions', 'ttsOptions')", - ) - ).rowCount ?? 0 + changed += await migrateConfiguration(client, codec) for (const table of ['snippets', 'webhooks']) { for (const row of ( await client.query(`SELECT id, secret FROM ${table} FOR UPDATE`) @@ -178,6 +114,10 @@ export async function prepareUpgrade( [PREPARED_MIGRATION_ID], ) } + await client.query( + 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', + [CONFIG_FORMAT_MIGRATION_ID], + ) await client.query(options.dryRun ? 'ROLLBACK' : 'COMMIT') return { status: options.dryRun ? 'would-be-ready' : 'prepared', diff --git a/apps/core/test/src/database/canonical-config.migration.spec.ts b/apps/core/test/src/database/canonical-config.migration.spec.ts new file mode 100644 index 00000000000..02b7fd6b0d7 --- /dev/null +++ b/apps/core/test/src/database/canonical-config.migration.spec.ts @@ -0,0 +1,135 @@ +import { createPgTestDatabase } from 'test/helper/pg-verify-url' + +import { AppMigrationsService } from '~/database/app-migrations/app-migrations.service' +import { EncryptUtil } from '~/utils/encrypt.util' + +const migrationId = '20261109-canonical-config' + +it('automatically migrates old configuration rows before release startup and preserves credentials on rerun', async () => { + const context = await createPgTestDatabase('mx_canonical_config') + try { + await context.pool.query( + `INSERT INTO options (id,name,value) VALUES ('100','membership',$1), ('101','thirdPartyServiceIntegration',$2), ('102','imageGenerationOptions',$3), ('103','ttsOptions',$4)`, + [ + { + enabled: true, + dodoApiKey: EncryptUtil.encrypt( + 'billing-private', + 'membership.dodoApiKey', + ), + }, + { + githubToken: EncryptUtil.encrypt( + 'github-private', + 'thirdPartyServiceIntegration.githubToken', + ), + }, + { + enable: true, + apiKey: EncryptUtil.encrypt( + 'media-private', + 'imageGenerationOptions.apiKey', + ), + endpoint: 'https://api.example.test/v1', + model: 'image-model', + }, + { + enable: true, + apiKey: 'media-private', + endpoint: 'https://api.example.test/v1', + model: 'speech-model', + voice: 'retained-voice', + }, + ], + ) + const runner = new AppMigrationsService(context.db, context.pool) + await runner.run() + const rows = ( + await context.pool.query('SELECT * FROM options ORDER BY name') + ).rows + const config = Object.fromEntries(rows.map((row) => [row.name, row.value])) + expect(config).not.toHaveProperty('imageGenerationOptions') + expect(config).not.toHaveProperty('ttsOptions') + expect(config.membership).not.toHaveProperty('dodoApiKey') + expect( + EncryptUtil.decrypt(config.membership.apiKey, 'membership.apiKey'), + ).toBe('billing-private') + expect( + EncryptUtil.decrypt( + config.thirdPartyServiceIntegration.github.token, + 'thirdPartyServiceIntegration.github.token', + ), + ).toBe('github-private') + expect(config.ai.providers).toHaveLength(1) + expect( + EncryptUtil.decrypt( + config.ai.providers[0].apiKey, + 'ai.providers.0.apiKey', + ), + ).toBe('media-private') + expect(config.ai.imageGeneration.model.model).toBe('image-model') + expect(config.ai.tts).toMatchObject({ + enable: true, + model: { model: 'speech-model' }, + voice: 'retained-voice', + }) + expect( + ( + await context.pool.query('SELECT id FROM _app_migrations WHERE id=$1', [ + migrationId, + ]) + ).rowCount, + ).toBe(1) + await runner.run() + expect( + (await context.pool.query('SELECT * FROM options ORDER BY name')).rows, + ).toEqual(rows) + } finally { + await context.close() + } +}) + +it.each(['conflict', 'bad-ciphertext'])( + 'blocks release and rolls back every configuration row on %s', + async (invalid) => { + const context = await createPgTestDatabase('mx_bad_config') + try { + await context.pool.query( + "INSERT INTO options (id,name,value) VALUES ('100','membership',$1), ('101','thirdPartyServiceIntegration',$2)", + [ + { + dodoApiKey: 'old-billing', + ...(invalid === 'conflict' ? { apiKey: 'different-billing' } : {}), + }, + { + githubToken: + invalid === 'bad-ciphertext' + ? '$${mx}$$v2.invalid.envelope' + : 'github-private', + }, + ], + ) + const before = ( + await context.pool.query('SELECT * FROM options ORDER BY name') + ).rows + await expect( + new AppMigrationsService(context.db, context.pool).run(), + ).rejects.toThrow( + invalid === 'conflict' ? 'CONFIG_SECRET_CONFLICT' : 'SECRET_UNREADABLE', + ) + expect( + (await context.pool.query('SELECT * FROM options ORDER BY name')).rows, + ).toEqual(before) + expect( + ( + await context.pool.query( + 'SELECT id FROM _app_migrations WHERE id=$1', + [migrationId], + ) + ).rowCount, + ).toBe(0) + } finally { + await context.close() + } + }, +) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 02b6df3810f..308e616d325 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -4,6 +4,8 @@ 修订(2026-11-08):用户明确要求本 PR 立即移除非加密运行和旧格式写入,覆盖下文“准备阶段保留逃生口”的安排。显式字段密钥、至少 32 字符 JWT 和固定 V2 写入在本分支立即生效;旧 true 开关仅作为弃用配置接受,false 拒绝启动。历史读取与显式迁移保留。其余 API、SDK 6 和数据库列删除仍按后续阶段执行,当前分支尚未发布。 +修订(2026-11-09):用户明确要求通过 DB 迁移一次性结束旧配置运行时兼容。本 PR 新增必跑的发布前配置数据迁移;先按原路径解密、转换并按新路径加密,删除旧配置行,成功后记录迁移账本。正常 ConfigsService、schema 和 PATCH 只接受规范结构;旧字段按 schema 严格拒绝。历史转换器只供发布迁移与离线 check/prepare、历史导入恢复使用。此配置契约切换需要先停止旧副本及后台写入者,不能把此 contract 当作任意 v17 副本仍在写时的滚动迁移;不涉及删除数据库物理列或其他 API 变更。 + 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 ## 意图 From 40cb6f0cf3da152c3294e7f94abeb9c93df3e6f2 Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 00:34:41 +0800 Subject: [PATCH 19/26] feat(config): remove runtime legacy configuration compatibility --- apps/core/README.md | 4 +- .../src/modules/configs/configs.current.ts | 106 ++++ .../src/modules/configs/configs.default.ts | 30 +- .../src/modules/configs/configs.interface.ts | 4 - .../src/modules/configs/configs.schema.ts | 117 ---- .../modules/configs/configs.secret-paths.ts | 18 +- .../src/modules/configs/configs.service.ts | 138 ++--- .../src/modules/configs/options.repository.ts | 12 +- apps/core/src/upgrade/config-migration.ts | 24 +- .../legacy-config.ts} | 156 +++--- apps/core/src/upgrade/preparation.ts | 10 +- apps/core/src/upgrade/readiness.ts | 50 +- apps/core/test/helper/pg-verify-url.ts | 8 + .../canonical-config.migration.spec.ts | 35 +- .../database/protect-stored-secrets.spec.ts | 4 +- .../modules/configs/configs.service.spec.ts | 501 +----------------- .../configs/configs.transaction.spec.ts | 12 +- .../modules/configs/current-config.spec.ts | 90 ++++ .../src/modules/configs/legacy-config.spec.ts | 53 +- apps/core/test/src/upgrade/readiness.spec.ts | 8 + ...-11-08-mx-core-v18-compatibility-design.md | 18 +- docs/security-secret-storage.md | 13 + docs/v18-preparation.md | 38 +- packages/api-client/models/setting.ts | 33 +- packages/e2e/src/helpers/e2e-app.ts | 3 + 25 files changed, 545 insertions(+), 940 deletions(-) create mode 100644 apps/core/src/modules/configs/configs.current.ts rename apps/core/src/{modules/configs/legacy-config.util.ts => upgrade/legacy-config.ts} (73%) create mode 100644 apps/core/test/src/modules/configs/current-config.spec.ts diff --git a/apps/core/README.md b/apps/core/README.md index 78d4cbaa18f..88c979e3368 100644 --- a/apps/core/README.md +++ b/apps/core/README.md @@ -246,7 +246,9 @@ pnpm test:watch # watch mode | `TZ` | Timezone | `Asia/Shanghai` | | `DISABLE_CACHE` | Disable Redis caching | `false` | -On this unreleased branch, field encryption and V2 writes are mandatory; `JWT_SECRET` must have at least 32 characters. See the [upgrade guide](../../docs/v18-preparation.md) before deploying to an existing instance. +On this unreleased branch, run the mandatory configuration data migration via `node migrate.mjs` before startup, after stopping all old replicas/background writers. Runtime configuration accepts only the current schema; retired sections and unknown declared fields are rejected. See the [migration guide](../../docs/v18-preparation.md). + +Field encryption and V2 writes are mandatory; `JWT_SECRET` must have at least 32 characters. See the [upgrade guide](../../docs/v18-preparation.md) before deploying to an existing instance. For managed Redis providers such as Upstash, pass the provider URI as `REDIS_CONNECTION_STRING=rediss://user:password@host:port`. The `rediss://` diff --git a/apps/core/src/modules/configs/configs.current.ts b/apps/core/src/modules/configs/configs.current.ts new file mode 100644 index 00000000000..af9f71e5a1f --- /dev/null +++ b/apps/core/src/modules/configs/configs.current.ts @@ -0,0 +1,106 @@ +import { cloneDeep } from 'es-toolkit/compat' +import type { z } from 'zod' + +import { generateDefaultConfig } from './configs.default' +import type { IConfig } from './configs.interface' +import { configSchemaMapping } from './configs.schema' +import { isEncryptedPath } from './configs.secret-paths' + +function unwrap(schema: z.ZodType): any { + let def = (schema as any)?._zod?.def + while (def?.innerType || (def?.type === 'pipe' && def.in)) { + schema = def.innerType ?? def.in + def = (schema as any)._zod?.def + } + return def +} + +/** Normalize declared keys only. Record/freeform payloads remain verbatim. */ +export function normalizeConfigPatch(schema: z.ZodType, value: any): any { + const def = unwrap(schema) + if (def?.type === 'array' && Array.isArray(value)) + return value.map((item) => normalizeConfigPatch(def.element, item)) + if ( + def?.type !== 'object' || + !value || + typeof value !== 'object' || + Array.isArray(value) + ) + return value + const shape = typeof def.shape === 'function' ? def.shape() : def.shape + return Object.fromEntries( + Object.entries(value).map(([key, item]) => { + const name = key.replaceAll(/_(\w)/g, (_, c) => c.toUpperCase()) + return [ + name, + shape[name] ? normalizeConfigPatch(shape[name], item) : item, + ] + }), + ) +} + +export function unknownConfigPaths( + schema: z.ZodType, + value: any, + prefix: string, +): string[] { + const def = unwrap(schema) + if (def?.type === 'array' && Array.isArray(value)) + return value.flatMap((item, index) => + unknownConfigPaths(def.element, item, `${prefix}.${index}`), + ) + if ( + def?.type !== 'object' || + !value || + typeof value !== 'object' || + Array.isArray(value) + ) + return [] + const shape = typeof def.shape === 'function' ? def.shape() : def.shape + return Object.entries(value).flatMap(([key, item]) => { + const path = `${prefix}.${key}` + return Object.hasOwn(shape, key) + ? unknownConfigPaths(shape[key], item, path) + : [path] + }) +} + +/** No historical transforms: a missed migration must fail before cache publication. */ +export function loadCurrentConfig(raw: Record): IConfig { + const requireMigration = (path: string): never => { + throw new Error( + `Configuration data migration is required (${path}); run node migrate.mjs before starting the app`, + ) + } + for (const name of ['imageGenerationOptions', 'ttsOptions']) + if (name in raw) requireMigration(name) + const config = generateDefaultConfig() + for (const [key, schema] of Object.entries(configSchemaMapping)) { + if (!(key in raw)) continue + const value = raw[key] + if (!value || typeof value !== 'object' || Array.isArray(value)) + requireMigration(key) + if (key === 'ai' && value.version !== 2) requireMigration('ai.version') + const unknown = unknownConfigPaths(schema, value, key) + if (unknown.length) requireMigration(unknown[0]) + ;(config as any)[key] = { ...config[key], ...cloneDeep(value) } + } + return config +} + +/** Empty optional credentials are legitimate stored defaults, not corrupt data. */ +export function isValidStoredConfig( + key: keyof typeof configSchemaMapping, + value: unknown, +): boolean { + const result = configSchemaMapping[key].safeParse(value) + if (result.success) return true + return result.error.issues.every((issue) => { + const field = issue.path.reduce((node, part) => node?.[part], value) + return ( + isEncryptedPath([key, ...issue.path].join('.')) && + ((issue.code === 'too_small' && field === '') || + (issue.code === 'invalid_type' && field === null)) + ) + }) +} diff --git a/apps/core/src/modules/configs/configs.default.ts b/apps/core/src/modules/configs/configs.default.ts index 82f799597eb..6be6d0c2fad 100644 --- a/apps/core/src/modules/configs/configs.default.ts +++ b/apps/core/src/modules/configs/configs.default.ts @@ -22,11 +22,9 @@ export const generateDefaultConfig: () => IConfig = () => ({ smtp: { user: '', pass: '', - options: { - host: '', - port: 465, - secure: true, - }, + host: '', + port: 465, + secure: true, }, resend: { apiKey: '', @@ -84,28 +82,6 @@ export const generateDefaultConfig: () => IConfig = () => ({ prefix: '', commentUploadPrefix: '', }, - imageGenerationOptions: { - enable: false, - provider: 'openrouter', - apiKey: '', - endpoint: '', - model: '', - defaultAspectRatio: '16:9', - defaultQuality: 'standard', - defaultFormat: 'png', - }, - ttsOptions: { - enable: false, - provider: 'openrouter', - apiKey: '', - endpoint: '', - model: '', - voice: '', - speed: 1, - maxCharsPerChunk: 1800, - concurrency: 3, - maxCharsPerRun: 120000, - }, fileUploadOptions: { enableCustomNaming: false, filenameTemplate: '{Y}{m}{d}/{md5-16}{ext}', diff --git a/apps/core/src/modules/configs/configs.interface.ts b/apps/core/src/modules/configs/configs.interface.ts index 5f22f089ad1..fc882ecccd9 100644 --- a/apps/core/src/modules/configs/configs.interface.ts +++ b/apps/core/src/modules/configs/configs.interface.ts @@ -15,7 +15,6 @@ import { type FeatureListSchema, type FileUploadOptionsSchema, type FriendLinkOptionsSchema, - type ImageGenerationOptionsSchema, type ImageStorageOptionsSchema, type MailOptionsSchema, type MediaProcessingOptionsSchema, @@ -23,7 +22,6 @@ import { type OAuthSchema, type SeoSchema, type ThirdPartyServiceIntegrationSchema, - type TtsOptionsSchema, type UrlSchema, } from './configs.schema' @@ -46,8 +44,6 @@ export abstract class IConfig { friendLinkOptions: Required> backupOptions: Required> imageStorageOptions: Required> - imageGenerationOptions: Required> - ttsOptions: Required> fileUploadOptions: Required> mediaProcessingOptions: z.infer commentUploadOptions: Required> diff --git a/apps/core/src/modules/configs/configs.schema.ts b/apps/core/src/modules/configs/configs.schema.ts index 24f4fd2e82e..80bbf4f111a 100644 --- a/apps/core/src/modules/configs/configs.schema.ts +++ b/apps/core/src/modules/configs/configs.schema.ts @@ -259,121 +259,6 @@ export const ImageStorageOptionsSchema = section('Image storage', { ), }) -// ==================== Image Generation Options ==================== -export const ImageGenerationOptionsSchema = section('AI image generation', { - enable: field.toggle(z.boolean().optional(), 'Enable AI image generation'), - provider: field.select( - z.enum(['openrouter', 'custom']).optional().default('openrouter'), - 'Provider', - [ - { label: 'OpenRouter', value: 'openrouter' }, - { label: 'Custom', value: 'custom' }, - ], - { - description: - 'Endpoint preset — determines where generation requests are sent. "custom" requires Endpoint to be set.', - }, - ), - apiKey: field.password( - z - .string() - .nullable() - .optional() - .transform((v) => (v === null ? '' : v)), - 'API Key', - ), - endpoint: field.plain( - z - .string() - .nullable() - .optional() - .transform((v) => (v === null ? '' : v)), - 'Endpoint', - { 'ui:options': { showWhen: { provider: 'custom' } } }, - ), - model: field.plain( - z - .string() - .nullable() - .optional() - .transform((v) => (v === null ? '' : v)), - 'Model', - { description: 'Image generation model id' }, - ), - defaultAspectRatio: field.halfGrid( - z.string().optional().default('16:9'), - 'Default aspect ratio', - ), - defaultQuality: field.select( - z.enum(['low', 'standard', 'high']).optional().default('standard'), - 'Default quality', - [ - { label: 'Low', value: 'low' }, - { label: 'Standard', value: 'standard' }, - { label: 'High', value: 'high' }, - ], - { 'ui:options': { halfGrid: true } }, - ), - defaultFormat: field.select( - z.enum(['png', 'jpeg', 'webp']).optional().default('png'), - 'Default format', - [ - { label: 'PNG', value: 'png' }, - { label: 'JPEG', value: 'jpeg' }, - { label: 'WebP', value: 'webp' }, - ], - { 'ui:options': { halfGrid: true } }, - ), -}) - -// ==================== TTS Options ==================== -export const TtsOptionsSchema = section('AI text to speech', { - enable: field.toggle(z.boolean().optional(), 'Enable AI narration'), - provider: field.select( - z.enum(['openrouter', 'openai', 'custom']).optional().default('openrouter'), - 'Provider', - [ - { label: 'OpenRouter', value: 'openrouter' }, - { label: 'OpenAI', value: 'openai' }, - { label: 'Custom', value: 'custom' }, - ], - { - description: - 'Endpoint preset. "custom" requires Endpoint to be set. The adapter appends /audio/speech.', - }, - ), - apiKey: field.password(nullableStorageText(), 'API Key'), - endpoint: field.plain(nullableStorageText(), 'Endpoint', { - 'ui:options': { showWhen: { provider: 'custom' } }, - }), - model: field.plain(nullableStorageText(), 'Model', { - description: 'Speech model id, e.g. openai/gpt-4o-mini-tts-2025-12-15', - }), - voice: field.plain(nullableStorageText(), 'Voice', { - description: 'Voice id. Availability depends on the model.', - }), - speed: field.number( - z.coerce.number().min(0.25).max(4).optional().default(1), - 'Speed', - { 'ui:options': { halfGrid: true } }, - ), - maxCharsPerChunk: field.number( - z.coerce.number().int().min(200).max(4000).optional().default(1800), - 'Max characters per request', - { 'ui:options': { halfGrid: true } }, - ), - concurrency: field.number( - z.coerce.number().int().min(1).max(8).optional().default(3), - 'Concurrent requests per task', - { 'ui:options': { halfGrid: true } }, - ), - maxCharsPerRun: field.number( - z.coerce.number().int().min(1000).max(1000000).optional().default(120000), - 'Max characters per run', - { 'ui:options': { halfGrid: true } }, - ), -}) - // ==================== Comment Upload Options ==================== export const DraftCleanupOptionsSchema = section('Draft cleanup', { enable: field.toggle( @@ -1269,8 +1154,6 @@ export const configSchemaMapping = { friendLinkOptions: FriendLinkOptionsSchema, backupOptions: BackupOptionsSchema, imageStorageOptions: ImageStorageOptionsSchema, - imageGenerationOptions: ImageGenerationOptionsSchema, - ttsOptions: TtsOptionsSchema, fileUploadOptions: FileUploadOptionsSchema, mediaProcessingOptions: MediaProcessingOptionsSchema, commentUploadOptions: CommentUploadOptionsSchema, diff --git a/apps/core/src/modules/configs/configs.secret-paths.ts b/apps/core/src/modules/configs/configs.secret-paths.ts index 0fdade2193e..4ec94d5b4fc 100644 --- a/apps/core/src/modules/configs/configs.secret-paths.ts +++ b/apps/core/src/modules/configs/configs.secret-paths.ts @@ -14,11 +14,7 @@ function getEncryptedPaths(): Set { return cachedEncryptedPaths } - cachedEncryptedPaths = new Set([ - 'thirdPartyServiceIntegration.githubToken', - 'membership.dodoApiKey', - 'membership.dodoWebhookKey', - ]) + cachedEncryptedPaths = new Set() for (const [sectionKey, schema] of Object.entries(configSchemaMapping)) { const paths = collectEncryptedPathsFromSchema(schema, sectionKey) @@ -213,18 +209,24 @@ export function visitConfigSecrets( value: T, prefix: string, visit: (secret: string, path: string) => string, + isSecretPath: (path: string) => boolean = isEncryptedPath, ): T { if (typeof value === 'string') - return (isEncryptedPath(prefix) ? visit(value, prefix) : value) as T + return (isSecretPath(prefix) ? visit(value, prefix) : value) as T if (Array.isArray(value)) return value.map((item, index) => - visitConfigSecrets(item, `${prefix}.${index}`, visit), + visitConfigSecrets(item, `${prefix}.${index}`, visit, isSecretPath), ) as T if (value && typeof value === 'object') return Object.fromEntries( Object.entries(value).map(([key, item]) => [ key, - visitConfigSecrets(item, prefix ? `${prefix}.${key}` : key, visit), + visitConfigSecrets( + item, + prefix ? `${prefix}.${key}` : key, + visit, + isSecretPath, + ), ]), ) as T return value diff --git a/apps/core/src/modules/configs/configs.service.ts b/apps/core/src/modules/configs/configs.service.ts index aee03120631..5377d168af0 100644 --- a/apps/core/src/modules/configs/configs.service.ts +++ b/apps/core/src/modules/configs/configs.service.ts @@ -19,9 +19,13 @@ import { } from '~/processors/redis/config-version.service' import { RedisService } from '~/processors/redis/redis.service' import { getRedisKey } from '~/utils/redis.util' -import { camelcaseKeys } from '~/utils/tool.util' import { backupEncryptionPublicKey } from '../backup/backup-crypto' +import { + loadCurrentConfig, + normalizeConfigPatch, + unknownConfigPaths, +} from './configs.current' import { generateDefaultConfig } from './configs.default' import { decryptObject, @@ -31,14 +35,6 @@ import { } from './configs.encrypt.util' import { configDtoMapping, IConfig } from './configs.interface' import type { OAuthConfig } from './configs.schema' -import { - canonicalConfigRows, - LegacyConfigConflictError, - legacySecretConflicts, - normalizeLegacyConfig, - normalizeLegacyPatch, - updateLegacyMedia, -} from './legacy-config.util' import { OptionsRepository } from './options.repository' const aggregateConfigKeys = new Set([ @@ -79,7 +75,6 @@ export class ConfigsService implements OnModuleInit { private readonly logger = new Logger(ConfigsService.name) private stageCache = false private stagedConfig?: IConfig - private sourceConfig: Record = {} private configInitd = false private configInitPromise?: Promise @@ -168,16 +163,17 @@ export class ConfigsService implements OnModuleInit { const rows = await this.optionsRepository.findAll() const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) const plaintext = decryptObject(cloneDeep(raw)) - this.sourceConfig = cloneDeep(plaintext) if (isDev) delete plaintext.url - await this.setConfig( - normalizeLegacyConfig(plaintext, { allowConflicts: true }), - ) + await this.setConfig(loadCurrentConfig(plaintext)) } public async get( key: T, ): Promise> { + if (!Object.hasOwn(configDtoMapping, key)) + throw createAppException(AppErrorCode.CONFIG_NOT_FOUND, { + id: key as string, + }) const config = await this.getStoredConfig() return decryptObject(config[key], key) } @@ -193,7 +189,7 @@ export class ConfigsService implements OnModuleInit { if (configCache) { try { - const configValue = JSON.parse(configCache) as IConfig + const configValue = loadCurrentConfig(JSON.parse(configCache)) return configValue } catch (error) { await this.ensureConfigInitialized(true) @@ -227,6 +223,10 @@ export class ConfigsService implements OnModuleInit { public async getForResponse( key: T, ): Promise> { + if (!Object.hasOwn(configDtoMapping, key)) + throw createAppException(AppErrorCode.CONFIG_NOT_FOUND, { + id: key as string, + }) const config = await this.getStoredConfig() const value = config[key] return sanitizeConfigForResponse(value as object, key) as IConfig[T] @@ -237,34 +237,6 @@ export class ConfigsService implements OnModuleInit { data: Partial, ): Promise { const config = await this.getConfig() - const conflicts = legacySecretConflicts(key, this.sourceConfig[key] ?? {}) - for (const path of conflicts) { - const replacement = path - .split('.') - .reduce((node, part) => node?.[part], data) - if (typeof replacement !== 'string' || !replacement) - throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { - message: `CONFIG_SECRET_CONFLICT (${key}.${path}); explicitly replace the canonical credential to resolve it`, - }) - } - if (key === 'imageGenerationOptions' || key === 'ttsOptions') { - const canonical = updateLegacyMedia(config as IConfig, key, data) - const storedAi = await this.optionsRepository.upsert( - 'ai', - encryptObject(cloneDeep(canonical.ai), 'ai'), - ) - await this.optionsRepository.deleteByName?.('imageGenerationOptions') - await this.optionsRepository.deleteByName?.('ttsOptions') - await this.setConfig(canonical) - await this.eventManager.emit( - EventBusEvents.ConfigChanged, - storedAi.value, - { - scope: EventScope.TO_SYSTEM, - }, - ) - return encryptObject(cloneDeep(canonical[key]), key) as IConfig[T] - } const updatedConfigRow = await this.optionsRepository.upsert( key as string, encryptObject( @@ -290,14 +262,10 @@ export class ConfigsService implements OnModuleInit { ), ) const newData = updatedConfigRow.value as IConfig[T] - this.sourceConfig[key] = decryptObject(cloneDeep(newData), key) - const mergedFullConfig = normalizeLegacyConfig( - Object.assign( - {}, - key === 'ai' ? canonicalConfigRows(config as IConfig) : config, - { [key]: decryptObject(cloneDeep(newData), key) }, - ), - ) + const mergedFullConfig = loadCurrentConfig({ + ...config, + [key]: decryptObject(cloneDeep(newData), key), + }) await this.setConfig(mergedFullConfig) this.eventManager.emit( @@ -378,15 +346,19 @@ export class ConfigsService implements OnModuleInit { key: T, value: Partial, ) { - try { - value = normalizeLegacyPatch(key, camelcaseKeys(value)) as any - } catch (error) { - if (error instanceof LegacyConfigConflictError) - throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { - message: error.message, - }) - throw error - } + const dto = Object.hasOwn(configDtoMapping, key) + ? configDtoMapping[key] + : undefined + if (!dto) + throw createAppException(AppErrorCode.CONFIG_NOT_FOUND, { + id: key as string, + }) + value = normalizeConfigPatch(dto, value) + const unknown = unknownConfigPaths(dto, value, key) + if (unknown.length) + throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { + message: `Unknown configuration fields: ${unknown.join(', ')}`, + }) value = removeEmptyEncryptedFields(value as object, key) as Partial< IConfig[T] > @@ -399,12 +371,6 @@ export class ConfigsService implements OnModuleInit { value = await this.hydrateAiProviderApiKeys(value as any) } - const dto = configDtoMapping[key] - if (!dto) { - throw createAppException(AppErrorCode.CONFIG_NOT_FOUND, { - id: key as string, - }) - } // If this is the comment settings and AI review is being enabled, validate the AI config if (key === 'commentOptions' && (value as any).aiReview === true) { const aiConfig = await this.get('ai') @@ -449,22 +415,6 @@ export class ConfigsService implements OnModuleInit { this.validateMailProvider(nextConfig) } - if (key === 'imageGenerationOptions') { - const nextConfig = await this.buildNextConfigForValidation( - key, - instanceValue, - ) - this.validateImageGenerationProvider(nextConfig) - } - - if (key === 'ttsOptions') { - const nextConfig = await this.buildNextConfigForValidation( - key, - instanceValue, - ) - this.validateTtsProvider(nextConfig) - } - switch (key) { case 'url': { const newValue = await this.patch(key, instanceValue as any) @@ -580,30 +530,6 @@ export class ConfigsService implements OnModuleInit { } } - private validateImageGenerationProvider(config: IConfig) { - const { imageGenerationOptions } = config - - if ( - imageGenerationOptions.provider === 'custom' && - !imageGenerationOptions.endpoint - ) { - throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { - message: - 'imageGenerationOptions.endpoint: required when provider is "custom"', - }) - } - } - - private validateTtsProvider(config: IConfig) { - const { ttsOptions } = config - - if (ttsOptions.provider === 'custom' && !ttsOptions.endpoint) { - throw createAppException(AppErrorCode.CONFIG_VALIDATION_FAILED, { - message: 'ttsOptions.endpoint: required when provider is "custom"', - }) - } - } - private validateAiFeatureRoutes(aiConfig: IConfig['ai']) { const providerIds = new Set() for (const provider of aiConfig.providers ?? []) { diff --git a/apps/core/src/modules/configs/options.repository.ts b/apps/core/src/modules/configs/options.repository.ts index 2eba5c0416b..2e63e4f1200 100644 --- a/apps/core/src/modules/configs/options.repository.ts +++ b/apps/core/src/modules/configs/options.repository.ts @@ -4,12 +4,13 @@ import { drizzle } from 'drizzle-orm/node-postgres' import type { Pool } from 'pg' import { PG_DB_TOKEN } from '~/constants/system.constant' -import { options } from '~/database/schema' +import { appMigrations, options } from '~/database/schema' import { BaseRepository, toEntityId, } from '~/processors/database/base.repository' import type { AppDatabase } from '~/processors/database/postgres.provider' +import { CONFIG_FORMAT_MIGRATION_ID } from '~/shared/config-format' import { CONFIG_UPGRADE_LOCK } from '~/shared/config-lock' import { type EntityId } from '~/shared/id/entity-id' import { SnowflakeService } from '~/shared/id/snowflake.service' @@ -72,6 +73,15 @@ export class OptionsRepository extends BaseRepository { } async findAll(): Promise { + const [current] = await this.db + .select({ id: appMigrations.id }) + .from(appMigrations) + .where(eq(appMigrations.id, CONFIG_FORMAT_MIGRATION_ID)) + .limit(1) + if (!current) + throw new Error( + 'Configuration data migration is required; run node migrate.mjs before starting the app', + ) const rows = await this.db.select().from(options).orderBy(options.name) return rows.map(mapRow) } diff --git a/apps/core/src/upgrade/config-migration.ts b/apps/core/src/upgrade/config-migration.ts index fd0325fcaeb..400b21d7325 100644 --- a/apps/core/src/upgrade/config-migration.ts +++ b/apps/core/src/upgrade/config-migration.ts @@ -3,12 +3,16 @@ import { isDeepStrictEqual } from 'node:util' import { SnowflakeGenerator } from '@mx-space/db-schema/id' import type { PoolClient } from 'pg' -import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' import { - canonicalConfigRows, - normalizeLegacyConfig, -} from '../modules/configs/legacy-config.util' + isValidStoredConfig, + loadCurrentConfig, +} from '../modules/configs/configs.current' +import { configSchemaMapping } from '../modules/configs/configs.schema' import type { SecretCodec } from '../utils/secret-codec' +import { + normalizeLegacyConfig, + visitLegacyConfigSecrets, +} from './legacy-config' export class ConfigMigrationError extends Error { constructor( @@ -56,12 +60,18 @@ export async function migrateConfiguration( const plaintext = Object.fromEntries( rows.map((row) => [ row.name, - visitConfigSecrets(row.value, row.name, (secret, path) => + visitLegacyConfigSecrets(row.value, row.name, (secret, path) => decrypt(secret, path), ), ]), ) - const canonical = canonicalConfigRows(normalizeLegacyConfig(plaintext)) + const canonical = loadCurrentConfig(normalizeLegacyConfig(plaintext)) + for (const key of Object.keys( + configSchemaMapping, + ) as (keyof typeof configSchemaMapping)[]) { + if (!isValidStoredConfig(key, canonical[key])) + throw new ConfigMigrationError('CONFIG_SHAPE_UNSUPPORTED', key) + } const names = new Set(rows.map((row) => row.name)) if (names.has('imageGenerationOptions') || names.has('ttsOptions')) names.add('ai') @@ -69,7 +79,7 @@ export async function migrateConfiguration( names.delete('ttsOptions') for (const name of names) { const value = canonical[name] ?? plaintext[name] - const encrypted = visitConfigSecrets(value, name, (secret, path) => { + const encrypted = visitLegacyConfigSecrets(value, name, (secret, path) => { const previous = path .split('.') .reduce((node, part) => node?.[part], raw) diff --git a/apps/core/src/modules/configs/legacy-config.util.ts b/apps/core/src/upgrade/legacy-config.ts similarity index 73% rename from apps/core/src/modules/configs/legacy-config.util.ts rename to apps/core/src/upgrade/legacy-config.ts index 1e399bdc338..6e0c6789ca4 100644 --- a/apps/core/src/modules/configs/legacy-config.util.ts +++ b/apps/core/src/upgrade/legacy-config.ts @@ -2,8 +2,39 @@ import { cloneDeep } from 'es-toolkit/compat' import { AIProviderType } from '~/modules/ai/ai.types' -import { generateDefaultConfig } from './configs.default' -import type { IConfig } from './configs.interface' +import { generateDefaultConfig } from '../modules/configs/configs.default' +import type { IConfig } from '../modules/configs/configs.interface' +import { + isEncryptedPath, + visitConfigSecrets, +} from '../modules/configs/configs.secret-paths' + +const legacyDefaults = { + image: { + enable: false, + provider: 'openrouter', + apiKey: '', + endpoint: '', + model: '', + defaultAspectRatio: '16:9', + defaultQuality: 'standard' as 'low' | 'standard' | 'high', + defaultFormat: 'png' as 'png' | 'jpeg' | 'webp', + }, + tts: { + enable: false, + provider: 'openrouter', + apiKey: '', + endpoint: '', + model: '', + voice: '', + speed: 1, + maxCharsPerChunk: 1800, + concurrency: 3, + maxCharsPerRun: 120000, + }, +} +type LegacyImageConfig = typeof legacyDefaults.image +type LegacyTtsConfig = typeof legacyDefaults.tts export class LegacyConfigConflictError extends Error { readonly code = 'CONFIG_SECRET_CONFLICT' @@ -12,7 +43,7 @@ export class LegacyConfigConflictError extends Error { } } -export function legacySecretConflicts( +function legacySecretConflicts( key: string, raw: Record, ): string[] { @@ -69,8 +100,8 @@ function extractGoogleVertexProjectId(endpoint: string): string | undefined { export function normalizeAiConfig( storedAiConfig: Record | undefined, aiConfig: IConfig['ai'], - legacyImageConfig: IConfig['imageGenerationOptions'], - legacyTtsConfig: IConfig['ttsOptions'], + legacyImageConfig: LegacyImageConfig, + legacyTtsConfig: LegacyTtsConfig, ): IConfig['ai'] { const providers = (aiConfig.providers ?? []).map((provider) => { const endpoint = provider.endpoint?.trim() ?? '' @@ -212,8 +243,8 @@ export function normalizeAiConfig( } /** - * Backward-compat: migrate old flat { githubToken } to new nested - * { github: { enabled, token } } shape at read time. + * Historical data conversion: flat { githubToken } to nested + * { github: { enabled, token } }. Never called by the configuration runtime. */ export function normalizeThirdPartyConfig( raw: Record, @@ -258,11 +289,8 @@ export function normalizeMembershipConfig( } /** Input must be decrypted at its original paths before calling this converter. */ -export function normalizeLegacyConfig( - raw: Record, - options: { allowConflicts?: boolean } = {}, -): IConfig { - if (!options.allowConflicts) { +export function normalizeLegacyConfig(raw: Record): IConfig { + { for (const key of ['membership', 'thirdPartyServiceIntegration']) { const conflict = legacySecretConflicts(key, raw[key] ?? {})[0] if (conflict) throw new LegacyConfigConflictError(`${key}.${conflict}`) @@ -287,93 +315,35 @@ export function normalizeLegacyConfig( delete (config.thirdPartyServiceIntegration as any).githubToken for (const key of ['dodoApiKey', 'dodoEnvironment', 'dodoWebhookKey']) delete (config.membership as any)[key] + if (raw.mailOptions?.smtp?.options) { + const { options, ...smtp } = raw.mailOptions.smtp + config.mailOptions.smtp = { ...options, ...smtp } + } config.ai = normalizeAiConfig( raw.ai, config.ai, - config.imageGenerationOptions, - config.ttsOptions, + { ...legacyDefaults.image, ...raw.imageGenerationOptions }, + { ...legacyDefaults.tts, ...raw.ttsOptions }, ) - { - for (const [section, feature, legacyId] of [ - [ - 'imageGenerationOptions', - 'imageGeneration', - '__legacy_image_generation__', - ], - ['ttsOptions', 'tts', '__legacy_tts__'], - ] as const) { - const route = config.ai[feature] - const provider = - (config.ai.providers ?? []).find( - (p) => p.id === route?.model?.providerId, - ) ?? (config.ai.providers ?? []).find((p) => p.id === legacyId) - const { model: assignment, ...settings } = route ?? {} - ;(config as any)[section] = { - ...config[section], - ...settings, - ...(provider - ? { - apiKey: provider.apiKey, - endpoint: provider.endpoint, - model: assignment?.model ?? provider.defaultModel, - provider: provider.endpoint?.includes('openrouter.ai') - ? 'openrouter' - : 'custom', - } - : {}), - } - } - } return config } -/** Canonical storage has one source for media credentials and assignments. */ -export function canonicalConfigRows(config: IConfig): Record { - const { - imageGenerationOptions: _image, - ttsOptions: _speech, - ...canonical - } = config - return canonical -} - -export function normalizeLegacyPatch(key: keyof IConfig, input: object) { - const value = cloneDeep(input) as Record - const conflict = legacySecretConflicts(key, value)[0] - if (conflict) throw new LegacyConfigConflictError(`${key}.${conflict}`) - if (key === 'thirdPartyServiceIntegration' && 'githubToken' in value) { - value.github = { - ...value.github, - token: legacySecret(value.github?.token, value.githubToken), - } - delete value.githubToken - } - if (key === 'membership') { - for (const [old, current] of [ - ['dodoApiKey', 'apiKey'], - ['dodoWebhookKey', 'webhookSigningKey'], - ['dodoEnvironment', 'environment'], - ]) { - if (old in value) - value[current] = legacySecret(value[current], value[old]) - delete value[old] - } - } - return value -} - -export function updateLegacyMedia( - config: IConfig, - section: 'imageGenerationOptions' | 'ttsOptions', - value: object, -): IConfig { - const updated = { ...config, [section]: { ...config[section], ...value } } - // Reuse the import converter, then expose the v17 view from the canonical result. - updated.ai = normalizeAiConfig( - undefined, - updated.ai, - updated.imageGenerationOptions, - updated.ttsOptions, +const oldSecretPaths = new Set([ + 'imageGenerationOptions.apiKey', + 'ttsOptions.apiKey', + 'thirdPartyServiceIntegration.githubToken', + 'membership.dodoApiKey', + 'membership.dodoWebhookKey', +]) +export function visitLegacyConfigSecrets( + value: T, + prefix: string, + visit: (secret: string, path: string) => string, +): T { + return visitConfigSecrets( + value, + prefix, + visit, + (path) => oldSecretPaths.has(path) || isEncryptedPath(path), ) - return normalizeLegacyConfig(updated) } diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts index 15a9f17e92c..6e9c6ffdc86 100644 --- a/apps/core/src/upgrade/preparation.ts +++ b/apps/core/src/upgrade/preparation.ts @@ -47,6 +47,7 @@ export async function prepareUpgrade( 'LEGACY_CONFIG', 'METRICS_INCOMPLETE', 'PREPARATION_MARKER_MISSING', + 'CONFIG_MIGRATION_MISSING', ]) const blocking = before.issues.find((issue) => !allowed.has(issue.code)) if (blocking) throw new UpgradeError(blocking.code, blocking.path) @@ -103,6 +104,10 @@ export async function prepareUpgrade( } } changed += await backfillAiMetrics(client) + await client.query( + 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', + [CONFIG_FORMAT_MIGRATION_ID], + ) const after = await readUpgradeState(client, config) const residual = after.issues.find( (issue) => issue.code !== 'PREPARATION_MARKER_MISSING', @@ -114,10 +119,7 @@ export async function prepareUpgrade( [PREPARED_MIGRATION_ID], ) } - await client.query( - 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', - [CONFIG_FORMAT_MIGRATION_ID], - ) + await client.query(options.dryRun ? 'ROLLBACK' : 'COMMIT') return { status: options.dryRun ? 'would-be-ready' : 'prepared', diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index e06f5fb158e..cfb96c45e2b 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -1,17 +1,22 @@ import type { Pool, PoolClient } from 'pg' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' -import { configSchemaMapping } from '../modules/configs/configs.schema' -import { visitConfigSecrets } from '../modules/configs/configs.secret-paths' import { - LegacyConfigConflictError, - normalizeLegacyConfig, -} from '../modules/configs/legacy-config.util' + isValidStoredConfig, + loadCurrentConfig, +} from '../modules/configs/configs.current' +import { configSchemaMapping } from '../modules/configs/configs.schema' +import { CONFIG_FORMAT_MIGRATION_ID } from '../shared/config-format' import { SecretCodec } from '../utils/secret-codec' import { inspectSecretStorageConfig, type SecretStorageConfig, } from '../utils/secret-storage-config' +import { + LegacyConfigConflictError, + normalizeLegacyConfig, + visitLegacyConfigSecrets, +} from './legacy-config' import { isEncryptedOAuth, unwrapOAuthToken } from './oauth-token' export { oauthKey } from './oauth-token' @@ -44,6 +49,10 @@ export function isLegacyConfig(name: string, value: any): boolean { return ( name === 'imageGenerationOptions' || name === 'ttsOptions' || + (name === 'mailOptions' && + value?.smtp && + typeof value.smtp === 'object' && + 'options' in value.smtp) || (name === 'ai' && value?.version !== 2) || (name === 'thirdPartyServiceIntegration' && value && @@ -97,9 +106,13 @@ export async function readUpgradeState( for (const row of options) { if (isLegacyConfig(row.name, row.value)) counts.legacyConfig++ if (row.value && typeof row.value === 'object') - visitConfigSecrets(row.value, row.name, inspect) + visitLegacyConfigSecrets(row.value, row.name, inspect) } - const knownRows = options.filter((row) => row.name in configSchemaMapping) + const knownRows = options.filter( + (row) => + row.name in configSchemaMapping || + ['imageGenerationOptions', 'ttsOptions'].includes(row.name), + ) for (const row of knownRows) { if (!row.value || typeof row.value !== 'object' || Array.isArray(row.value)) issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: row.name }) @@ -116,17 +129,17 @@ export async function readUpgradeState( const raw = Object.fromEntries( knownRows.map((row) => [ row.name, - visitConfigSecrets(row.value, row.name, (secret, path) => + visitLegacyConfigSecrets(row.value, row.name, (secret, path) => codec.decrypt(secret, path), ), ]), ) - const normalized = normalizeLegacyConfig(raw) - for (const row of knownRows) { - if ( - !configSchemaMapping[row.name].safeParse(normalized[row.name]).success - ) - issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: row.name }) + const normalized = loadCurrentConfig(normalizeLegacyConfig(raw)) + for (const key of Object.keys( + configSchemaMapping, + ) as (keyof typeof configSchemaMapping)[]) { + if (!isValidStoredConfig(key, normalized[key])) + issues.push({ code: 'CONFIG_SHAPE_UNSUPPORTED', path: key }) } } catch (error) { issues.push( @@ -223,6 +236,15 @@ export async function readUpgradeState( issues.push({ code: 'CREDENTIALS_NEED_PREPARATION' }) if (counts.legacyConfig) issues.push({ code: 'LEGACY_CONFIG' }) if (counts.metricsMissing) issues.push({ code: 'METRICS_INCOMPLETE' }) + if ( + installation === 'existing' && + !( + await database.query('SELECT 1 FROM _app_migrations WHERE id = $1', [ + CONFIG_FORMAT_MIGRATION_ID, + ]) + ).rowCount + ) + issues.push({ code: 'CONFIG_MIGRATION_MISSING' }) if (installation === 'existing' && !prepared) issues.push({ code: 'PREPARATION_MARKER_MISSING' }) return { diff --git a/apps/core/test/helper/pg-verify-url.ts b/apps/core/test/helper/pg-verify-url.ts index 58f6b11bc21..9fbff802827 100644 --- a/apps/core/test/helper/pg-verify-url.ts +++ b/apps/core/test/helper/pg-verify-url.ts @@ -5,6 +5,7 @@ import { Pool } from 'pg' import * as schema from '~/database/schema' import { runSchemaMigrationFiles } from '~/processors/database/schema-migrator' +import { CONFIG_FORMAT_MIGRATION_ID } from '~/shared/config-format' export function requirePgVerifyUrl() { const verifyUrl = process.env.PG_VERIFY_URL @@ -18,6 +19,7 @@ export function requirePgVerifyUrl() { interface PgTestDatabaseOptions { migrate?: boolean + configCurrent?: boolean } export interface PgTestDatabase { @@ -73,6 +75,12 @@ export async function createPgTestDatabase( '../../src/database/migrations', ) await runSchemaMigrationFiles(pool, migrationsFolder) + // Runtime configuration fixtures opt in to the current app-data marker. + // Schema-only and migration regression fixtures remain unprepared. + if (options.configCurrent === true) + await pool.query('INSERT INTO _app_migrations (id) VALUES ($1)', [ + CONFIG_FORMAT_MIGRATION_ID, + ]) } return { diff --git a/apps/core/test/src/database/canonical-config.migration.spec.ts b/apps/core/test/src/database/canonical-config.migration.spec.ts index 02b7fd6b0d7..fa911ffcaf5 100644 --- a/apps/core/test/src/database/canonical-config.migration.spec.ts +++ b/apps/core/test/src/database/canonical-config.migration.spec.ts @@ -1,12 +1,30 @@ import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { AppMigrationsService } from '~/database/app-migrations/app-migrations.service' +import { OptionsRepository } from '~/modules/configs/options.repository' +import { SnowflakeService } from '~/shared/id/snowflake.service' import { EncryptUtil } from '~/utils/encrypt.util' const migrationId = '20261109-canonical-config' +it('blocks configuration startup until the release migration has completed, including a new installation', async () => { + const context = await createPgTestDatabase('mx_config_guard', { + configCurrent: false, + }) + try { + const repository = new OptionsRepository(context.db, new SnowflakeService()) + await expect(repository.findAll()).rejects.toThrow('migration') + await new AppMigrationsService(context.db, context.pool).run() + await expect(repository.findAll()).resolves.toEqual([]) + } finally { + await context.close() + } +}) + it('automatically migrates old configuration rows before release startup and preserves credentials on rerun', async () => { - const context = await createPgTestDatabase('mx_canonical_config') + const context = await createPgTestDatabase('mx_canonical_config', { + configCurrent: false, + }) try { await context.pool.query( `INSERT INTO options (id,name,value) VALUES ('100','membership',$1), ('101','thirdPartyServiceIntegration',$2), ('102','imageGenerationOptions',$3), ('103','ttsOptions',$4)`, @@ -89,10 +107,12 @@ it('automatically migrates old configuration rows before release startup and pre } }) -it.each(['conflict', 'bad-ciphertext'])( +it.each(['conflict', 'bad-ciphertext', 'bad-shape'])( 'blocks release and rolls back every configuration row on %s', async (invalid) => { - const context = await createPgTestDatabase('mx_bad_config') + const context = await createPgTestDatabase('mx_bad_config', { + configCurrent: false, + }) try { await context.pool.query( "INSERT INTO options (id,name,value) VALUES ('100','membership',$1), ('101','thirdPartyServiceIntegration',$2)", @@ -100,6 +120,9 @@ it.each(['conflict', 'bad-ciphertext'])( { dodoApiKey: 'old-billing', ...(invalid === 'conflict' ? { apiKey: 'different-billing' } : {}), + ...(invalid === 'bad-shape' + ? { apiKey: { malformed: 'private-value' } } + : {}), }, { githubToken: @@ -115,7 +138,11 @@ it.each(['conflict', 'bad-ciphertext'])( await expect( new AppMigrationsService(context.db, context.pool).run(), ).rejects.toThrow( - invalid === 'conflict' ? 'CONFIG_SECRET_CONFLICT' : 'SECRET_UNREADABLE', + invalid === 'conflict' + ? 'CONFIG_SECRET_CONFLICT' + : invalid === 'bad-ciphertext' + ? 'SECRET_UNREADABLE' + : 'CONFIG_SHAPE_UNSUPPORTED', ) expect( (await context.pool.query('SELECT * FROM options ORDER BY name')).rows, diff --git a/apps/core/test/src/database/protect-stored-secrets.spec.ts b/apps/core/test/src/database/protect-stored-secrets.spec.ts index f05301caf24..44c746b19c7 100644 --- a/apps/core/test/src/database/protect-stored-secrets.spec.ts +++ b/apps/core/test/src/database/protect-stored-secrets.spec.ts @@ -21,7 +21,9 @@ import { oauthKey } from '~/upgrade/oauth-token' import { EncryptUtil } from '~/utils/encrypt.util' it('upgrades real credential rows without changing their usable values, and can run twice', async () => { - const context = await createPgTestDatabase('mx_secret_verify') + const context = await createPgTestDatabase('mx_secret_verify', { + configCurrent: true, + }) __setTestPostgresInstance(context.pool, context.db) try { const cipher = crypto.createCipheriv( diff --git a/apps/core/test/src/modules/configs/configs.service.spec.ts b/apps/core/test/src/modules/configs/configs.service.spec.ts index 18277a2a889..7a2be1e9a42 100644 --- a/apps/core/test/src/modules/configs/configs.service.spec.ts +++ b/apps/core/test/src/modules/configs/configs.service.spec.ts @@ -21,57 +21,6 @@ function createDeferred() { } describe('ConfigsService', () => { - it('persists legacy media writes through canonical AI and retains blank secrets', async () => { - let cache: string | null = null - const rows = new Map() - const repository = { - findAll: async () => [...rows].map(([name, value]) => ({ name, value })), - upsert: vi.fn(async (name, value) => { - rows.set(name, value) - return { name, value } - }), - deleteByName: vi.fn(async (name) => rows.delete(name)), - } - const service = new ConfigsService( - repository as any, - { - waitForReady: async () => {}, - getClient: () => ({ - get: async () => cache, - set: async (_key, value) => { - cache = value - }, - }), - } as any, - {} as any, - { emit: vi.fn() } as any, - ) - await service.patchAndValid('imageGenerationOptions', { - enable: true, - apiKey: 'image-secret', - model: 'image-v1', - }) - expect(rows.has('imageGenerationOptions')).toBe(false) - expect(JSON.stringify(rows.get('ai'))).not.toContain('image-secret') - await service.patchAndValid('imageGenerationOptions', { - apiKey: '', - model: 'image-v2', - }) - expect(await service.get('imageGenerationOptions')).toMatchObject({ - apiKey: 'image-secret', - model: 'image-v2', - }) - await service.patchAndValid('ai', { imageGeneration: { enable: false } }) - expect((await service.get('imageGenerationOptions')).enable).toBe(false) - await service.patchAndValid('membership', { - dodoApiKey: 'billing-secret', - } as any) - expect(decryptObject(rows.get('membership'), 'membership').apiKey).toBe( - 'billing-secret', - ) - expect(rows.get('membership')).not.toHaveProperty('dodoApiKey') - }) - it('requires a public key to enable backup encryption and never accepts private keys in settings', async () => { const pair = generateKeyPairSync('rsa', { modulusLength: 2048, @@ -203,258 +152,6 @@ describe('ConfigsService', () => { ) }) - it('reads imageGenerationOptions defaults via get()', async () => { - const redisClient = { - get: vi.fn().mockResolvedValue(JSON.stringify(generateDefaultConfig())), - set: vi.fn().mockResolvedValue('OK'), - } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([]), - } - - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - { emit: vi.fn() } as any, - ) - - await expect(service.get('imageGenerationOptions')).resolves.toEqual({ - enable: false, - provider: 'openrouter', - apiKey: '', - endpoint: '', - model: '', - defaultAspectRatio: '16:9', - defaultQuality: 'standard', - defaultFormat: 'png', - }) - }) - - it('imports legacy image and speech credentials into capability-routed AI providers', async () => { - const redisClient = { set: vi.fn().mockResolvedValue('OK') } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([ - { name: 'ai', value: { providers: [] } }, - { - name: 'imageGenerationOptions', - value: { - enable: true, - provider: 'openrouter', - apiKey: 'legacy-image-key', - model: 'google/gemini-image', - }, - }, - { - name: 'ttsOptions', - value: { - enable: true, - provider: 'openai', - apiKey: 'legacy-speech-key', - model: 'gpt-4o-mini-tts', - voice: 'alloy', - }, - }, - ]), - } - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - { emit: vi.fn() } as any, - ) - - await service.onModuleInit() - - const cachedConfig = decryptObject( - JSON.parse(redisClient.set.mock.calls[0][1]), - ) - const imageProvider = cachedConfig.ai.providers.find( - (provider: any) => provider.id === '__legacy_image_generation__', - ) - const speechProvider = cachedConfig.ai.providers.find( - (provider: any) => provider.id === '__legacy_tts__', - ) - expect(cachedConfig.ai.version).toBe(2) - expect(imageProvider).toMatchObject({ - apiKey: 'legacy-image-key', - capabilities: expect.objectContaining({ - text: false, - image: true, - speech: false, - }), - endpoint: 'https://openrouter.ai/api/v1', - }) - expect(speechProvider).toMatchObject({ - apiKey: 'legacy-speech-key', - capabilities: { text: false, image: false, speech: true }, - endpoint: 'https://api.openai.com/v1', - }) - expect(cachedConfig.ai.imageGeneration.model).toEqual({ - providerId: imageProvider.id, - model: 'google/gemini-image', - }) - expect(cachedConfig.ai.tts.model).toEqual({ - providerId: speechProvider.id, - model: 'gpt-4o-mini-tts', - }) - }) - - it('deduplicates legacy media settings that share one provider connection', async () => { - const redisClient = { set: vi.fn().mockResolvedValue('OK') } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([ - { name: 'ai', value: { providers: [] } }, - { - name: 'imageGenerationOptions', - value: { - provider: 'custom', - apiKey: 'shared-key', - endpoint: 'https://media.example.com/v1/', - model: 'image-model', - }, - }, - { - name: 'ttsOptions', - value: { - provider: 'custom', - apiKey: 'shared-key', - endpoint: 'https://media.example.com/v1', - model: 'speech-model', - }, - }, - ]), - } - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - { emit: vi.fn() } as any, - ) - - await service.onModuleInit() - - const cachedConfig = decryptObject( - JSON.parse(redisClient.set.mock.calls[0][1]), - ) - expect(cachedConfig.ai.providers).toHaveLength(1) - expect(cachedConfig.ai.providers[0].capabilities).toMatchObject({ - text: false, - image: true, - speech: true, - }) - expect(cachedConfig.ai.imageGeneration.model.providerId).toBe( - cachedConfig.ai.providers[0].id, - ) - expect(cachedConfig.ai.tts.model.providerId).toBe( - cachedConfig.ai.providers[0].id, - ) - }) - - it('preserves a legacy media credential even when no model was assigned', async () => { - const redisClient = { set: vi.fn().mockResolvedValue('OK') } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([ - { name: 'ai', value: { providers: [] } }, - { - name: 'imageGenerationOptions', - value: { - provider: 'custom', - apiKey: 'credential-without-model', - endpoint: 'https://images.example.com/v1', - model: '', - }, - }, - ]), - } - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - { emit: vi.fn() } as any, - ) - - await service.onModuleInit() - - const cachedConfig = decryptObject( - JSON.parse(redisClient.set.mock.calls[0][1]), - ) - expect(cachedConfig.ai.providers).toEqual([ - expect.objectContaining({ - apiKey: 'credential-without-model', - defaultModel: '', - endpoint: 'https://images.example.com/v1', - capabilities: expect.objectContaining({ - text: false, - image: true, - speech: false, - }), - }), - ]) - expect(cachedConfig.ai.imageGeneration.model).toBeUndefined() - }) - - it('migrates legacy Dodo credential names into provider-neutral fields', async () => { - const redisClient = { - set: vi.fn().mockResolvedValue('OK'), - } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([ - { - name: 'membership', - value: { - enabled: true, - provider: 'dodo', - dodoApiKey: 'legacy-api-key', - dodoWebhookKey: 'legacy-webhook-key', - dodoEnvironment: 'test_mode', - }, - }, - ]), - } - - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - { emit: vi.fn() } as any, - ) - - await service.onModuleInit() - - const cachedConfig = decryptObject( - JSON.parse(redisClient.set.mock.calls[0][1]), - ) - expect(cachedConfig.membership).toMatchObject({ - apiKey: 'legacy-api-key', - environment: 'test_mode', - webhookSigningKey: 'legacy-webhook-key', - }) - expect(cachedConfig.membership).not.toHaveProperty('dodoApiKey') - expect(cachedConfig.membership).not.toHaveProperty('dodoWebhookKey') - expect(cachedConfig.membership).not.toHaveProperty('dodoEnvironment') - }) - it('waits for redis before reading config cache', async () => { const initReady = Promise.resolve() const readReady = createDeferred() @@ -521,6 +218,7 @@ describe('ConfigsService', () => { { name: 'ai', value: { + version: 2, providers: [ { ...config.ai.providers[0], @@ -616,161 +314,6 @@ describe('ConfigsService', () => { }) }) - describe('imageGenerationOptions provider validation', () => { - function createService( - currentConfig: ReturnType, - ) { - const redisClient = { - get: vi.fn().mockResolvedValue(JSON.stringify(currentConfig)), - set: vi.fn().mockResolvedValue('OK'), - } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([]), - upsert: vi.fn(async (name: string, value: unknown) => ({ - id: '1' as any, - name, - value, - })), - } - const eventManager = { emit: vi.fn() } - - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - eventManager as any, - ) - - return { service } - } - - it('rejects provider: "custom" with a blank endpoint', async () => { - const { service } = createService(generateDefaultConfig()) - - await expect( - service.patchAndValid('imageGenerationOptions', { - provider: 'custom', - }), - ).rejects.toMatchObject({ code: AppErrorCode.CONFIG_VALIDATION_FAILED }) - }) - - it('accepts provider: "custom" when an endpoint is already configured', async () => { - const currentConfig = generateDefaultConfig() - currentConfig.imageGenerationOptions.endpoint = - 'https://existing.example.com/v1' - const { service } = createService(currentConfig) - - const result = await service.patchAndValid('imageGenerationOptions', { - provider: 'custom', - }) - - expect(result.provider).toBe('custom') - }) - - it('accepts provider: "custom" together with a newly-supplied endpoint in the same patch', async () => { - const { service } = createService(generateDefaultConfig()) - - const result = await service.patchAndValid('imageGenerationOptions', { - provider: 'custom', - endpoint: 'https://new-custom.example.com/v1', - }) - - expect(result.provider).toBe('custom') - expect(result.endpoint).toBe('https://new-custom.example.com/v1') - }) - - it('leaves the default "openrouter" provider unaffected by the custom-endpoint check', async () => { - const { service } = createService(generateDefaultConfig()) - - const result = await service.patchAndValid('imageGenerationOptions', { - provider: 'openrouter', - }) - - expect(result.provider).toBe('openrouter') - }) - }) - - describe('ttsOptions provider validation', () => { - function createService( - currentConfig: ReturnType, - ) { - const redisClient = { - get: vi.fn().mockResolvedValue(JSON.stringify(currentConfig)), - set: vi.fn().mockResolvedValue('OK'), - } - const redisService = { - getClient: vi.fn(() => redisClient), - waitForReady: vi.fn().mockResolvedValue(undefined), - } - const optionsRepository = { - findAll: vi.fn().mockResolvedValue([]), - upsert: vi.fn(async (name: string, value: unknown) => ({ - id: '1' as any, - name, - value, - })), - } - const eventManager = { emit: vi.fn() } - - const service = new ConfigsService( - optionsRepository as any, - redisService as any, - {} as any, - eventManager as any, - ) - - return { service } - } - - it('rejects provider: "custom" with a blank endpoint', async () => { - const { service } = createService(generateDefaultConfig()) - - await expect( - service.patchAndValid('ttsOptions', { - provider: 'custom', - }), - ).rejects.toMatchObject({ code: AppErrorCode.CONFIG_VALIDATION_FAILED }) - }) - - it('accepts provider: "custom" when an endpoint is already configured', async () => { - const currentConfig = generateDefaultConfig() - currentConfig.ttsOptions.endpoint = 'https://existing.example.com/v1' - const { service } = createService(currentConfig) - - const result = await service.patchAndValid('ttsOptions', { - provider: 'custom', - }) - - expect(result.provider).toBe('custom') - }) - - it('accepts provider: "custom" together with a newly-supplied endpoint in the same patch', async () => { - const { service } = createService(generateDefaultConfig()) - - const result = await service.patchAndValid('ttsOptions', { - provider: 'custom', - endpoint: 'https://new-custom.example.com/v1', - }) - - expect(result.provider).toBe('custom') - expect(result.endpoint).toBe('https://new-custom.example.com/v1') - }) - - it('leaves the default "openrouter" provider unaffected by the custom-endpoint check', async () => { - const { service } = createService(generateDefaultConfig()) - - const result = await service.patchAndValid('ttsOptions', { - provider: 'openrouter', - }) - - expect(result.provider).toBe('openrouter') - }) - }) - describe('AI provider and capability route validation', () => { function createService( currentConfig: ReturnType, @@ -982,7 +525,7 @@ describe('ConfigsService', () => { }) }) - describe('S3 and image-generation option save round-trips', () => { + describe('S3 option save round-trips', () => { function createService( currentConfig: ReturnType, ) { @@ -1014,46 +557,6 @@ describe('ConfigsService', () => { return { service } } - it('accepts the section re-submitted verbatim as fetched, with apiKey/endpoint/model still "" — the exact shape the admin form sends when it never touches a hidden field', async () => { - const { service } = createService(generateDefaultConfig()) - const fetched = await service.get('imageGenerationOptions') - - const result = await service.patchAndValid('imageGenerationOptions', { - ...fetched, - enable: true, - }) - - expect(result).toMatchObject({ - enable: true, - apiKey: '', - endpoint: '', - model: '', - }) - }) - - it('accepts an explicit null for apiKey/endpoint/model and normalizes each to "" rather than throwing', async () => { - const { service } = createService(generateDefaultConfig()) - - const result = await service.patchAndValid('imageGenerationOptions', { - apiKey: null, - endpoint: null, - model: null, - } as any) - - expect(result).toMatchObject({ apiKey: '', endpoint: '', model: '' }) - }) - - it('still rejects provider: "custom" when endpoint arrives as null (normalizes to "" first, then the custom-endpoint check still fires)', async () => { - const { service } = createService(generateDefaultConfig()) - - await expect( - service.patchAndValid('imageGenerationOptions', { - provider: 'custom', - endpoint: null, - } as any), - ).rejects.toMatchObject({ code: AppErrorCode.CONFIG_VALIDATION_FAILED }) - }) - it('accepts legacy null S3 values in backupOptions', async () => { const { service } = createService(generateDefaultConfig()) diff --git a/apps/core/test/src/modules/configs/configs.transaction.spec.ts b/apps/core/test/src/modules/configs/configs.transaction.spec.ts index f46d01de0af..d41c4580043 100644 --- a/apps/core/test/src/modules/configs/configs.transaction.spec.ts +++ b/apps/core/test/src/modules/configs/configs.transaction.spec.ts @@ -22,7 +22,9 @@ describe('configuration commit boundary', () => { const emit = vi.fn() const bump = vi.fn() beforeAll(async () => { - context = await createPgTestDatabase('mx_config_commit') + context = await createPgTestDatabase('mx_config_commit', { + configCurrent: true, + }) repository = new OptionsRepository(context.db, new SnowflakeService()) await context.pool .query(`CREATE FUNCTION reject_config_commit() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.value->>'title' = 'reject-commit' OR NEW.value->>'webUrl' = 'https://reject-commit.example' THEN RAISE EXCEPTION 'test commit rejection'; END IF; RETURN NEW; END $$; @@ -75,22 +77,18 @@ describe('configuration commit boundary', () => { expect(await repository.get('seo')).toMatchObject({ title: 'after-commit' }) expect(JSON.parse(cache).seo.title).toBe('after-commit') }) - it('keeps conflicting stored secrets readable and requires an explicit replacement to save', async () => { + it('refuses unmigrated configuration without modifying stored conflicting secrets', async () => { await repository.upsert('membership', { apiKey: 'canonical-key', dodoApiKey: 'legacy-key', }) await expect( service.patchAndValid('membership', { enabled: true }), - ).rejects.toMatchObject({ code: 'CONFIG_VALIDATION_FAILED' }) + ).rejects.toThrow('migration') expect(await repository.get('membership')).toMatchObject({ apiKey: 'canonical-key', dodoApiKey: 'legacy-key', }) - await service.patchAndValid('membership', { apiKey: 'chosen-key' }) - const stored = await repository.get>('membership') - expect(stored).not.toHaveProperty('dodoApiKey') - expect((await service.get('membership')).apiKey).toBe('chosen-key') }) it('orders cache publication before allowing the next configuration writer', async () => { let entered!: () => void diff --git a/apps/core/test/src/modules/configs/current-config.spec.ts b/apps/core/test/src/modules/configs/current-config.spec.ts new file mode 100644 index 00000000000..7d044693b7c --- /dev/null +++ b/apps/core/test/src/modules/configs/current-config.spec.ts @@ -0,0 +1,90 @@ +import { ConfigsService } from '~/modules/configs/configs.service' + +function createService(rows: { name: string; value: unknown }[] = []) { + let cache: string | null = null + const repository = { + findAll: async () => rows, + upsert: vi.fn(async (name, value) => ({ name, value })), + } + const service = new ConfigsService( + repository as any, + { + waitForReady: async () => {}, + getClient: () => ({ + get: async () => cache, + set: async (_key, value) => { + cache = value + }, + }), + } as any, + { bump: vi.fn() } as any, + { emit: vi.fn() } as any, + ) + return { service, repository, cache: () => cache } +} + +it('exposes only current configuration and rejects retired sections without writes', async () => { + const { service, repository } = createService() + const config = await service.getConfigForResponse() + expect(config).not.toHaveProperty('imageGenerationOptions') + expect(config).not.toHaveProperty('ttsOptions') + for (const section of ['imageGenerationOptions', 'ttsOptions']) { + await expect(service.get(section as any)).rejects.toMatchObject({ + code: 'CONFIG_NOT_FOUND', + }) + await expect(service.getForResponse(section as any)).rejects.toMatchObject({ + code: 'CONFIG_NOT_FOUND', + }) + await expect( + service.patchAndValid( + section as any, + { apiKey: 'must-not-write' } as any, + ), + ).rejects.toMatchObject({ code: 'CONFIG_NOT_FOUND' }) + } + expect(repository.upsert).not.toHaveBeenCalled() +}) + +it.each([ + ['membership', { dodoApiKey: 'must-not-write' }], + ['thirdPartyServiceIntegration', { githubToken: 'must-not-write' }], + [ + 'thirdPartyServiceIntegration', + { github: { unexpected: 'must-not-write' } }, + ], +])( + 'rejects unknown and retired fields in %s instead of translating or ignoring them', + async (key, value) => { + const { service, repository } = createService() + await expect( + service.patchAndValid(key as any, value as any), + ).rejects.toMatchObject({ code: 'CONFIG_VALIDATION_FAILED' }) + expect(repository.upsert).not.toHaveBeenCalled() + }, +) + +it.each([ + { name: 'membership', value: { dodoApiKey: 'stored-private' } }, + { name: 'imageGenerationOptions', value: { apiKey: 'stored-private' } }, + { name: 'ai', value: { providers: [] } }, +])( + 'requires DB migration rather than converting old %s values in startup/cache', + async (row) => { + const { service, cache } = createService([row]) + await expect(service.onModuleInit()).rejects.toThrow('migration') + expect(cache()).toBeNull() + }, +) + +it('preserves freeform OAuth keys and values during partial updates', async () => { + const { service, repository } = createService() + await service.patchAndValid('oauth', { + public: { google: { custom_dynamic_key: 'exact-value' } }, + }) + expect(repository.upsert).toHaveBeenCalledWith( + 'oauth', + expect.objectContaining({ + public: { google: { custom_dynamic_key: 'exact-value' } }, + }), + ) +}) diff --git a/apps/core/test/src/modules/configs/legacy-config.spec.ts b/apps/core/test/src/modules/configs/legacy-config.spec.ts index ff37bd234be..83a20168e84 100644 --- a/apps/core/test/src/modules/configs/legacy-config.spec.ts +++ b/apps/core/test/src/modules/configs/legacy-config.spec.ts @@ -1,13 +1,30 @@ import { describe, expect, it } from 'vitest' +import { encryptObject } from '~/modules/configs/configs.encrypt.util' import { - decryptObject, - encryptObject, -} from '~/modules/configs/configs.encrypt.util' -import { normalizeLegacyConfig } from '~/modules/configs/legacy-config.util' + normalizeLegacyConfig, + visitLegacyConfigSecrets, +} from '~/upgrade/legacy-config' import { EncryptUtil } from '~/utils/encrypt.util' describe('canonical legacy config conversion', () => { + it('migrates nested SMTP connection options into the declared current fields', () => { + const config = normalizeLegacyConfig({ + mailOptions: { + smtp: { + user: 'sender', + options: { host: 'smtp.example.test', port: 2525, secure: false }, + }, + }, + }) + expect(config.mailOptions.smtp).toMatchObject({ + user: 'sender', + host: 'smtp.example.test', + port: 2525, + secure: false, + }) + expect(config.mailOptions.smtp).not.toHaveProperty('options') + }) it('preserves populated legacy secrets when canonical fields are blank', () => { const value = normalizeLegacyConfig({ membership: { @@ -69,7 +86,9 @@ describe('canonical legacy config conversion', () => { }, } const converted = normalizeLegacyConfig( - decryptObject(structuredClone(original)), + visitLegacyConfigSecrets(structuredClone(original), '', (value, path) => + EncryptUtil.decrypt(value, path), + ), ) expect(converted.thirdPartyServiceIntegration.github.token).toBe( 'github-secret', @@ -90,26 +109,14 @@ describe('canonical legacy config conversion', () => { expect(normalizeLegacyConfig(converted)).toEqual(converted) }) - it('retains legacy media views from canonical AI configuration', () => { + it('removes historical media sections while retaining credentials without an assignment', () => { const config = normalizeLegacyConfig({ - imageGenerationOptions: { - enable: true, - apiKey: 'legacy-image', - model: 'image-model', - endpoint: 'https://api.example.com/v1', - }, - }) - const { - imageGenerationOptions: _image, - ttsOptions: _speech, - ...stored - } = config - expect(normalizeLegacyConfig(stored).imageGenerationOptions).toMatchObject({ - enable: true, - apiKey: 'legacy-image', - model: 'image-model', - endpoint: 'https://api.example.com/v1', + imageGenerationOptions: { apiKey: 'unassigned-private' }, }) + expect(config).not.toHaveProperty('imageGenerationOptions') + expect(config).not.toHaveProperty('ttsOptions') + expect(config.ai.providers[0].apiKey).toBe('unassigned-private') + expect(config.ai.imageGeneration?.model).toBeUndefined() }) it('retains leftover media credentials alongside an already-v2 AI row', () => { const converted = normalizeLegacyConfig({ diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts index a1399fc7f82..80c7eb19c58 100644 --- a/apps/core/test/src/upgrade/readiness.spec.ts +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -89,4 +89,12 @@ describe('standalone upgrade readiness', () => { prepared: false, }) }) + it('requires the current configuration migration even when an older preparation marker exists', async () => { + await context.pool.query( + "DELETE FROM options; INSERT INTO categories (id,name,slug) VALUES ('700','Existing','existing') ON CONFLICT DO NOTHING; INSERT INTO _app_migrations (id) VALUES ('20261108-v18-prepared') ON CONFLICT DO NOTHING", + ) + const report = await readUpgradeState(context.pool, config) + expect(report.status).toBe('needs-action') + expect(report.issues).toContainEqual({ code: 'CONFIG_MIGRATION_MISSING' }) + }) }) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 308e616d325..0b93dd8a56a 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -1,6 +1,6 @@ # mx-core v18:兼容性收口与文档迁移 -状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。本次完成 **v17 准备阶段** 的本地实现与验证,尚未发布;不发布 v18,不操作真实站点数据。 +状态:两轮共 13 项范围及完整设计已确认;用户于 2026-11-08 明确要求开始实施。当前开发分支已实施强制加密及配置 DB 迁移与运行时收口,尚未发布;不发布 v18,不操作真实站点数据。 修订(2026-11-08):用户明确要求本 PR 立即移除非加密运行和旧格式写入,覆盖下文“准备阶段保留逃生口”的安排。显式字段密钥、至少 32 字符 JWT 和固定 V2 写入在本分支立即生效;旧 true 开关仅作为弃用配置接受,false 拒绝启动。历史读取与显式迁移保留。其余 API、SDK 6 和数据库列删除仍按后续阶段执行,当前分支尚未发布。 @@ -43,7 +43,7 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 5. 完成 v17 事实文档纠错、v15–v17 迁移入口及 v18 预告,提供可执行准备说明。 6. 行为测试、真实临时 PostgreSQL 升级演练、构建和独立整体验证;保留原有工作。 -准备阶段不切换普通 AI GET 的既有默认行为,不拒绝现有合法旧参数,不删除 SDK 5.x 的公开签名,不删除数据库列,不修改线上环境或发布标签。 +准备阶段不切换普通 AI GET 的既有默认行为,配置 PATCH 已严格拒绝旧/未知字段,其他既有参数暂保持,不删除 SDK 5.x 的公开签名,不删除数据库列,不修改线上环境或发布标签。 ## 安全配置与迁移契约 @@ -63,7 +63,7 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 顺序:识别旧记录 → 用原路径解密 → 转换旧 AI/第三方/会员配置 → 按新路径加密并持久化 → 转换 Snippet/Webhook/API Key/OAuth → 验证残留及指标覆盖 → 记录准备完成。API Key 原始字符串仍可用,不重复哈希,不把哈希当作客户端凭据。 -准备版对仍需兼容的旧配置写请求在落库边界转换为新形状;旧读取可由新形状派生,不能让旧客户端把旧结构写回。转换和常规 PATCH 串行保护同一配置。旧配置转换留在迁移/导入/恢复共用模块,v18 常规读取不再归一化旧结构。 +已按 2026-11-09 修订结束运行时旧结构兼容:发布前必跑配置数据迁移,普通读取与 PATCH 仅处理规范结构。转换器仅在发布迁移与离线工具中复用;迁移和正常配置写入使用同一个数据库锁。 坏密钥、坏密文或未知格式必须失败,不回落为明文成功。报告只带字段路径/记录 ID/计数,不带值。原实例若曾用 machine-id 加密,准备期先取得等效密钥并显式保存;不能生成新随机密钥替代。 @@ -176,7 +176,7 @@ flowchart TD - 独立整体验证发现的 2 项 Critical、2 项 Important 已通过先失败后通过的行为测试修复;未再次分派审阅。另补「仅有分类等业务数据时不得判为空库」回归。缓存加密验证比较解密后的原值,不把随机 GCM 密文字节相同当作契约。 - 官方文档工作区 `/Users/innei/git/innei-repo/mx-docs` 的 `545e5c8`、`0719e79` 已本地提交,build/check/links/pangu/lint 与 3 项配置生成测试通过。保留原有 1 条 Astro hint、1 条既有死链;真实预览浏览器和临时 Compose 容器验证了复制映射、字符串类型与美元符号保留,生成的 llms 内容已核对。 -裁量保持:先抽纯加密/字段路径模块以支持独立诊断,仅改变实现顺序;启动强制已按用户修订在本分支实施;严格配置、只读 AI 默认行为、SDK 6 清理和物理列删除仍属后续阶段。当前没有自动隔离恢复入口,继续使用离线恢复到准备版再准备的路径;回退最低公开版本与消费者矩阵仍须发布后实测确定。原有 Agent 测试夹具另行处理。 +裁量保持:先抽纯加密/字段路径模块以支持独立诊断,仅改变实现顺序;启动强制、配置 DB 迁移和严格配置已按用户修订在本分支实施;只读 AI 默认行为、SDK 6 清理和物理列删除仍属后续阶段。当前没有自动隔离恢复入口,继续使用离线恢复到准备版再准备的路径;回退最低公开版本与消费者矩阵仍须发布后实测确定。原有 Agent 测试夹具另行处理。 ## 强制加密修订验证(2026-11-08) @@ -189,3 +189,13 @@ flowchart TD - mx-docs 的 3 项测试、build/check/links/pangu/lint 通过,生成 llms 内容包含新要求;v17.0.4 页面明确固定历史行为,当前开发分支说明明确强制加密已实现。保留既有 hint 和死链。 最终独立审阅:无 Critical、1 项 Important、无 Minor。保留的旧凭据迁移会将历史 Better Auth hex 密文当作明文二次加密;已复用严格解封装逻辑,新增历史 token 保留/幂等、错误原 JWT、截断密文整批回滚 3 项行为回归,均先失败后通过。修复后完整回归为 431 个文件、3576 项通过、11 项跳过,排除项不变;重新构建和类型检查通过。未再次派发审阅,未操作生产。 + +## 配置 DB 迁移与运行时收口验证(2026-11-09) + +新增必跑发布前数据迁移 `20261109-canonical-config`,以数据库事务及配置锁完成旧 GitHub、会员、图片/TTS 和嵌套 SMTP 配置转换,按原路径解密并按新路径加密,删除旧媒体配置行。旧转换器已从 `modules/configs/legacy-config.util.ts` 移到 `upgrade/legacy-config.ts`,仅由发布迁移及离线工具调用;运行时服务、schema、defaults 不再转换或派生旧结构。启动检查配置迁移账本,旧分区读写明确失败,声明字段严格校验,自由 record、大小写归一化、部分更新和空秘密保留仍有效。SDK 配置类型已同步,大版本配套发布时使用,不以 5.x patch 发布。 + +- 新增真实 PostgreSQL 发布 runner 自动迁移、凭据保留/AAD、重复执行、冲突/坏密文/坏数据形状回滚、缺账本启动拦截,以及新配置服务拒绝旧字段/分区和自由 OAuth JSON 的行为回归,均已验证先失败后通过。原有旧接口接受测试已改为迁移或拒绝契约测试。 +- 完整 Core 回归使用 CI 的 UTC 时区与 3 workers:433 个文件、3574 项通过,11 项跳过;继续显式排除原有 Agent 缺 DI 夹具。4 workers 的两次尝试遇到既有 Redis 端口分配冲突,相关文件独立重跑通过;本地零点日期用例在 UTC 通过,未修改这些无关用例。 +- SDK 207 项通过;Core/Admin 生产构建、Core/Admin/SDK/E2E 类型检查、API envelope/验证配对检查、50 个 SQL 迁移 lint 通过。E2E 后端准备在 app.init 前运行真实 app-data runner。 +- mx-docs build/check/links/pangu/lint 与 3 项配置生成测试通过,llms 产物包含维护切换与新配置契约;既有 1 hint、1 死链保持。 +- 维护窗口是此次配置 contract 的部署条件:先停全部旧副本及后台写入者,再运行 migrate,完成准备后启动新服务。未执行线上迁移、未发布版本、未删除物理数据库列。 diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index 2b83331359e..d2ae751a46e 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -37,6 +37,19 @@ After conversion, roll back only to a tested reader of the new formats. Neither disabling encryption nor switching back to old-format writes exists. Preserve original field/JWT secrets and snapshots for recovery. +## Configuration migration before startup + +The current-only config runtime requires the mandatory +`20261109-canonical-config` data migration via `node migrate.mjs` (source: +`pnpm -C apps/core migrate`). Stop all old replicas and background writers +before this contract migration. It converts old GitHub/membership/media and +nested SMTP options, encrypts credentials at their new AAD paths and removes +retired configuration rows. Invalid shapes, unreadable secrets and conflicting +credentials roll back the config batch and block deployment. No activation +flag is needed. Startup requires the migration ledger; configuration reads +and PATCH no longer convert legacy data. Retired sections/unknown fields fail +explicitly. Historical conversion remains in release/offline upgrade code. + ## Storage and API behavior - Sensitive configuration is encrypted **after merging**, before PostgreSQL and Redis writes. Blank secret fields retain the existing value. Responses sanitize stored objects without decrypting them, including when reading historical plaintext. diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index 5920b152b4f..96316f954ad 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -2,8 +2,8 @@ This unreleased branch now requires field encryption, an explicit hex64 key and a JWT of at least 32 characters. Plaintext/ECB writes, encryption-disable -flags and machine-ID fallback have been removed. SDK 6 cleanup and column -removal remain future work. These commands are not part of released v17.0.4. +flags and machine-ID fallback have been removed. Canonical configuration migration and strict configuration PATCH validation +are also enforced here. SDK 6 cleanup and column removal remain future work. These commands are not part of released v17.0.4. No minimum released preparation tag is established yet. Build and deploy this preparation code to every replica before using it; do not publish or migrate production as part of reviewing this branch. @@ -58,11 +58,24 @@ requires recovering the original secret or reauthorizing the account. ## Explicit preparation -Obtain this build without starting business replicas. Run the standalone -check and rotate a short JWT during maintenance first. Remove old disable -flags, then deploy the required secrets and this runtime to all replicas. -Every new write is V2 automatically. Confirm no old writer remains. Use the original field key; only an instance that -has never encrypted data may generate its first random key. +Obtain this build without starting business replicas. Run check and rotate a +short JWT during maintenance first. Stop every old replica/background writer, +remove old encryption flags and use the persisted field/JWT secrets. Run +`node migrate.mjs` (source: `pnpm -C apps/core migrate`) before startup. The +mandatory `20261109-canonical-config` app-data migration converts old config +rows atomically, rebinds secrets to new paths, deletes retired rows and then +records completion. No activation flag is needed. This contract step requires +a maintenance cutover: old versions must not write afterwards. + +The current runtime has no config conversion or synthetic media sections. +Retired section reads/writes fail with CONFIG_NOT_FOUND; old/unknown declared +fields fail with CONFIG_VALIDATION_FAILED. Partial updates, blank secret +retention and freeform record keys remain supported. Startup refuses to load +configuration without the migration record. Historical conversion lives only +in release/offline upgrade code, including nested legacy SMTP options. + +Continue the explicit credential/metrics preparation before starting new +replicas. All writers must already be stopped or use the current formats. ```sh pnpm -C apps/core upgrade:prepare --all-replicas-v2 --dry-run @@ -77,7 +90,7 @@ and membership options to canonical storage, encrypts at new paths, hashes API keys once, protects OAuth, and fills missing AI metrics. Existing client API-key strings continue to verify. It writes the new marker only after no residuals remain. Malformed data or wrong secrets roll the whole transaction -back. Unknown fields retained by v17 are not made strict in this phase. +back. Current configuration PATCH validation is strict. Dry-run executes and rolls back the complete transaction. `would-be-ready` does not mean a marker was persisted. Repeated successful preparation is @@ -87,11 +100,10 @@ authentication, OAuth, configuration consumption and AI metadata. Do not flush all Redis state. A blank canonical credential falls back to a populated legacy credential. -Different nonempty old/new values report `CONFIG_SECRET_CONFLICT` with the -field path and stop preparation. Reads stay compatible; to resolve the -conflict, explicitly save the intended canonical credential in settings, -then run check again. An unrelated partial save of that section is rejected -so it cannot silently delete the other secret. +Different nonempty old/new values report CONFIG_SECRET_CONFLICT and roll back +the configuration batch. Resolve the values in the old installation or an +isolated maintenance environment, then rerun; the new runtime cannot repair +old shapes via PATCH. Never overwrite a lost key with a newly generated key. ## Built artifacts and containers diff --git a/packages/api-client/models/setting.ts b/packages/api-client/models/setting.ts index b84de1a1430..ff7137154da 100644 --- a/packages/api-client/models/setting.ts +++ b/packages/api-client/models/setting.ts @@ -11,15 +11,12 @@ export declare class UrlOptionModel { serverUrl: string wsUrl: string } -declare class SmtpOptionsModel { - port?: number - host?: string - secure?: boolean -} declare class SmtpConfigModel { user?: string pass?: string - options?: SmtpOptionsModel + host?: string + port?: number + secure?: boolean } declare class ResendConfigModel { apiKey?: string @@ -67,7 +64,29 @@ export declare class AdminExtraModel { } export declare class ThirdPartyServiceIntegrationModel { - githubToken?: string + github?: { enabled?: boolean; token?: string } + tmdb?: { enabled?: boolean; apiKey?: string } + bangumi?: { enabled?: boolean; accessToken?: string } + neodb?: { enabled?: boolean } + arxiv?: { enabled?: boolean } + leetcode?: { enabled?: boolean } + neteaseMusic?: { enabled?: boolean } + qqMusic?: { enabled?: boolean } + openGraph?: { + enabled?: boolean + fetchMode?: 'fetch' | 'browser' + timeoutMs?: number + maxBodyBytes?: number + screenshot?: { + enabled?: boolean + maxItems?: number + maxTotalBytes?: number + maxBytesPerImage?: number + webpQuality?: number + } + } + twelveData?: { enabled?: boolean; apiKey?: string } + polygon?: { enabled?: boolean; apiKey?: string } } export interface IConfig { diff --git a/packages/e2e/src/helpers/e2e-app.ts b/packages/e2e/src/helpers/e2e-app.ts index e49739dec1b..2fdc1062e44 100644 --- a/packages/e2e/src/helpers/e2e-app.ts +++ b/packages/e2e/src/helpers/e2e-app.ts @@ -78,6 +78,9 @@ export async function createE2EBackend(): Promise { // was set. Match production bootstrap; do not pull in platform-socket.io. app.useWebSocketAdapter(createWsAdapter(app)) + const { AppMigrationsService } = + await import('~/database/app-migrations/app-migrations.service') + await app.get(AppMigrationsService).run() await app.init() await app.getHttpAdapter().getInstance().ready() await app.listen(0, '127.0.0.1') From d8debe9ba11f37ad78faf1b4deb8730949f726b6 Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 00:49:09 +0800 Subject: [PATCH 20/26] fix(config): preserve raw configuration and sibling credentials during migration --- apps/core/src/upgrade/config-migration.ts | 4 +- apps/core/src/upgrade/legacy-config.ts | 29 ++---- .../canonical-config.migration.spec.ts | 94 +++++++++++++++++++ ...-11-08-mx-core-v18-compatibility-design.md | 4 + 4 files changed, 109 insertions(+), 22 deletions(-) diff --git a/apps/core/src/upgrade/config-migration.ts b/apps/core/src/upgrade/config-migration.ts index 400b21d7325..a2faa773a84 100644 --- a/apps/core/src/upgrade/config-migration.ts +++ b/apps/core/src/upgrade/config-migration.ts @@ -45,9 +45,7 @@ export async function migrateConfiguration( ).rows const raw = Object.fromEntries(rows.map((row) => [row.name, row.value])) for (const key of [ - 'ai', - 'membership', - 'thirdPartyServiceIntegration', + ...Object.keys(configSchemaMapping), 'imageGenerationOptions', 'ttsOptions', ]) { diff --git a/apps/core/src/upgrade/legacy-config.ts b/apps/core/src/upgrade/legacy-config.ts index 6e0c6789ca4..5ec0ea07b2e 100644 --- a/apps/core/src/upgrade/legacy-config.ts +++ b/apps/core/src/upgrade/legacy-config.ts @@ -249,26 +249,17 @@ export function normalizeAiConfig( export function normalizeThirdPartyConfig( raw: Record, ): IConfig['thirdPartyServiceIntegration'] { - // Already new shape - if (raw?.github && typeof raw.github === 'object') { - return { - ...raw, - github: { - ...raw.github, - token: legacySecret(raw.github.token, raw.githubToken), - }, - } as IConfig['thirdPartyServiceIntegration'] - } - // Old flat shape — convert + const { githubToken, ...current } = raw + if (!Object.hasOwn(raw, 'githubToken')) + return current as IConfig['thirdPartyServiceIntegration'] + // Converting GitHub must preserve every independent integration and secret. return { - github: { enabled: true, token: raw.githubToken || '' }, - tmdb: { enabled: false, apiKey: '' }, - bangumi: { enabled: true, accessToken: '' }, - neodb: { enabled: true }, - arxiv: { enabled: true }, - leetcode: { enabled: true }, - neteaseMusic: { enabled: true }, - qqMusic: { enabled: true }, + ...current, + github: { + enabled: true, + ...raw.github, + token: legacySecret(raw.github?.token, githubToken), + }, } as IConfig['thirdPartyServiceIntegration'] } diff --git a/apps/core/test/src/database/canonical-config.migration.spec.ts b/apps/core/test/src/database/canonical-config.migration.spec.ts index fa911ffcaf5..89727a4e03e 100644 --- a/apps/core/test/src/database/canonical-config.migration.spec.ts +++ b/apps/core/test/src/database/canonical-config.migration.spec.ts @@ -7,6 +7,100 @@ import { EncryptUtil } from '~/utils/encrypt.util' const migrationId = '20261109-canonical-config' +it.each([false, true])( + 'preserves configured sibling integrations when converting a section without GitHub (legacy flat token=%s)', + async (legacy) => { + const context = await createPgTestDatabase('mx_integration_preservation', { + configCurrent: false, + }) + try { + await context.pool.query( + "INSERT INTO options (id,name,value) VALUES ('100','thirdPartyServiceIntegration',$1)", + [ + { + tmdb: { + enabled: true, + apiKey: EncryptUtil.encrypt( + 'existing-tmdb-private', + 'thirdPartyServiceIntegration.tmdb.apiKey', + ), + }, + bangumi: { enabled: true, accessToken: 'existing-bangumi-private' }, + ...(legacy ? { githubToken: 'imported-github-private' } : {}), + }, + ], + ) + await new AppMigrationsService(context.db, context.pool).run() + const value = ( + await context.pool.query( + "SELECT value FROM options WHERE name='thirdPartyServiceIntegration'", + ) + ).rows[0].value + expect(value.tmdb.enabled).toBe(true) + expect( + EncryptUtil.decrypt( + value.tmdb.apiKey, + 'thirdPartyServiceIntegration.tmdb.apiKey', + ), + ).toBe('existing-tmdb-private') + expect( + EncryptUtil.decrypt( + value.bangumi.accessToken, + 'thirdPartyServiceIntegration.bangumi.accessToken', + ), + ).toBe('existing-bangumi-private') + expect(value).not.toHaveProperty('githubToken') + if (legacy) + expect( + EncryptUtil.decrypt( + value.github.token, + 'thirdPartyServiceIntegration.github.token', + ), + ).toBe('imported-github-private') + } finally { + await context.close() + } + }, +) + +it.each(['mailOptions', 'seo', 'backupOptions'])( + 'rejects malformed raw %s without defaulting away its contents', + async (section) => { + const context = await createPgTestDatabase('mx_raw_config_preservation', { + configCurrent: false, + }) + try { + await context.pool.query( + "INSERT INTO options (id,name,value) VALUES ('100',$1,$2),('101','membership',$3)", + [ + section, + JSON.stringify(['unknown-private-content']), + { dodoApiKey: 'preserve-billing-private' }, + ], + ) + const before = ( + await context.pool.query('SELECT * FROM options ORDER BY name') + ).rows + await expect( + new AppMigrationsService(context.db, context.pool).run(), + ).rejects.toThrow('CONFIG_SHAPE_UNSUPPORTED') + expect( + (await context.pool.query('SELECT * FROM options ORDER BY name')).rows, + ).toEqual(before) + expect( + ( + await context.pool.query( + 'SELECT id FROM _app_migrations WHERE id=$1', + [migrationId], + ) + ).rowCount, + ).toBe(0) + } finally { + await context.close() + } + }, +) + it('blocks configuration startup until the release migration has completed, including a new installation', async () => { const context = await createPgTestDatabase('mx_config_guard', { configCurrent: false, diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 0b93dd8a56a..916e0505f2f 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -199,3 +199,7 @@ flowchart TD - SDK 207 项通过;Core/Admin 生产构建、Core/Admin/SDK/E2E 类型检查、API envelope/验证配对检查、50 个 SQL 迁移 lint 通过。E2E 后端准备在 app.init 前运行真实 app-data runner。 - mx-docs build/check/links/pangu/lint 与 3 项配置生成测试通过,llms 产物包含维护切换与新配置契约;既有 1 hint、1 死链保持。 - 维护窗口是此次配置 contract 的部署条件:先停全部旧副本及后台写入者,再运行 migrate,完成准备后启动新服务。未执行线上迁移、未发布版本、未删除物理数据库列。 + +最终配置审阅:无 Critical、2 项 Important、无 Minor。已修复「未覆盖的原始分区坏形状被默认值覆盖」和「缺少可选 GitHub 时其他第三方凭据被重置」两处数据保留问题,新增 5 项真实数据库回归并验证先失败后通过;修复后的配置/升级定向回归 67 项通过,构建、类型检查通过。未再次派发审阅。 + +修复后完整 UTC/3 workers 回归实际结果:3576 项通过,11 项既定跳过,另 1 个未触及的 Push Relay SQL 迁移测试文件因 Redis 端口竞争在 setup 阶段失败、3 项未执行;该文件独立重跑 3 项全部通过。合计 3579 项行为已验证,所有代码断言通过;原完整命令因基础设施错误退出非零,不能称该次单命令全绿。Agent 排除项不变。67 项配置/升级定向回归及重新构建、类型检查通过;无未修复 Critical/Important 或延后 Minor,未再次派发审阅。 From c9d7baac000122d2f7db4ac6fb158d06f8ea0451 Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 01:07:33 +0800 Subject: [PATCH 21/26] refactor(db): move shared migration helpers out of upgrade --- .../20261018-protect-stored-secrets.ts | 2 +- .../app-migrations/20261109-canonical-config.ts | 2 +- .../app-migrations/helpers}/config-migration.ts | 7 ++++--- .../app-migrations/helpers}/legacy-config.ts | 7 +++---- .../app-migrations/helpers}/oauth-token.ts | 0 apps/core/src/upgrade/jwt-rotation.ts | 5 ++++- apps/core/src/upgrade/preparation.ts | 4 ++-- apps/core/src/upgrade/readiness.ts | 17 ++++++++++------- .../src/database/protect-stored-secrets.spec.ts | 2 +- .../src/modules/configs/legacy-config.spec.ts | 4 ++-- apps/core/test/src/upgrade/jwt-rotation.spec.ts | 2 +- ...26-11-08-mx-core-v18-compatibility-design.md | 6 +++++- 12 files changed, 34 insertions(+), 24 deletions(-) rename apps/core/src/{upgrade => database/app-migrations/helpers}/config-migration.ts (94%) rename apps/core/src/{upgrade => database/app-migrations/helpers}/legacy-config.ts (98%) rename apps/core/src/{upgrade => database/app-migrations/helpers}/oauth-token.ts (100%) diff --git a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts index 140fc72bd23..c4e955e7138 100644 --- a/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts +++ b/apps/core/src/database/app-migrations/20261018-protect-stored-secrets.ts @@ -12,9 +12,9 @@ import { } from '~/database/schema' import { HASHED_API_KEY_CONFIG_ID } from '~/modules/auth/auth.constant' import { encryptObject } from '~/modules/configs/configs.encrypt.util' -import { unwrapOAuthToken } from '~/upgrade/oauth-token' import { EncryptUtil } from '~/utils/encrypt.util' +import { unwrapOAuthToken } from './helpers/oauth-token' import type { AppMigration } from './types' export const migration: AppMigration = { diff --git a/apps/core/src/database/app-migrations/20261109-canonical-config.ts b/apps/core/src/database/app-migrations/20261109-canonical-config.ts index c6c3ed49d06..291d9106db6 100644 --- a/apps/core/src/database/app-migrations/20261109-canonical-config.ts +++ b/apps/core/src/database/app-migrations/20261109-canonical-config.ts @@ -1,9 +1,9 @@ import { ENCRYPT } from '~/app.config' import { CONFIG_FORMAT_MIGRATION_ID } from '~/shared/config-format' import { CONFIG_UPGRADE_LOCK } from '~/shared/config-lock' -import { migrateConfiguration } from '~/upgrade/config-migration' import { SecretCodec } from '~/utils/secret-codec' +import { migrateConfiguration } from './helpers/config-migration' import type { AppMigration } from './types' export const migration: AppMigration = { diff --git a/apps/core/src/upgrade/config-migration.ts b/apps/core/src/database/app-migrations/helpers/config-migration.ts similarity index 94% rename from apps/core/src/upgrade/config-migration.ts rename to apps/core/src/database/app-migrations/helpers/config-migration.ts index a2faa773a84..6d85b542b6a 100644 --- a/apps/core/src/upgrade/config-migration.ts +++ b/apps/core/src/database/app-migrations/helpers/config-migration.ts @@ -6,9 +6,10 @@ import type { PoolClient } from 'pg' import { isValidStoredConfig, loadCurrentConfig, -} from '../modules/configs/configs.current' -import { configSchemaMapping } from '../modules/configs/configs.schema' -import type { SecretCodec } from '../utils/secret-codec' +} from '~/modules/configs/configs.current' +import { configSchemaMapping } from '~/modules/configs/configs.schema' +import type { SecretCodec } from '~/utils/secret-codec' + import { normalizeLegacyConfig, visitLegacyConfigSecrets, diff --git a/apps/core/src/upgrade/legacy-config.ts b/apps/core/src/database/app-migrations/helpers/legacy-config.ts similarity index 98% rename from apps/core/src/upgrade/legacy-config.ts rename to apps/core/src/database/app-migrations/helpers/legacy-config.ts index 5ec0ea07b2e..13a5258f40f 100644 --- a/apps/core/src/upgrade/legacy-config.ts +++ b/apps/core/src/database/app-migrations/helpers/legacy-config.ts @@ -1,13 +1,12 @@ import { cloneDeep } from 'es-toolkit/compat' import { AIProviderType } from '~/modules/ai/ai.types' - -import { generateDefaultConfig } from '../modules/configs/configs.default' -import type { IConfig } from '../modules/configs/configs.interface' +import { generateDefaultConfig } from '~/modules/configs/configs.default' +import type { IConfig } from '~/modules/configs/configs.interface' import { isEncryptedPath, visitConfigSecrets, -} from '../modules/configs/configs.secret-paths' +} from '~/modules/configs/configs.secret-paths' const legacyDefaults = { image: { diff --git a/apps/core/src/upgrade/oauth-token.ts b/apps/core/src/database/app-migrations/helpers/oauth-token.ts similarity index 100% rename from apps/core/src/upgrade/oauth-token.ts rename to apps/core/src/database/app-migrations/helpers/oauth-token.ts diff --git a/apps/core/src/upgrade/jwt-rotation.ts b/apps/core/src/upgrade/jwt-rotation.ts index 54b3a325b33..e7071593de4 100644 --- a/apps/core/src/upgrade/jwt-rotation.ts +++ b/apps/core/src/upgrade/jwt-rotation.ts @@ -3,7 +3,10 @@ import { readFile, stat } from 'node:fs/promises' import { symmetricEncrypt } from 'better-auth/crypto' import type { Pool } from 'pg' -import { oauthKey, unwrapOAuthToken } from './oauth-token' +import { + oauthKey, + unwrapOAuthToken, +} from '../database/app-migrations/helpers/oauth-token' import { UpgradeError } from './preparation' import { CONFIG_UPGRADE_LOCK } from './readiness' diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts index 6e9c6ffdc86..7715316a818 100644 --- a/apps/core/src/upgrade/preparation.ts +++ b/apps/core/src/upgrade/preparation.ts @@ -3,11 +3,11 @@ import { symmetricEncrypt } from 'better-auth/crypto' import type { Pool } from 'pg' import { backfillAiMetrics } from '../database/app-migrations/20261108-ai-metrics-coverage' +import { migrateConfiguration } from '../database/app-migrations/helpers/config-migration' +import { unwrapOAuthToken } from '../database/app-migrations/helpers/oauth-token' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' import { CONFIG_FORMAT_MIGRATION_ID } from '../shared/config-format' import { SecretCodec } from '../utils/secret-codec' -import { migrateConfiguration } from './config-migration' -import { unwrapOAuthToken } from './oauth-token' import { CONFIG_UPGRADE_LOCK, oauthKey, diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index cfb96c45e2b..b5762802101 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -1,5 +1,14 @@ import type { Pool, PoolClient } from 'pg' +import { + LegacyConfigConflictError, + normalizeLegacyConfig, + visitLegacyConfigSecrets, +} from '../database/app-migrations/helpers/legacy-config' +import { + isEncryptedOAuth, + unwrapOAuthToken, +} from '../database/app-migrations/helpers/oauth-token' import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' import { isValidStoredConfig, @@ -12,14 +21,8 @@ import { inspectSecretStorageConfig, type SecretStorageConfig, } from '../utils/secret-storage-config' -import { - LegacyConfigConflictError, - normalizeLegacyConfig, - visitLegacyConfigSecrets, -} from './legacy-config' -import { isEncryptedOAuth, unwrapOAuthToken } from './oauth-token' -export { oauthKey } from './oauth-token' +export { oauthKey } from '../database/app-migrations/helpers/oauth-token' export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' export { CONFIG_UPGRADE_LOCK } from '../shared/config-lock' diff --git a/apps/core/test/src/database/protect-stored-secrets.spec.ts b/apps/core/test/src/database/protect-stored-secrets.spec.ts index 44c746b19c7..34691a68416 100644 --- a/apps/core/test/src/database/protect-stored-secrets.spec.ts +++ b/apps/core/test/src/database/protect-stored-secrets.spec.ts @@ -6,6 +6,7 @@ import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { ENCRYPT, SECURITY } from '~/app.config' import { migration } from '~/database/app-migrations/20261018-protect-stored-secrets' +import { oauthKey } from '~/database/app-migrations/helpers/oauth-token' import { CreateAuth } from '~/modules/auth/auth.implement' import { AuthRepository } from '~/modules/auth/auth.repository' import { AuthService } from '~/modules/auth/auth.service' @@ -17,7 +18,6 @@ import { SnippetRepository } from '~/modules/snippet/snippet.repository' import { WebhookRepository } from '~/modules/webhook/webhook.repository' import { __setTestPostgresInstance } from '~/processors/database/postgres.provider' import { SnowflakeService } from '~/shared/id/snowflake.service' -import { oauthKey } from '~/upgrade/oauth-token' import { EncryptUtil } from '~/utils/encrypt.util' it('upgrades real credential rows without changing their usable values, and can run twice', async () => { diff --git a/apps/core/test/src/modules/configs/legacy-config.spec.ts b/apps/core/test/src/modules/configs/legacy-config.spec.ts index 83a20168e84..0c35d1ef69e 100644 --- a/apps/core/test/src/modules/configs/legacy-config.spec.ts +++ b/apps/core/test/src/modules/configs/legacy-config.spec.ts @@ -1,10 +1,10 @@ import { describe, expect, it } from 'vitest' -import { encryptObject } from '~/modules/configs/configs.encrypt.util' import { normalizeLegacyConfig, visitLegacyConfigSecrets, -} from '~/upgrade/legacy-config' +} from '~/database/app-migrations/helpers/legacy-config' +import { encryptObject } from '~/modules/configs/configs.encrypt.util' import { EncryptUtil } from '~/utils/encrypt.util' describe('canonical legacy config conversion', () => { diff --git a/apps/core/test/src/upgrade/jwt-rotation.spec.ts b/apps/core/test/src/upgrade/jwt-rotation.spec.ts index 309fd20ddf3..de626195cb3 100644 --- a/apps/core/test/src/upgrade/jwt-rotation.spec.ts +++ b/apps/core/test/src/upgrade/jwt-rotation.spec.ts @@ -6,8 +6,8 @@ import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' import { createPgTestDatabase } from 'test/helper/pg-verify-url' import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' +import { oauthKey } from '~/database/app-migrations/helpers/oauth-token' import { rotateJwtSecret } from '~/upgrade/jwt-rotation' -import { oauthKey } from '~/upgrade/oauth-token' describe('maintenance JWT rotation', () => { let context: Awaited> diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 916e0505f2f..b7d2248be60 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -192,7 +192,7 @@ flowchart TD ## 配置 DB 迁移与运行时收口验证(2026-11-09) -新增必跑发布前数据迁移 `20261109-canonical-config`,以数据库事务及配置锁完成旧 GitHub、会员、图片/TTS 和嵌套 SMTP 配置转换,按原路径解密并按新路径加密,删除旧媒体配置行。旧转换器已从 `modules/configs/legacy-config.util.ts` 移到 `upgrade/legacy-config.ts`,仅由发布迁移及离线工具调用;运行时服务、schema、defaults 不再转换或派生旧结构。启动检查配置迁移账本,旧分区读写明确失败,声明字段严格校验,自由 record、大小写归一化、部分更新和空秘密保留仍有效。SDK 配置类型已同步,大版本配套发布时使用,不以 5.x patch 发布。 +新增必跑发布前数据迁移 `20261109-canonical-config`,以数据库事务及配置锁完成旧 GitHub、会员、图片/TTS 和嵌套 SMTP 配置转换,按原路径解密并按新路径加密,删除旧媒体配置行。旧转换器已从 `modules/configs/legacy-config.util.ts` 移到 `database/app-migrations/helpers/legacy-config.ts`,仅由发布迁移及离线工具调用;运行时服务、schema、defaults 不再转换或派生旧结构。启动检查配置迁移账本,旧分区读写明确失败,声明字段严格校验,自由 record、大小写归一化、部分更新和空秘密保留仍有效。SDK 配置类型已同步,大版本配套发布时使用,不以 5.x patch 发布。 - 新增真实 PostgreSQL 发布 runner 自动迁移、凭据保留/AAD、重复执行、冲突/坏密文/坏数据形状回滚、缺账本启动拦截,以及新配置服务拒绝旧字段/分区和自由 OAuth JSON 的行为回归,均已验证先失败后通过。原有旧接口接受测试已改为迁移或拒绝契约测试。 - 完整 Core 回归使用 CI 的 UTC 时区与 3 workers:433 个文件、3574 项通过,11 项跳过;继续显式排除原有 Agent 缺 DI 夹具。4 workers 的两次尝试遇到既有 Redis 端口分配冲突,相关文件独立重跑通过;本地零点日期用例在 UTC 通过,未修改这些无关用例。 @@ -203,3 +203,7 @@ flowchart TD 最终配置审阅:无 Critical、2 项 Important、无 Minor。已修复「未覆盖的原始分区坏形状被默认值覆盖」和「缺少可选 GitHub 时其他第三方凭据被重置」两处数据保留问题,新增 5 项真实数据库回归并验证先失败后通过;修复后的配置/升级定向回归 67 项通过,构建、类型检查通过。未再次派发审阅。 修复后完整 UTC/3 workers 回归实际结果:3576 项通过,11 项既定跳过,另 1 个未触及的 Push Relay SQL 迁移测试文件因 Redis 端口竞争在 setup 阶段失败、3 项未执行;该文件独立重跑 3 项全部通过。合计 3579 项行为已验证,所有代码断言通过;原完整命令因基础设施错误退出非零,不能称该次单命令全绿。Agent 排除项不变。67 项配置/升级定向回归及重新构建、类型检查通过;无未修复 Critical/Important 或延后 Minor,未再次派发审阅。 + +共享模块归位(2026-11-09):配置迁移、历史配置转换和 OAuth 解封装已统一归入 `database/app-migrations/helpers`。正式 app-data migrations 只依赖该目录,独立升级 CLI 也调用这些函数;`src/upgrade` 保留检查、准备和 JWT 轮换命令逻辑,不再被正式迁移反向引用。本次仅调整模块归属与引用路径。 + +模块归位验证:现有历史转换行为测试先改用新模块路径,验证迁移前因模块未就位而失败;移动后 11 个文件、71 项定向测试通过,包含真实发布迁移、凭据保留、JWT 轮换和构建后 CLI 恢复演练。Core 构建、Core/E2E 类型检查、响应/验证检查、50 个迁移 lint 通过;正式迁移不再引用 `src/upgrade`。本次没有更改数据转换算法或迁移 ID。 From 665a0d608f751b506081b6251754e04b626cf77c Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 01:38:45 +0800 Subject: [PATCH 22/26] refactor(db): consolidate v18 preparation into release migrations --- apps/core/README.md | 12 +- apps/core/package.json | 2 - .../20261108-ai-metrics-coverage.ts | 2 +- .../20261109-required-secret-storage.ts | 10 + .../src/database/app-migrations/registry.ts | 4 +- apps/core/src/upgrade/cli.ts | 22 +- apps/core/src/upgrade/jwt-rotation.ts | 4 +- apps/core/src/upgrade/preparation.ts | 138 ------------ apps/core/src/upgrade/readiness.ts | 16 +- apps/core/src/upgrade/upgrade.error.ts | 8 + .../required-secret-storage.migration.spec.ts | 164 ++++++++++++++ .../src/upgrade/archive-rehearsal.spec.ts | 56 +++-- apps/core/test/src/upgrade/cli.spec.ts | 18 ++ .../core/test/src/upgrade/preparation.spec.ts | 203 ------------------ apps/core/test/src/upgrade/readiness.spec.ts | 19 +- ...-11-08-mx-core-v18-compatibility-design.md | 30 ++- docs/security-secret-storage.md | 13 +- docs/v18-preparation.md | 54 +++-- 18 files changed, 322 insertions(+), 453 deletions(-) create mode 100644 apps/core/src/database/app-migrations/20261109-required-secret-storage.ts delete mode 100644 apps/core/src/upgrade/preparation.ts create mode 100644 apps/core/src/upgrade/upgrade.error.ts create mode 100644 apps/core/test/src/database/required-secret-storage.migration.spec.ts delete mode 100644 apps/core/test/src/upgrade/preparation.spec.ts diff --git a/apps/core/README.md b/apps/core/README.md index 88c979e3368..2394daa6c2e 100644 --- a/apps/core/README.md +++ b/apps/core/README.md @@ -305,13 +305,15 @@ Since 2021-08-31. ## Preparing for v18 (unreleased) -This branch adds standalone `upgrade:check`, `upgrade:prepare` and -`upgrade:rotate-jwt` tooling. These commands are not in v17.0.4; a minimum +This branch adds standalone `upgrade:check` and `upgrade:rotate-jwt` tooling. +Configuration, credential and metric backfills run through the mandatory +release-phase `migrate` command. These changes are not in v17.0.4; a minimum released preparation tag and client matrix will be recorded after release -verification. v17 behavior remains compatible, including legacy GET actions -and SDK 5.x signatures. No columns are dropped in this phase. +verification. Legacy GET actions and SDK 5.x signatures remain, while this +branch requires encryption and canonical configuration. No columns are +dropped in this phase. See the [preparation runbook](../../docs/v18-preparation.md) for read-only -diagnostics, dry-run/transactional preparation, original-secret requirements +diagnostics, release-phase migration, original-secret requirements and offline JWT rotation. Do not change an existing encryption key or JWT to fix decryption failures. diff --git a/apps/core/package.json b/apps/core/package.json index e5ae771862b..0f8ce2fde88 100644 --- a/apps/core/package.json +++ b/apps/core/package.json @@ -19,7 +19,6 @@ "build": "npm run bundle", "dev": "cross-env NODE_ENV=development nodemon --signal SIGKILL --watch src --ext ts,json --exec \"vite-node src/dev.ts\"", "migrate": "vite-node src/migrate.ts", - "migrate:secrets": "cross-env MX_MIGRATE_SECRETS=1 vite-node src/app-migrate.ts", "migrate:app": "cross-env NODE_ENV=development vite-node src/app-migrate.ts", "lint:migrations": "tsx --tsconfig tsconfig.json scripts/lint-migrations.ts", "openapi:export": "vite-node scripts/export-openapi.ts", @@ -46,7 +45,6 @@ "docs": "npx @compodoc/compodoc -p tsconfig.json -s -d docs", "publish": "bump", "upgrade:check": "vite-node --config vite.upgrade.config.ts src/upgrade.ts check", - "upgrade:prepare": "vite-node --config vite.upgrade.config.ts src/upgrade.ts prepare", "upgrade:rotate-jwt": "vite-node --config vite.upgrade.config.ts src/upgrade.ts rotate-jwt" }, "bump": { diff --git a/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts b/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts index b75f5a47b95..d7cb991b151 100644 --- a/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts +++ b/apps/core/src/database/app-migrations/20261108-ai-metrics-coverage.ts @@ -2,7 +2,7 @@ import type { Pool, PoolClient } from 'pg' import type { AppMigration } from './types' -/** Reusable for explicit preparation; never overwrites a recorded generation. */ +/** Backfill missing metadata without overwriting a recorded generation. */ export async function backfillAiMetrics( client: Pick, ): Promise { diff --git a/apps/core/src/database/app-migrations/20261109-required-secret-storage.ts b/apps/core/src/database/app-migrations/20261109-required-secret-storage.ts new file mode 100644 index 00000000000..8ad0984144e --- /dev/null +++ b/apps/core/src/database/app-migrations/20261109-required-secret-storage.ts @@ -0,0 +1,10 @@ +import { migration as protectStoredSecrets } from './20261018-protect-stored-secrets' +import type { AppMigration } from './types' + +/** Recheck at the cutover even if the opt-in historical migration was applied. */ +export const migration: AppMigration = { + id: '20261109-required-secret-storage', + description: + 'Require encrypted credentials and hashed API keys before v18 startup', + up: protectStoredSecrets.up, +} diff --git a/apps/core/src/database/app-migrations/registry.ts b/apps/core/src/database/app-migrations/registry.ts index 35dcd52fc5f..5758bcddb64 100644 --- a/apps/core/src/database/app-migrations/registry.ts +++ b/apps/core/src/database/app-migrations/registry.ts @@ -1,9 +1,9 @@ import { migration as recentlyDropEnrichmentColumns } from './20260515-recently-drop-enrichment-columns' import { migration as treeContentRevisions } from './20260831-tree-content-revisions' -import { migration as protectStoredSecrets } from './20261018-protect-stored-secrets' import { migration as mintLoroBlockIds } from './20261026-mint-loro-block-ids' import { migration as aiMetricsCoverage } from './20261108-ai-metrics-coverage' import { migration as canonicalConfig } from './20261109-canonical-config' +import { migration as requiredSecretStorage } from './20261109-required-secret-storage' import type { AppMigration } from './types' /** @@ -17,8 +17,8 @@ import type { AppMigration } from './types' export const migrations: AppMigration[] = [ recentlyDropEnrichmentColumns, treeContentRevisions, - ...(process.env.MX_MIGRATE_SECRETS === '1' ? [protectStoredSecrets] : []), mintLoroBlockIds, aiMetricsCoverage, canonicalConfig, + requiredSecretStorage, ] diff --git a/apps/core/src/upgrade/cli.ts b/apps/core/src/upgrade/cli.ts index 3a7fdc1beee..66943d42a43 100644 --- a/apps/core/src/upgrade/cli.ts +++ b/apps/core/src/upgrade/cli.ts @@ -4,8 +4,8 @@ import { Pool } from 'pg' import { secretStorageConfigFromEnv } from '../utils/secret-storage-config' import { rotateJwtSecret } from './jwt-rotation' -import { prepareUpgrade, UpgradeError } from './preparation' import { readUpgradeState, type UpgradeConfig } from './readiness' +import { UpgradeError } from './upgrade.error' export function upgradeConfigFromEnv(env: NodeJS.ProcessEnv): UpgradeConfig { return secretStorageConfigFromEnv(env) @@ -23,12 +23,6 @@ export async function runUpgradeCli( args: flags, options: { json: { type: 'boolean' }, - ...(command === 'prepare' - ? { - 'dry-run': { type: 'boolean' as const }, - 'all-replicas-v2': { type: 'boolean' as const }, - } - : {}), ...(command === 'rotate-jwt' ? { 'maintenance-confirmed': { type: 'boolean' as const }, @@ -42,9 +36,9 @@ export async function runUpgradeCli( write('Invalid upgrade arguments') return 1 } - if (!['check', 'prepare', 'rotate-jwt'].includes(command)) { + if (!['check', 'rotate-jwt'].includes(command)) { write( - 'Usage: upgrade check [--json] | prepare --all-replicas-v2 [--dry-run] [--json]', + 'Usage: upgrade check [--json] | rotate-jwt --maintenance-confirmed --old-jwt-file --new-jwt-file ', ) return 1 } @@ -74,14 +68,6 @@ export async function runUpgradeCli( write(JSON.stringify(result)) return 0 } - if (command === 'prepare') { - const result = await prepareUpgrade(pool, upgradeConfigFromEnv(env), { - allReplicasV2: flags.includes('--all-replicas-v2'), - dryRun: flags.includes('--dry-run'), - }) - write(JSON.stringify(result)) - return 0 - } const client = await pool.connect() try { await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY') @@ -109,7 +95,7 @@ export async function runUpgradeCli( code: command === 'check' ? 'UPGRADE_CHECK_FAILED' - : 'UPGRADE_PREPARATION_FAILED', + : 'JWT_ROTATION_FAILED', }, ], }), diff --git a/apps/core/src/upgrade/jwt-rotation.ts b/apps/core/src/upgrade/jwt-rotation.ts index e7071593de4..e52b561106a 100644 --- a/apps/core/src/upgrade/jwt-rotation.ts +++ b/apps/core/src/upgrade/jwt-rotation.ts @@ -7,8 +7,8 @@ import { oauthKey, unwrapOAuthToken, } from '../database/app-migrations/helpers/oauth-token' -import { UpgradeError } from './preparation' -import { CONFIG_UPGRADE_LOCK } from './readiness' +import { CONFIG_UPGRADE_LOCK } from '../shared/config-lock' +import { UpgradeError } from './upgrade.error' async function readSecretFile(file: string) { try { diff --git a/apps/core/src/upgrade/preparation.ts b/apps/core/src/upgrade/preparation.ts deleted file mode 100644 index 7715316a818..00000000000 --- a/apps/core/src/upgrade/preparation.ts +++ /dev/null @@ -1,138 +0,0 @@ -import { defaultKeyHasher } from '@better-auth/api-key' -import { symmetricEncrypt } from 'better-auth/crypto' -import type { Pool } from 'pg' - -import { backfillAiMetrics } from '../database/app-migrations/20261108-ai-metrics-coverage' -import { migrateConfiguration } from '../database/app-migrations/helpers/config-migration' -import { unwrapOAuthToken } from '../database/app-migrations/helpers/oauth-token' -import { HASHED_API_KEY_CONFIG_ID } from '../modules/auth/auth.constant' -import { CONFIG_FORMAT_MIGRATION_ID } from '../shared/config-format' -import { SecretCodec } from '../utils/secret-codec' -import { - CONFIG_UPGRADE_LOCK, - oauthKey, - PREPARED_MIGRATION_ID, - readUpgradeState, - type UpgradeConfig, -} from './readiness' - -export class UpgradeError extends Error { - constructor( - public readonly code: string, - public readonly path?: string, - ) { - super(path ? `${code} (${path})` : code) - } -} - -export async function prepareUpgrade( - pool: Pool, - config: UpgradeConfig, - options: { allReplicasV2: boolean; dryRun?: boolean }, -) { - if (!options.allReplicasV2) - throw new UpgradeError('REPLICA_CONFIRMATION_REQUIRED') - const client = await pool.connect() - const codec = new SecretCodec(config) - let changed = 0 - try { - await client.query('BEGIN') - await client.query("SET LOCAL lock_timeout = '10s'") - await client.query('SELECT pg_advisory_xact_lock($1::bigint)', [ - CONFIG_UPGRADE_LOCK, - ]) - const before = await readUpgradeState(client, config) - const allowed = new Set([ - 'CREDENTIALS_NEED_PREPARATION', - 'LEGACY_CONFIG', - 'METRICS_INCOMPLETE', - 'PREPARATION_MARKER_MISSING', - 'CONFIG_MIGRATION_MISSING', - ]) - const blocking = before.issues.find((issue) => !allowed.has(issue.code)) - if (blocking) throw new UpgradeError(blocking.code, blocking.path) - changed += await migrateConfiguration(client, codec) - for (const table of ['snippets', 'webhooks']) { - for (const row of ( - await client.query(`SELECT id, secret FROM ${table} FOR UPDATE`) - ).rows) { - if (!row.secret) continue - const secret = codec.encrypt(row.secret, `${table}.${row.id}.secret`) - if (secret === row.secret) continue - await client.query(`UPDATE ${table} SET secret = $1 WHERE id = $2`, [ - secret, - row.id, - ]) - changed++ - } - } - for (const row of ( - await client.query( - 'SELECT id, key, config_id, user_id, reference_id, start FROM api_keys FOR UPDATE', - ) - ).rows) { - if (row.config_id === HASHED_API_KEY_CONFIG_ID) continue - await client.query( - 'UPDATE api_keys SET key = $1, config_id = $2, reference_id = $3, start = $4 WHERE id = $5', - [ - await defaultKeyHasher(row.key), - HASHED_API_KEY_CONFIG_ID, - row.reference_id || row.user_id, - row.start || row.key.slice(0, 6), - row.id, - ], - ) - changed++ - } - for (const row of ( - await client.query( - 'SELECT id, access_token, refresh_token FROM accounts FOR UPDATE', - ) - ).rows) { - for (const field of ['access_token', 'refresh_token']) { - const token = row[field] - if (!token || token.startsWith('$ba$')) continue // Authenticated by readUpgradeState. - const encrypted = await symmetricEncrypt({ - key: oauthKey(config.jwtSecret), - data: await unwrapOAuthToken(token, config.jwtSecret), - }) - await client.query(`UPDATE accounts SET ${field} = $1 WHERE id = $2`, [ - encrypted, - row.id, - ]) - changed++ - } - } - changed += await backfillAiMetrics(client) - await client.query( - 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', - [CONFIG_FORMAT_MIGRATION_ID], - ) - const after = await readUpgradeState(client, config) - const residual = after.issues.find( - (issue) => issue.code !== 'PREPARATION_MARKER_MISSING', - ) - if (residual) throw new UpgradeError(residual.code, residual.path) - if (!after.prepared) { - await client.query( - 'INSERT INTO _app_migrations (id) VALUES ($1) ON CONFLICT (id) DO NOTHING', - [PREPARED_MIGRATION_ID], - ) - } - - await client.query(options.dryRun ? 'ROLLBACK' : 'COMMIT') - return { - status: options.dryRun ? 'would-be-ready' : 'prepared', - dryRun: !!options.dryRun, - changed, - requiresRestart: !options.dryRun, - } - } catch (error) { - await client.query('ROLLBACK').catch(() => {}) - throw error instanceof UpgradeError - ? error - : new UpgradeError('UPGRADE_PREPARATION_FAILED') - } finally { - client.release() - } -} diff --git a/apps/core/src/upgrade/readiness.ts b/apps/core/src/upgrade/readiness.ts index b5762802101..c8ef0f06d4a 100644 --- a/apps/core/src/upgrade/readiness.ts +++ b/apps/core/src/upgrade/readiness.ts @@ -22,17 +22,12 @@ import { type SecretStorageConfig, } from '../utils/secret-storage-config' -export { oauthKey } from '../database/app-migrations/helpers/oauth-token' - -export const PREPARED_MIGRATION_ID = '20261108-v18-prepared' -export { CONFIG_UPGRADE_LOCK } from '../shared/config-lock' export type UpgradeConfig = SecretStorageConfig export type UpgradeIssue = { code: string; path?: string } export type UpgradeReport = { target: 18 status: 'ready' | 'needs-action' | 'error' installation: 'empty' | 'existing' - prepared: boolean counts: { plaintext: number ecb: number @@ -197,12 +192,6 @@ export async function readUpgradeState( ) counts.metricsMissing += result.rows[0].count } - const prepared = - ( - await database.query('SELECT 1 FROM _app_migrations WHERE id = $1', [ - PREPARED_MIGRATION_ID, - ]) - ).rowCount !== 0 // A new database may contain migration ledgers, but no application data. const tables = ( await database.query( @@ -236,7 +225,7 @@ export async function readUpgradeState( counts.plainApiKeys || counts.plainOAuth ) - issues.push({ code: 'CREDENTIALS_NEED_PREPARATION' }) + issues.push({ code: 'CREDENTIALS_NEED_MIGRATION' }) if (counts.legacyConfig) issues.push({ code: 'LEGACY_CONFIG' }) if (counts.metricsMissing) issues.push({ code: 'METRICS_INCOMPLETE' }) if ( @@ -248,13 +237,10 @@ export async function readUpgradeState( ).rowCount ) issues.push({ code: 'CONFIG_MIGRATION_MISSING' }) - if (installation === 'existing' && !prepared) - issues.push({ code: 'PREPARATION_MARKER_MISSING' }) return { target: 18, status: issues.length ? 'needs-action' : 'ready', installation, - prepared, counts, issues, } diff --git a/apps/core/src/upgrade/upgrade.error.ts b/apps/core/src/upgrade/upgrade.error.ts new file mode 100644 index 00000000000..a0ef74f2113 --- /dev/null +++ b/apps/core/src/upgrade/upgrade.error.ts @@ -0,0 +1,8 @@ +export class UpgradeError extends Error { + constructor( + public readonly code: string, + public readonly path?: string, + ) { + super(path ? `${code} (${path})` : code) + } +} diff --git a/apps/core/test/src/database/required-secret-storage.migration.spec.ts b/apps/core/test/src/database/required-secret-storage.migration.spec.ts new file mode 100644 index 00000000000..f51eb5cbe4b --- /dev/null +++ b/apps/core/test/src/database/required-secret-storage.migration.spec.ts @@ -0,0 +1,164 @@ +import { defaultKeyHasher } from '@better-auth/api-key' +import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' +import { legacySecret } from 'test/helper/legacy-secret' +import { createPgTestDatabase } from 'test/helper/pg-verify-url' + +import { ENCRYPT, SECURITY } from '~/app.config' +import { AppMigrationsService } from '~/database/app-migrations/app-migrations.service' +import { oauthKey } from '~/database/app-migrations/helpers/oauth-token' +import { readUpgradeState } from '~/upgrade/readiness' +import { EncryptUtil } from '~/utils/encrypt.util' + +it.each([false, true])( + 'release migration protects every credential without opt-in, including after an old credential migration (already applied=%s)', + async (previouslyApplied) => { + const context = await createPgTestDatabase('mx_required_secrets') + try { + if (previouslyApplied) + await context.pool.query( + "INSERT INTO _app_migrations (id) VALUES ('20261018-protect-stored-secrets')", + ) + await context.pool.query( + "INSERT INTO options (id,name,value) VALUES ('100','membership',$1)", + [{ dodoApiKey: legacySecret('retained-billing', ENCRYPT.key) }], + ) + await context.pool.query( + "INSERT INTO readers (id,name,email) VALUES ('owner','Owner','owner@example.test')", + ) + await context.pool.query( + "INSERT INTO api_keys (id,user_id,key) VALUES ('key','owner','original-client-key')", + ) + await context.pool.query( + "INSERT INTO accounts (id,user_id,provider_id,access_token) VALUES ('oauth','owner','github',$1)", + [ + await symmetricEncrypt({ + key: SECURITY.jwtSecret!, + data: 'provider-token', + }), + ], + ) + await context.pool.query( + "INSERT INTO snippets (id,raw,path,secret) VALUES ('101','{}','sample','snippet-secret')", + ) + await context.pool.query( + "INSERT INTO webhooks (id,payload_url,events,secret) VALUES ('102','https://example.test','{}','webhook-secret')", + ) + await context.pool.query( + "INSERT INTO ai_translations (id,ref_id,ref_type,hash,lang,source_lang,title,text,ai_model,ai_provider) VALUES ('103','104','post','h','ja','zh','Title','Body','legacy-model','legacy-provider')", + ) + const runner = new AppMigrationsService(context.db, context.pool) + await runner.run() + expect( + await readUpgradeState(context.pool, { + ...ENCRYPT, + jwtSecret: SECURITY.jwtSecret, + }), + ).toMatchObject({ + status: 'ready', + counts: { + plaintext: 0, + ecb: 0, + legacyConfig: 0, + plainApiKeys: 0, + plainOAuth: 0, + metricsMissing: 0, + }, + }) + const membership = ( + await context.pool.query( + "SELECT value FROM options WHERE name='membership'", + ) + ).rows[0].value + expect(EncryptUtil.decrypt(membership.apiKey, 'membership.apiKey')).toBe( + 'retained-billing', + ) + expect( + (await context.pool.query('SELECT key,config_id FROM api_keys')) + .rows[0], + ).toEqual({ + key: await defaultKeyHasher('original-client-key'), + config_id: 'mx-hashed-v1', + }) + const token = ( + await context.pool.query('SELECT access_token FROM accounts') + ).rows[0].access_token + expect( + await symmetricDecrypt({ + key: oauthKey(SECURITY.jwtSecret!), + data: token, + }), + ).toBe('provider-token') + expect( + EncryptUtil.decrypt( + (await context.pool.query('SELECT secret FROM snippets')).rows[0] + .secret, + 'snippets.101.secret', + ), + ).toBe('snippet-secret') + expect( + EncryptUtil.decrypt( + (await context.pool.query('SELECT secret FROM webhooks')).rows[0] + .secret, + 'webhooks.102.secret', + ), + ).toBe('webhook-secret') + const snapshot = async () => + Promise.all( + [ + 'options', + 'api_keys', + 'accounts', + 'snippets', + 'webhooks', + 'ai_generation_metrics', + '_app_migrations', + ].map( + async (table) => + (await context.pool.query(`SELECT * FROM ${table} ORDER BY id`)) + .rows, + ), + ) + const before = await snapshot() + await runner.run() + expect(await snapshot()).toEqual(before) + } finally { + await context.close() + } + }, +) + +it('blocks release and rolls back the credential batch without recording completion when OAuth cannot be decrypted', async () => { + const context = await createPgTestDatabase('mx_required_secrets_failure') + try { + await context.pool.query( + "INSERT INTO readers (id,name,email) VALUES ('owner','Owner','owner@example.test')", + ) + await context.pool.query( + "INSERT INTO api_keys (id,user_id,key) VALUES ('key','owner','original-client-key')", + ) + await context.pool.query( + "INSERT INTO snippets (id,raw,path,secret) VALUES ('100','{}','sample','snippet-secret')", + ) + await context.pool.query( + "INSERT INTO accounts (id,user_id,provider_id,access_token) VALUES ('oauth','owner','github','$ba$1$invalid-ciphertext')", + ) + await expect( + new AppMigrationsService(context.db, context.pool).run(), + ).rejects.toThrow() + expect( + (await context.pool.query('SELECT key FROM api_keys')).rows[0].key, + ).toBe('original-client-key') + expect( + (await context.pool.query('SELECT secret FROM snippets')).rows[0].secret, + ).toBe('snippet-secret') + expect( + ( + await context.pool.query( + "SELECT id FROM _app_migrations WHERE id='20261109-required-secret-storage'", + ) + ).rows, + ).toEqual([]) + } finally { + await context.close() + } +}) diff --git a/apps/core/test/src/upgrade/archive-rehearsal.spec.ts b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts index bc0d1ef6ce6..4b3492cafd6 100644 --- a/apps/core/test/src/upgrade/archive-rehearsal.spec.ts +++ b/apps/core/test/src/upgrade/archive-rehearsal.spec.ts @@ -16,9 +16,12 @@ import { SecretCodec } from '~/utils/secret-codec' const entry = fileURLToPath( new URL('../../../out/upgrade.mjs', import.meta.url), ) +const migrateEntry = fileURLToPath( + new URL('../../../out/migrate.mjs', import.meta.url), +) -describe.skipIf(!existsSync(entry))( - 'historical archive → private PG → built upgrade CLI', +describe.skipIf(!existsSync(entry) || !existsSync(migrateEntry))( + 'historical archive → private PG → built release migration', () => { let source: Awaited> let target: Awaited> @@ -34,7 +37,7 @@ describe.skipIf(!existsSync(entry))( if (cliCwd) rmSync(cliCwd, { recursive: true }) }) - it('restores ZIP/v1 contents using original keys, then prepares through the shipped entry', async (test) => { + it('restores ZIP/v1 contents using original keys, then migrates through the shipped release entry', async (test) => { const url = new URL(source.connectionString) const container = execFileSync( 'docker', @@ -134,21 +137,30 @@ describe.skipIf(!existsSync(entry))( MX_ENCRYPT_ENABLE: undefined, MX_SECRET_STORAGE_V2: undefined, PG_URL: target.connectionString, + PG_CONNECTION_STRING: target.connectionString, MX_ENCRYPT_KEY: key, JWT_SECRET: 'private-rehearsal-jwt-123456789012345', + NODE_ENV: 'production', + SNOWFLAKE_WORKER_ID: '1', + MIGRATIONS_DIR: fileURLToPath( + new URL('../../../src/database/migrations', import.meta.url), + ), + } + const run = (args: string[]) => { + try { + return JSON.parse( + execFileSync(process.execPath, [entry, ...args], { + cwd: cliCwd, + env, + encoding: 'utf8', + }), + ) + } catch (error) { + // Diagnostic reports intentionally omit stored secret values. + const { stdout, stderr } = error as { stdout: string; stderr: string } + throw new Error(`${stdout}\n${stderr}`, { cause: error }) + } } - const run = (args: string[]) => - JSON.parse( - execFileSync(process.execPath, [entry, ...args], { - cwd: cliCwd, - env, - encoding: 'utf8', - }), - ) - expect(run(['prepare', '--all-replicas-v2', '--dry-run'])).toMatchObject({ - status: 'would-be-ready', - dryRun: true, - }) expect( ( await target.pool.query( @@ -156,10 +168,18 @@ describe.skipIf(!existsSync(entry))( ) ).rows[0].value, ).toHaveProperty('dodoApiKey') - expect(run(['prepare', '--all-replicas-v2'])).toMatchObject({ - status: 'prepared', + const migrationOutput = execFileSync(process.execPath, [migrateEntry], { + cwd: cliCwd, + env, + encoding: 'utf8', + }) + const report = run(['check', '--json']) + expect( + report, + `${JSON.stringify(report)}\n${migrationOutput}`, + ).toMatchObject({ + status: 'ready', }) - expect(run(['check', '--json'])).toMatchObject({ status: 'ready' }) const canonical = ( await target.pool.query( "SELECT value FROM options WHERE name = 'membership'", diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts index 0f3ea241f86..df459c8f782 100644 --- a/apps/core/test/src/upgrade/cli.spec.ts +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -34,6 +34,24 @@ describe('upgrade CLI isolated process', () => { }) afterAll(async () => context?.close()) + it('rejects the removed prepare command without modifying the database', async () => { + const output: string[] = [] + expect( + await runUpgradeCli( + ['prepare', '--all-replicas-v2'], + { + PG_URL: context.connectionString, + MX_ENCRYPT_KEY: 'ab'.repeat(32), + JWT_SECRET: 'long-enough-test-jwt-123456789012345', + }, + (value) => output.push(value), + ), + ).toBe(1) + expect( + (await context.pool.query('SELECT id FROM _app_migrations')).rows, + ).toEqual([]) + }) + it('reports bad runtime settings in a fresh process without bootstrap failure', async () => { const result = await promisify(execFile)( 'node_modules/.bin/vite-node', diff --git a/apps/core/test/src/upgrade/preparation.spec.ts b/apps/core/test/src/upgrade/preparation.spec.ts deleted file mode 100644 index d4f9c7abf5a..00000000000 --- a/apps/core/test/src/upgrade/preparation.spec.ts +++ /dev/null @@ -1,203 +0,0 @@ -import { defaultKeyHasher } from '@better-auth/api-key' -import { symmetricDecrypt, symmetricEncrypt } from 'better-auth/crypto' -import { legacySecret } from 'test/helper/legacy-secret' -import { createPgTestDatabase } from 'test/helper/pg-verify-url' -import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest' - -import { prepareUpgrade } from '~/upgrade/preparation' -import { - CONFIG_UPGRADE_LOCK, - oauthKey, - readUpgradeState, -} from '~/upgrade/readiness' -import { SecretCodec } from '~/utils/secret-codec' - -describe('transactional v18 preparation', () => { - let context: Awaited> - const config = { - key: 'ab'.repeat(32), - jwtSecret: 'long-test-jwt-secret-1234567890123456789', - } - const codec = new SecretCodec(config) - beforeAll(async () => { - context = await createPgTestDatabase('mx_upgrade_prepare') - }) - afterAll(async () => context?.close()) - beforeEach(async () => { - await context.pool.query( - 'TRUNCATE options, snippets, webhooks, api_keys, accounts, readers, _app_migrations CASCADE', - ) - await context.pool.query( - "INSERT INTO options (id, name, value) VALUES ('100', 'membership', $1), ('101', 'thirdPartyServiceIntegration', $2), ('102', 'imageGenerationOptions', $3)", - [ - { - dodoApiKey: codec.encrypt('billing-secret', 'membership.dodoApiKey'), - }, - { - githubToken: legacySecret('github-secret', config.key), - }, - { enable: true, apiKey: 'image-secret', model: 'image-v1' }, - ], - ) - await context.pool.query( - "INSERT INTO snippets (id, raw, path, secret) VALUES ('200', '{}', 'sample', 'snippet-secret')", - ) - await context.pool.query( - "INSERT INTO webhooks (id, payload_url, events, secret) VALUES ('201', 'https://example.test', '{}', 'webhook-secret')", - ) - await context.pool.query( - "INSERT INTO readers (id, email, name) VALUES ('owner', 'owner@example.test', 'Owner')", - ) - await context.pool.query( - "INSERT INTO api_keys (id, user_id, key) VALUES ('key', 'owner', 'raw-client-key')", - ) - await context.pool.query( - "INSERT INTO accounts (id, user_id, account_id, provider_id, access_token) VALUES ('account', 'owner', 'remote', 'github', 'provider-token')", - ) - }) - async function snapshot() { - return Promise.all( - [ - 'options', - 'snippets', - 'webhooks', - 'api_keys', - 'accounts', - '_app_migrations', - ].map( - async (table) => - (await context.pool.query(`SELECT * FROM ${table} ORDER BY id`)).rows, - ), - ) - } - it('requires an explicit all-replicas confirmation and leaves data untouched', async () => { - const before = await snapshot() - await expect( - prepareUpgrade(context.pool, config, { allReplicasV2: false }), - ).rejects.toThrow('REPLICA_CONFIRMATION_REQUIRED') - expect(await snapshot()).toEqual(before) - }) - it('dry-run proves conversion and rolls back every write', async () => { - const before = await snapshot() - expect( - await prepareUpgrade(context.pool, config, { - allReplicasV2: true, - dryRun: true, - }), - ).toMatchObject({ dryRun: true, status: 'would-be-ready' }) - expect(await snapshot()).toEqual(before) - }) - it('converts at original paths, hashes once and preserves OAuth/provider secrets', async () => { - await prepareUpgrade(context.pool, config, { allReplicasV2: true }) - expect(await readUpgradeState(context.pool, config)).toMatchObject({ - status: 'ready', - prepared: true, - counts: { - legacyConfig: 0, - ecb: 0, - plaintext: 0, - plainApiKeys: 0, - plainOAuth: 0, - }, - }) - const rows = Object.fromEntries( - (await context.pool.query('SELECT name, value FROM options')).rows.map( - (row) => [row.name, row.value], - ), - ) - expect(codec.decrypt(rows.membership.apiKey, 'membership.apiKey')).toBe( - 'billing-secret', - ) - expect( - codec.decrypt( - rows.thirdPartyServiceIntegration.github.token, - 'thirdPartyServiceIntegration.github.token', - ), - ).toBe('github-secret') - expect( - codec.decrypt(rows.ai.providers[0].apiKey, 'ai.providers.0.apiKey'), - ).toBe('image-secret') - const key = (await context.pool.query('SELECT key FROM api_keys')).rows[0] - .key - expect(key).toBe(await defaultKeyHasher('raw-client-key')) - expect( - await symmetricDecrypt({ - key: oauthKey(config.jwtSecret), - data: (await context.pool.query('SELECT access_token FROM accounts')) - .rows[0].access_token, - }), - ).toBe('provider-token') - const before = await snapshot() - await prepareUpgrade(context.pool, config, { allReplicasV2: true }) - expect(await snapshot()).toEqual(before) - }) - it('rolls back and omits a marker if original decryption fails', async () => { - const before = await snapshot() - await expect( - prepareUpgrade( - context.pool, - { ...config, key: 'cd'.repeat(32) }, - { allReplicasV2: true }, - ), - ).rejects.toThrow('SECRET_UNREADABLE') - expect(await snapshot()).toEqual(before) - }) - it('rolls back when mixed config contains conflicting secrets', async () => { - await context.pool.query( - "UPDATE options SET value = $1 WHERE name = 'membership'", - [{ apiKey: 'canonical-private', dodoApiKey: 'legacy-private' }], - ) - const before = await snapshot() - await expect( - prepareUpgrade(context.pool, config, { allReplicasV2: true }), - ).rejects.toThrow('CONFIG_SECRET_CONFLICT') - expect(await snapshot()).toEqual(before) - }) - it('rolls back on truncated legacy OAuth ciphertext rather than wrapping it as plaintext', async () => { - const bare = await symmetricEncrypt({ - key: config.jwtSecret, - data: 'original-provider-token', - }) - await context.pool.query('UPDATE accounts SET access_token = $1', [ - bare.slice(0, -1), - ]) - const before = await snapshot() - await expect( - prepareUpgrade(context.pool, config, { allReplicasV2: true }), - ).rejects.toThrow('OAUTH_UNREADABLE') - expect(await snapshot()).toEqual(before) - }) - it('upgrades a legacy Better Auth bare-hex token without double encryption', async () => { - const bare = await symmetricEncrypt({ - key: config.jwtSecret, - data: 'original-provider-token', - }) - await context.pool.query('UPDATE accounts SET access_token = $1', [bare]) - await prepareUpgrade(context.pool, config, { allReplicasV2: true }) - const stored = ( - await context.pool.query('SELECT access_token FROM accounts') - ).rows[0].access_token - expect( - await symmetricDecrypt({ key: oauthKey(config.jwtSecret), data: stored }), - ).toBe('original-provider-token') - }) - it('serializes preparation with normal configuration writes', async () => { - const writer = await context.pool.connect() - await writer.query('BEGIN') - await writer.query('SELECT pg_advisory_xact_lock($1::bigint)', [ - CONFIG_UPGRADE_LOCK, - ]) - let finished = false - const preparing = prepareUpgrade(context.pool, config, { - allReplicasV2: true, - }).then(() => { - finished = true - }) - await new Promise((resolve) => setTimeout(resolve, 100)) - expect(finished).toBe(false) - await writer.query('COMMIT') - writer.release() - await preparing - expect(finished).toBe(true) - }) -}) diff --git a/apps/core/test/src/upgrade/readiness.spec.ts b/apps/core/test/src/upgrade/readiness.spec.ts index 80c7eb19c58..85fdee6049d 100644 --- a/apps/core/test/src/upgrade/readiness.spec.ts +++ b/apps/core/test/src/upgrade/readiness.spec.ts @@ -19,7 +19,6 @@ describe('standalone upgrade readiness', () => { expect(await readUpgradeState(context.pool, config)).toMatchObject({ installation: 'empty', status: 'ready', - prepared: false, }) }) @@ -71,7 +70,7 @@ describe('standalone upgrade readiness', () => { }) it('does not trust a marker over malformed canonical configuration', async () => { await context.pool.query( - "DELETE FROM options; INSERT INTO _app_migrations (id) VALUES ('20261108-v18-prepared'); INSERT INTO options (id, name, value) VALUES ('300', 'membership', '42'::jsonb)", + "DELETE FROM options; INSERT INTO _app_migrations (id) VALUES ('20261109-canonical-config'); INSERT INTO options (id, name, value) VALUES ('300', 'membership', '42'::jsonb)", ) expect( (await readUpgradeState(context.pool, config)).issues.map( @@ -86,15 +85,25 @@ describe('standalone upgrade readiness', () => { expect(await readUpgradeState(context.pool, config)).toMatchObject({ installation: 'existing', status: 'needs-action', - prepared: false, }) }) - it('requires the current configuration migration even when an older preparation marker exists', async () => { + it('requires the current configuration migration for an existing installation', async () => { await context.pool.query( - "DELETE FROM options; INSERT INTO categories (id,name,slug) VALUES ('700','Existing','existing') ON CONFLICT DO NOTHING; INSERT INTO _app_migrations (id) VALUES ('20261108-v18-prepared') ON CONFLICT DO NOTHING", + "DELETE FROM options; INSERT INTO categories (id,name,slug) VALUES ('700','Existing','existing') ON CONFLICT DO NOTHING", ) const report = await readUpgradeState(context.pool, config) expect(report.status).toBe('needs-action') expect(report.issues).toContainEqual({ code: 'CONFIG_MIGRATION_MISSING' }) }) + + it('accepts current existing data after release migration without a separate preparation marker', async () => { + await context.pool.query( + "DELETE FROM options; DELETE FROM _app_migrations; INSERT INTO _app_migrations (id) VALUES ('20261109-canonical-config')", + ) + expect(await readUpgradeState(context.pool, config)).toMatchObject({ + installation: 'existing', + status: 'ready', + issues: [], + }) + }) }) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index b7d2248be60..e69262675c2 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -4,7 +4,9 @@ 修订(2026-11-08):用户明确要求本 PR 立即移除非加密运行和旧格式写入,覆盖下文“准备阶段保留逃生口”的安排。显式字段密钥、至少 32 字符 JWT 和固定 V2 写入在本分支立即生效;旧 true 开关仅作为弃用配置接受,false 拒绝启动。历史读取与显式迁移保留。其余 API、SDK 6 和数据库列删除仍按后续阶段执行,当前分支尚未发布。 -修订(2026-11-09):用户明确要求通过 DB 迁移一次性结束旧配置运行时兼容。本 PR 新增必跑的发布前配置数据迁移;先按原路径解密、转换并按新路径加密,删除旧配置行,成功后记录迁移账本。正常 ConfigsService、schema 和 PATCH 只接受规范结构;旧字段按 schema 严格拒绝。历史转换器只供发布迁移与离线 check/prepare、历史导入恢复使用。此配置契约切换需要先停止旧副本及后台写入者,不能把此 contract 当作任意 v17 副本仍在写时的滚动迁移;不涉及删除数据库物理列或其他 API 变更。 +修订(2026-11-09):用户明确要求通过 DB 迁移一次性结束旧配置运行时兼容。本 PR 新增必跑的发布前配置数据迁移;先按原路径解密、转换并按新路径加密,删除旧配置行,成功后记录迁移账本。正常 ConfigsService、schema 和 PATCH 只接受规范结构;旧字段按 schema 严格拒绝。历史转换器只供发布迁移、只读检查与历史导入恢复使用。此配置契约切换需要先停止旧副本及后台写入者,不能把此 contract 当作任意 v17 副本仍在写时的滚动迁移;不涉及删除数据库物理列或其他 API 变更。 + +升级入口收口(2026-11-09):按用户要求删除独立 prepare、dry-run 和准备标记。配置、凭据及 AI 指标统一由正式 `migrate` 完成;新增必跑 `20261109-required-secret-storage`,复用历史凭据转换行为,并以新迁移 ID 保证旧 opt-in 账本不会跳过本次回填。删除凭据迁移开关及专用脚本。独立 CLI 仅保留只读 check 和需要旧、新秘密文件的 JWT 维护轮换。 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 @@ -39,7 +41,7 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 1. 增加后续需要的 POST 入口,并迁移 Admin 和兼容 SDK;保留旧 GET。 2. 将旧配置转换抽为可复用模块,支持持久化准备与旧接口写入转换,保持 v17 的已承诺形状。 3. 补齐 AI 指标;让服务端运行时不再读写旧元数据列,旧 v17 响应从新指标派生,暂不删除物理列。 -4. 提供只读 `upgrade:check`、显式 `upgrade:prepare`、安全的 JWT 轮换工具。命令支持构建后的交付与本地源码执行。 +4. 提供只读 `upgrade:check` 和安全的 JWT 轮换工具;所有数据转换统一使用正式 `migrate`。命令支持构建后的交付与本地源码执行。 5. 完成 v17 事实文档纠错、v15–v17 迁移入口及 v18 预告,提供可执行准备说明。 6. 行为测试、真实临时 PostgreSQL 升级演练、构建和独立整体验证;保留原有工作。 @@ -53,15 +55,15 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 - 新装生成每实例随机密钥并持久保存;现有密钥原样保留。生产模板不能内置公共有效密钥。 - 工程选择:旧 `ENCRYPT_ENABLE=true`、`MX_SECRET_STORAGE_V2=true` 暂接受并提示可删除;显式 false 拒绝启动。新写入固定 V2,`ENCRYPT_ALGORITHM` 只服务历史读取。 - 新装空库由发布前迁移初始化当前状态,不要求 v17 历史;既有实例需要准备完成。 -- 普通启动只检查配置、schema 和准备状态,不做业务数据迁移。 +- 普通生产启动检查配置和 schema,不做业务数据迁移;部署前统一运行正式迁移。 -### 检查与显式准备 +### 检查与正式迁移 -`upgrade:check` 不启动 HTTP/业务模块,不因普通配置初始化提前退出而丢失诊断。输出人可读摘要或 JSON:新装识别、配置缺失、密文类型计数、旧配置计数、指标覆盖和准备状态。退出状态区分 ready、needs-action 和检查失败;不得输出秘密值。 +`upgrade:check` 不启动 HTTP/业务模块,不因普通配置初始化提前退出而丢失诊断。输出人可读摘要或 JSON:新装识别、配置缺失、密文类型计数、旧配置计数、指标覆盖和正式迁移状态。退出状态区分 ready、needs-action 和检查失败;不得输出秘密值。 -`upgrade:prepare` 使用 slim runner、数据库 advisory lock 和迁移账本,支持 dry-run、事务失败保护和幂等重试。运行需要显式确认所有副本已经停止旧格式写入;进程不能凭自己猜测集群状态。只在转换成功、残留检查通过后写入新的 v18 准备标记,旧凭据迁移账本行不能充当完成证明。 +停掉全部旧副本和后台写入者后,运行 `node migrate.mjs`(源码为 `pnpm -C apps/core migrate`)。迁移使用 slim runner、数据库 advisory lock 和每项独立账本,支持失败阻断与幂等重试。没有独立准备标记或凭据迁移开关;旧凭据迁移账本不能跳过新的 v18 凭据迁移。失败批次回滚,先前完成的迁移保留账本。 -顺序:识别旧记录 → 用原路径解密 → 转换旧 AI/第三方/会员配置 → 按新路径加密并持久化 → 转换 Snippet/Webhook/API Key/OAuth → 验证残留及指标覆盖 → 记录准备完成。API Key 原始字符串仍可用,不重复哈希,不把哈希当作客户端凭据。 +顺序:迁移 schema → 回填 AI 指标 → 按原路径解密并转换旧配置、按新路径加密 → 转换 Snippet/Webhook/API Key/OAuth → 运行只读检查 → 启动新业务。API Key 原始字符串仍可用,不重复哈希,不把哈希当作客户端凭据。 已按 2026-11-09 修订结束运行时旧结构兼容:发布前必跑配置数据迁移,普通读取与 PATCH 仅处理规范结构。转换器仅在发布迁移与离线工具中复用;迁移和正常配置写入使用同一个数据库锁。 @@ -146,12 +148,12 @@ flowchart TD 回退最低版本是实测准备版(能读 V2/新配置且不再引用旧列),不是任意 v17。回退代码不回退 schema;更老版本需要完整旧快照与当时的秘密。区分代码回退、数据库恢复、文件恢复。 -历史备份恢复先验证归档与当时的字段密钥/JWT/恢复私钥,隔离验证转换后再服务。环境不能自动隔离预检时提供离线“恢复到准备版 → prepare → v18”路径,不能报告成功后把未准备数据交给业务。保留恢复前快照;不得持久保存新的公开秘密副本。 +历史备份恢复先验证归档与当时的字段密钥/JWT/恢复私钥,隔离验证转换后再服务。环境不能自动隔离预检时提供离线“隔离恢复 → migrate → check → v18”路径,不能报告成功后把未迁移数据交给业务。保留恢复前快照;不得持久保存新的公开秘密副本。 ## 验收 - 新装、历史明文、ECB、V2;缺密钥/坏密钥/短 JWT/显式禁用的独立诊断。 -- 错误原密钥/JWT、损坏密文、未知形状失败不写完成标记;dry-run 无持久写入,重跑不重复哈希或重置 Token。 +- 错误原密钥/JWT、损坏密文、未知形状失败不写对应迁移完成记录;失败批次回滚,重跑不重复哈希或重置 Token。 - AI 普通 GET 缺结果不调用模型/任务/写仓库;权限、SSE 字节和显式生成保持。 - 新 POST 与旧 GET 在准备版并存;鉴权、下载、Admin 操作、SDK 调用真实通过。 - 严格配置与规范排序在 v18 生效;准备版保证旧契约,SDK 转换后的实际排序正确。 @@ -204,6 +206,14 @@ flowchart TD 修复后完整 UTC/3 workers 回归实际结果:3576 项通过,11 项既定跳过,另 1 个未触及的 Push Relay SQL 迁移测试文件因 Redis 端口竞争在 setup 阶段失败、3 项未执行;该文件独立重跑 3 项全部通过。合计 3579 项行为已验证,所有代码断言通过;原完整命令因基础设施错误退出非零,不能称该次单命令全绿。Agent 排除项不变。67 项配置/升级定向回归及重新构建、类型检查通过;无未修复 Critical/Important 或延后 Minor,未再次派发审阅。 -共享模块归位(2026-11-09):配置迁移、历史配置转换和 OAuth 解封装已统一归入 `database/app-migrations/helpers`。正式 app-data migrations 只依赖该目录,独立升级 CLI 也调用这些函数;`src/upgrade` 保留检查、准备和 JWT 轮换命令逻辑,不再被正式迁移反向引用。本次仅调整模块归属与引用路径。 +共享模块归位(2026-11-09):配置迁移、历史配置转换和 OAuth 解封装已统一归入 `database/app-migrations/helpers`。正式 app-data migrations 只依赖该目录;`src/upgrade` 在后续入口收口后仅保留只读检查和 JWT 轮换,不再被正式迁移反向引用。 模块归位验证:现有历史转换行为测试先改用新模块路径,验证迁移前因模块未就位而失败;移动后 11 个文件、71 项定向测试通过,包含真实发布迁移、凭据保留、JWT 轮换和构建后 CLI 恢复演练。Core 构建、Core/E2E 类型检查、响应/验证检查、50 个迁移 lint 通过;正式迁移不再引用 `src/upgrade`。本次没有更改数据转换算法或迁移 ID。 + +## 删除独立 prepare 的验证(2026-11-09) + +- 删除命令、实现、dry-run/副本参数、准备标记和 opt-in 凭据迁移入口。正式发布链必跑 `20261109-required-secret-storage`,配置转换和指标回填继续由各自正式迁移完成。 +- 5 项新增行为回归先失败,再通过:没有开关也回填凭据、历史凭据迁移记录不会跳过本次回填、损坏 OAuth 阻断发布并回滚凭据批次、已删除命令不写库、检查不再要求准备标记。34 项定向测试通过。 +- 历史 ZIP/v1 归档恢复到隔离 PostgreSQL 后,经构建产物 `migrate.mjs` 和只读 check 验证原凭据保留。演练 fixture 同步覆盖两个 PG 连接别名,避免继承测试环境连接到其他库。 +- 原始全量命令确认既有 `admin-agent.e2e-spec.ts` 缺 `AiAgentConversationRepository`,初始化及清理失败;初次归档演练的连接别名问题修正后单独通过。最终明确排除该既有 Agent 夹具重跑:433 个文件通过、3 个文件跳过,3576 项通过、11 项跳过,退出 0。没有声称未排除夹具的原始命令全绿。 +- Core 构建、Core/E2E 类型检查、API envelope/验证检查、50 个迁移 lint、定向 ESLint/Prettier 和 diff 检查通过。官方文档 build/check/links/pangu/lint 与 3 项测试通过,生成 llms 已同步;保留既有 1 条 hint 与 1 条死链。 diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index d2ae751a46e..5747469fb67 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -21,17 +21,18 @@ V2 encryption, API-key writes always hash, and OAuth writes stay encrypted. The published v17.0.4 escape hatch is historical behavior, removed here. This branch is not released yet. The [preparation runbook](v18-preparation.md) -explains standalone check/prepare and maintenance JWT rotation. The tools can +explains release migration, standalone check and maintenance JWT rotation. The tools can inspect invalid settings without booting the application. Short JWT values must be rotated with the original secret before starting this runtime; do not replace their environment value alone. Before persistent backfill, stop old-format writers on every replica and -background worker. `upgrade:prepare --all-replicas-v2` requires an explicit -operator attestation and performs transactional conversion/verification. -`migrate:secrets` remains an opt-in credential-only historical migration; its -ledger is not proof that configuration and AI metadata are prepared. It no -longer needs a V2 activation flag. +background worker. `node migrate.mjs` (source: `pnpm -C apps/core migrate`) +performs the mandatory configuration, credential and AI-metric migrations. +The v18 credential migration has its own ledger ID, so an older opt-in +credential backfill cannot bypass it. There is no separate prepare command, +preparation marker or credential-migration activation flag. Run the read-only +check after migration, then start the new business replicas. After conversion, roll back only to a tested reader of the new formats. Neither disabling encryption nor switching back to old-format writes exists. diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index 96316f954ad..f479247eb2a 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -4,9 +4,9 @@ This unreleased branch now requires field encryption, an explicit hex64 key and a JWT of at least 32 characters. Plaintext/ECB writes, encryption-disable flags and machine-ID fallback have been removed. Canonical configuration migration and strict configuration PATCH validation are also enforced here. SDK 6 cleanup and column removal remain future work. These commands are not part of released v17.0.4. -No minimum released preparation tag is established yet. Build and deploy this -preparation code to every replica before using it; do not publish or migrate -production as part of reviewing this branch. +No minimum released preparation tag is established yet. Obtain this build +without starting business replicas; do not publish or migrate production as +part of reviewing this branch. ## Preconditions and checks @@ -32,7 +32,8 @@ pnpm -C apps/core upgrade:check --json inspection failed. Reports contain counts, record paths and stable codes, never secret values. Missing/bad encryption settings and short JWT values do not prevent diagnostics. Empty installations do not need a historical marker. -Existing installations require a new marker plus current data verification; +Existing installations require the canonical configuration migration record +and current data verification. There is no separate preparation marker; an old secret-migration ledger row alone is insufficient. ## Short JWT maintenance @@ -56,7 +57,7 @@ must sign in again. The operator then applies the same new JWT to every replica before starting them. Missing original JWT plus encrypted tokens requires recovering the original secret or reauthorizing the account. -## Explicit preparation +## Release migration Obtain this build without starting business replicas. Run check and rotate a short JWT during maintenance first. Stop every old replica/background writer, @@ -74,30 +75,27 @@ retention and freeform record keys remain supported. Startup refuses to load configuration without the migration record. Historical conversion lives only in release/offline upgrade code, including nested legacy SMTP options. -Continue the explicit credential/metrics preparation before starting new -replicas. All writers must already be stopped or use the current formats. +The same release runner backfills AI metrics and always runs the v18 +`20261109-required-secret-storage` migration. It protects Snippet/Webhook +secrets, hashes API keys and encrypts OAuth tokens. This uses a new migration +ID so a historical opt-in credential migration cannot skip the v18 backfill. +No activation flag or separate preparation command is required. ```sh -pnpm -C apps/core upgrade:prepare --all-replicas-v2 --dry-run -pnpm -C apps/core upgrade:prepare --all-replicas-v2 +pnpm -C apps/core migrate pnpm -C apps/core upgrade:check --json ``` -`--all-replicas-v2` is an operator attestation, not automatic cluster discovery. -Preparation holds a PostgreSQL advisory lock shared by configuration PATCH. -It decrypts secrets at their old AAD paths, converts legacy AI/media, GitHub -and membership options to canonical storage, encrypts at new paths, hashes -API keys once, protects OAuth, and fills missing AI metrics. Existing client -API-key strings continue to verify. It writes the new marker only after no -residuals remain. Malformed data or wrong secrets roll the whole transaction -back. Current configuration PATCH validation is strict. - -Dry-run executes and rolls back the complete transaction. `would-be-ready` -does not mean a marker was persisted. Repeated successful preparation is -idempotent and does not rehash keys or rotate tokens. After a real commit, -restart business replicas to retire old caches, then check original API-key -authentication, OAuth, configuration consumption and AI metadata. Do not flush -all Redis state. +Configuration conversion holds the database lock shared by configuration +PATCH. Each migration records completion only after success; a failed batch +rolls back and blocks deployment. Earlier successful migrations retain their +ledger records and are skipped on retry. Existing API-key strings continue to +verify; rerunning migrations does not rehash keys or rotate existing tokens. +There is no independent prepare, preparation marker or dry-run command. + +After migration and a successful check, start new business replicas and verify +original API-key authentication, OAuth, configuration consumption and AI +metadata. Do not flush all Redis state. A blank canonical credential falls back to a populated legacy credential. Different nonempty old/new values report CONFIG_SECRET_CONFLICT and roll back @@ -107,8 +105,8 @@ old shapes via PATCH. Never overwrite a lost key with a newly generated key. ## Built artifacts and containers -Run `node upgrade.mjs check --json`, `node upgrade.mjs prepare` or -`node upgrade.mjs rotate-jwt` with the same arguments in the built output. +Run `node migrate.mjs` for the release migration, `node upgrade.mjs check --json` +for diagnostics or `node upgrade.mjs rotate-jwt` for maintenance rotation. Docker copies this entry and its shared chunks with the main application. A future preparation image can run the tool without starting app services: @@ -144,7 +142,7 @@ Older versions require their old complete snapshot and secrets. The restore service does not currently provide automatic v18 isolation or preflight. Restore historical ZIP/v1 encrypted archives into an isolated -preparation installation using original archive/field/JWT secrets, run schema -migration and check/prepare, verify credentials and files, and only then plan +maintenance installation using original archive/field/JWT secrets, run the +release migration and check, verify credentials and files, and only then plan a v18 switch. Keep a pre-restore snapshot. Database replacement is atomic, file copying is separate; failures must not be reported as a complete restore. From 8f0f90fae891dbc5cd96f0750d3b0cdd9ee6451a Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 02:00:58 +0800 Subject: [PATCH 23/26] fix(configs): preserve schema field casing in admin settings --- apps/admin/src/api/options.test.ts | 67 +++++++++++++++++++ apps/admin/src/api/options.ts | 34 +++++++++- .../settings/components/SystemSettings.tsx | 2 +- .../src/modules/configs/configs.current.ts | 6 +- .../modules/configs/current-config.spec.ts | 15 +++++ ...-11-08-mx-core-v18-compatibility-design.md | 8 +++ 6 files changed, 128 insertions(+), 4 deletions(-) create mode 100644 apps/admin/src/api/options.test.ts diff --git a/apps/admin/src/api/options.test.ts b/apps/admin/src/api/options.test.ts new file mode 100644 index 00000000000..971c46feb12 --- /dev/null +++ b/apps/admin/src/api/options.test.ts @@ -0,0 +1,67 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' + +import { type ConfigFormSchema, getAllOptions } from './options' + +afterEach(() => vi.unstubAllGlobals()) + +describe('settings values and form schema', () => { + it('restores declared acronym keys after wire camelization so fields display and save the existing values', async () => { + vi.stubGlobal( + 'fetch', + vi.fn( + async () => + new Response( + JSON.stringify({ + data: { + comment_upload_options: { + single_file_size_mb: 7, + reader_total_active_bytes_mb: 70, + enable: true, + }, + opaque: { customField: { APIKey: 'preserved' } }, + }, + }), + { status: 200, headers: { 'content-type': 'application/json' } }, + ), + ), + ) + const schema: ConfigFormSchema = { + title: 'Settings', + defaults: {}, + groups: [ + { + key: 'storage', + title: 'Storage', + description: '', + icon: 'database', + sections: [ + { + key: 'commentUploadOptions', + title: 'Comment uploads', + fields: [ + { + key: 'singleFileSizeMB', + title: 'File limit', + ui: { component: 'number' }, + }, + { + key: 'readerTotalActiveBytesMB', + title: 'Total limit', + ui: { component: 'number' }, + }, + ], + }, + ], + }, + ], + } + expect(await getAllOptions(schema)).toEqual({ + commentUploadOptions: { + singleFileSizeMB: 7, + readerTotalActiveBytesMB: 70, + enable: true, + }, + opaque: { customField: { APIKey: 'preserved' } }, + }) + }) +}) diff --git a/apps/admin/src/api/options.ts b/apps/admin/src/api/options.ts index 3a536b2615a..b2c3b79f8bb 100644 --- a/apps/admin/src/api/options.ts +++ b/apps/admin/src/api/options.ts @@ -88,8 +88,38 @@ export interface UpdateOwnerData { username?: string } -export function getAllOptions() { - return getJson('/options') +function restoreFormFieldKeys( + value: unknown, + fields: ConfigFormField[], +): unknown { + if (!value || typeof value !== 'object' || Array.isArray(value)) return value + const source = value as Record + const keys = new Map( + Object.keys(source).map((key) => [key.toLowerCase(), key]), + ) + const result = { ...source } + for (const field of fields) { + const key = keys.get(field.key.toLowerCase()) + if (key === undefined) continue + result[field.key] = field.fields?.length + ? restoreFormFieldKeys(source[key], field.fields) + : source[key] + if (key !== field.key) delete result[key] + } + return result +} + +export async function getAllOptions(schema?: ConfigFormSchema) { + const options = await getJson('/options') + if (!schema) return options + const result = { ...options } + for (const group of schema.groups) + for (const section of group.sections) + result[section.key] = restoreFormFieldKeys( + options[section.key], + section.fields, + ) + return result } export function getFormSchema() { diff --git a/apps/admin/src/features/settings/components/SystemSettings.tsx b/apps/admin/src/features/settings/components/SystemSettings.tsx index 4bcb5b51d3c..83624421b30 100644 --- a/apps/admin/src/features/settings/components/SystemSettings.tsx +++ b/apps/admin/src/features/settings/components/SystemSettings.tsx @@ -42,7 +42,7 @@ export function SystemSettings(props: { const optionsQuery = useQuery({ enabled: Boolean(props.schema), - queryFn: getAllOptions, + queryFn: () => getAllOptions(props.schema), queryKey: adminQueryKeys.settings.options(), }) diff --git a/apps/core/src/modules/configs/configs.current.ts b/apps/core/src/modules/configs/configs.current.ts index af9f71e5a1f..eeab362be1b 100644 --- a/apps/core/src/modules/configs/configs.current.ts +++ b/apps/core/src/modules/configs/configs.current.ts @@ -28,9 +28,13 @@ export function normalizeConfigPatch(schema: z.ZodType, value: any): any { ) return value const shape = typeof def.shape === 'function' ? def.shape() : def.shape + const declaredNames = new Map( + Object.keys(shape).map((name) => [name.toLowerCase(), name]), + ) return Object.fromEntries( Object.entries(value).map(([key, item]) => { - const name = key.replaceAll(/_(\w)/g, (_, c) => c.toUpperCase()) + const camelName = key.replaceAll(/_(\w)/g, (_, c) => c.toUpperCase()) + const name = declaredNames.get(camelName.toLowerCase()) ?? camelName return [ name, shape[name] ? normalizeConfigPatch(shape[name], item) : item, diff --git a/apps/core/test/src/modules/configs/current-config.spec.ts b/apps/core/test/src/modules/configs/current-config.spec.ts index 7d044693b7c..4885f537914 100644 --- a/apps/core/test/src/modules/configs/current-config.spec.ts +++ b/apps/core/test/src/modules/configs/current-config.spec.ts @@ -23,6 +23,21 @@ function createService(rows: { name: string; value: unknown }[] = []) { return { service, repository, cache: () => cache } } +it.each([ + { single_file_size_mb: 7, reader_total_active_bytes_mb: 70 }, + { singleFileSizeMb: 7, readerTotalActiveBytesMb: 70 }, +])( + 'accepts serialized acronym fields from the settings client: %j', + async (value) => { + const { service } = createService() + await service.patchAndValid('commentUploadOptions', value as any) + expect(await service.get('commentUploadOptions')).toMatchObject({ + singleFileSizeMB: 7, + readerTotalActiveBytesMB: 70, + }) + }, +) + it('exposes only current configuration and rejects retired sections without writes', async () => { const { service, repository } = createService() const config = await service.getConfigForResponse() diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index e69262675c2..a341fd55d85 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -217,3 +217,11 @@ flowchart TD - 历史 ZIP/v1 归档恢复到隔离 PostgreSQL 后,经构建产物 `migrate.mjs` 和只读 check 验证原凭据保留。演练 fixture 同步覆盖两个 PG 连接别名,避免继承测试环境连接到其他库。 - 原始全量命令确认既有 `admin-agent.e2e-spec.ts` 缺 `AiAgentConversationRepository`,初始化及清理失败;初次归档演练的连接别名问题修正后单独通过。最终明确排除该既有 Agent 夹具重跑:433 个文件通过、3 个文件跳过,3576 项通过、11 项跳过,退出 0。没有声称未排除夹具的原始命令全绿。 - Core 构建、Core/E2E 类型检查、API envelope/验证检查、50 个迁移 lint、定向 ESLint/Prettier 和 diff 检查通过。官方文档 build/check/links/pangu/lint 与 3 项测试通过,生成 llms 已同步;保留既有 1 条 hint 与 1 条死链。 + +## Admin 设置迁移验收(2026-11-09) + +- 通过当前分支启动隔离 PostgreSQL、Redis、Core 和 Admin,先写入旧 GitHub、会员、嵌套 SMTP、图片/TTS 配置,再执行正式迁移并以真实 owner 登录。 +- 18 个设置分区使用真实 Admin API 客户端读回保存,发现评论上传的 `singleFileSizeMB` / `readerTotalActiveBytesMB` 经 wire camelization 变成 `Mb`,导致输入为空且严格 PATCH 返回 422。新增 3 项回归先失败,再修复:后端按声明 schema 恢复字段大小写,Admin 按已有表单 schema 恢复值的键名,不增加请求或改变 wire/DB 名称;未知、退役字段仍拒绝。 +- 修复后 18 个分区全部通过;实测 GitHub、会员、SMTP 与 AI 编辑器编辑、保存、刷新回显,5 个凭据原值保留且仍以 V2 认证加密。评论上传值改为 7 / 70 后刷新和数据库均一致。 +- Admin 73 项定向测试、Core 40 项定向测试通过;最终 Core 回归 3578 项通过、11 项跳过,明确排除既有 Agent 缺 DI 夹具。Core/Admin 构建及类型检查、API guards、定向 lint/format 与 diff 检查通过。 +- 实际截图和录像:[Admin 设置验收页](https://ryugi.innei.dev/a/48IrQ95Vj1VecQTZtWKtf)。文件与布局检查干净;页面最终确认等待用户。验收覆盖设置的显示、提交和持久化,使用虚构凭据,不代表真实邮件、支付或模型服务商连通性验证。 From eea9e2f7bb8efcdc6786adadb7fa4012cfd2c2de Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 02:08:01 +0800 Subject: [PATCH 24/26] docs: record admin settings acceptance --- docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index a341fd55d85..02bc738a3df 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -224,4 +224,4 @@ flowchart TD - 18 个设置分区使用真实 Admin API 客户端读回保存,发现评论上传的 `singleFileSizeMB` / `readerTotalActiveBytesMB` 经 wire camelization 变成 `Mb`,导致输入为空且严格 PATCH 返回 422。新增 3 项回归先失败,再修复:后端按声明 schema 恢复字段大小写,Admin 按已有表单 schema 恢复值的键名,不增加请求或改变 wire/DB 名称;未知、退役字段仍拒绝。 - 修复后 18 个分区全部通过;实测 GitHub、会员、SMTP 与 AI 编辑器编辑、保存、刷新回显,5 个凭据原值保留且仍以 V2 认证加密。评论上传值改为 7 / 70 后刷新和数据库均一致。 - Admin 73 项定向测试、Core 40 项定向测试通过;最终 Core 回归 3578 项通过、11 项跳过,明确排除既有 Agent 缺 DI 夹具。Core/Admin 构建及类型检查、API guards、定向 lint/format 与 diff 检查通过。 -- 实际截图和录像:[Admin 设置验收页](https://ryugi.innei.dev/a/48IrQ95Vj1VecQTZtWKtf)。文件与布局检查干净;页面最终确认等待用户。验收覆盖设置的显示、提交和持久化,使用虚构凭据,不代表真实邮件、支付或模型服务商连通性验证。 +- 实际截图和录像:[Admin 设置验收页](https://ryugi.innei.dev/a/48IrQ95Vj1VecQTZtWKtf)。文件与布局检查干净;用户已在验收页及对话中确认通过。验收覆盖设置的显示、提交和持久化,使用虚构凭据,不代表真实邮件、支付或模型服务商连通性验证。 From 162517d15aaf1ff5aa412536d06409082c6dedfd Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 02:24:18 +0800 Subject: [PATCH 25/26] feat(core)!: remove encryption algorithm selection BREAKING CHANGE: Remove ENCRYPT_ALGORITHM, --encrypt_algorithm and YAML encrypt_algorithm. Field writes use fixed AES-256-GCM; only the default ECB legacy reader remains. Custom legacy algorithms require conversion on the prior version before upgrading. --- apps/core/src/app.config.ts | 9 ++--- apps/core/src/utils/secret-codec.ts | 8 +--- apps/core/src/utils/secret-storage-config.ts | 11 +++-- .../test/fixtures/read-security-config.ts | 6 +++ apps/core/test/src/upgrade/cli.spec.ts | 23 +++++++++++ apps/core/test/src/utils/encrypt.util.spec.ts | 34 ++++++++++++++++ .../src/utils/required-secret-config.spec.ts | 40 ++++++++++++++++++- docker-entrypoint.sh | 2 +- ...-11-08-mx-core-v18-compatibility-design.md | 12 +++++- docs/security-secret-storage.md | 9 ++++- docs/v18-preparation.md | 14 ++++++- packages/e2e/src/helpers/core-app-config.ts | 1 - 12 files changed, 148 insertions(+), 21 deletions(-) diff --git a/apps/core/src/app.config.ts b/apps/core/src/app.config.ts index f878600192f..a6e938fd376 100644 --- a/apps/core/src/app.config.ts +++ b/apps/core/src/app.config.ts @@ -126,10 +126,6 @@ const commander = program ).default(ENCRYPT_KEY_FROM_ENV, '[configured]'), ) .option('--encrypt_enable', 'deprecated no-op; encryption is always required') - .option( - '--encrypt_algorithm ', - 'historical ciphertext read algorithm, default is aes-256-ecb', - ) // throttle .option('--throttle_ttl ', 'throttle ttl') .option('--throttle_limit ', 'throttle limit') @@ -274,11 +270,14 @@ export const THROTTLE_OPTIONS = { const secretStorageConfig = secretStorageConfigFromEnv(process.env) export const ENCRYPT = { key: argv.encrypt_key || '', - algorithm: argv.encrypt_algorithm || 'aes-256-ecb', } assertSecretStorageConfig({ ...ENCRYPT, jwtSecret: SECURITY.jwtSecret || '', + removedOptions: { + ...secretStorageConfig.removedOptions, + encrypt_algorithm: argv.encrypt_algorithm, + }, deprecatedFlags: { ...secretStorageConfig.deprecatedFlags, encrypt_enable: argv.encrypt_enable, diff --git a/apps/core/src/utils/secret-codec.ts b/apps/core/src/utils/secret-codec.ts index 482c8466d0d..b216c5c3257 100644 --- a/apps/core/src/utils/secret-codec.ts +++ b/apps/core/src/utils/secret-codec.ts @@ -5,7 +5,6 @@ const V2_PREFIX = `${PREFIX}v2.` export interface SecretCodecOptions { key: string - algorithm?: string } /** No runtime config imports: usable by offline upgrade tools. */ @@ -101,11 +100,8 @@ export class SecretCodec { if (!/^[\d+/a-z]+={0,2}$/i.test(data.slice(PREFIX.length))) { throw new Error('Unsupported encrypted secret format') } - const decipher = crypto.createDecipheriv( - this.options.algorithm || 'aes-256-ecb', - key, - '', - ) + // Read-only compatibility for the historical default format. + const decipher = crypto.createDecipheriv('aes-256-ecb', key, '') return Buffer.concat([ decipher.update(Buffer.from(data.slice(PREFIX.length), 'base64')), decipher.final(), diff --git a/apps/core/src/utils/secret-storage-config.ts b/apps/core/src/utils/secret-storage-config.ts index f27c8ced19e..0c274702d00 100644 --- a/apps/core/src/utils/secret-storage-config.ts +++ b/apps/core/src/utils/secret-storage-config.ts @@ -1,8 +1,7 @@ export interface SecretStorageConfig { key: string jwtSecret: string - /** Used only when reading historical ECB ciphertext. */ - algorithm?: string + removedOptions?: Record deprecatedFlags?: Record } @@ -12,7 +11,7 @@ export function secretStorageConfigFromEnv( return { key: env.MX_ENCRYPT_KEY || env.ENCRYPT_KEY || '', jwtSecret: env.JWT_SECRET || env.JWTSECRET || '', - algorithm: env.ENCRYPT_ALGORITHM || 'aes-256-ecb', + removedOptions: { ENCRYPT_ALGORITHM: env.ENCRYPT_ALGORITHM }, deprecatedFlags: { MX_ENCRYPT_ENABLE: env.MX_ENCRYPT_ENABLE, ENCRYPT_ENABLE: env.ENCRYPT_ENABLE, @@ -28,6 +27,10 @@ export function inspectSecretStorageConfig(config: SecretStorageConfig) { issues.push({ code: 'FIELD_KEY_INVALID' }) if (typeof config.jwtSecret !== 'string' || config.jwtSecret.length < 32) issues.push({ code: 'JWT_TOO_SHORT' }) + for (const [path, value] of Object.entries(config.removedOptions ?? {})) { + if (value !== undefined) + issues.push({ code: 'REMOVED_ENCRYPTION_OPTION', path }) + } for (const [path, value] of Object.entries(config.deprecatedFlags ?? {})) { if ( value !== undefined && @@ -51,6 +54,8 @@ export function assertSecretStorageConfig(config: SecretStorageConfig) { return 'JWT_SECRET must be at least 32 characters; preserve existing secrets or use upgrade:rotate-jwt during maintenance' if (code === 'DEPRECATED_ENCRYPTION_FLAG') return `${path} is no longer supported: encryption and V2 writes are required. Remove this deprecated flag` + if (code === 'REMOVED_ENCRYPTION_OPTION') + return `${path} has been removed; field encryption is fixed to AES-256-GCM. Remove this option` return 'Encryption requires an explicit 64-character hexadecimal MX_ENCRYPT_KEY or ENCRYPT_KEY; preserve your existing key' }) .join('\n'), diff --git a/apps/core/test/fixtures/read-security-config.ts b/apps/core/test/fixtures/read-security-config.ts index d8cf2caa06d..5e2c7d29fbf 100644 --- a/apps/core/test/fixtures/read-security-config.ts +++ b/apps/core/test/fixtures/read-security-config.ts @@ -6,11 +6,17 @@ for (const name of [ 'MX_ENCRYPT_ENABLE', 'ENCRYPT_ENABLE', 'MX_SECRET_STORAGE_V2', + 'ENCRYPT_ALGORITHM', 'JWT_SECRET', 'JWTSECRET', ]) delete process.env[name] Object.assign(process.env, JSON.parse(process.env.SECURITY_CONFIG_TEST_ENV!)) +process.argv.splice( + 2, + process.argv.length - 2, + ...JSON.parse(process.env.SECURITY_CONFIG_TEST_ARGS || '[]'), +) await import('../../src/app.config') process.stdout.write('SECURITY_CONFIG_READY') diff --git a/apps/core/test/src/upgrade/cli.spec.ts b/apps/core/test/src/upgrade/cli.spec.ts index df459c8f782..a74b836bf92 100644 --- a/apps/core/test/src/upgrade/cli.spec.ts +++ b/apps/core/test/src/upgrade/cli.spec.ts @@ -34,6 +34,29 @@ describe('upgrade CLI isolated process', () => { }) afterAll(async () => context?.close()) + it('diagnoses the removed algorithm setting through the standalone read-only check', async () => { + const output: string[] = [] + expect( + await runUpgradeCli( + ['check', '--json'], + { + PG_URL: context.connectionString, + MX_ENCRYPT_KEY: 'ab'.repeat(32), + JWT_SECRET: 'long-enough-test-jwt-123456789012345', + ENCRYPT_ALGORITHM: 'aes-256-ecb', + }, + (value) => output.push(value), + ), + ).toBe(2) + expect(JSON.parse(output[0]).issues).toContainEqual({ + code: 'REMOVED_ENCRYPTION_OPTION', + path: 'ENCRYPT_ALGORITHM', + }) + expect( + (await context.pool.query('SELECT id FROM _app_migrations')).rows, + ).toEqual([]) + }) + it('rejects the removed prepare command without modifying the database', async () => { const output: string[] = [] expect( diff --git a/apps/core/test/src/utils/encrypt.util.spec.ts b/apps/core/test/src/utils/encrypt.util.spec.ts index 56bc3e46157..21601e85c65 100644 --- a/apps/core/test/src/utils/encrypt.util.spec.ts +++ b/apps/core/test/src/utils/encrypt.util.spec.ts @@ -2,6 +2,7 @@ import crypto from 'node:crypto' import { ENCRYPT } from '~/app.config' import { EncryptUtil } from '~/utils/encrypt.util' +import { SecretCodec } from '~/utils/secret-codec' const legacyEncrypt = (value: string) => { const cipher = crypto.createCipheriv( @@ -18,6 +19,39 @@ const legacyEncrypt = (value: string) => { } describe('EncryptUtil', () => { + it('uses the fixed legacy reader and upgrades to interoperable AES-256-GCM despite obsolete codec options', () => { + const options = { + key: ENCRYPT.key, + algorithm: 'aes-256-ctr', + } + const codec = new SecretCodec(options) + const encrypted = codec.encrypt( + legacyEncrypt('retained-secret'), + 'owner.field', + ) + const [, keyId, nonce, tag, ciphertext] = encrypted.split('.') + const key = Buffer.from(ENCRYPT.key, 'hex') + const expectedId = crypto + .createHash('sha256') + .update(key) + .digest('hex') + .slice(0, 16) + const decipher = crypto.createDecipheriv( + 'aes-256-gcm', + key, + Buffer.from(nonce, 'base64url'), + ) + decipher.setAAD(Buffer.from(`mx:v2:${expectedId}:owner.field`)) + decipher.setAuthTag(Buffer.from(tag, 'base64url')) + expect(keyId).toBe(expectedId) + expect( + Buffer.concat([ + decipher.update(Buffer.from(ciphertext, 'base64url')), + decipher.final(), + ]).toString('utf8'), + ).toBe('retained-secret') + }) + it('randomizes ciphertext and round-trips nested data', () => { const data = { value: ['a secret', null, 42] } const encrypted = EncryptUtil.encrypt(data) diff --git a/apps/core/test/src/utils/required-secret-config.spec.ts b/apps/core/test/src/utils/required-secret-config.spec.ts index d7c001ae51f..8cdc513bc57 100644 --- a/apps/core/test/src/utils/required-secret-config.spec.ts +++ b/apps/core/test/src/utils/required-secret-config.spec.ts @@ -1,4 +1,7 @@ import { execFile } from 'node:child_process' +import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join, relative } from 'node:path' import { fileURLToPath } from 'node:url' import { promisify } from 'node:util' @@ -6,7 +9,8 @@ const validEnv = { MX_ENCRYPT_KEY: 'ab'.repeat(32), JWT_SECRET: 'runtime-jwt-secret-for-testing-1234567890', } -async function boot(overrides: NodeJS.ProcessEnv = {}) { +const coreCwd = fileURLToPath(new URL('../../../', import.meta.url)) +async function boot(overrides: NodeJS.ProcessEnv = {}, args: string[] = []) { const env = { ...process.env } for (const name of [ 'MX_ENCRYPT_KEY', @@ -14,6 +18,7 @@ async function boot(overrides: NodeJS.ProcessEnv = {}) { 'MX_ENCRYPT_ENABLE', 'ENCRYPT_ENABLE', 'MX_SECRET_STORAGE_V2', + 'ENCRYPT_ALGORITHM', 'JWT_SECRET', 'JWTSECRET', ]) @@ -27,11 +32,12 @@ async function boot(overrides: NodeJS.ProcessEnv = {}) { 'test/fixtures/read-security-config.ts', ], { - cwd: fileURLToPath(new URL('../../../', import.meta.url)), + cwd: coreCwd, env: { ...env, NODE_ENV: 'development', SECURITY_CONFIG_TEST_ENV: JSON.stringify(overrides), + SECURITY_CONFIG_TEST_ARGS: JSON.stringify(args), }, }, ) @@ -75,3 +81,33 @@ it('keeps key/JWT aliases and accepts old true flags as deprecated no-ops', asyn expect(result.stdout, result.stderr).toContain('SECURITY_CONFIG_READY') expect(result.stderr).toContain('deprecated') }) + +it.each(['', 'aes-256-ecb', 'aes-256-gcm'])( + 'rejects the removed algorithm environment option even when its value is %j', + async (algorithm) => { + const result = await boot({ ...validEnv, ENCRYPT_ALGORITHM: algorithm }) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') + expect(result.stderr).toContain('ENCRYPT_ALGORITHM') + }, +) + +it('rejects the removed algorithm command-line option', async () => { + const result = await boot(validEnv, ['--encrypt_algorithm', 'aes-256-ecb']) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') +}) + +it('rejects the removed algorithm YAML option', async () => { + const directory = await mkdtemp(join(tmpdir(), 'mx-algorithm-config-')) + try { + const file = join(directory, 'config.yml') + await writeFile(file, 'encrypt_algorithm: aes-256-ecb\n') + const result = await boot(validEnv, ['--config', relative(coreCwd, file)]) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') + expect(result.stderr).toContain('encrypt_algorithm') + } finally { + await rm(directory, { recursive: true }) + } +}) diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index 798993cadd0..38973ab0462 100755 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -172,7 +172,7 @@ if [ -z "$encrypt_enable" ] && [ -n "${MX_ENCRYPT_KEY:-${ENCRYPT_KEY:-}}" ]; the fi log_kv "Encryption" "${encrypt_enable:-false}" "env:MX_ENCRYPT_ENABLE/ENCRYPT_ENABLE (+key implies true)" log_kv "Encrypt Key" "$(mask_secret_hint "${MX_ENCRYPT_KEY:-${ENCRYPT_KEY:-}}")" "env:MX_ENCRYPT_KEY/ENCRYPT_KEY" -log_kv "Encrypt Algorithm" "${ENCRYPT_ALGORITHM:-}" "$(env_source ENCRYPT_ALGORITHM '')" +log_kv "Encrypt Algorithm" "AES-256-GCM" "fixed" cluster_enable="$(parse_booleanish "${CLUSTER:-}")" log_kv "Cluster" "${cluster_enable:-false}" "$(env_source CLUSTER false)" diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 02bc738a3df..3714a66ad4a 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -8,6 +8,8 @@ 升级入口收口(2026-11-09):按用户要求删除独立 prepare、dry-run 和准备标记。配置、凭据及 AI 指标统一由正式 `migrate` 完成;新增必跑 `20261109-required-secret-storage`,复用历史凭据转换行为,并以新迁移 ID 保证旧 opt-in 账本不会跳过本次回填。删除凭据迁移开关及专用脚本。独立 CLI 仅保留只读 check 和需要旧、新秘密文件的 JWT 维护轮换。 +算法收口(2026-11-09):按用户要求删除 `ENCRYPT_ALGORITHM`、CLI `--encrypt_algorithm` 和 YAML `encrypt_algorithm`,字段新写入固定 AES-256-GCM(随机 96-bit nonce、128-bit tag、路径 AAD)。旧配置即使等于旧默认值或 GCM 也明确拒绝,check 对环境变量报告 `REMOVED_ENCRYPTION_OPTION`。历史读取固定为默认 AES-256-ECB;使用自定义历史算法的实例必须先用旧版、原密钥和原算法将相关凭据转换为 V2 并验证,再升级;旧快照隔离恢复仍需保留原设置。 + 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 ## 意图 @@ -53,7 +55,7 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 - 显式 `ENCRYPT_KEY`/`MX_ENCRYPT_KEY` 为 64 位十六进制;`JWT_SECRET` 至少 32 字符。保留既有别名优先级。 - 新装生成每实例随机密钥并持久保存;现有密钥原样保留。生产模板不能内置公共有效密钥。 -- 工程选择:旧 `ENCRYPT_ENABLE=true`、`MX_SECRET_STORAGE_V2=true` 暂接受并提示可删除;显式 false 拒绝启动。新写入固定 V2,`ENCRYPT_ALGORITHM` 只服务历史读取。 +- 工程选择:旧 `ENCRYPT_ENABLE=true`、`MX_SECRET_STORAGE_V2=true` 暂接受并提示可删除;显式 false 拒绝启动。新写入固定 V2 AES-256-GCM,算法选择已删除,历史读取固定 AES-256-ECB。 - 新装空库由发布前迁移初始化当前状态,不要求 v17 历史;既有实例需要准备完成。 - 普通生产启动检查配置和 schema,不做业务数据迁移;部署前统一运行正式迁移。 @@ -225,3 +227,11 @@ flowchart TD - 修复后 18 个分区全部通过;实测 GitHub、会员、SMTP 与 AI 编辑器编辑、保存、刷新回显,5 个凭据原值保留且仍以 V2 认证加密。评论上传值改为 7 / 70 后刷新和数据库均一致。 - Admin 73 项定向测试、Core 40 项定向测试通过;最终 Core 回归 3578 项通过、11 项跳过,明确排除既有 Agent 缺 DI 夹具。Core/Admin 构建及类型检查、API guards、定向 lint/format 与 diff 检查通过。 - 实际截图和录像:[Admin 设置验收页](https://ryugi.innei.dev/a/48IrQ95Vj1VecQTZtWKtf)。文件与布局检查干净;用户已在验收页及对话中确认通过。验收覆盖设置的显示、提交和持久化,使用虚构凭据,不代表真实邮件、支付或模型服务商连通性验证。 + +## 删除算法选择的验证(2026-11-09) + +- 删除 codec/config 的算法属性、CLI 参数及环境算法来源;env/YAML 旧项明确拒绝,Docker 日志固定标示 AES-256-GCM。历史默认 ECB 读取固定,不再受传入对象中的旧算法属性影响。 +- 7 项新增行为回归先失败后通过:环境变量为空/旧默认/GCM 均拒绝、CLI/YAML 拒绝、独立只读检查诊断且不写库、默认 ECB 升级后可用独立 Node AES-256-GCM 解密。25 项定向测试通过。 +- 完整回归 `TZ=UTC pnpm -C apps/core test run --maxWorkers=3 --exclude '**/admin-agent.e2e-spec.ts'` 退出 0:433 个文件通过、3 个跳过,3585 项通过、11 项跳过,包含构建产物的历史归档恢复及正式迁移演练。既有 Agent 缺 DI 夹具继续明确排除。 +- Core 生产构建、Core/E2E 类型检查、API envelope/验证检查、定向 ESLint/Prettier、入口脚本语法与 diff 检查通过。没有修改 Admin 设置流程或撤销已确认的验收。 +- mx-docs 同步环境变量、加密、v18 迁移及 Agent 参考;build/check/links/pangu/lint 与 3 项测试通过。生成的 llms 已核对算法删除和自定义历史算法的升级前转换要求;既有 1 hint、1 死链保持。未发版或运行生产迁移。 diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index 5747469fb67..bb49e6767a6 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -17,7 +17,14 @@ Historical true values are accepted with a deprecation warning; false or other unsupported values fail before business startup. These flags no longer select a storage mode. Config/Snippet/Webhook writes always use authenticated V2 encryption, API-key writes always hash, and OAuth writes stay encrypted. -`ENCRYPT_ALGORITHM` is used only to read historical ECB ciphertext. +Field writes use fixed AES-256-GCM with a fresh 96-bit nonce and a 128-bit +authentication tag. `ENCRYPT_ALGORITHM`, `--encrypt_algorithm` and YAML +`encrypt_algorithm` have been removed; supplying them fails even when set to +the old default or AES-256-GCM. Remove the option rather than changing its value. +The historical reader is fixed to the old default AES-256-ECB. Instances that +used another legacy algorithm must convert all affected credentials to V2 on +the prior version with their original algorithm and key before upgrading. +Keep those original settings with historical backups for isolated recovery. The published v17.0.4 escape hatch is historical behavior, removed here. This branch is not released yet. The [preparation runbook](v18-preparation.md) diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index f479247eb2a..0f9afbdcc21 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -2,7 +2,8 @@ This unreleased branch now requires field encryption, an explicit hex64 key and a JWT of at least 32 characters. Plaintext/ECB writes, encryption-disable -flags and machine-ID fallback have been removed. Canonical configuration migration and strict configuration PATCH validation +flags, algorithm selection and machine-ID fallback have been removed. Field +writes are fixed to AES-256-GCM. Canonical configuration migration and strict configuration PATCH validation are also enforced here. SDK 6 cleanup and column removal remain future work. These commands are not part of released v17.0.4. No minimum released preparation tag is established yet. Obtain this build without starting business replicas; do not publish or migrate production as @@ -15,6 +16,17 @@ archive recovery private key. Never replace existing secrets with newly generated ones to resolve a decryption error. Older instances complete their historical schema/data migrations before using this runner. +Remove `ENCRYPT_ALGORITHM`, CLI `--encrypt_algorithm` and YAML +`encrypt_algorithm`; these are rejected, including an empty environment value +or the old default. `check` reports `REMOVED_ENCRYPTION_OPTION` for the environment +variable. The historical field reader is fixed to AES-256-ECB; new writes use +AES-256-GCM with a random 96-bit nonce and 128-bit authentication tag. If an +instance used a different legacy algorithm, first convert its affected +credentials to V2 on the prior version using the original key and algorithm, +then verify the credentials before upgrading. Keep the original algorithm and +key with old snapshots for isolated recovery; removing an option does not +convert custom-algorithm ciphertext. + The independent CLI consumes environment configuration, not the application's CLI/YAML flags. Supply the same effective `PG_URL` (preferred), `PG_CONNECTION_STRING` or `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE`, diff --git a/packages/e2e/src/helpers/core-app-config.ts b/packages/e2e/src/helpers/core-app-config.ts index 3b3c8da0aa9..6ab7bb92a92 100644 --- a/packages/e2e/src/helpers/core-app-config.ts +++ b/packages/e2e/src/helpers/core-app-config.ts @@ -102,7 +102,6 @@ export const ENCRYPT = { key: process.env.MX_ENCRYPT_KEY || '593f62860255feb0a914534a43814b9809cc7534da7f5485cd2e3d3c8609acab', - algorithm: 'aes-256-ecb', } export const TELEMETRY = { From 988ff353dec5bf216d930db62b3b168fe0e7b220 Mon Sep 17 00:00:00 2001 From: Innei Date: Mon, 9 Nov 2026 02:50:15 +0800 Subject: [PATCH 26/26] feat(core): persist automatic development encryption keys --- apps/core/src/app.config.ts | 11 +- .../src/utils/development-encryption-key.ts | 43 ++++++ .../fixtures/development-migrations.config.ts | 11 ++ .../test/fixtures/development-migrations.ts | 19 +++ .../test/fixtures/read-security-config.ts | 13 +- .../src/utils/development-encryption.spec.ts | 72 +++++++++ .../src/utils/required-secret-config.spec.ts | 145 +++++++++++++++++- ...-11-08-mx-core-v18-compatibility-design.md | 41 +++-- docs/security-secret-storage.md | 13 +- docs/v18-preparation.md | 11 +- 10 files changed, 356 insertions(+), 23 deletions(-) create mode 100644 apps/core/src/utils/development-encryption-key.ts create mode 100644 apps/core/test/fixtures/development-migrations.config.ts create mode 100644 apps/core/test/fixtures/development-migrations.ts create mode 100644 apps/core/test/src/utils/development-encryption.spec.ts diff --git a/apps/core/src/app.config.ts b/apps/core/src/app.config.ts index a6e938fd376..23ba327ba15 100644 --- a/apps/core/src/app.config.ts +++ b/apps/core/src/app.config.ts @@ -6,6 +6,7 @@ import { Option, program } from 'commander' import { load as yamlLoad } from 'js-yaml' import { isDebugMode, isDev } from './global/env.global' +import { resolveDevelopmentEncryptionKey } from './utils/development-encryption-key' import { parseRedisConnectionString, resolveRedisConnectionStringEnv, @@ -269,7 +270,11 @@ export const THROTTLE_OPTIONS = { const secretStorageConfig = secretStorageConfigFromEnv(process.env) export const ENCRYPT = { - key: argv.encrypt_key || '', + key: resolveDevelopmentEncryptionKey( + argv.encrypt_key || '', + isDev, + path.join(process.cwd(), 'tmp'), + ), } assertSecretStorageConfig({ ...ENCRYPT, @@ -283,6 +288,10 @@ assertSecretStorageConfig({ encrypt_enable: argv.encrypt_enable, }, }) +if (isDev && !argv.encrypt_key) + console.warn( + 'Using the local development encryption key at tmp/dev-encryption.key; preserve this file with your development database', + ) export const TELEMETRY = { enable: !parseBooleanishValue(argv.disable_telemetry ?? MX_DISABLE_TELEMETRY), diff --git a/apps/core/src/utils/development-encryption-key.ts b/apps/core/src/utils/development-encryption-key.ts new file mode 100644 index 00000000000..7c80c028f51 --- /dev/null +++ b/apps/core/src/utils/development-encryption-key.ts @@ -0,0 +1,43 @@ +import { randomBytes } from 'node:crypto' +import { + linkSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs' +import { join } from 'node:path' + +/** Resolve a persistent local key only when a development build has no explicit key. */ +export function resolveDevelopmentEncryptionKey( + configuredKey: string, + development: boolean, + directory: string, +): string { + if (configuredKey || !development) return configuredKey + const keyFile = join(directory, 'dev-encryption.key') + try { + return readFileSync(keyFile, 'utf8').trim() + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error + } + mkdirSync(directory, { recursive: true, mode: 0o700 }) + const temporary = mkdtempSync(join(directory, '.dev-encryption-')) + try { + const candidate = join(temporary, 'key') + writeFileSync(candidate, randomBytes(32).toString('hex'), { + flag: 'wx', + mode: 0o600, + }) + try { + // Publish a complete key atomically; concurrent workers reuse the winner. + linkSync(candidate, keyFile) + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error + } + } finally { + rmSync(temporary, { recursive: true }) + } + return readFileSync(keyFile, 'utf8').trim() +} diff --git a/apps/core/test/fixtures/development-migrations.config.ts b/apps/core/test/fixtures/development-migrations.config.ts new file mode 100644 index 00000000000..b55caf8e9c9 --- /dev/null +++ b/apps/core/test/fixtures/development-migrations.config.ts @@ -0,0 +1,11 @@ +import { defineConfig, mergeConfig } from 'vite' + +import coreConfig from '../../vite.config' + +// Keep production transforms/metadata while disabling browser dependency +// prebundling in this one-shot development migration process. +export default defineConfig((environment) => + mergeConfig(coreConfig(environment), { + optimizeDeps: { noDiscovery: true }, + }), +) diff --git a/apps/core/test/fixtures/development-migrations.ts b/apps/core/test/fixtures/development-migrations.ts new file mode 100644 index 00000000000..b3b4d707e2a --- /dev/null +++ b/apps/core/test/fixtures/development-migrations.ts @@ -0,0 +1,19 @@ +// Resolve Vite dependencies from the project, then isolate runtime storage and +// credentials from its local .env before importing application configuration. +for (const name of [ + 'MX_ENCRYPT_KEY', + 'ENCRYPT_KEY', + 'MX_ENCRYPT_ENABLE', + 'ENCRYPT_ENABLE', + 'MX_SECRET_STORAGE_V2', + 'ENCRYPT_ALGORITHM', + 'JWT_SECRET', + 'JWTSECRET', +]) + delete process.env[name] +Object.assign(process.env, JSON.parse(process.env.DEV_MIGRATION_TEST_ENV!)) +process.chdir(process.env.SECURITY_CONFIG_TEST_CWD!) +process.argv.splice(2) +const { runAppMigrations } = await import('../../src/app-migrate') +await runAppMigrations() +process.exit(0) diff --git a/apps/core/test/fixtures/read-security-config.ts b/apps/core/test/fixtures/read-security-config.ts index 5e2c7d29fbf..e397076cee0 100644 --- a/apps/core/test/fixtures/read-security-config.ts +++ b/apps/core/test/fixtures/read-security-config.ts @@ -17,6 +17,15 @@ process.argv.splice( process.argv.length - 2, ...JSON.parse(process.env.SECURITY_CONFIG_TEST_ARGS || '[]'), ) -await import('../../src/app.config') +if (process.env.SECURITY_CONFIG_TEST_CWD) + process.chdir(process.env.SECURITY_CONFIG_TEST_CWD) +const { ENCRYPT } = await import('../../src/app.config') +const { SecretCodec } = await import('../../src/utils/secret-codec') +const codec = new SecretCodec(ENCRYPT) +const encrypted = + process.env.SECURITY_CONFIG_TEST_CIPHERTEXT || + codec.encrypt('fixture-secret', 'fixture.field') -process.stdout.write('SECURITY_CONFIG_READY') +process.stdout.write( + `SECURITY_CONFIG_READY\n${JSON.stringify({ encrypted, decrypted: codec.decrypt(encrypted, 'fixture.field') })}`, +) diff --git a/apps/core/test/src/utils/development-encryption.spec.ts b/apps/core/test/src/utils/development-encryption.spec.ts new file mode 100644 index 00000000000..9ca73adff47 --- /dev/null +++ b/apps/core/test/src/utils/development-encryption.spec.ts @@ -0,0 +1,72 @@ +import { execFile } from 'node:child_process' +import { mkdtemp, readFile, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' + +import { createPgTestDatabase } from 'test/helper/pg-verify-url' + +import { SecretCodec } from '~/utils/secret-codec' + +it('runs development migrations without a configured field key and preserves encrypted credentials on restart', async () => { + const context = await createPgTestDatabase('mx_development_key') + const directory = await mkdtemp(join(tmpdir(), 'mx-dev-migrations-')) + const core = fileURLToPath(new URL('../../../', import.meta.url)) + try { + await context.pool.query( + "INSERT INTO options (id,name,value) VALUES ('100','membership',$1)", + [{ dodoApiKey: 'retained-development-token' }], + ) + const overrides = { + NODE_ENV: 'development', + PG_URL: context.connectionString, + PG_CONNECTION_STRING: context.connectionString, + MIGRATIONS_DIR: join(core, 'src/database/migrations'), + JWT_SECRET: 'development-migration-jwt-at-least-32-characters', + } + const run = () => + promisify(execFile)( + join(core, 'node_modules/.bin/vite-node'), + [ + '--config', + join(core, 'test/fixtures/development-migrations.config.ts'), + join(core, 'test/fixtures/development-migrations.ts'), + ], + { + cwd: core, + env: { + ...process.env, + DEV_MIGRATION_TEST_ENV: JSON.stringify(overrides), + SECURITY_CONFIG_TEST_CWD: directory, + }, + }, + ).catch((error) => { + throw new Error(`${error.message}\n${error.stdout}`) + }) + await run() + const keyFile = join(directory, 'tmp/dev-encryption.key') + const key = await readFile(keyFile, 'utf8') + const rows = () => + context.pool.query('SELECT name,value FROM options ORDER BY name') + const first = (await rows()).rows + const membership = first.find((row) => row.name === 'membership')!.value + expect(membership.apiKey).toMatch(/^\$\$\{mx\}\$\$v2\./) + expect( + new SecretCodec({ key }).decrypt(membership.apiKey, 'membership.apiKey'), + ).toBe('retained-development-token') + expect( + ( + await context.pool.query( + "SELECT id FROM _app_migrations WHERE id='20261109-required-secret-storage'", + ) + ).rowCount, + ).toBe(1) + await run() + expect((await rows()).rows).toEqual(first) + expect(await readFile(keyFile, 'utf8')).toBe(key) + } finally { + await context.close() + await rm(directory, { recursive: true }) + } +}, 60000) diff --git a/apps/core/test/src/utils/required-secret-config.spec.ts b/apps/core/test/src/utils/required-secret-config.spec.ts index 8cdc513bc57..ca35327944a 100644 --- a/apps/core/test/src/utils/required-secret-config.spec.ts +++ b/apps/core/test/src/utils/required-secret-config.spec.ts @@ -1,5 +1,14 @@ import { execFile } from 'node:child_process' -import { mkdtemp, rm, writeFile } from 'node:fs/promises' +import { existsSync } from 'node:fs' +import { + mkdir, + mkdtemp, + readdir, + readFile, + rm, + stat, + writeFile, +} from 'node:fs/promises' import { tmpdir } from 'node:os' import { join, relative } from 'node:path' import { fileURLToPath } from 'node:url' @@ -35,7 +44,7 @@ async function boot(overrides: NodeJS.ProcessEnv = {}, args: string[] = []) { cwd: coreCwd, env: { ...env, - NODE_ENV: 'development', + NODE_ENV: 'production', SECURITY_CONFIG_TEST_ENV: JSON.stringify(overrides), SECURITY_CONFIG_TEST_ARGS: JSON.stringify(args), }, @@ -111,3 +120,135 @@ it('rejects the removed algorithm YAML option', async () => { await rm(directory, { recursive: true }) } }) + +it('persists a private development key and decrypts data across concurrent boots and restarts', async () => { + const directory = await mkdtemp(join(tmpdir(), 'mx-dev-security-')) + const env = { + NODE_ENV: 'development', + JWT_SECRET: validEnv.JWT_SECRET, + SECURITY_CONFIG_TEST_CWD: directory, + } + try { + const results = await Promise.all([boot(env), boot(env)]) + for (const result of results) + expect(result.stdout, result.stderr).toContain('SECURITY_CONFIG_READY') + const keyFile = join(directory, 'tmp', 'dev-encryption.key') + const key = (await readFile(keyFile, 'utf8')).trim() + expect(key).toMatch(/^[\da-f]{64}$/i) + expect((await stat(keyFile)).mode & 0o777).toBe(0o600) + expect(await readdir(join(directory, 'tmp'))).toEqual([ + 'dev-encryption.key', + ]) + const first = JSON.parse(results[0].stdout.split('\n')[1]) + const restarted = await boot({ + ...env, + SECURITY_CONFIG_TEST_CIPHERTEXT: first.encrypted, + }) + expect(JSON.parse(restarted.stdout.split('\n')[1])).toEqual({ + encrypted: first.encrypted, + decrypted: 'fixture-secret', + }) + expect(await readFile(keyFile, 'utf8')).toBe(key) + } finally { + await rm(directory, { recursive: true }) + } +}, 30000) + +it('preserves an explicit development key without creating or replacing a local key', async () => { + const directory = await mkdtemp(join(tmpdir(), 'mx-explicit-security-')) + try { + const result = await boot({ + ...validEnv, + NODE_ENV: 'development', + SECURITY_CONFIG_TEST_CWD: directory, + }) + expect(result.stdout, result.stderr).toContain('SECURITY_CONFIG_READY') + expect(await readdir(directory)).toEqual([]) + } finally { + await rm(directory, { recursive: true }) + } +}) + +it.each(['production', 'test', 'staging'])( + 'never generates a field key outside development (%s)', + async (mode) => { + const directory = await mkdtemp(join(tmpdir(), 'mx-strict-security-')) + try { + const result = await boot({ + NODE_ENV: mode, + JWT_SECRET: validEnv.JWT_SECRET, + SECURITY_CONFIG_TEST_CWD: directory, + }) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') + expect(await readdir(directory)).toEqual([]) + } finally { + await rm(directory, { recursive: true }) + } + }, +) + +it.each(['local', 'explicit'])( + 'refuses an invalid %s development key without replacing persisted secrets', + async (source) => { + const directory = await mkdtemp(join(tmpdir(), 'mx-invalid-dev-security-')) + const persisted = + source === 'local' ? 'invalid-local-key' : validEnv.MX_ENCRYPT_KEY + try { + await mkdir(join(directory, 'tmp')) + const keyFile = join(directory, 'tmp', 'dev-encryption.key') + await writeFile(keyFile, persisted) + const result = await boot({ + NODE_ENV: 'development', + JWT_SECRET: validEnv.JWT_SECRET, + ...(source === 'explicit' + ? { MX_ENCRYPT_KEY: 'invalid-explicit-key' } + : {}), + SECURITY_CONFIG_TEST_CWD: directory, + }) + expect(result).toHaveProperty('code', 1) + expect(result.stdout).not.toContain('SECURITY_CONFIG_READY') + expect(result.stderr).not.toContain('invalid-local-key') + expect(result.stderr).not.toContain('invalid-explicit-key') + expect(await readFile(keyFile, 'utf8')).toBe(persisted) + expect(await readdir(join(directory, 'tmp'))).toEqual([ + 'dev-encryption.key', + ]) + } finally { + await rm(directory, { recursive: true }) + } + }, +) + +const builtEntry = fileURLToPath( + new URL('../../../out/main.mjs', import.meta.url), +) +it.skipIf(!existsSync(builtEntry))( + 'keeps release builds strict even with NODE_ENV=development and a local development key', + async () => { + const directory = await mkdtemp(join(tmpdir(), 'mx-built-security-')) + try { + await mkdir(join(directory, 'tmp')) + const keyFile = join(directory, 'tmp', 'dev-encryption.key') + await writeFile(keyFile, validEnv.MX_ENCRYPT_KEY) + const env: NodeJS.ProcessEnv = { + ...process.env, + NODE_ENV: 'development', + JWT_SECRET: validEnv.JWT_SECRET, + } + delete env.MX_ENCRYPT_KEY + delete env.ENCRYPT_KEY + const result = await promisify(execFile)('node', [builtEntry], { + cwd: directory, + env, + }).catch( + (error) => error as { code: number; stdout: string; stderr: string }, + ) + expect(result).toHaveProperty('code', 1) + expect(result.stderr).toContain('Encryption requires an explicit') + expect(await readFile(keyFile, 'utf8')).toBe(validEnv.MX_ENCRYPT_KEY) + } finally { + await rm(directory, { recursive: true }) + } + }, +) diff --git a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md index 3714a66ad4a..16653c3dea1 100644 --- a/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md +++ b/docs/ryugi/specs/2026-11-08-mx-core-v18-compatibility-design.md @@ -10,6 +10,8 @@ 算法收口(2026-11-09):按用户要求删除 `ENCRYPT_ALGORITHM`、CLI `--encrypt_algorithm` 和 YAML `encrypt_algorithm`,字段新写入固定 AES-256-GCM(随机 96-bit nonce、128-bit tag、路径 AAD)。旧配置即使等于旧默认值或 GCM 也明确拒绝,check 对环境变量报告 `REMOVED_ENCRYPTION_OPTION`。历史读取固定为默认 AES-256-ECB;使用自定义历史算法的实例必须先用旧版、原密钥和原算法将相关凭据转换为 V2 并验证,再升级;旧快照隔离恢复仍需保留原设置。 +开发体验修订(2026-11-09):用户选择开发环境不配字段密钥也能启动,自动生成并持久保存本地开发密钥。源码 dev 首次在当前工作目录 `tmp/dev-encryption.key` 以 `0600` 保存随机 hex64 密钥,多进程原子发布并复用;显式密钥优先,错误密钥不覆盖。开发环境仍写 AES-256-GCM,JWT 要求不变。生产构建和独立升级检查仍要求显式字段密钥,生产产物即使设置 `NODE_ENV=development` 也不能启用自动密钥。开发库转生产需显式提供原本地密钥。 + 基线:Core `v17.0.4`(`c6551f15a`);SDK `5.13.1`;官方文档仓库 `mx-space/docs` 的 `main`(`cc0f960`,Astro + Fumadocs)。版本号仅是核对基线,准备版和消费者的最低发布 tag 要以实际验证结果确定。 ## 意图 @@ -18,21 +20,21 @@ ## 已确认范围 -| 项目 | 决定 | -| ----------- | ------------------------------------------------------------------------------------------- | -| 加密必填 | v18 要求显式 64 位十六进制字段密钥,结束关闭加密与机器标识回退 | -| V2 写入 | v18 统一配置/Snippet 的 GCM、Webhook 加密、API Key 哈希与 OAuth 加密,退役旧格式写入开关 | -| 历史读取 | 保留历史明文/ECB 的读取和迁移、旧 ZIP/v1 加密备份恢复、旧密码登录后升级 | -| 升级基线 | 只有完成准备的 v17.x 可直接升 v18;更老版本先完成历史迁移 | -| 已废弃入口 | 移除评论列表的 `state` 查询别名和速记表态 GET;先迁 SDK、CLI、官方主题 | -| 旧配置 | AI、第三方服务、会员配置先持久化迁移,再移除运行时读取兼容;清理无效参数和 MongoDB 日志残留 | -| 文档节奏 | 先纠正 v17 和补 v15–v17 指南;v18 发布时切默认示例,并保留 v17 关键入口 | -| AI 普通读取 | v18 摘要/精读 GET 只读;缺结果返回 `{ data: null }`,生成显式触发 | -| 执行类 GET | 非流式备份/清缓存/测试邮件等改 POST,保留现有 GET SSE 和帧协议 | -| 配置验证 | v18 拒绝未知配置字段,保留大小写归一化、部分更新和自由 JSON | -| AI 指标 | 先补齐 generationMetrics 并停止旧列引用,v18 再删除旧字段和物理列 | -| 排序 | 服务端统一 `asc`/`desc`;兼容 SDK 先转换 `1`/`-1` 的调用 | -| SDK 与别名 | 协调 SDK 6.x 清理确认废弃项;保留 SDK legacy、CLI V2 适配和其他有效路由别名、长期 URL | +| 项目 | 决定 | +| ----------- | ----------------------------------------------------------------------------------------------- | +| 加密必填 | v18 生产要求显式 64 位十六进制字段密钥,源码 dev 可生成持久本地密钥;结束关闭加密与机器标识回退 | +| V2 写入 | v18 统一配置/Snippet 的 GCM、Webhook 加密、API Key 哈希与 OAuth 加密,退役旧格式写入开关 | +| 历史读取 | 保留历史明文/ECB 的读取和迁移、旧 ZIP/v1 加密备份恢复、旧密码登录后升级 | +| 升级基线 | 只有完成准备的 v17.x 可直接升 v18;更老版本先完成历史迁移 | +| 已废弃入口 | 移除评论列表的 `state` 查询别名和速记表态 GET;先迁 SDK、CLI、官方主题 | +| 旧配置 | AI、第三方服务、会员配置先持久化迁移,再移除运行时读取兼容;清理无效参数和 MongoDB 日志残留 | +| 文档节奏 | 先纠正 v17 和补 v15–v17 指南;v18 发布时切默认示例,并保留 v17 关键入口 | +| AI 普通读取 | v18 摘要/精读 GET 只读;缺结果返回 `{ data: null }`,生成显式触发 | +| 执行类 GET | 非流式备份/清缓存/测试邮件等改 POST,保留现有 GET SSE 和帧协议 | +| 配置验证 | v18 拒绝未知配置字段,保留大小写归一化、部分更新和自由 JSON | +| AI 指标 | 先补齐 generationMetrics 并停止旧列引用,v18 再删除旧字段和物理列 | +| 排序 | 服务端统一 `asc`/`desc`;兼容 SDK 先转换 `1`/`-1` 的调用 | +| SDK 与别名 | 协调 SDK 6.x 清理确认废弃项;保留 SDK legacy、CLI V2 适配和其他有效路由别名、长期 URL | ## 不纳入 @@ -53,7 +55,7 @@ API v4、wire 大小写/envelope 重做、SSE 帧协议重做、强制加密全 ### 当前分支已生效的启动条件 -- 显式 `ENCRYPT_KEY`/`MX_ENCRYPT_KEY` 为 64 位十六进制;`JWT_SECRET` 至少 32 字符。保留既有别名优先级。 +- 生产显式 `ENCRYPT_KEY`/`MX_ENCRYPT_KEY` 为 64 位十六进制;源码 dev 可自动生成持久本地密钥。`JWT_SECRET` 至少 32 字符,保留既有别名优先级。 - 新装生成每实例随机密钥并持久保存;现有密钥原样保留。生产模板不能内置公共有效密钥。 - 工程选择:旧 `ENCRYPT_ENABLE=true`、`MX_SECRET_STORAGE_V2=true` 暂接受并提示可删除;显式 false 拒绝启动。新写入固定 V2 AES-256-GCM,算法选择已删除,历史读取固定 AES-256-ECB。 - 新装空库由发布前迁移初始化当前状态,不要求 v17 历史;既有实例需要准备完成。 @@ -235,3 +237,10 @@ flowchart TD - 完整回归 `TZ=UTC pnpm -C apps/core test run --maxWorkers=3 --exclude '**/admin-agent.e2e-spec.ts'` 退出 0:433 个文件通过、3 个跳过,3585 项通过、11 项跳过,包含构建产物的历史归档恢复及正式迁移演练。既有 Agent 缺 DI 夹具继续明确排除。 - Core 生产构建、Core/E2E 类型检查、API envelope/验证检查、定向 ESLint/Prettier、入口脚本语法与 diff 检查通过。没有修改 Admin 设置流程或撤销已确认的验收。 - mx-docs 同步环境变量、加密、v18 迁移及 Agent 参考;build/check/links/pangu/lint 与 3 项测试通过。生成的 llms 已核对算法删除和自定义历史算法的升级前转换要求;既有 1 hint、1 死链保持。未发版或运行生产迁移。 + +## 本地开发密钥验证(2026-11-09) + +- 无显式字段密钥的源码 dev 自动生成并持久保存随机 hex64 密钥;使用 `0600` 文件及完整文件的原子发布,多进程首次启动共享同一结果。重启复用,显式密钥优先,坏密钥报错且不覆盖。生产构建即使 `NODE_ENV=development` 仍要求显式密钥。 +- 跨进程 dev 首次启动测试先因缺密钥失败,实现后通过。22 项启动边界回归通过,覆盖并发启动、重启解密、权限、显式密钥优先、无效密钥保留、非 dev 环境和真实生产产物拒绝回退;另 1 项真实 PostgreSQL 演练验证无配置密钥时正式迁移成功、凭据仍为 V2 且重跑不变。 +- 完整 UTC/3 workers 回归退出 0:434 个文件通过、3 个跳过;3594 项通过、11 项跳过。继续显式排除既有 `admin-agent.e2e-spec.ts` 缺 DI 夹具。Core 构建、Core/E2E 类型检查、API guards、定向 ESLint/Prettier 和 diff 检查通过。 +- 临时工作目录的迁移演练显式提供项目依赖解析、禁用浏览器依赖预打包并指定迁移目录;未为测试修改业务迁移。官方文档 build/check/links/pangu/lint 和 3 项测试通过,llms 已包含本地密钥生命周期与生产边界;保留既有 1 hint、1 死链。开发库转生产或运行独立检查须显式提供原本地密钥,JWT 校验保持不变。 diff --git a/docs/security-secret-storage.md b/docs/security-secret-storage.md index bb49e6767a6..e39697cc109 100644 --- a/docs/security-secret-storage.md +++ b/docs/security-secret-storage.md @@ -4,7 +4,7 @@ Core protects recoverable credentials with authenticated encryption and stores A ## Required configuration (unreleased branch) -Field encryption is the only runtime mode on this branch. Supply an explicit +Field encryption is the only runtime mode on this branch. Production builds require an explicit 64-character hexadecimal `MX_ENCRYPT_KEY` or `ENCRYPT_KEY`, and a stable `JWT_SECRET` (alias `JWTSECRET`) of at least 32 characters. There is no machine-ID fallback or short-key derivation. Keep existing effective secrets; @@ -12,6 +12,17 @@ only an instance that has never encrypted data may generate its first random field key with `openssl rand -hex 32`. Persist secrets before starting and share the same values between app and migration services. +Source development startup may omit the field key. It creates a random hex64 +key in `tmp/dev-encryption.key` relative to the working directory, with mode +`0600`, and reuses it across restarts and concurrent workers. An explicit +CLI/YAML/environment key always takes precedence. Preserve the local key with +the development database; an invalid local or explicit key fails instead of +being replaced. AES-256-GCM writes and the JWT requirements stay the same. +Release builds never use this fallback, even with `NODE_ENV=development`. +Standalone upgrade diagnostics still require the effective key in environment +configuration; copy the local key into the secret store when promoting a +development database to production. + Remove `ENCRYPT_ENABLE`, `MX_ENCRYPT_ENABLE` and `MX_SECRET_STORAGE_V2`. Historical true values are accepted with a deprecation warning; false or other unsupported values fail before business startup. These flags no longer diff --git a/docs/v18-preparation.md b/docs/v18-preparation.md index 0f9afbdcc21..37f61e252e8 100644 --- a/docs/v18-preparation.md +++ b/docs/v18-preparation.md @@ -1,6 +1,6 @@ # v18 preparation (unreleased) -This unreleased branch now requires field encryption, an explicit hex64 key +This unreleased branch now requires field encryption, an explicit hex64 key in production and a JWT of at least 32 characters. Plaintext/ECB writes, encryption-disable flags, algorithm selection and machine-ID fallback have been removed. Field writes are fixed to AES-256-GCM. Canonical configuration migration and strict configuration PATCH validation @@ -9,6 +9,15 @@ No minimum released preparation tag is established yet. Obtain this build without starting business replicas; do not publish or migrate production as part of reviewing this branch. +Source development startup can omit the field key: a random private key is +persisted in `tmp/dev-encryption.key` and reused on restart. Explicit keys +take precedence; corrupt local keys are never replaced automatically. Preserve +that file with the development database. Production builds and independent +upgrade diagnostics still require an explicit effective field key; even +`NODE_ENV=development` cannot enable this fallback in a release build. Supply +the same local key explicitly before promoting a development database or +checking it with the standalone CLI. JWT requirements are unchanged. + ## Preconditions and checks Save database/files snapshots, the original field key, original JWT, and any