diff --git a/docs/producer-receiver-contract.md b/docs/producer-receiver-contract.md index 6edc9e34..77753e13 100644 --- a/docs/producer-receiver-contract.md +++ b/docs/producer-receiver-contract.md @@ -1,6 +1,6 @@ # Offline producer → receiver contracts -This test-only gate executes the **real upstream projection and strip functions**, +This offline gate executes the **real upstream projection and strip functions**, then sends their unchanged synthetic output through QRS's authenticated sync handler, local Miniflare SQLite persistence, read model and balance history. It does not run either publisher CLI, use credentials, contact a broker/cloud, @@ -17,12 +17,12 @@ change publisher stripping, deploy, merge, or establish runtime adoption. - [Schwab 5b563ad3609eefd9cf431ead82d0f30c586645db](https://github.com/QuantStrategyLab/CharlesSchwabPlatform/blob/5b563ad3609eefd9cf431ead82d0f30c586645db/scripts/publish_account_facts_from_reports.py): `project_schwab_account_facts_history` plus `strip_archive_only_keys`. The stripped contract is accepted, balances/history remain readable, and absent - cash stays absent. **The archive contract is incompatible**: each projected - holding has `currency_source: "owner_confirmed"`, while QRS's exact-key position - validator rejects that extra key with HTTP 400 `invalid_account_facts_positions`. - The negative test preserves this known blocker and proves rejection does not - overwrite accepted data. A green CI job does **not** mean Schwab holdings can - be enabled. The test must change with any separately approved contract repair. + cash stays absent. The unchanged archive contract is now accepted: each + projected holding carries `currency_source: "owner_confirmed"`, explicitly + validated and preserved by QRS in latest readback and durable daily history. + Unknown keys and malformed provenance are rejected without changing accepted + data. A newer stripped payload clears latest holdings and retains older evidence. + This is **future publication compatibility**, not enabled publication. The receiver baseline is QRS [6070efb73cde35fd5ffafd5d3da196e04b462e42](https://github.com/QuantStrategyLab/QuantRuntimeSettings/commit/6070efb73cde35fd5ffafd5d3da196e04b462e42). @@ -31,16 +31,30 @@ visible against these pinned producer contracts. This complements the existing `account_facts_validation.mjs` malformed-payload and rollback coverage, rather than replacing it or duplicating the full validator matrix. -### Schwab decision required before future publication - -A separate authorized design decision must choose either a publisher boundary -mapping that deliberately removes position currency provenance, or explicit -receiver support that validates and preserves `owner_confirmed` semantics. -Do not silently strip the field in this test or imply USD was broker-confirmed. -Neither change is included here. Keep current stripping enabled. See -[holdings rollout and rollback](account-facts-holdings-rollout.md) for the separate -runtime-evidence, publication and receiver-downgrade gates. This suite does not -claim an actual pre-holdings receiver downgrade was executed. +### Schwab provenance compatibility decision + +QRS accepts the optional holding field `currency_source` only for Schwab, only +with the exact value `owner_confirmed`, and only alongside an explicit `USD` +currency. It does not infer a currency or add provenance to legacy rows lacking +this field. All other unknown row keys remain invalid, and IBKR's contract is +unchanged. Optional `avg_cost` remains independently decimal-validated. + +The pinned producer's `_project_positions` accepts runtime currency `None` or +`USD` and explicitly marks the resulting USD as owner-confirmed. A boundary +projection that drops this field would discard that distinction. Preserving it +requires only a narrow receiver normalization/type extension, no producer edits, +new dependencies, storage migration or changes to broker execution. Existing +Schwab holdings without provenance remain readable for backward compatibility. + +Keep current publisher stripping enabled. Receiver release needs separate +approval (merging console runtime changes to `main` triggers automatic deploy), +followed by deployed-revision/read-only health evidence and actual restricted +runtime-report evidence, before any separately approved publisher enablement. +See [holdings rollout and rollback](account-facts-holdings-rollout.md). After new +rows have been stored, a receiver downgrade that rejects `currency_source` can +also reject retained history; retain this compatibility unless a separate +migration/downgrade plan is approved. This suite does not establish production +adoption or execute an actual receiver downgrade. ## Reproduce @@ -92,14 +106,19 @@ or successful coverage. No generated account payload is uploaded as an artifact. - IBKR:当前剥离载荷可接收;归档扩展载荷的 `stocks_only` 也可接收,但这只是 未来发布契约兼容性。更新的剥离载荷清除最新持仓,旧日期证据保留,未知资产 仍为历史缺口而不是零。 -- Schwab:当前剥离载荷可接收;扩展载荷的每行有 - `currency_source: owner_confirmed`,接收端精确键校验以 HTTP 400 拒绝。 - 测试明确验证这一未修复阻塞以及原有余额和历史不受影响。CI 通过不代表可以 - 开启 Schwab 持仓发布,不能修改测试数据掩盖不兼容。 - -未来需要单独批准:在发布边界明确舍弃货币来源字段,或让接收端显式校验并保留 -该来源语义。本次不实施任一方案。继续保持剥离,运行时证据、发布与降级仍遵守 -独立发布门槛。本测试未执行旧接收端降级。 +- Schwab:当前剥离载荷与未修改的扩展归档载荷均可接收;每行显式 + `currency_source: owner_confirmed` 会保留在最新读模型及每日历史。未知字段、 + 非法来源或来源搭配非 USD 货币均拒绝,且不覆盖已接收的数据。新的剥离载荷 + 清除最新持仓,但保留旧日期证据。CI 通过不代表已开启持仓发送。 + +采用接收端窄范围兼容:该可选字段仅允许 Schwab 的显式 USD 与 +`owner_confirmed`,不补货币、不推断来源;无来源的旧行仍可读,IBKR 不变。 +发布器将运行时未知货币投影为所有者确认的 USD,删除字段会丢失这一区别, +因此显式校验并保留;不改发布器、固定版本、存储结构或部署流程。 +继续保持剥离。接收端合并会自动部署,需另行批准;再核验已部署版本、只读健康 +和受限真实运行报告,最后单独批准发布器启用。写入带来源的历史后,直接降级到 +拒绝该字段的接收端会影响历史读取,需独立兼容或迁移方案。合成测试不证明生产 +采用,也未执行真实降级。 源清单记录仓库、完整 SHA、文件路径和内容摘要;生成器检查版本及字节一致, 不会复制生产代码或增加运行依赖。默认固定时间用于重复生成比对,集成测试在 diff --git a/tests/account_facts_validation.mjs b/tests/account_facts_validation.mjs index a613034c..2f7b10c4 100644 --- a/tests/account_facts_validation.mjs +++ b/tests/account_facts_validation.mjs @@ -1280,6 +1280,34 @@ const schwabWithPositions = normalizeAccountFactsHistoryPayload(schwabHistory({ broker_reported_positions_scope: "strategy_symbols_only", })); assert.equal(schwabWithPositions.broker_reported_positions_scope, "strategy_symbols_only"); +assert.equal(Object.hasOwn(schwabWithPositions.broker_reported_positions[0], "currency_source"), false, + "legacy holdings do not acquire inferred provenance"); +const schwabPosition = schwabWithPositions.broker_reported_positions[0]; +for (const avgCost of [{}, { avg_cost: "90" }]) { + const row = { ...schwabPosition, ...avgCost, currency_source: "owner_confirmed" }; + const normalized = normalizeAccountFactsHistoryPayload(schwabHistory({ + broker_reported_positions: [row], broker_reported_positions_scope: "strategy_symbols_only", + })); + assert.deepEqual(normalized.broker_reported_positions, [row]); +} +for (const patch of [ + ...[null, "", "broker", "owner_confirmed_wrongly", 1, true, {}, []].map(currency_source => ({ currency_source })), + { currency_source: "owner_confirmed", currency: "HKD" }, + { currency_source: "owner_confirmed", currency: null }, + { currency_source: "owner_confirmed", currency: undefined }, + { currency_source: "owner_confirmed", unexpected: "field" }, + { currency_source: "owner_confirmed", avg_cost: "invalid" }, +]) { + assert.throws(() => normalizeAccountFactsHistoryPayload(schwabHistory({ + broker_reported_positions: [{ ...schwabPosition, ...patch }], + broker_reported_positions_scope: "strategy_symbols_only", + })), /invalid_account_facts_positions/); +} +assert.throws(() => normalizeAccountFactsHistoryPayload(ibkrHistory({ + broker_reported_positions: [{ ...ibkrPositions[0], currency_source: "owner_confirmed" }], + broker_reported_positions_scope: "stocks_only", +})), /invalid_account_facts_positions/); + assert.throws(() => normalizeAccountFactsHistoryPayload(schwabHistory({ broker_reported_positions: [{ symbol: "TQQQ", quantity: "10", market_value: "900", currency: "USD" }], broker_reported_positions_scope: "stocks_only", diff --git a/tests/producer_receiver_contract_validation.mjs b/tests/producer_receiver_contract_validation.mjs index 72baf743..7b0796f4 100644 --- a/tests/producer_receiver_contract_validation.mjs +++ b/tests/producer_receiver_contract_validation.mjs @@ -85,24 +85,30 @@ try { // Pre-strip archive output is a FUTURE publication contract, never evidence of activation. assert.equal(extended.broker_reported_positions_scope, expectedScope); const stub = namespace.get(namespace.idFromName("runtime-instances")); - const compatibleArchive = platform === "ibkr"; - if (compatibleArchive) { - await accept(extended); - assert.deepEqual((await read()).positions, extended.broker_reported_positions); - assert.equal((await read()).positions_scope, expectedScope); - assert.equal((await accept(extended)).unchanged, true); - assert.equal((await read()).balances[0].net_assets, null); - } else { - // Real incompatibility: the producer adds currency_source, but the receiver - // has exact position keys. Do not sanitize producer output to make this pass. + await accept(extended); + assert.deepEqual((await read()).positions, extended.broker_reported_positions); + assert.equal((await read()).positions_scope, expectedScope); + assert.equal((await accept(extended)).unchanged, true); + if (platform === "ibkr") assert.equal((await read()).balances[0].net_assets, null); + else { assert.equal(extended.broker_reported_positions[0].currency_source, "owner_confirmed"); - const rejected = await post(extended); + assert.deepEqual((await read()).cash, []); + } + // Reject malformed provenance and unknown keys without contaminating accepted data. + const invalidRows = [ + { ...extended.broker_reported_positions[0], unexpected: "field" }, + { ...extended.broker_reported_positions[0], currency_source: "broker" }, + { ...extended.broker_reported_positions[0], currency_source: null }, + ...(platform === "schwab" ? [{ ...extended.broker_reported_positions[0], currency: "HKD" }] : + [{ ...extended.broker_reported_positions[0], currency_source: "owner_confirmed" }]), + ]; + for (const row of invalidRows) { + const rejected = await post({ ...extended, broker_reported_positions: [row] }); assert.equal(rejected.status, 400); assert.equal((await rejected.json()).error, "invalid_account_facts_positions"); - assert.equal(Object.hasOwn(await read(), "positions"), false); - assert.deepEqual((await read()).cash, []); + assert.deepEqual((await read()).positions, extended.broker_reported_positions); } - const valid = compatibleArchive ? extended : published; + const valid = extended; for (const [body, status] of [ [{ ...valid, source_binding: { ...valid.source_binding, id: "0".repeat(64) } }, 409], [{ ...valid, target_id: "wrong-target" }, 409], @@ -111,8 +117,7 @@ try { [{ ...valid, account_key: "caller-assigned" }, 400], ]) { assert.equal((await post(body)).status, status); - if (compatibleArchive) assert.deepEqual((await read()).positions, extended.broker_reported_positions); - else assert.equal(Object.hasOwn(await read(), "positions"), false); + assert.deepEqual((await read()).positions, extended.broker_reported_positions); } assert.equal((await post(extended, "wrong-token")).status, 401); // Real newer stripped output removes latest holdings without erasing previous-day evidence. @@ -133,14 +138,9 @@ try { assert.equal(response.status, 200, await response.clone().text()); const history = await response.json(); assert.ok(history.days.length >= 2); - if (compatibleArchive) { - const old = history.days.find(day => day.payload.observation_date === extended.observation_date).payload; - assert.deepEqual(old.broker_reported_positions, extended.broker_reported_positions); - assert.equal(old.broker_reported_positions_scope, expectedScope); - } else { - assert.ok(history.days.every(day => !Object.hasOwn(day.payload, "broker_reported_positions")), - "rejected Schwab archive output must not contaminate durable history"); - } + const old = history.days.find(day => day.payload.observation_date === extended.observation_date).payload; + assert.deepEqual(old.broker_reported_positions, extended.broker_reported_positions); + assert.equal(old.broker_reported_positions_scope, expectedScope); const series = await get(`/api/account-facts/history?platform=${platform}&account_key=${account.key}¤cy=USD`); if (platform === "ibkr") { assert.deepEqual(series.series.points, []); @@ -152,7 +152,7 @@ try { } } assert.equal(outboundCalls, 0, "neither receiver nor projections may make external HTTP calls"); - console.log("Pinned contracts passed: stripped IBKR/Schwab accepted; IBKR archive accepted; Schwab archive BLOCKED by currency_source (synthetic/offline)."); + console.log("Pinned contracts passed: stripped IBKR/Schwab accepted; IBKR/Schwab archive accepted with provenance preserved (synthetic/offline)."); } finally { await mf.dispose(); globalThis.fetch = nativeFetch; diff --git a/web/strategy-switch-console/account_facts.js b/web/strategy-switch-console/account_facts.js index 1f779f8b..361896bf 100644 --- a/web/strategy-switch-console/account_facts.js +++ b/web/strategy-switch-console/account_facts.js @@ -130,16 +130,23 @@ function normalizeBrokerReportedPositions(platform, raw) { if (!Array.isArray(rows) || rows.length === 0 || rows.length > MAX_POSITIONS) reject("invalid_account_facts_positions"); const seen = new Set(); const positions = rows.map((item) => { - const keys = Object.prototype.hasOwnProperty.call(item || {}, "avg_cost") ? [...POSITION_KEYS, "avg_cost"] : POSITION_KEYS; + const hasAvgCost = Object.prototype.hasOwnProperty.call(item || {}, "avg_cost"); + const hasCurrencySource = Object.prototype.hasOwnProperty.call(item || {}, "currency_source"); + const keys = [...POSITION_KEYS, ...(hasAvgCost ? ["avg_cost"] : []), + ...(hasCurrencySource ? ["currency_source"] : [])]; if (!exactKeys(item, keys) || typeof item.symbol !== "string" || !POSITION_SYMBOL_RE.test(item.symbol) || seen.has(item.symbol) || typeof item.currency !== "string" || !CURRENCY_RE.test(item.currency) - || keys.slice(1).some((key) => key !== "currency" && (typeof item[key] !== "string" || !DECIMAL_RE.test(item[key])))) { + || (hasCurrencySource && (platform !== "schwab" || item.currency_source !== "owner_confirmed" || item.currency !== "USD")) + || ["quantity", "market_value", ...(hasAvgCost ? ["avg_cost"] : [])] + .some((key) => typeof item[key] !== "string" || !DECIMAL_RE.test(item[key]))) { reject("invalid_account_facts_positions"); } seen.add(item.symbol); const row = { symbol: item.symbol, quantity: item.quantity, market_value: item.market_value, currency: item.currency }; - if (keys.length > POSITION_KEYS.length) row.avg_cost = item.avg_cost; + if (hasAvgCost) row.avg_cost = item.avg_cost; + // Preserve the producer's explicit owner-confirmed USD provenance; never infer it. + if (hasCurrencySource) row.currency_source = item.currency_source; return row; }); return { positions, scope }; diff --git a/web/strategy-switch-console/frontend/src/types.ts b/web/strategy-switch-console/frontend/src/types.ts index 3e872768..5064259d 100644 --- a/web/strategy-switch-console/frontend/src/types.ts +++ b/web/strategy-switch-console/frontend/src/types.ts @@ -108,6 +108,7 @@ export type AccountFactsPosition = { market_value: string; currency: string; avg_cost?: string; + currency_source?: "owner_confirmed"; }; export type AccountFactsTotals = {