test: pin real offline producer-to-receiver contracts

.github/workflows/producer-receiver-contract.yml

@@ -0,0 +1,54 @@
+name: Offline producer receiver contracts
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  pinned-producer-roundtrip:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - name: Checkout exact IBKR projection source
+        uses: actions/checkout@v6
+        with:
+          repository: QuantStrategyLab/InteractiveBrokersPlatform
+          ref: bd5c1a12d9cd0df8dbd88c1eb8b058b7b849990f
+          path: .contract-sources/ibkr
+          persist-credentials: false
+          sparse-checkout: scripts/publish_account_facts_from_report.py
+          sparse-checkout-cone-mode: false
+      - name: Checkout exact Schwab projection source
+        uses: actions/checkout@v6
+        with:
+          repository: QuantStrategyLab/CharlesSchwabPlatform
+          ref: 5b563ad3609eefd9cf431ead82d0f30c586645db
+          path: .contract-sources/schwab
+          persist-credentials: false
+          sparse-checkout: scripts/publish_account_facts_from_reports.py
+          sparse-checkout-cone-mode: false
+      - uses: actions/setup-node@v6
+        with:
+          node-version: "22"
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      - name: Install existing receiver test runtime
+        run: npm ci --prefix web/strategy-switch-console
+      - name: Verify deterministic synthetic projection regeneration
+        run: |
+          python3 tests/generate_producer_contract.py --ibkr .contract-sources/ibkr --schwab .contract-sources/schwab > "$RUNNER_TEMP/contract-a.json"
+          python3 tests/generate_producer_contract.py --ibkr .contract-sources/ibkr --schwab .contract-sources/schwab > "$RUNNER_TEMP/contract-b.json"
+          cmp "$RUNNER_TEMP/contract-a.json" "$RUNNER_TEMP/contract-b.json"
+      - name: Exercise offline authenticated receiver and durable history
+        env:
+          CLOUDFLARE_CF_FETCH_ENABLED: "false"
+          npm_config_update_notifier: "false"
+        run: node tests/producer_receiver_contract_validation.mjs .contract-sources/ibkr .contract-sources/schwab

docs/producer-receiver-contract.md

@@ -0,0 +1,108 @@
+# Offline producer → receiver contracts
+
+This test-only gate executes the **real upstream projection and strip functions**,
+then sends their unchanged synthetic output through QRS's authenticated sync
+handler, local Miniflare SQLite persistence, read model and balance history.
+It does not run either publisher CLI, use credentials, contact a broker/cloud,
+change publisher stripping, deploy, merge, or establish runtime adoption.
+
+## Pinned evidence and current result
+
+- [IBKR bd5c1a12d9cd0df8dbd88c1eb8b058b7b849990f](https://github.com/QuantStrategyLab/InteractiveBrokersPlatform/blob/bd5c1a12d9cd0df8dbd88c1eb8b058b7b849990f/scripts/publish_account_facts_from_report.py):
+  `project_ibkr_account_facts_history` plus `strip_archive_only_keys`.
+  The stripped contract is accepted. The pre-strip archive contract is accepted
+  with `stocks_only`; this is **future publication compatibility**, not enabled
+  publication. A newer real stripped projection clears latest holdings while
+  previous-day holdings remain in durable history. Null net assets remain gaps.
+- [Schwab 5b563ad3609eefd9cf431ead82d0f30c586645db](https://github.com/QuantStrategyLab/CharlesSchwabPlatform/blob/5b563ad3609eefd9cf431ead82d0f30c586645db/scripts/publish_account_facts_from_reports.py):
+  `project_schwab_account_facts_history` plus `strip_archive_only_keys`.
+  The stripped contract is accepted, balances/history remain readable, and absent
+  cash stays absent. **The archive contract is incompatible**: each projected
+  holding has `currency_source: "owner_confirmed"`, while QRS's exact-key position
+  validator rejects that extra key with HTTP 400 `invalid_account_facts_positions`.
+  The negative test preserves this known blocker and proves rejection does not
+  overwrite accepted data. A green CI job does **not** mean Schwab holdings can
+  be enabled. The test must change with any separately approved contract repair.
+
+The receiver baseline is QRS
+[6070efb73cde35fd5ffafd5d3da196e04b462e42](https://github.com/QuantStrategyLab/QuantRuntimeSettings/commit/6070efb73cde35fd5ffafd5d3da196e04b462e42).
+It is exercised from the current checkout so future receiver changes remain
+visible against these pinned producer contracts. This complements the existing
+`account_facts_validation.mjs` malformed-payload and rollback coverage, rather
+than replacing it or duplicating the full validator matrix.
+
+### Schwab decision required before future publication
+
+A separate authorized design decision must choose either a publisher boundary
+mapping that deliberately removes position currency provenance, or explicit
+receiver support that validates and preserves `owner_confirmed` semantics.
+Do not silently strip the field in this test or imply USD was broker-confirmed.
+Neither change is included here. Keep current stripping enabled. See
+[holdings rollout and rollback](account-facts-holdings-rollout.md) for the separate
+runtime-evidence, publication and receiver-downgrade gates. This suite does not
+claim an actual pre-holdings receiver downgrade was executed.
+
+## Reproduce
+
+The two report templates under `tests/fixtures/producer_contract/` contain only
+invented account identifiers, balances and holdings. Service/profile strings that
+upstream treats as constants are contract identifiers, not evidence of access.
+The source manifests record repository, exact Git revision, path and SHA-256 of
+the imported file. No production source is vendored and no runtime dependency or
+runtime configuration is added. Regeneration fails on a different checkout HEAD
+or changed projection bytes. Deliberate pin updates require reviewing both source
+and digest changes, then rerunning this test; `main` is never used in CI.
+
+After obtaining these two repositories and checking out the exact revisions
+above (network setup only), run from QRS:
+
+```sh
+npm ci --prefix web/strategy-switch-console
+python3 tests/generate_producer_contract.py --ibkr ../InteractiveBrokersPlatform --schwab ../CharlesSchwabPlatform > /tmp/contract-a.json
+python3 tests/generate_producer_contract.py --ibkr ../InteractiveBrokersPlatform --schwab ../CharlesSchwabPlatform > /tmp/contract-b.json
+cmp /tmp/contract-a.json /tmp/contract-b.json
+CLOUDFLARE_CF_FETCH_ENABLED=false node tests/producer_receiver_contract_validation.mjs ../InteractiveBrokersPlatform ../CharlesSchwabPlatform
+node tests/account_facts_validation.mjs
+python3 python/scripts/runtime_settings.py validate
+python3 -m unittest discover -s python/tests -v
+git diff --check
+```
+
+The generator's default `--now` is fixed for reproducibility. The roundtrip test
+passes the current UTC time **into synthetic report generation before projection**
+to exercise production freshness gates without rewriting projected output.
+All report times and report-path provenance are generated consistently. Previous
+observations are within the receiver's 36-hour window and on older UTC days.
+
+Both imported scripts have stdlib-only imports and a guarded CLI entry point.
+Before importing them, the generator clears the environment and installs
+fail-on-call guards for sockets, HTTP and subprocess creation. It checks that no
+guard was called, even if upstream catches a raised error. Receiver outbound
+fetches are likewise blocked and counted. Only Miniflare's local transport is
+used. CI fetch/install steps require network; projection and roundtrip assertions
+are offline. Source checkout failures fail CI; they are never reported as skipped
+or successful coverage. No generated account payload is uploaded as an artifact.
+
+## 中文说明
+
+这项测试仅使用合成报表，执行固定 SHA 的真实投影函数和字段剥离函数，再经过
+接收端鉴权、本地 SQLite、读模型和历史查询。不会运行发布器 CLI、访问实盘、
+使用凭据、改变发布开关、合并或部署。
+
+- IBKR：当前剥离载荷可接收；归档扩展载荷的 `stocks_only` 也可接收，但这只是
+  未来发布契约兼容性。更新的剥离载荷清除最新持仓，旧日期证据保留，未知资产
+  仍为历史缺口而不是零。
+- Schwab：当前剥离载荷可接收；扩展载荷的每行有
+  `currency_source: owner_confirmed`，接收端精确键校验以 HTTP 400 拒绝。
+  测试明确验证这一未修复阻塞以及原有余额和历史不受影响。CI 通过不代表可以
+  开启 Schwab 持仓发布，不能修改测试数据掩盖不兼容。
+
+未来需要单独批准：在发布边界明确舍弃货币来源字段，或让接收端显式校验并保留
+该来源语义。本次不实施任一方案。继续保持剥离，运行时证据、发布与降级仍遵守
+独立发布门槛。本测试未执行旧接收端降级。
+
+源清单记录仓库、完整 SHA、文件路径和内容摘要；生成器检查版本及字节一致，
+不会复制生产代码或增加运行依赖。默认固定时间用于重复生成比对，集成测试在
+投影前生成当前时间的合成报表以验证时效。导入前阻断网络和子进程并清空环境，
+接收端外部请求同样阻断；CI 仅检出与依赖安装需要网络。源码取不到就失败，
+不会把未执行说成通过，也不上传生成的账户载荷。

tests/fixtures/producer_contract/ibkr-report.json

@@ -0,0 +1,44 @@
+{
+  "schema_version": "runtime_report.v1",
+  "platform": "interactive_brokers",
+  "deploy_target": "cloud_run",
+  "project_id": "example-project",
+  "service_name": "synthetic-service",
+  "account_scope": "live-synthetic",
+  "runtime_target": {
+    "account_selector": [
+      "U00000001"
+    ],
+    "deployment_selector": "synthetic-deployment"
+  },
+  "diagnostics": {
+    "runtime_revision": "synthetic-00001-abc"
+  },
+  "summary": {
+    "account_facts": {
+      "schema_version": "ibkr_account_snapshot.v1",
+      "account_ids": [
+        "U00000001"
+      ],
+      "currency": "USD",
+      "net_assets": null,
+      "cash": [
+        {
+          "currency": "USD",
+          "cash_balance": "12.50",
+          "source_tag": "CashBalance"
+        }
+      ],
+      "broker_reported_positions": [
+        {
+          "symbol": "SOXL",
+          "quantity": "2",
+          "market_value": "25.50",
+          "currency": "USD",
+          "avg_cost": "10"
+        }
+      ],
+      "broker_reported_positions_scope": "stocks_only"
+    }
+  }
+}

tests/fixtures/producer_contract/ibkr-source.json

@@ -0,0 +1,6 @@
+{
+  "repository": "QuantStrategyLab/InteractiveBrokersPlatform",
+  "revision": "bd5c1a12d9cd0df8dbd88c1eb8b058b7b849990f",
+  "path": "scripts/publish_account_facts_from_report.py",
+  "sha256": "37464f92228caa795eae6e71f7acc48f8a1e46fbd8bb21bbc3c707e2afdb58f1"
+}

tests/fixtures/producer_contract/schwab-report.json

@@ -0,0 +1,38 @@
+{
+  "schema_version": "runtime_report.v1",
+  "platform": "charles_schwab",
+  "project_id": "charlesschwabquant",
+  "service_name": "synthetic-service",
+  "strategy_profile": "soxl_soxx_trend_income",
+  "account_scope": null,
+  "status": "ok",
+  "errors": [],
+  "runtime_target": {
+    "strategy_profile": "soxl_soxx_trend_income",
+    "account_scope": "live",
+    "account_selector": [
+      "live"
+    ]
+  },
+  "diagnostics": {
+    "runtime_revision": "synthetic-00001-abc"
+  },
+  "summary": {
+    "account_observation": {
+      "account_hash": "synthetic-account-hash",
+      "currency": null,
+      "net_assets": "123.45",
+      "net_assets_source": "liquidationValue",
+      "net_assets_currency": "USD",
+      "net_assets_currency_source": "owner_confirmed",
+      "broker_reported_positions": [
+        {
+          "symbol": "SOXL",
+          "quantity": "2",
+          "market_value": "25.50"
+        }
+      ],
+      "broker_reported_positions_scope": "strategy_symbols_only"
+    }
+  }
+}

tests/fixtures/producer_contract/schwab-source.json

@@ -0,0 +1,6 @@
+{
+  "repository": "QuantStrategyLab/CharlesSchwabPlatform",
+  "revision": "5b563ad3609eefd9cf431ead82d0f30c586645db",
+  "path": "scripts/publish_account_facts_from_reports.py",
+  "sha256": "a32adbd35762037c5bc2954cb80d741d928d3861336c6901e9185ba9798ca350"
+}

tests/generate_producer_contract.py

@@ -0,0 +1,114 @@
+"""Run exact upstream projections offline; stdout contains synthetic data only."""
+from __future__ import annotations
+
+import argparse
+from contextlib import ExitStack
+from copy import deepcopy
+from datetime import datetime, timedelta, timezone
+import hashlib
+import importlib.util
+import json
+from pathlib import Path
+import subprocess
+from unittest.mock import patch
+
+FIXTURES = Path(__file__).parent / "fixtures" / "producer_contract"
+
+
+def load_source(platform, root):
+    provenance = json.loads((FIXTURES / f"{platform}-source.json").read_text())
+    head = subprocess.check_output(["git", "-C", str(root), "rev-parse", "HEAD"], text=True).strip()
+    if head != provenance["revision"]:
+        raise ValueError(f"{platform}: checkout is not the pinned revision")
+    source = root / provenance["path"]
+    if hashlib.sha256(source.read_bytes()).hexdigest() != provenance["sha256"]:
+        raise ValueError(f"{platform}: projection source differs from pinned bytes")
+    return source, provenance
+
+
+def iso(value):
+    return value.isoformat().replace("+00:00", "Z")
+
+
+def generate(platform, source, now):
+    spec = importlib.util.spec_from_file_location(f"contract_{platform}", source)
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    template = json.loads((FIXTURES / f"{platform}-report.json").read_text())
+    slot = "account_facts" if platform == "ibkr" else "account_observation"
+
+    def project(report, observed):
+        report["started_at"] = iso(observed - timedelta(seconds=1))
+        report["finished_at"] = iso(observed + timedelta(seconds=1))
+        report["run_id"] = (observed - timedelta(seconds=1)).strftime("%Y%m%dT%H%M%SZ")
+        report["summary"][slot]["observed_at"] = iso(observed)
+        if platform == "ibkr":
+            return module.project_ibkr_account_facts_history(
+                report, target_id="synthetic-target",
+                expected_report_prefix="gs://example-bucket/ibkr",
+                source_report_uri="gs://example-bucket/ibkr/synthetic.json",
+                expected_project_id="example-project", expected_service_name="synthetic-service",
+                expected_runtime_revision="synthetic-00001-abc",
+                expected_account_scope="live-synthetic", expected_account_selector=["U00000001"],
+                expected_deployment_selector="synthetic-deployment",
+            )
+        prefix = "gs://example-bucket/execution-reports/charles_schwab/soxl_soxx_trend_income/"
+        return module.project_schwab_account_facts_history(
+            report, source_report_uri=prefix + report["started_at"][:7] + "/" + report["run_id"] + ".json",
+            report_prefix=prefix, expected_service_name="synthetic-service",
+            expected_runtime_revision="synthetic-00001-abc", expected_target_id="synthetic-target", now=now,
+        )
+
+    expanded = project(deepcopy(template), now - timedelta(hours=24, minutes=1))
+    current = project(deepcopy(template), now - timedelta(minutes=1))
+    assert "broker_reported_positions" in expanded, expanded
+    assert "broker_reported_positions" in current, current
+    result = {"archive_future_contract": expanded,
+              "published_current_contract": module.strip_archive_only_keys(
+                  project(deepcopy(template), now - timedelta(hours=25))),
+              "published_newer_contract": module.strip_archive_only_keys(current)}
+    # Producer omission is not a fabricated zero-holdings statement.
+    for label, change in (("missing", None), ("empty", []), ("invalid", [{"symbol": "SOXL", "quantity": "NaN"}])):
+        report = deepcopy(template)
+        if change is None:
+            del report["summary"][slot]["broker_reported_positions"]
+            del report["summary"][slot]["broker_reported_positions_scope"]
+        else:
+            report["summary"][slot]["broker_reported_positions"] = change
+        body = project(report, now - timedelta(minutes=1))
+        assert "schema_version" in body, body
+        assert "broker_reported_positions" not in body
+        assert "broker_reported_positions_scope" not in body
+        result[label] = body
+    wrong_identity = deepcopy(template)
+    wrong_identity["service_name"] = "wrong-service"
+    rejected = project(wrong_identity, now - timedelta(minutes=1))
+    assert rejected["status"] == "skipped", rejected
+    result["producer_identity_rejection"] = rejected
+    return result
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--ibkr", type=Path, required=True)
+    parser.add_argument("--schwab", type=Path, required=True)
+    parser.add_argument("--now", default="2026-10-01T12:00:00Z")
+    args = parser.parse_args()
+    now = datetime.fromisoformat(args.now.replace("Z", "+00:00")).astimezone(timezone.utc).replace(microsecond=0)
+    sources = {key: load_source(key, getattr(args, key)) for key in ("ibkr", "schwab")}
+    # Install before importing upstream modules. No broker SDK, cloud CLI, HTTP,
+    # subprocess, credentials or IO mocks that manufacture projection output.
+    with ExitStack() as stack:
+        stack.enter_context(patch.dict("os.environ", {}, clear=True))
+        guards = [stack.enter_context(patch(target, side_effect=AssertionError("offline boundary attempted")))
+                  for target in ("socket.socket.connect", "socket.create_connection", "subprocess.Popen",
+                                 "urllib.request.urlopen", "urllib.request.build_opener")]
+        output = {key: {"provenance": provenance, **generate(key, source, now)}
+                  for key, (source, provenance) in sources.items()}
+        for guard in guards:
+            guard.assert_not_called()
+    print(json.dumps(output, sort_keys=True, indent=2))
+
+
+if __name__ == "__main__":
+    main()