diff --git a/docs/market-data-access-layer-design.md b/docs/market-data-access-layer-design.md new file mode 100644 index 0000000..f65c919 --- /dev/null +++ b/docs/market-data-access-layer-design.md @@ -0,0 +1,111 @@ +# Market-data access-layer design + +Status: proposal. Not yet adopted by any consumer. + +The [standard](market-data-standard.md) defines the semantics; the [inventory](market-data-inventory.md) lists the identified consumers. This document describes the shared contract, configuration and migration order for a future implementation. No consumer implementation, dependency upgrade, runtime change or deployment is included. + +## 1. Smallest shared contract in QPK + +Extend the existing `src/quant_platform_kit/data/` and keep network acquisition in the current pipelines and broker adapter modules. Do not create a market-data service, scheduler, database, broker registry or second research orchestrator. + +Reuse these existing boundaries: + +- `common/ports.py`: `DecisionDataArtifactPort` loads immutable, digest-bound decision inputs; `ExecutionQuotePort` handles short-lived execution quotes. The legacy mixed `MarketDataPort` remains a migration boundary. +- `data/manifest.py`, `data/version.py`, `data/research_input.py`, `data/decision_data_binding.py`, `data/decision_price_series_artifact.py`: extend the existing versioned manifests and binding validation with semantic metadata. Preserve existing canonicalizations and frozen roots. +- `data/multisource_assurance.py`: preserve independently acquired source observations and current VERIFIED/DEGRADED/PARKED behaviour. Selecting a fallback must not bypass a candidate's requirement for two agreeing sources. +- `ibkr/market_data.py`, `schwab/market_data.py`, `longbridge/market_data.py`, `binance/market_data.py`: retain provider SDK differences; promote only the common result contract. Indicator calculations remain separate from normalized acquisition. + +The first implementation should prove two existing US research acquisition callers share semantics before factoring provider code. Ordinary strategies consume a supplied frame or artifact without importing provider SDKs. + +## 2. Interfaces and ownership + +These signatures describe future interfaces; they are not exported APIs. + +| Interface | Contract | Owner | +|---|---|---| +| `HistoricalRequest` | Instrument IDs, UTC bounds, frequency, mandatory `price_basis`, purpose, decision cutoff, market/session profile, source policy, expected coverage | QPK data contract, constructed by the pipeline/research consumer | +| `SourceObservation` | Actual provider/feed/mode, retained source root/digest, source timestamps, unmodified native fields, status/reason | Thin provider adapter in the existing producer module | +| `NormalizedDataset` | Typed bars/series plus standard metadata, actual coverage, action/calendar versions, source/content digests, manifest binding | QPK normalization/validation plus provider-specific mapping | +| `fetch_history(request, *, adapter, store, clock)` | Returns a READY dataset or a typed unavailable/invalid/unsupported/coverage status; no strategy, account or order side effects | Small explicit acquisition function | +| `load_verified_price_series(symbol, *, binding_id, binding_sha256)` | Existing artifact port; verifies frozen input identity and request semantics before returning values | Existing decision-data adapter | +| `get_execution_quote(symbol)` | Existing execution quote port; broker source time, bid/ask/last, age and tradability checks; never synthesizes a quote from adjusted history | Existing platform adapter | + +Historical decision prices and execution quotes remain separate types and purposes, even when they share the same instrument. Do not silently add fields to the legacy `PriceSeries` and claim every consumer validates them. A versioned manifest-backed envelope carries semantics while a thin checked adapter returns the existing numeric structure. A naked numeric series is not conformance evidence. + +Provider adapters map capabilities explicitly: + +| Adapter | Native mode and design issue | +|---|---| +| Alpaca | Preserve `feed` and `adjustment`; apply the Basic SIP delay before fetching; record effective bounds; paginate every symbol; verify native daily session scope. | +| IBKR | Pin `whatToShow`, `useRTH`, contract identity, bar size and API date format. `TRADES` cannot satisfy raw or dividend-adjusted requests. Never take over a production Gateway session. | +| Schwab | Keep the existing SDK and retry behaviour; expose its documented/verified history basis separately from execution quotes. Preserve a timestamp's source rather than treating client fetch time as quote time. | +| LongPort | Explicit adjustment enum and trade-session enum, timezone conversion and source close semantics. US signal and HK research profiles are distinct. | +| Yahoo/yfinance | One shared normalization recipe for existing callers after conformance tests; explicit `auto_adjust` choice, source/library version, symbol aliases and action support. Preserve research-only licensing/quality restrictions. | +| Twelve Data | Existing split-only source observations and independent Yahoo cross-check; respect endpoint/plan limits and assurance gates. Not an interchangeable total-return feed. | +| AkShare | Underlying endpoint/version and `qfq` factors/actions; explicitly identify the Yahoo fallback. Preserve unavailable history and lack of strict historical point-in-time. | +| Binance | Venue/pair/currency identity, UTC interval, pagination and completed-kline filter; raw spot prices with token events separate. | +| Injected CSV/context | Verify the existing manifest/root, not just column names. Reject unknown provider/basis/time semantics for conforming new research. | + +## 3. Configuration + +Use one non-secret `market_data` config object, validated before network or storage access. These are proposed shared schema fields, not new environment aliases. No configuration is written by this proposal. + +| Field | Default / requirement | +|---|---| +| `schema_version` | `qsl.market_data_config.v1` | +| `purpose` | Required: `research`, `decision`, `display` or `execution_quote`; no inherited live authority | +| `instrument_ids`, `price_basis` | Required, no implicit basis; quote requests require contemporaneous `raw_unadjusted` plus their quote type | +| `primary.provider`, `primary.feed` | Market/purpose profile from the standard; US history `alpaca`/`sip`, US recent `alpaca`/`iex`; actual values retained in result | +| `fallback.providers` | Ordered candidates from the standard, empty until explicitly selected | +| `fallback.allow_source_change` | `false`; a selected fallback still creates a separately identified source result | +| `provider_options.alpaca.plan` | `basic`; entitlement declaration is not proof of a paid subscription | +| `provider_options.alpaca` delay/feed adapter mapping | Maps `ALPACA_SIP_MIN_DELAY_MINUTES`, `ALPACA_FEED_HISTORICAL`, `ALPACA_FEED_RECENT`; historical SIP default/minimum `16`, recent default `iex` | +| `provider_options..base_url` | Official public provider endpoint from the current adapter; override must preserve provider identity and approved transport policy. No private host defaults. | +| `market_profile`, `market_timezone`, `calendar_id` | Market defaults in the standard; timezone/venue mismatches rejected | +| `calendar_version`, `calendar_schedule_digest` | Required pinned version/schedule for a frozen dataset; no moving `latest` default | +| `session_scope`, `close_kind` | `regular`, `last_eligible_trade` for a daily signal profile; adapter capability check required; unproven native bars remain `provider_native` | +| `finalization_delay_seconds` | Explicit per-adapter/profile before acquisition; no universal zero-latency default | +| `storage.object_root` | Required approved cloud root; no default bucket/project or local path | +| `storage.namespace` | `market-data/v1`, configurable; existing frozen candidate paths retain their contracts | +| `storage.retention_policy_id` | Required rights/retention policy; no automatic deletion | +| `cache.enabled` | `false` until approved cloud storage is injected; request-local memory reuse may be enabled independently | +| `cache.namespace` | `market-data-cache/v1`, configurable | +| `cache.completed_history` | Immutable versioned entries; a new vendor revision creates a new entry | +| `cache.mutable_ttl_seconds` | `0` (no persistent reuse); a positive TTL never extends source freshness or quote validity | +| `validation.expected_sessions`, `validation.max_age_seconds` | Derived from the explicit request/profile and existing consumer policy; no relaxed global default | + +Resolution order: versioned package/market-profile defaults, then versioned consumer config, then explicitly documented non-secret environment/CLI overrides. Record the resolved non-secret configuration and its binding. Overrides cannot change a frozen candidate's identity in place or bypass minimum-delay/entitlement, basis, calendar, risk or permission checks. + +Alpaca segment rules: the feed-window planner in UsEquitySnapshotPipelines maps these three environment names once; the shared layer adds no competing aliases. The planner returns disjoint inclusive time windows, with the recent segment starting one microsecond after the historical bound. Native daily bar timestamps can still overlap across segments; do not merge their OHLCV. `ALPACA_FEED_RECENT` accepts `iex` or `none`; there is no recent SIP and no retry-to-IEX after a 401/403. + +## 4. Acquisition, cache and validation flow + +1. Resolve and bind the instrument/profile, required basis, UTC coverage, decision cutoff and approved source policy. Validate capabilities and entitlements without attempting an unauthorized endpoint. +2. Compute an identity from provider, feed, endpoint/adjustment mode, instrument/venue/currency, frequency, requested/effective bounds, session/close policy, calendar/action versions, transformation/adapter version and access-policy/retention scope. A symbol/date-only key is insufficient. +3. Load an exact immutable cloud object or acquire through the selected adapter in approved cloud compute. Store native responses in approved storage, then normalize. No real data is materialized on developer machines or committed to Git. +4. Validate schema, finite numeric fields, units, OHLC relationships, sorting/duplicates, expected sessions/inception/suspensions, completed bars, availability cutoff, action coverage and requested basis. Missing is not zero and does not become a non-trading day without evidence. +5. Verify source/content/manifest digests, the existing decision binding and any required multisource assurance. Atomically publish a new immutable generation. Cache-hit validation verifies identity, bytes, license scope and freshness; it never changes fetch/availability time. +6. Return values with their bound provenance and quality. Consumers receive unavailable/incomplete on a failed contract. A stale cache or IEX value cannot satisfy a SIP/full-market request without an explicitly new source policy and dataset identity. + +Error handling: classify 401/403, quota, transient transport and invalid payload separately. Respect the existing bounded retry policy; the shared layer does not create another retry loop around it. A 401/403 stops the affected acquisition; do not change feed and retry after an entitlement failure. A preplanned IEX segment cannot satisfy a trading decision that requires SIP. + +## 5. Migration order and risk + +This is a future plan, not authorization to execute it. Each consumer migrates in its own pull request with explicit source/basis pins, and is confirmed by the maintainer before adoption. The default remains the existing path; there is no global switch. + +| Wave | Consumers | Risk and acceptance evidence | +|---|---|---| +| 0 | QPK manifest/normalization/ports and existing source assurance | Medium: changing identity/serialization can invalidate evidence. Synthetic golden vectors, old-root compatibility and rejection cases first. No remote data fetch. | +| 1 | Existing US research collectors, then HK/CN/crypto collectors | Medium/high: source rights, adjusted basis and calendar differences. Run source conformance in approved cloud only when separately authorized. Preserve old objects and independent-source gates. | +| 2 | Research loaders/runners, strategy mathematical cores and plugin replay paths | Medium: changed signals/returns and lost point-in-time. Migrate one frozen input first, compare on identical permitted inputs, rerun only the affected evidence under its own authorization. | +| 3 | Benchmark publisher, revision policy, worker storage/read model and website chart projection | Medium: misleading benchmark comparison or accidental history overwrite. Keep price/total-return and ETF/index IDs visible and preserve current revision authorization. | +| 4 | Platform historical input adapters in an isolated no-order/offline path, one platform at a time | High: platform signal history differs from research. Verify semantic mapping and dependency pin using synthetic or separately approved cloud replay. | +| 5 (last) | Live decision consumers for IBKR, Schwab, LongBridge, Firstrade and Binance | Highest: explicit per-consumer provider/feed/basis/calendar/adapter/binding pins and a separately authorized adoption change. Any future QMT or Alpaca live consumer follows the same rule. No test orders. | + +A future adoption must explicitly bind `provider + feed + price_basis + session/calendar profile + adapter version + QPK version + decision artifact identity`. Within wave 2, migrate an artifact loader before its replay, strategy or plot consumers. Within wave 5, the maintainer chooses the account and platform order. There is no implicit fallback on an exception. Existing order-path protections (reconcile-only mode, the unknown-result resend guard, duplicate-order protection) and risk limits are not changed by any migration step. Rollback returns to the previously approved compatible input/profile; it does not erase evidence, change ledger history or resend unknown orders. + +## 6. Verification plan + +Before a shared implementation is claimed conforming, use synthetic fixtures for split/reverse-split, distribution double-counting, missing adjusted-close entries, float/decimal serialization, action revisions, DST, early closes, HK/CN lunch/half-days, crypto day completion, pagination, inception/delisting, source-change cache separation, stale/invalid cache, truncated SIP coverage and late data. Perturb future events and ensure earlier decisions are unchanged. + +Then compare only separately authorized cloud observations under identical instrument/basis/session policies, with predeclared tolerances. Report differences in provider trade filtering, corporate actions and timestamps instead of averaging them away. Empirical IBKR comparisons may guide a specific adoption but do not establish universal provider equivalence. diff --git a/docs/market-data-inventory.md b/docs/market-data-inventory.md new file mode 100644 index 0000000..90418ea --- /dev/null +++ b/docs/market-data-inventory.md @@ -0,0 +1,98 @@ +# Market-data source inventory (summary) + +A static source scan of 25 repositories read 1783 source files and identified 393 consumer/contract/acquisition boundaries and 30 findings. No runtime, account or market data was read. The full per-file inventory is kept outside the repository and is available from the maintainers on request. + +## Per-repository consumer counts by migration wave + +| Repository | Rows | W0 | W1 | W2 | W3 | W4 | W5 | +|---|---:|---:|---:|---:|---:|---:|---:| +| BinancePlatform | 12 | | | 2 | | | 10 | +| CharlesSchwabPlatform | 10 | | | 2 | | | 8 | +| CnEquitySnapshotPipelines | 7 | | 2 | 5 | | | | +| CnEquityStrategies | 32 | | | 32 | | | | +| CryptoLivePoolPipelines | 10 | | 1 | 9 | | | | +| CryptoStrategies | 13 | | | 13 | | | | +| FirstradePlatform | 4 | | | | | | 4 | +| HkEquitySnapshotPipelines | 5 | | 2 | 3 | | | | +| HkEquityStrategies | 16 | | 1 | 15 | | | | +| InteractiveBrokersPlatform | 11 | | | 2 | | | 9 | +| LongBridgePlatform | 8 | | | | | | 8 | +| MarketSignalSources | 12 | | 1 | 11 | | | | +| PoliticalEventTrackingResearch | 2 | | | 2 | | | | +| QmtPlatform | 2 | | | | | 2 | | +| QuantAdvisorResearch | 3 | | 1 | 2 | | | | +| QuantPlatformKit | 25 | 25 | | | | | | +| QuantRuntimeSettings | 8 | | | | 8 | | | +| QuantStrategyPlugins | 26 | | 1 | 25 | | | | +| ResearchSignalContextPipelines | 9 | | 1 | 8 | | | | +| UsEquitySnapshotPipelines | 116 | | 16 | 100 | | | | +| UsEquityStrategies | 62 | | | 62 | | | | + +Repositories with no rows: AIAuditBridge, AlpacaPlatform, IBKRGatewayManager, SchwabTokenAutoRefresher. + +Waves: + +- W0: shared contract in QPK. +- W1: research acquisition. +- W2: research and injected consumers. +- W3: website projections and benchmark publisher. +- W4: isolated platform inputs. +- W5: live decision paths (last). + +## Role counts + +| Role | Count | +|---|---:| +| series/feature consumer | 270 | +| artifact/CSV input boundary | 75 | +| provider acquisition/normalization | 21 | +| provider acquisition entry/contract | 9 | +| benchmark/FX display or publication boundary | 8 | +| manifest/binding/validation contract | 6 | +| historical universe/membership input boundary | 1 | +| price artifact and synthetic-fixture contract | 1 | +| synthetic history transformation | 1 | +| execution quote wrapper | 1 | + +## Findings + +| ID | Category | Finding | Main location | +|---|---|---|---| +| DS-01 | basis conflict | QPK tries `ADJUSTED_LAST` then `TRADES` and returns the same series type; strict acquisition forbids this fallback. | QuantPlatformKit/src/quant_platform_kit/ibkr/market_data.py | +| DS-02 | basis conflict | The IBKR signal proxy suppresses `ADJUSTED_LAST` and uses `TRADES`, so signal and research inputs need separate identities. | InteractiveBrokersPlatform/application/runtime_strategy_adapters.py | +| DS-03 | basis conflict | `auto_adjust=False` selects split-only Close, but falls back to Adj Close while other OHLC columns stay independently taken. | InteractiveBrokersPlatform/application/runtime_strategy_adapters.py | +| DS-04 | basis conflict | Raw, split-only and dividend-aware datasets coexist for the same US instruments and are not interchangeable. | UsEquitySnapshotPipelines/scripts/acquire_batch_a_v2_price_snapshots_alpaca.py | +| DS-05 | duplicate code | Two Yahoo download modules contain 8 identical top-level functions with diverging download policies. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/yfinance_prices.py | +| DS-06 | duplicate code | Several repos own separate Yahoo request and normalization policies. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/yfinance_prices.py | +| DS-07 | duplicate code | Several Alpaca collectors each own their own request or publication logic. | UsEquitySnapshotPipelines/scripts/acquire_batch_a_v2_price_snapshots_alpaca.py | +| DS-08 | basis conflict | QAR prefers `adjclose` but takes `quote.close` when a value is absent, changing dividend treatment without a per-row basis marker. | QuantAdvisorResearch/src/quant_advisor_research/market_confirmation.py | +| DS-09 | basis conflict | The context downloader falls back from `adjclose` to `close` and does not bind a canonical `price_basis`. | ResearchSignalContextPipelines/src/research_signal_context_pipelines/context_bundle.py | +| DS-10 | basis conflict | AkShare `qfq`, Tencent `qfq` and Yahoo `adjclose` are summarized as `adjusted_close_equivalent` without proven factor equivalence. | CnEquitySnapshotPipelines/src/cn_equity_snapshot_pipelines/akshare_market_history.py | +| DS-11 | basis conflict | LongBridge `ForwardAdjust`/Intraday and yfinance `auto_adjust=True` differ in dividend reinvestment, factor anchors and session semantics. | HkEquitySnapshotPipelines/src/hk_equity_snapshot_pipelines/low_vol_dividend_longbridge_snapshot.py | +| DS-12 | provenance | `PricePoint`/`PriceSeries` do not require provider, feed, basis, calendar or digests, unlike the newer decision-data contracts. | QuantPlatformKit/src/quant_platform_kit/common/models.py | +| DS-13 | calendar | The parser uses `America/New_York` but the read model returns `Asia/Shanghai`, which can mislead joins and display. | QuantRuntimeSettings/web/strategy-switch-console/infrastructure/market_benchmark/market_benchmark.js | +| DS-14 | calendar | The website publisher drops all current New York dates while research checks the actual session close. | QuantRuntimeSettings/scripts/sync_market_benchmark_closes.mjs | +| DS-15 | calendar | US session logic exists as pinned exchange calendars and local holiday rules, separately maintained with different versions. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/lifecycle/soxl_pit_input_packager.py | +| DS-16 | calendar | Universe audit and a LongBridge fallback use weekday ranges, which include holidays and cannot prove actual sessions. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/universe_audit_engine.py | +| DS-17 | calendar | The CN calendar is a finite hand-maintained holiday set with a stated date range, distinct from the exchange-calendar profiles. | QuantPlatformKit/src/quant_platform_kit/common/cn_equity_calendar.py | +| DS-18 | calendar | Some consumers strip timezones or derive dates in UTC rather than market-local time. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/yfinance_prices.py | +| DS-19 | calendar | Binance helpers take the last returned daily row without proving that bar has closed. | QuantPlatformKit/src/quant_platform_kit/binance/market_data.py | +| DS-20 | provenance | A per-symbol CSV cache is updated in place with no provider/basis/calendar identity in the filename. | CryptoLivePoolPipelines/src/binance_client.py | +| DS-21 | provenance | Schwab `QuoteSnapshot` stamps `as_of` from the client clock, conflating fetch time with quote time. | QuantPlatformKit/src/quant_platform_kit/schwab/market_data.py | +| DS-22 | basis conflict | HK staging chooses volume or turnover or 0, while QAR uses 0 when volume is absent; these are different units and missing is not zero. | HkEquitySnapshotPipelines/src/hk_equity_snapshot_pipelines/low_vol_dividend_longbridge_snapshot.py | +| DS-23 | calendar | The HK strategy downloader drops dates missing any instrument, hiding inception, suspension or missing-data periods. | HkEquityStrategies/src/hk_equity_strategies/backtest/yfinance_market_data.py | +| DS-24 | basis conflict | `total_return_adjusted` and `SPLIT_ADJUSTED_PRICE_RETURN` contracts coexist; adjusted prices alone do not specify a reinvested index. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/lifecycle/soxl_core_only_p1_binding.py | +| DS-25 | provenance | Website price indices and dividend-aware ETF research are different series and must state that difference in comparisons. | QuantRuntimeSettings/web/strategy-switch-console/infrastructure/market_benchmark/market_benchmark.js | +| DS-26 | synthetic | The legacy TECL mode applies 3x daily return minus fixed annual expense/252 with a -99% loss clamp, lacking the financing and real/synthetic/bridge contract. | UsEquitySnapshotPipelines/src/us_equity_snapshot_pipelines/tecl_xlk_trend_income_research_inputs.py | +| DS-27 | point-in-time | Current-only constituents and factor snapshots are not historical point-in-time; adjusted prices cannot fix universe survivorship. | CnEquitySnapshotPipelines/src/cn_equity_snapshot_pipelines/index_membership.py | +| DS-28 | config | The website benchmark publisher's sync URL is configurable but its source default is tied to a personal deployment. | QuantRuntimeSettings/scripts/sync_market_benchmark_closes.mjs | +| DS-29 | config | The Alpaca SIP access diagnostic passes the date cutoff as the end bound with `feed=sip` and has no configurable free-plan delay or feed policy. | UsEquitySnapshotPipelines/.github/workflows/alpaca-sip-access-diagnostic.yml | +| DS-30 | config (resolved) | The Alpaca feed-window planner enforces a minimum SIP delay of 16 minutes, matching the standard. | UsEquitySnapshotPipelines/scripts/alpaca_feed_windows.py | + +## Known limits + +The static scan did not verify: broker entitlements, source licenses and retention, RTH equivalence of native daily bars, corporate-action factor equivalence, pre-2016 source coverage, deployed versions, or dynamic plugin inputs. + +## Per-row fields + +The full inventory records these fields per consumer: `consumer_id, repo, revision, file, consumer_role, provider, plan_constraints, price_basis, adjustment_application, timezone, session_close, trading_calendar, corporate_actions, cache_storage, config_keys_env_vars, evidence, migration_wave, migration_risk`. diff --git a/docs/market-data-standard.md b/docs/market-data-standard.md new file mode 100644 index 0000000..eda8e86 --- /dev/null +++ b/docs/market-data-standard.md @@ -0,0 +1,113 @@ +# Market-data standard + +Status: proposal. Not yet adopted by any consumer. + +This standard defines how market data is named, priced, timed, fingerprinted and labelled in QuantPlatformKit (QPK) and the pipelines that use it. Read it with the [access-layer design](market-data-access-layer-design.md) and the [source inventory](market-data-inventory.md). + +## 1. Per-market primary source and fallback + +| Market and purpose | Primary | Fallback and limits | +|---|---|---| +| US equity/ETF completed historical sessions | Alpaca historical SIP, Basic (free) plan | IBKR historical adapter in an isolated, approved research environment with verified entitlement and the same basis/session contract. Yahoo is an optional, separately configured research source where rights permit. | +| US recent observations | Alpaca IEX, labelled single-exchange | Delayed SIP only when the consumer explicitly accepts its age. Broker execution quotes keep their current provider. | +| HK equity/ETF | LongPort/LongBridge with authorized entitlement and a pinned adjustment policy | Yahoo research inputs only under an explicit source profile and rights review. | +| CN equity/ETF research | AkShare source family with the endpoint and `qfq` policy named | Yahoo adapter only for supported instruments, separately identified. | +| Crypto spot | Binance spot for Binance instruments; quote currency and venue fixed | No default cross-exchange fallback. A different venue or pair is a different instrument. | +| Indices, rates, constituents, context | Explicit instrument- or dataset-specific source | An ETF is not an index substitute without a different benchmark ID. Today's constituents and revised values do not reconstruct historical point-in-time inputs. | + +## 2. Alpaca free (Basic) plan + +The Basic plan provides recent IEX observations; historical SIP requests must exclude the most recent 15 minutes. The operational safety margin is 16 minutes. For SIP, `effective_end = min(requested_end, now_utc - configured_delay)` with a Basic-plan minimum delay of 16 minutes. Record both bounds and any truncation. A strict request that loses required coverage returns unavailable; it must not claim to have fulfilled the original interval. The delay does not by itself prove that a daily bar is complete. + +IEX and SIP are different sources. Changing IEX to SIP is a dataset/source change, never a retry with equivalent data. No paid upgrade is required or assumed. Provider coverage is not a promise of pre-2016 history. [Alpaca plan documentation](https://docs.alpaca.markets/us/docs/about-market-data-api) + +The Alpaca feed-window planner in UsEquitySnapshotPipelines already uses these configuration names. Reuse them exactly: + +| Name | Default | Meaning | +|---|---|---| +| `ALPACA_SIP_MIN_DELAY_MINUTES` | `16` | Minimum SIP delay for the Basic plan. | +| `ALPACA_FEED_HISTORICAL` | `sip` | Historical feed: `sip` or explicitly selected `iex`. | +| `ALPACA_FEED_RECENT` | `iex` | Recent feed: `iex` or `none`. No recent SIP and no retry-to-IEX after a 401/403. | + +## 3. Fallback + +Fallback is request-wide by default. It creates a new immutable dataset identity, records the reason and actual source, and is allowed only by the consumer's profile. Do not fill individual missing rows from another source, switch price basis, or bypass an existing two-source assurance requirement. Frozen candidates retain their original source contract; adopting this standard does not rewrite their evidence. + +## 4. Price basis + +Every request, normalized dataset, cache key, decision binding, research manifest and public summary that describes prices must carry `price_basis`. A generic `adjusted`, `close`, `auto_adjust`, provider name or filename is insufficient. Missing legacy metadata is `unknown_legacy` during inventory only; new conforming inputs reject it. + +| `price_basis` | Meaning | Adjustment/accounting requirements | +|---|---|---| +| `raw_unadjusted` | Contemporaneous quoted/traded units; no retrospective corporate-action adjustment | Splits change units and quantities; dividends are separate cash events. Executable prices and quantities must be in contemporaneous units. | +| `split_adjusted` | Historical prices restated for splits and reverse splits only | Record factor orientation, anchor/as-of date, source and action revision. Dividend cash is not included in returns unless separately accounted for. Volume adjustment is separately declared. | +| `split_dividend_adjusted` | Provider-adjusted history incorporating splits and distributions | Preserve the exact vendor mode and method. It is a research price series, not an executable price. Do not book the same dividends a second time. | +| `total_return_index` | An explicit reinvested wealth index | Record reinvestment date/price, gross/net tax treatment, currency, base value/date and action inputs. Not an OHLC or trade-price series. | + +Use `return_basis=price_return|total_return` separately from `price_basis`, with a concrete return-construction method. A provider's adjusted-close ratio can support a specified total-return approximation, but is not automatically an audited total-return index. Existing `total_return_adjusted` labels in candidate contracts are legacy names: keep them, record their actual vendor method in a new manifest, and do not treat them as a reinvested wealth index. + +For new general US historical research, the default profile requests `price_basis=split_dividend_adjusted` with Alpaca `adjustment=all`. Every request must still name its basis. Existing split-only, raw-unit and frozen candidate profiles keep their declared semantics until they are migrated separately. + +Adapter mappings must be tested against vendor semantics: + +- Alpaca: `raw` maps to `raw_unadjusted`, `split` to `split_adjusted`, `all` to `split_dividend_adjusted`, with any additional supported action types recorded. `dividend` alone needs an explicitly described policy; it is not equivalent to `all`. Store the actual request parameter. [Historical bars parameters](https://docs.alpaca.markets/us/reference/stockbars) +- IBKR: `TRADES` historical equity bars are split adjusted, not raw contemporaneous history. `ADJUSTED_LAST` includes split and dividend adjustment. No automatic interchange. A raw request unsupported by the selected endpoint must fail as unsupported. Streaming `close`/prior-close fields have their own corporate-action rules. [Historical bars](https://www.interactivebrokers.com/docs/tws-api/doc/md/historical-bars), [streaming fields](https://www.interactivebrokers.com/docs/tws-api/doc/md/streaming-data) +- Yahoo/yfinance: pass `auto_adjust` explicitly and retain the library version. The chart `quote.close` path is treated by the existing split-only assurance adapter as split adjusted; the `adjclose/close` path scales OHLC for dividend-aware research. Neither is assumed to be raw historical trade units. Null adjusted values must not silently change rows to another basis. [yfinance download reference](https://ranaroussi.github.io/yfinance/reference/api/yfinance.download.html) +- LongPort `ForwardAdjust`, AkShare `qfq`, and Schwab's split-only convention remain named provider policies until their action/factor semantics are verified. Do not translate the word "forward" into `total_return_index`. + +Do not split-adjust volume with a dividend factor. Preserve `volume_basis`, native volume units, and whether missing volume is unknown or zero. Raw OHLC, split-adjusted OHLC and a wealth index must have distinct dataset identities. Currency, listing/venue, instrument type, multiplier and symbol-validity dates are also part of identity; a ticker alone is insufficient. + +## 5. Time, sessions and calendars + +Store instants as timezone-aware UTC. Store `session_date` separately as the exchange-local trading date, plus `market_timezone`, `calendar_id`, pinned calendar package/version and the relevant schedule revision/digest. Reject ambiguous naive timestamps at new adapter boundaries. Provider-native daily date labels are dates, not midnight execution timestamps. + +The defaults below are selectable profile values, not assumptions hard-coded in consumers. Exchange exceptions override ordinary hours; do not use a fixed 16:00 or weekday-only test for US sessions. + +| Market | Timezone / calendar | Session-close rule | +|---|---|---| +| US cash equities | `America/New_York`; XNYS | Regular-session close from the pinned schedule, including DST, holidays and early closes. A scheduled close is not proof of the primary exchange's official auction price. | +| HK equities | `Asia/Hong_Kong`; XHKG | Include lunch break, applicable closing auction and exceptional notices. Full-day and half-day schedules remain distinct. | +| CN equities | `Asia/Shanghai`; XSHG/XSHE | Include lunch, holidays, suspension and applicable auction rules. | +| Crypto spot | UTC; `24x7` profile | Daily interval `[00:00Z, next 00:00Z)`, only after its closing instant plus declared provider finalization latency. No 252-session assumption. | + +Required times: `bar_start_at`, `bar_end_at`, `session_close_at`, `provider_timestamp`, `fetched_at`, `available_at`, and `decision_as_of` where applicable. `available_at` must be supported by the source/observation policy; it is not reconstructed by setting it equal to a historical session close. + +Define `session_scope=regular|extended|provider_native` and `close_kind=last_eligible_trade|official_auction|provider_native|wealth_index`. The canonical daily signal profile targets completed regular sessions and last eligible trade. An adapter must demonstrate that its native bars satisfy that scope or explicitly aggregate eligible intraday observations under a versioned recipe. Unverified native daily bars remain `provider_native`. [NYSE trading hours](https://www.nyse.com/markets/hours-calendars) + +Only complete, available observations may feed a decision. A signal that uses session t's close can execute no earlier than the next permitted execution time, unless an existing frozen strategy proves a valid earlier information cutoff. Data latency is never permission to submit orders. Cross-market joins use availability as of the decision instant. Missing sessions remain explicit; never silently forward-fill a halt or delisting, and do not drop stress periods by intersecting available dates. + +## 6. Corporate actions and revisions + +Keep versioned split, dividend/distribution, symbol-change, merger, spin-off, delisting and settlement events, including effective, announcement/availability, revision and fetch times. Preserve original instrument identity across symbol changes. Factors identify their source, orientation and anchor. Unknown actions or ambiguous factor mappings make a requested conversion unavailable. + +Prefer the requested vendor basis when its documented semantics match the request. A raw-plus-actions reconstruction is a separate, versioned transformation with its own digest and tests. Never reverse a provider adjustment without the required action/factor inputs. Observed market prices generally already reflect fund expenses; do not deduct fund expense ratios twice. Preserve old objects when vendors revise history; new generations get new identities. + +## 7. Provenance and fingerprints + +Each immutable result carries: + +- Identity: schema version, instrument/listing/venue/currency, frequency, requested coverage, actual coverage, `price_basis`, `return_basis`, `volume_basis`, session scope and close kind. +- Actual source: `provider`, `feed`, endpoint class, vendor adjustment parameters, adapter/source code revision, provider/library version, plan/entitlement classification, acquisition bounds and fallback reason. No credential, secret reference name, private account or private host appears in public metadata. +- Time, calendar and action metadata, plus declared gaps, duplicates, revisions, finality and point-in-time limitations. +- `source_digest`: SHA-256 of the exact source-response bytes or the existing immutable source artifact's exact bytes, with immutable cloud object generation/version. Paged requests use an ordered page-digest manifest. No transport headers or authenticated URLs are included. For SDK sources without retained original response bytes, name the deterministic source-envelope serialization; do not call it a raw-wire digest. +- `content_digest`: SHA-256 of the versioned canonical normalized records. `manifest_digest`: SHA-256 of the canonical metadata, content/source references and transformations, excluding its own digest field. + +Proposed canonical content encoding `qsl.market_records.jsonl.v1`: UTF-8 without BOM; one compact object per line, LF including the last line; sorted ASCII field names; records sorted by instrument ID, bar end/session and source type; UTC timestamps with fixed microsecond precision and `Z`; prices/volumes as finite canonical decimal strings; integers in decimal, dates ISO-8601, explicit null only for permitted missing fields. Preserve parsed source precision rather than serialize binary float representations. Duplicate identity keys reject. A future implementation must supply synthetic cross-language golden vectors before this encoding is used for frozen inputs. Existing artifacts keep their original digest algorithm and bytes. + +Fetch timestamps and private storage locators belong to manifests, not the price-content digest. Different feed/basis/session policies cannot share a cache identity even if their numerical content hashes equally. + +## 8. Synthetic history + +Each record carries `source_type=real|synthetic_history|synthetic_bridge`. A synthetic fixture is separately labelled `evidence_class=synthetic_fixture`, never `real`. + +The synthetic manifest records the real underlying/proxy identity and digest, leverage, daily reset rule, fee and financing sources and day counts, drag model/version, target-index regime, train/validation windows and calibration version/status. Before the first verified real anchor t0, modelled history may be used; from t0 onward keep real product levels and only real returns. Use one scale factor for the splice, no smoothing. A return crossing into t0 without a prior real observation is `synthetic_bridge` and counts as synthetic. + +Report dates, proportions and limitations in tables, exports and charts. Produce independently rerun full-synthetic and real-only strategy results with their own initial capital, warmup and plugin ablations. Cropping the tail of an extended equity curve is not a real-only rerun. Daily close models cannot invent historical OHLC, volume, spread, options chains or next-open/stop fills; when the required inputs are missing, the affected execution replay is reported as not comparable. Keep full 2000-2002, 2008, 2020 and 2022 coverage where the underlying exists, and report a window as coverage-blocked when data is missing versus non-existent when the underlying did not exist. Synthetic history never counts as realized performance or untouched real out-of-sample evidence. Passing calibration qualifies the model; it does not promote a strategy. + +## 9. Configurability, retention and publication + +All provider endpoints, market profiles, calendars, dataset names, cloud roots, cache prefixes, retention settings and adapter selection are configuration values with documented defaults. No personal domains, accounts, cloud projects, buckets, secret names or machine paths are defaults. + +Real market and account data stays in approved cloud storage and compute, never on developer machines or in Git. Cache content inherits the source's license, retention and redistribution rules; a free or public endpoint is not a redistribution license. Retention is explicit per dataset and must not erase frozen evidence. Public documents contain generic references such as `` and aggregate coverage metadata, never prices, accounts or licensed response bodies. + +Legacy datasets are not overwritten or retrospectively recertified. This standard does not change order-path protections or risk limits.