diff --git a/docs/research-backtest-contract.md b/docs/research-backtest-contract.md new file mode 100644 index 00000000..52fc3dbc --- /dev/null +++ b/docs/research-backtest-contract.md @@ -0,0 +1,90 @@ +# Research backtest timing and cost contract + +Status: **characterization only; model defects remain open**. Related issue: +[#366](https://github.com/QuantStrategyLab/BinancePlatform/issues/366). +Baseline inspected: `44a0cd6e54c12cabf8944b7fbed9080840451297`. +This change adds synthetic tests and documentation only. It does not correct the +simulator, generate real signals, select a strategy, or change runtime, orders, +risk decisions, or notifications. + +## Current behavior versus required acceptance + +| Area | Observed in `research/backtest.py` | Required before treating results as comparable evidence | +| --- | --- | --- | +| Daily information boundary | `prepare_*_daily_indicators` includes the final close of day D; `align_symbol_data` joins on D's date, exposing it at D 00:00 and subsequent hours. | Persist bar open/close and availability timestamps; a decision cannot read an unfinished daily bar. Specify the UTC cutoff, boundary inclusion and source finality/latency. | +| Hourly signal and fill | The hourly close is used to form the snapshot, select weights, and fill buys/sells on that same row. Binance kline `time` is parsed from open time; close time is discarded. | Declare signal availability and execution timestamp/price. A next-bar contract needs an actually available next quote; a same-close model must explicitly declare that idealization and cannot be silently compared with next-open fills. | +| Missing data | Hourly closes are forward-filled after the first observed quote, with no quote-age flag. Leading gaps stay NaN. Filled prices can open or liquidate positions. | Distinguish mark-only carry-forward from executable quotes, reject stale fills, specify maximum age, outages, delistings and calendar coverage. | +| Buy costs | Both trend and BTC DCA calculate quantity as `budget * 0.985 / price`, deduct only `quantity * price` and retain the buffer as cash. | Deduct explicit per-side fees and execution friction, with declared rates and provenance. The buffer is not a 150-bps fee. | +| Sell costs | Trend exits, circuit-breaker liquidation and BTC DCA reductions credit `quantity * price` without costs. | Apply the chosen cost contract on every exit path as well as entry; preserve an auditable cash/quantity/fee ledger. | +| Drawdown anchor | Curve records only post-processing rows; `cummax()` starts at the first recorded value. Initial capital is used for return but is not a separate pre-fill curve anchor. | Include initial capital before the first execution so first-fill costs/losses count in drawdown. Specify observation frequency and cash-flow treatment. | +| Cost stress | No fee/slippage inputs, turnover series, or recomputed cost scenarios are returned. | Supply audited turnover and explicit base assumptions before computing 1×/2×/3× scenarios. Do not invent fee tiers or substitute cash buffers. | + +Current output must be labelled **未计成本 / costs not included; timing not +validated**. Corrected net returns, cost-stress results, and causal timing +validation are **not computed**. Existing headline return/drawdown output is not +promotion evidence or a valid same-window net-performance comparison. The tests +below do not change the script's printed labels; downstream use must honor this +restriction until separately authorized implementation fixes the model. + +## Executable characterization + +Run offline after installing the repository's pinned dependencies: + +```sh +python -m unittest discover -s tests -p test_research_backtest_contract.py -v +python -m unittest discover -s tests -v +PYTHONPATH=. python -m pytest -q tests +``` + +`tests/test_research_backtest_contract.py` calls the real alignment, indicator +preparation and simulation functions. It replaces universe, weight selection and +budget helpers with deterministic fixtures to isolate accounting and timing. +No exchange client is created; `build_client` is guarded to fail. It does not +validate strategy rankings or dependency allocation policy. + +The tests deliberately PASS on current limitations: + +- Changing only the final daily close changes that day's first-hour SMA20 and + BTC MA200; the previous hour is unchanged. +- A selected first-row buy is filled at that row's close. +- A missing-hour quote carried forward can execute a new buy. +- A flat-price trend buy/sell round trip leaves all initial equity intact. +- A 1,000-unit budget buys 9.85 units at 100, leaving 9,015 cash; selling at 110 + gives 10,098.5. These are arbitrary fixture units, not investment returns or + a real cost assumption. +- BTC DCA has the same cash-buffer behavior. +- The curve contains only simulation-row timestamps and no pre-fill capital + observation. The zero-cost fixture does not manufacture a first-fill loss. + +These assertions are tripwires for explicit contract changes, not acceptance +criteria for a correct backtester. When a future model repair changes them, +replace the affected characterization with causal/cost acceptance tests rather +than preserving the defect. There are no skipped or expected-failure tests +masquerading as model validation. + +## Future acceptance fixtures (not implemented or claimed passing) + +1. Daily boundary: changing D's final close must not change decisions before its + finalized availability time. Test the instant before, at and after that time, + plus the final unfinished daily bar. +2. Next-fill timing: a close-derived signal cannot fill at that close under a + next-bar contract. Test a price jump and missing next quote. Retain separate, + explicitly labelled cases if a same-close approximation is chosen. +3. Stale market: allow separately labelled valuation carry-forward if intended, + but no execution on a fabricated quote. Cover both entry and all exit paths. +4. Cost conservation: with arbitrary test-only fee `f`, a flat-price notional + `q * p` round trip loses exactly `2 * q * p * f` when that is the declared + per-side fee rule and the budget is sufficient. Test fee units, spread and + slippage separately, both trend and BTC, including circuit-breaker exits. + This algebra is a proposed test oracle, not today's model or an account rate. +5. Initial loss: starting equity 100 and a first-fill cost 1 imply first observed + equity 99 and at least -1% drawdown relative to the initial anchor, even if + all later observations stay at 99. Current zero-cost fills cannot exercise + this acceptance case; it is not marked fixed by the curve-shape test. +6. Cost scenarios and comparability: verify turnover units and per-side cost + accounting before recomputing 1×/2×/3× stress. Require matching data, period, + universe, availability, execution, valuation, initial capital and costs. + +Implementation of these fixtures against a corrected model is separate work. +Issue #366 remains open; this document does not authorize a redesign or live +configuration changes. diff --git a/tests/test_research_backtest_contract.py b/tests/test_research_backtest_contract.py new file mode 100644 index 00000000..f9b1b62d --- /dev/null +++ b/tests/test_research_backtest_contract.py @@ -0,0 +1,159 @@ +"""Offline characterization of known research limitations, not model acceptance. + +See docs/research-backtest-contract.md and issue #366. These passing tests +intentionally describe today's behavior; a model fix must revise the assertions. +""" + +import unittest +from unittest.mock import patch + +import numpy as np +import pandas as pd + +from research import backtest + + +class ResearchBacktestContractTests(unittest.TestCase): + def setUp(self): + # Any accidental data loading must fail rather than reach Binance. + self.client_patch = patch.object( + backtest, "build_client", side_effect=AssertionError("offline fixtures only") + ) + self.client_patch.start() + self.addCleanup(self.client_patch.stop) + self.times = pd.date_range("2025-01-01", periods=3, freq="h", tz="UTC") + + @staticmethod + def daily_bars(): + times = pd.date_range("2024-01-01", periods=201, freq="D", tz="UTC") + return pd.DataFrame({ + "time": times, "open": 100.0, "high": 100.0, + "low": 100.0, "close": 100.0, "vol": 10.0, + }) + + def test_final_daily_close_changes_same_day_first_hour_indicator(self): + for prepare, field, expected in ( + (backtest.prepare_trend_daily_indicators, "sma20", 110.0), + (backtest.prepare_btc_daily_indicators, "ma200", 101.0), + ): + with self.subTest(field=field): + daily = self.daily_bars() + day = daily.time.iloc[-1] + timeline = pd.date_range(day - pd.Timedelta(hours=1), periods=3, freq="h") + hourly = pd.DataFrame({"time": timeline, "close": 100.0}) + baseline = backtest.align_symbol_data(hourly, prepare(daily), timeline) + # Only the future final daily close changes; hourly history is identical. + daily.loc[daily.index[-1], "close"] = 300.0 + changed = backtest.align_symbol_data(hourly, prepare(daily), timeline) + self.assertEqual(changed[field].iloc[0], baseline[field].iloc[0]) + self.assertEqual(baseline[field].iloc[1], 100.0) + self.assertEqual(changed[field].iloc[1], expected) + self.assertEqual(changed[field].iloc[2], expected) + # Known lookahead: D's final close is visible at D 00:00. + + def test_missing_hour_close_is_forward_filled_but_leading_gap_is_not(self): + hourly = pd.DataFrame({"time": [self.times[1]], "close": [100.0]}) + aligned = backtest.align_symbol_data(hourly, pd.DataFrame(), self.times) + self.assertTrue(pd.isna(aligned.close.iloc[0])) + self.assertEqual(aligned.close.iloc[1:].tolist(), [100.0, 100.0]) + + def market(self, prices): + data = {} + for symbol in ("BTCUSDT", "ETHUSDT"): + data[symbol] = pd.DataFrame({ + "time": self.times[:len(prices)], + "close": prices if symbol == "ETHUSDT" else 100.0, + "sma20": 50.0, "sma60": 50.0, "sma200": 50.0, + "atr14": 1000.0, "ma200": 50.0, "ma200_slope": 1.0, + "geom200": np.nan, "zscore": np.nan, "sell_trigger": np.nan, + }) + return data + + def run_fixture(self, data, weights, *, dca=False): + def allocation(total, cash, trend, btc): + return {"trend_usdt_pool": 0.0 if dca else 1000.0, + "trend_layer_equity": trend, + "dca_usdt_pool": 1000.0 if dca else 0.0} + + with ( + patch.object(backtest, "ALL_SYMBOLS", ["BTCUSDT", "ETHUSDT"]), + patch.object(backtest, "FIXED_POOL", ["ETHUSDT"]), + patch.object(backtest, "select_rotation_weights", side_effect=weights), + patch.object(backtest, "compute_allocation_budgets", side_effect=allocation), + patch.object(backtest, "allocate_trend_buy_budget", + side_effect=lambda active, eligible, budget: { + symbol: budget for symbol in eligible + }), + patch.object(backtest, "get_dynamic_btc_base_order", return_value=1000.0), + ): + return backtest.simulate_window( + {"name": "synthetic", "label": "synthetic", "pool_mode": "fixed"}, + "synthetic", data["BTCUSDT"].time.iloc[0], + data["BTCUSDT"].time.iloc[-1], data, capture_details=True, + ) + + def test_same_row_signal_buys_at_that_rows_close(self): + data = self.market([100.0, 110.0]) + observed = [] + + def weights(strategy, rows, gate, ts, state): + observed.append((ts, rows["ETHUSDT"]["close"])) + return {"ETHUSDT": 1.0} + + result = self.run_fixture(data, weights) + self.assertEqual(observed[0], (self.times[0], 100.0)) + self.assertEqual(result["trades"], 1) + # 985 / first-row close = 9.85 units; a next-row fill gives no gain here. + self.assertEqual(result["equity_series"].tolist(), [10000.0, 10098.5]) + + def test_flat_trend_round_trip_deducts_no_cost(self): + result = self.run_fixture(self.market([100.0, 100.0]), [{"ETHUSDT": 1.0}, {}]) + self.assertEqual(result["trades"], 2) + self.assertEqual(result["final_equity"], backtest.INITIAL_CAPITAL) + self.assertEqual(result["total_return"], 0.0) + # The 1.5% buy budget buffer stays cash; it is not a fee. + + def test_trend_round_trip_retains_budget_buffer_as_cash(self): + result = self.run_fixture(self.market([100.0, 110.0]), [{"ETHUSDT": 1.0}, {}]) + self.assertEqual(result["trades"], 2) + self.assertEqual(result["final_equity"], 9015.0 + 9.85 * 110.0) + + def test_dca_budget_buffer_is_also_unspent_cash(self): + data = self.market([100.0, 100.0]) + data["BTCUSDT"]["close"] = [100.0, 110.0] + data["BTCUSDT"]["geom200"] = 100.0 + result = self.run_fixture(data, [{}, {}], dca=True) + self.assertEqual(result["trades"], 1) + self.assertEqual(result["equity_series"].tolist(), [10000.0, 10098.5]) + + def test_flat_dca_buy_and_partial_sell_deduct_no_cost(self): + data = self.market([100.0, 100.0]) + data["BTCUSDT"]["geom200"] = 100.0 + data["BTCUSDT"]["zscore"] = [0.0, 3.0] + data["BTCUSDT"]["sell_trigger"] = 2.5 + result = self.run_fixture(data, [{}, {}], dca=True) + self.assertEqual(result["trades"], 2) + self.assertEqual(result["equity_series"].tolist(), [10000.0, 10000.0]) + + def test_forward_filled_quote_can_be_used_for_a_new_trade(self): + data = self.market([100.0, 100.0, 110.0]) + raw = pd.DataFrame({"time": self.times[[0, 2]], "close": [100.0, 110.0]}) + aligned = backtest.align_symbol_data(raw, pd.DataFrame(), self.times) + data["ETHUSDT"]["close"] = aligned.close + self.assertNotIn(self.times[1], raw.time.tolist()) + result = self.run_fixture(data, [{}, {"ETHUSDT": 1.0}, {"ETHUSDT": 1.0}]) + self.assertEqual(result["trades"], 1) + self.assertEqual(result["equity_series"].tolist(), [10000.0, 10000.0, 10098.5]) + + def test_equity_curve_has_only_post_processing_rows_no_initial_anchor(self): + result = self.run_fixture(self.market([100.0, 90.0]), [{"ETHUSDT": 1.0}] * 2) + curve = result["equity_series"] + self.assertEqual(curve.index.tolist(), self.times[:2].tolist()) + self.assertEqual(len(curve), 2) + self.assertAlmostEqual(result["max_drawdown"], 9901.5 / 10000.0 - 1.0) + # This zero-cost fixture cannot expose a first-fill loss. The missing + # pre-fill capital anchor remains a future cost-model acceptance case. + + +if __name__ == "__main__": + unittest.main()