Adding an existing security at a finer resolution after Initialize() never creates the data subscription (backtest keeps the original resolution only)

[AlexCatarino]

Summary

Adding an already-subscribed security again at a finer resolution after Initialize() (e.g. AddCryptoFuture(ticker, Resolution.Minute) from a scheduled event for a symbol that was added at Resolution.Hour during Initialize()) registers the new SubscriptionDataConfigs but never creates a data-feed subscription for them in backtesting. The algorithm keeps receiving the hourly bars only; the minute bars never arrive. The same call made during Initialize() works.

Reproduced on LEAN 2.5.0.0.18057 (cloud backtest, Python), Binance crypto futures, 2024-06-01 to 2024-06-04.

Reproduction

class RuntimeResolutionUpgradeRepro(QCAlgorithm):
    def initialize(self):
        self.set_start_date(2024, 6, 1)
        self.set_end_date(2024, 6, 4)
        self.set_account_currency("USDT")
        self.set_cash(50_000)
        self._counts = {}
        # init-time dual resolution: works (minute bars delivered)
        for ticker in ["LINKUSDT", "AVAXUSDT"]:
            h = self.add_crypto_future(ticker, Resolution.HOUR, Market.BINANCE, fill_forward=False)
            self.add_crypto_future(ticker, Resolution.MINUTE, Market.BINANCE)
            self._counts[h.symbol] = {"minute": 0, "hour": 0}
        # runtime upgrade: minute config registered, no minute data ever delivered
        for ticker in ["1000BONKUSDT", "DOTUSDT"]:
            h = self.add_crypto_future(ticker, Resolution.HOUR, Market.BINANCE, fill_forward=False)
            self._counts[h.symbol] = {"minute": 0, "hour": 0}
        self.schedule.on(self.date_rules.on(2024, 6, 2), self.time_rules.at(2, 20), self._promote)

    def _promote(self):
        for ticker in ["1000BONKUSDT", "DOTUSDT"]:
            s = self.add_crypto_future(ticker, Resolution.MINUTE, Market.BINANCE)
            cfgs = self.subscription_manager.subscription_data_config_service.get_subscription_data_configs(s.symbol)
            self.log(f"{ticker}: configs now " + ", ".join(f"{c.resolution}/{c.tick_type}" for c in cfgs))

    def on_data(self, data):
        for symbol, bar in data.bars.items():
            e = self._counts.get(symbol)
            if e is None:
                continue
            if bar.period == timedelta(minutes=1):
                e["minute"] += 1
            elif bar.period == timedelta(hours=1):
                e["hour"] += 1

    def on_end_of_algorithm(self):
        for symbol, e in self._counts.items():
            self.log(f"{symbol.value}: minute={e['minute']} hour={e['hour']}")

Output:

1000BONKUSDT: configs now HOUR/TRADE, HOUR/QUOTE, HOUR/QUOTE, MINUTE/TRADE, MINUTE/QUOTE, MINUTE/QUOTE
DOTUSDT: configs now HOUR/TRADE, HOUR/QUOTE, HOUR/QUOTE, MINUTE/TRADE, MINUTE/QUOTE, MINUTE/QUOTE
LINKUSDT: minute=5761 hour=0
AVAXUSDT: minute=5761 hour=0
1000BONKUSDT: minute=0 hour=97
DOTUSDT: minute=0 hour=97

The init-time pair gets its minute bars (and, as designed in TimeSliceFactory, the slice keeps the finer bar when both exist). The runtime-upgraded pair keeps delivering hourly bars only, even though the minute configs are registered in the SubscriptionDataConfigService.

Where it goes wrong

• QCAlgorithm.AddSecurity -> AddToUserDefinedUniverse (Algorithm/QCAlgorithm.Universe.cs, ~L580) queues a UserDefinedUniverseUpdate with the new configs; ProcessUniverseChanges (~L149) then calls UserDefinedUniverse.Add(SubscriptionDataConfig) for each, which adds the config to _subscriptionDataConfigs and raises CollectionChanged because it is new.
• The data feed re-runs UniverseSelection.ApplyUniverseSelection (Engine/DataFeeds/UniverseSelection.cs), but the "find new selections" loop (~L255) does if (universe.Securities.ContainsKey(symbol)) continue; - the symbol is already a member, so universe.GetSubscriptionRequests(...) (which would return both the hour and the new minute configs) is never called and _dataManager.AddSubscription never sees the minute config.
• At start-up the same path works because the symbol is not yet a member when selection first runs, and UserDefinedUniverse.GetSubscriptionRequests (Common/Data/UniverseSelection/UserDefinedUniverse.cs, ~L225) returns every config registered for the symbol.

So a second resolution added for an existing member silently becomes a config with no subscription behind it. Nothing warns the user.

Proposed change

In UniverseSelection.ApplyUniverseSelection, when a selected symbol is already a universe member, still ask a UserDefinedUniverse for its subscription requests and add any SubscriptionDataConfig that the data manager does not already have a subscription for (the security.Subscriptions.Contains(request.Configuration) / _dataManager.AddSubscription(request) block that already runs for new members). Alternatively, AddToUserDefinedUniverse could treat a new config for an existing member as a remove + re-add of the member. Either way, a regression algorithm along the lines of the reproduction above (hour at init, minute added at runtime, assert minute bars arrive) would cover it; ForexMultiResolutionRegressionAlgorithm only covers the init-time case.

Context

Reported via Intercom conversation 215475797776311. The reporter also states the equivalent runtime upgrade does deliver minute bars in live/paper trading (not verified by us).