Load model systematically underestimates on sites with large PV #739

Symptom

A site running active arbitrage planned every afternoon slot against a house load of 383 W. The house was drawing 7.9 kW. The plan was internally consistent — 383 W − 11 720 W + 1 060 W = −10 277 W, matching the plan's own GRID −10.29 kW — so the optimizer solved the problem it was given correctly. It was given the wrong house.

Consecutive slots from that plan:

The user's own energy export puts real consumption in that window at ~3.0 kW (consumer_use 756.6 Wh per 15 min bucket), against a forecast of 383 W. The next day's plan predicted 523 / 771 / 1230 W for the same hours.

Downstream, the forecast error drives the visible complaint: real load misses the forecast by more than TwinDriftLoadW (200 W, mpc/service.go:255) on essentially every tick, so reactive replans fire constantly and the live setpoint keeps diverging from whatever plan the dashboard is showing.

Mechanism

Service.sampleAt computes the training sample and drops it when it comes out negative — loadmodel/service.go:344:

loadW := gridW - pvW - batW - evW - v2xW
if loadW < 0 {
    // Almost always a transient — during a PI step the measured
    // flow can briefly appear negative. Skip rather than train
    // on a physically impossible value.
    return
}

Model.Update applies the same guard again (loadmodel/model.go:240).

The guard is right about the individual sample: negative load is not physical. But the discarded samples are not a random subset. On a site with 11.7 kW of PV, loadW is a small difference between two large, independently-sampled quantities. Meter and inverter readings are not synchronised and each carries its own noise, so the residual scatters around the true load with an error comparable to the load itself. Every draw that lands below zero is deleted; every draw that lands low-but-positive is kept and trained on.

That is censored sampling, and the resulting estimator is biased low by construction. The bias scales with PV size relative to house load — which is why this site, with the largest array, shows it worst, and why a small-PV site never notices.

Two things then lock it in:

1. The outlier filter (model.go:277) rejects residuals beyond max(MAE × 10, 200) once Samples > 50. After the mean has converged low, MAE is small, so genuine high-load samples start looking like outliers and are rejected too. The model becomes confidently wrong: small reported error, large real error.
2. PV drivers that go offline are skipped in the PV sum while gridW is still used (service.go:334). With one of several strings down, pvW is too small, loadW too small, and the sample is still trained on if it stays positive.

repairPoisonedBuckets (model.go:185) already exists to undo a related poisoning from the heating-subtraction bug, so the failure mode is not new to this file — just not covered for this cause.

Why it matters

Every planner strategy sizes its slots against this forecast. An 8× underestimate does not degrade the plan gracefully; it produces a plan for a different building. The optimizer, the dispatcher and the safety layer then all behave correctly and the result still looks broken to the user — which is exactly what makes it expensive to diagnose from the outside.

Directions

Not prescribing a fix, but the options that look sound:

• Keep the censored samples as information. A negative residual is evidence that the true load is low, not evidence to be thrown away. Clamping to zero and training on it is biased in the other direction, but a Tobit-style update, or tracking the discard rate and correcting the mean by it, uses the signal without inventing data.
• Weight by measurement confidence. When |pvW| greatly exceeds the residual, the sample carries little information about load; down-weight rather than accept it at full strength.
• Reject on staleness, not on sign. Much of the negative-residual scatter is meter and inverter readings from different instants. Requiring the contributing readings to be close in time removes the cause instead of filtering the symptom.
• Surface the discard rate. slog.Debug("loadmodel: skip (neg load)") is invisible in production. A counter on /api/loadmodel would have made this diagnosable without reading the source.

Detection

The help report added in #TBD flags this at the top of its Findings when the model's prediction and live load diverge by more than 60%, with both numbers quoted. That makes the symptom visible; it does not fix the model.

Repro

Any site where PV materially exceeds house load for a large part of the day. Compare /api/loadmodel (samples, mae_w, quality) with load_w versus load_w_predicted from /api/status across a sunny afternoon. A small mae_w alongside a large live gap is the signature.