The latent loop detector works by encoding observations into a 16D hyperspherical latent space, then checking whether the agent has visited a "nearby" latent state before. Two failure modes exist:
| Failure mode | What happens | Root cause |
|---|---|---|
| False negative | Agent is stuck in a loop but the detector doesn't fire | Different observations of the same game state map too far apart in latent space, or the detection threshold is too tight |
| False positive | Agent is playing normally but the detector fires and overrides the action | Different game states collapse to nearby points in latent space, or the detection threshold is too loose |
Both are harmful — false negatives defeat the purpose of the detector, while false positives degrade a good policy by injecting unnecessary action overrides.
We don't have a labelled dataset of "the agent is stuck here" vs "the agent is progressing." Instead we construct a proxy evaluation from the offline transition dataset using two pools of latent-distance pairs:
NOOP transitions — transitions where the agent took the NOOP action. The game
state doesn't change; only distractors (ghosts, timers, animations) move. A good
encoder should map z_t and z_{t+1} to nearly identical points:
||z_t − z_{t+1}|| should be ≈ 0 when action = NOOP
If the detector can't even recognise that a NOOP didn't change the state, it will certainly fail at recognising real loops (where the agent revisits a state after several steps).
Random pairs — we sample random (z_i, z_j) from different timesteps. Most
random pairs correspond to genuinely different game states, so a good encoder
should map them far apart:
||z_i − z_j|| should be > threshold for random i ≠ j
If random states land too close together, the detector will fire on a good policy that is simply revisiting a region of the observation space — the false positive problem.
For a given trained encoder, we sweep a detection threshold from 0.01 to 1.0 and at each threshold compute standard binary-classification metrics treating NOOP pairs as positives and random pairs as negatives:
| Metric | Definition | What it measures |
|---|---|---|
| True positive (TP) | NOOP pair with ` | |
| False positive (FP) | Random pair with ` | |
| False negative (FN) | NOOP pair with ` | |
| Precision | TP / (TP + FP) |
"When the detector fires, how often is it right?" |
| Recall | TP / (TP + FN) |
"Of all real loops, how many does it catch?" |
| F1 | 2 · Precision · Recall / (Precision + Recall) |
Harmonic mean — balances both errors |
The threshold that maximizes F1 is reported as the best detection threshold for that encoder configuration.
The F1 at the optimal threshold. This single number captures how well the encoder separates same-state pairs from different-state pairs. High F1 means the encoder has found a latent structure where there exists a threshold that simultaneously catches real loops (high recall) without triggering on normal play (high precision).
At the optimal threshold, what fraction of reidentification events are correct.
Directly measures the false-positive rate: FP rate = 1 − Precision.
A model with 0.95 precision fires incorrectly 5% of the time. On a good policy that takes thousands of steps, even 5% false positives means dozens of unnecessary action overrides per episode.
At the optimal threshold, what fraction of NOOP same-state pairs are correctly
reidentified. Directly measures the false-negative rate: FN rate = 1 − Recall.
Fraction of random (negative) pairs whose latent distance exceeds the detection threshold. This is the "headroom" metric — a model with 0.99 margin satisfaction has its different-state embeddings well-separated from the threshold boundary, making the detector robust to noise.
Poor margin satisfaction is the root cause of false positives: different states that land near the threshold boundary will sometimes be misidentified.
Fraction of correctly predicted actions from (z_t, z_{t+1}) pairs. The inverse
model acts as a regularizer during training — high inverse accuracy indicates the
latent space has an orderly structure where transitions are action-discriminative
(different actions produce distinguishable latent transitions).
Mean squared error between predicted ẑ_{t+1} = forward(z_t, a_t) and actual
z_{t+1} = encoder(s_{t+1}). Lower is better. Good forward prediction means
the latent dynamics are learnable and the encoder isn't encoding noise/distractors.
Reported as fwd_quality = 1 / (1 + 10 · forward_mse) to squash into [0, 1].
The composite score is the single scalar that Optuna maximizes. It is a weighted sum designed to balance false-positive and false-negative control:
composite = 0.35 × F1
+ 0.15 × Precision
+ 0.10 × Recall
+ 0.20 × Margin satisfaction
+ 0.10 × Inverse accuracy
+ 0.10 × Forward quality
-
F1 (35%) — Primary signal. Naturally balances precision and recall. Prevents the optimizer from gaming one at the expense of the other.
-
Precision (15%) + Margin satisfaction (20%) = 35% total toward false-positive control. This is intentional: the current detector has too many false positives (triggering on a good policy). Margin satisfaction is weighted higher than precision because it measures the structural separation rather than just the threshold-dependent classification.
-
Recall (10%) — Still present but lower-weighted. We don't want to ignore false negatives entirely, but the F1 term already incorporates recall.
-
Inverse accuracy (10%) + Forward quality (10%) — Latent representation quality. These ensure the optimizer doesn't find a degenerate encoder that happens to separate NOOP pairs but has useless dynamics.
| Score range | Quality |
|---|---|
| 0.00 – 0.20 | Poor — likely collapsed latent space or random encoder |
| 0.20 – 0.40 | Weak — some structure but unreliable detection |
| 0.40 – 0.60 | Moderate — usable but expect noticeable error rate |
| 0.60 – 0.80 | Good — balanced detection with manageable error rates |
| 0.80 – 1.00 | Excellent — tight clustering, strong separation |
| Parameter | Range | Why it matters |
|---|---|---|
lr |
1e-5 to 5e-3 (log) | Controls convergence speed and final loss landscape region |
margin |
0.05 to 2.0 | Contrastive push distance — too large spreads everything apart (hurts recall), too small collapses states (hurts precision) |
w_margin |
0.1 to 10.0 (log) | Relative weight of contrastive separation loss |
w_inverse |
0.1 to 10.0 (log) | Relative weight of inverse model regularization |
w_forward |
0.1 to 10.0 (log) | Relative weight of forward dynamics loss |
latent_dim |
8, 16, 32 | Capacity of the latent space — higher dims give more room for separation but risk encoding noise |
hidden_dim_inverse |
16, 32, 64, 128 | Inverse model capacity |
hidden_dim_forward |
128, 256, 512 | Forward model capacity |
batch_size |
64, 128, 256 | Larger batches give better contrastive gradients (more negative pairs per update) |
epochs |
Fixed via --epochs (default 15) |
Same training length for every trial |
updates_per_epoch |
100, 250, 500 | Gradient steps per epoch |
The detection threshold is not an Optuna parameter — it's swept internally after each trial's training completes, and the best threshold is reported alongside the model. This avoids wasting trials on threshold-only variation.
The current detector only compares latent distances, which is position-sensitive but not orientation-sensitive. An agent facing left vs right at the same position may map to the same latent point, causing the detector to fire even though the agent is making progress (a false positive). Orientation-aware detection is planned as a follow-up — it will likely require augmenting the latent state with a directional component or tracking short trajectory windows rather than single states.