Skip to content

[BUG] slotsMissed_int counts standby interval as missed slots after forging is disabled and re-enabled #6698

Description

@Straightpool

Internal/External
External

Area
Other

Summary
cardano_node_metrics_slotsMissed_int appears to incorrectly count slots that elapsed while forging was intentionally disabled as missed slots when forging is subsequently re-enabled.

The issue does not occur on the first transition from the node's initial non-producing state into producing mode.

It occurs after the following sequence:

initial non-producing -> producing -> non-producing -> producing

After approximately 4 minutes in non-producing mode, slotsMissed_int jumped from 0 to 241 when forging was re-enabled and then remained stable during normal producing operation.

Relevant test log lines:

2026-09-20T22:22:02+02:00 slotsMissed=0 forging_enabled=1
2026-09-20T22:22:03+02:00 slotsMissed=0 forging_enabled=0

...

2026-09-20T22:26:00+02:00 slotsMissed=0 forging_enabled=0
2026-09-20T22:26:01+02:00 slotsMissed=0 forging_enabled=1
2026-09-20T22:26:02+02:00 slotsMissed=241 forging_enabled=1
2026-09-20T22:26:03+02:00 slotsMissed=241 forging_enabled=1
2026-09-20T22:26:04+02:00 slotsMissed=241 forging_enabled=1

Steps to reproduce

Forging is switched between active and standby states by making the forging credentials available/unavailable and sending SIGHUP to the running cardano-node process (kill -s HUP <pid>). The cardano-node process must remain running throughout the entire non-producing -> producing -> non-producing -> producing sequence; restarting the node between state changes does not reproduce the issue.

  1. Start a block-producing node with valid KES/VRF/opcert credentials using --non-producing-mode.

  2. Verify that the node is initially non-producing.

  3. With the forging credentials already present from startup, send SIGHUP to the running cardano-node process (kill -s HUP <pid>) to dynamically enable forging.

  4. Verify that:

    cardano_node_metrics_forging_enabled_int = 1
    cardano_node_metrics_slotsMissed_int = 0
    
  5. Make the forging credentials unavailable and send SIGHUP again to dynamically disable forging.

  6. Verify that:

    cardano_node_metrics_forging_enabled_int = 0
    cardano_node_metrics_slotsMissed_int = 0
    
  7. Leave the node in this intentionally non-producing state for several minutes.

  8. Make the forging credentials available again and send SIGHUP to dynamically re-enable forging.

  9. Observe the metrics immediately after cardano_node_metrics_forging_enabled_int changes from 0 to 1.

  10. Observe that cardano_node_metrics_slotsMissed_int increases approximately by the number of slots elapsed during the last period in which forging was disabled.

    In my reproduction test:

    2026-09-20T22:22:02+02:00 slotsMissed=0 forging_enabled=1
    2026-09-20T22:22:03+02:00 slotsMissed=0 forging_enabled=0
    
    ...
    
    2026-09-20T22:26:00+02:00 slotsMissed=0 forging_enabled=0
    2026-09-20T22:26:01+02:00 slotsMissed=0 forging_enabled=1
    2026-09-20T22:26:02+02:00 slotsMissed=241 forging_enabled=1
    2026-09-20T22:26:03+02:00 slotsMissed=241 forging_enabled=1
    
  11. Leave the node producing normally for several minutes.

  12. Observe that cardano_node_metrics_slotsMissed_int remains stable after the initial jump.

Expected behavior
Slots that elapse while forging is intentionally disabled should not be counted as missed leadership checks.

After forging is re-enabled, cardano_node_metrics_slotsMissed_int should continue from its previous value and should only increase for leadership checks that are actually missed while the node is in producing mode.

In the reproduction above, the expected value of cardano_node_metrics_slotsMissed_int after re-enabling forging should therefore still be 0, not 241.

System info:

  • OS Name: Ubuntu

  • OS Version 24.04.4 LTS

  • Node version:

      cardano-node 11.1.2 - linux-x86_64 - ghc-9.6
      git rev fef83fed01d7926f3de83b3b917be5a4a48768b5
    
  • CLI version

     cardano-cli 11.2.3.0 - linux-x86_64 - ghc-9.6
     git rev fef83fed01d7926f3de83b3b917be5a4a48768b5
    

Screenshots and attachments
Historical Grafana view of cardano_node_metrics_slotsMissed_int on my failover block producer. The metric shows large discrete increases after extended flat periods. The size of such an increase approximately matches the duration for which forging had intentionally been disabled, as both axes track seconds.

Image

Additional context
A related dynamic-forging metrics issue was previously reported in #6113. That issue affected failover operation itself, while this issue is limited to incorrect slotsMissed_int monitoring after forging is disabled and re-enabled.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageIssue / PR needs to be triaged.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions