feat: add Controller Internals Grafana dashboard

[rawadhossain]

Description

This PR adds the Controller Internals section, fifth of five planned dashboard sections for node-readiness-controller.

Questions it answers:

• Is the controller up and reachable?
• Is it keeping up with incoming reconcile work?
• Are workqueues building up or taking too long to process?
• Are reconciliations failing?
• Which controller version is currently running?

Most of these panels use metrics from controller-runtime and kube-state-metrics.

Panels in this Section

• Controller Up - controller availability.
• Workqueue Depth - queued reconcile work per controller.
• Workqueue Latency (p99) - p99 time items spend waiting in the queue.
• Reconcile Errors - failed reconcile attempts per second.
• Bootstrap Completions - total bootstrap completions since the controller started.
• Build Info - deployed version and pod identity.

Screenshot

Video demo

Section.5.mp4

Related to Issue #459

Type of Change

/kind feature

Testing

• Verified all panels render correctly against live data in a kind + kwok test cluster
• Confirmed the setup is reproducible from a clean cluster

[netlify]

✅ Deploy Preview for node-readiness-controller canceled.

Name	Link
🔨 Latest commit	9dcb599
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a9fc8afc77dd0000832cae1

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please assign mrunalp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rawadhossain]

/cc @AvineshTripathi

[AvineshTripathi]

initial look! I will need some time here to understand. Too many variables here!

[AvineshTripathi]

There can be a different name for the job for e.g. https://github.com/kubernetes-sigs/node-readiness-controller/blob/main/charts/node-readiness-controller/templates/servicemetrics.yaml#L5 which might break this

[rawadhossain]

my bad, that was for my local testing. Forgot to change it.
Fixed it. Should be fine now.

[AvineshTripathi]

we would need a text box for this or dropdown to select job?

[rawadhossain]

Hey @AvineshTripathi, here's a quick breakdown so its easier to skim.

{
  "id": 10,                    → unique panel id, ignore it
  "type": "timeseries",          → line graph (vs "stat" = big number, "table" = table)
  "title": "Workqueue Depth",
  "gridPos": {...},               → x/y position + size on the grid, just layout
  
  "targets": [
    { "expr": "sum by (controller) (workqueue_depth{job=\"$job\"})" }
  ],                 → the actual PromQL query — this is the only line that defines what the panel shows
 
 "fieldConfig": {
    "defaults": {
      "unit": "short",              → how the number's displayed
      "min": 0, "max": 1200,   → axis bounds
      "thresholds": {
        "steps": [            → color bands, kicks in once value crosses these
          green → null (default),
          yellow → 500,
          red → 1000
        ]
      }                        
    },
  
  "overrides": [
      "node" line → solid,
      "nodereadiness-controller" line → dashed
    ]               → per-series exceptions, so the two lines are visually distinguishable even if same color
  },
  "options": {...}        → position, tooltip behavior, cosmetic only
}

so overall, it's just: one query per panel, wrapped in a bit of formatting (colors, units, layout) that repeats the same way across all of them.

[AvineshTripathi]

am not an expert in this but it LGTM