Skip to content

HYPERFLEET-1482 - feat: add API auth failure observability #291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kuudori wants to merge 1 commit into
base: main
Choose a base branch
from
Open

HYPERFLEET-1482 - feat: add API auth failure observability #291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/alerts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,25 @@ annotations:
description: "More than 10% of events are failing for {{ $labels.component }}."
```

### HyperFleet API Authentication or Authorization Failures

```yaml
alert: HyperFleetAdapterAPIAuthFailures
expr: |
sum by (component, version, adapter_name, status_code) (
increase(hyperfleet_adapter_api_auth_failures_total[5m])
) >= 3
labels:
severity: critical
annotations:
summary: "HyperFleet Adapter API authentication or authorization failures"
description: "The HyperFleet API returned HTTP {{ $labels.status_code }} to {{ $labels.adapter_name }}. Check its service-account token, API gateway configuration, and subject allowlist. When tenant enforcement is enabled, verify tenant dimensions are present and non-empty and tenant headers are propagated."
```

**Impact:** The affected adapter cannot complete API-backed parameter extraction, preconditions, or post-actions until its identity is accepted.

**Response:** For HTTP 403 responses with tenant enforcement enabled, verify the gateway-injected tenant dimensions before changing the service-account token or subject allowlist. See [API Authentication and Authorization Failures](runbook.md#api-authentication-and-authorization-failures).

### No Events Processed (Dead Man's Switch)

```yaml
Expand Down
9 changes: 9 additions & 0 deletions docs/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ All adapter metrics include `component` and `version` as constant labels. Event-
| `hyperfleet_adapter_events_processed_total` | Counter | `component`, `version`, `adapter_name`, `status` | Total CloudEvents processed. Status: `success`, `failed`, `skipped` |
| `hyperfleet_adapter_event_processing_duration_seconds` | Histogram | `component`, `version`, `adapter_name` | End-to-end event processing duration |
| `hyperfleet_adapter_errors_total` | Counter | `component`, `version`, `adapter_name`, `error_type` | Total errors by execution phase |
| `hyperfleet_adapter_api_auth_failures_total` | Counter | `component`, `version`, `adapter_name`, `status_code` | Total HyperFleet API authentication and authorization failures. `status_code` is always `401` or `403` |

#### Status Values

Expand Down Expand Up @@ -103,6 +104,14 @@ Error rate by phase:
sum by (error_type) (rate(hyperfleet_adapter_errors_total[5m]))
```

HyperFleet API authentication and authorization failures:

```promql
sum by (status_code) (
rate(hyperfleet_adapter_api_auth_failures_total{status_code=~"401|403"}[5m])
)
```

## Broker Metrics

The adapter automatically registers Prometheus metrics from the [hyperfleet-broker](https://github.com/openshift-hyperfleet/hyperfleet-broker) library.
Expand Down
17 changes: 13 additions & 4 deletions docs/runbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,14 +181,23 @@ All event failures are ACKed (not retried) to avoid infinite loops on non-transi

The adapter retries on 5xx, 408 (Request Timeout), and 429 (Too Many Requests) with configurable backoff (exponential, linear, or constant).

#### API Authentication and Authorization Failures

**Symptoms:** `hyperfleet_adapter_api_auth_failures_total{status_code=~"401|403"}` is increasing. Error logs include `http_status` and either `phase` (parameter extraction or preconditions) or `post_action`, with the affected resource context.

HTTP 401 and 403 failures are ACKed under the normal event-failure policy; they are not redelivered automatically. Redelivery cannot repair invalid credentials, subject allowlists, or tenant dimensions.

**Remediation:** For HTTP 403 responses with tenant enforcement enabled, verify the gateway-injected tenant dimensions before changing the projected service-account token or subject allowlist. If they are correct, check the token and allowlist, then rely on normal upstream reconciliation.

@ciaranRoche ciaranRoche Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"rely on normal upstream reconciliation" is vague for someone on call at 3am. The thing that resyncs the resource whose status write was dropped is the HyperFleet operator, so say that: "once credentials are fixed, the HyperFleet operator will reconcile the affected resources; no redelivery is needed." That also tells them where to look if convergence does not happen.


**Steps:**
1. Check HyperFleet API health: `kubectl get pods -l app=hyperfleet-api`
2. Check API response times from the adapter pod:
1. With tenant enforcement enabled, verify gateway-injected tenant dimensions match the adapter's tenant.
2. Check HyperFleet API health: `kubectl get pods -l app=hyperfleet-api`
3. Check API response times from the adapter pod:
```bash
kubectl exec <pod> -- curl -s -o /dev/null -w "%{http_code} %{time_total}s" http://hyperfleet-api:8000/healthz
```
3. Check for resource exhaustion on the API service
4. Review `retryAttempts` and `timeout` in adapter config
4. Check for resource exhaustion on the API service
5. Review `retryAttempts` and `timeout` in adapter config

---

Expand Down
Loading