OLS-3990 Restore OTEL Collector reconciliation and add service-ca trust for tracing

.ai/spec/what/audit-logging.md

@@ -6,14 +6,14 @@ Collector / Postgres storage and OTEL hub behavior: see `what/templog.md`.
 
 ## Architecture
 
-Audit configuration is split between **service** (stdout JSON events) and **collector** (Postgres storage, optional external trace forwarding). The operator generates `olsconfig.yaml` for lightspeed-service from service fields only; `spec.audit` is collector-only. Service trace export to the in-cluster collector is currently **disabled** (OLS-3737).
+Audit configuration is split between **service** (stdout JSON events, trace export to in-cluster collector) and **collector** (Postgres storage, optional external trace forwarding). The operator generates `olsconfig.yaml` for lightspeed-service from service fields only; `spec.audit` is collector-only.
 
-```yaml
+```
 spec.ols.auditEventsEnabled  →  olsconfig.yaml audit.logging (Enabled|Disabled)
                                →  stdout compliance JSON
 
-(disabled — OLS-3737)          →  olsconfig.yaml audit.otel.endpoint
-                               →  NOT injected until e2e coverage exists
+always                         →  olsconfig.yaml audit.otel.endpoint
+                               →  lightspeed-otel-collector.<ns>.svc:4317 (TLS)
 
 spec.audit.logging             →  collector Postgres pipeline (OLS-3510+)
 spec.audit.tracingEndpoint     →  collector external trace export (OLS-3510+)
@@ -47,15 +47,17 @@ spec:
 | olsconfig.yaml key | Source | Default |
 |---|---|---|
 | `audit.logging` | `spec.ols.auditEventsEnabled` | `Enabled` |
+| `audit.otel.endpoint` | operator constant | `lightspeed-otel-collector.<namespace>.svc:4317` |
+| `audit.otel.tls_mode` | operator constant | `Secure` |
 
-7. **OLS-3737**: OTEL endpoint injection (`audit.otel.endpoint`, `audit.otel.tls_mode`) is **disabled** until e2e tests prove the collector pipeline works end-to-end. The service falls back to a no-op tracer when the otel section is absent. Re-enablement is tracked in OLS-3737 Phase 3.
+7. Trace export to the in-cluster collector is **always** configured (gRPC OTLP). External backends are reached via the collector (`spec.audit.tracingEndpoint`), not by setting a service endpoint on the CR.
 8. `spec.audit` MUST NOT affect generated `olsconfig.yaml` audit settings.
 9. Changes to `spec.ols.auditEventsEnabled` MUST trigger reconciliation that regenerates `olsconfig.yaml` and rolls the app-server deployment.
-10. The operator mounts the OpenShift service-ca bundle (`openshift-service-ca.crt`) at `/etc/certs/otel-collector-ca/service-ca.crt` in the app-server, adds it to `extra_ca`, and sets `OTEL_EXPORTER_OTLP_CERTIFICATE` to that path. These mounts are **retained** even while OTEL export is disabled (OLS-3737) to simplify Phase 3 re-enablement. See `tls.md`.
+10. The operator mounts the OpenShift service-ca bundle (`openshift-service-ca.crt`) at `/etc/certs/otel-collector-ca/service-ca.crt` in the app-server, adds it to `extra_ca`, and sets `OTEL_EXPORTER_OTLP_CERTIFICATE` to that path for OTLP/gRPC trust. See `tls.md`.
 
 ### Reconciliation
 
-11. The operator does not emit its own audit events. Its responsibilities are CRD schema and `olsconfig.yaml` generation for stdout audit config. OTEL Collector operand reconciliation (`OtelCollectorReady`) and in-cluster trace export are currently **disabled** (OLS-3737); see `reconciliation.md` and `templog.md`.
+11. The operator does not emit its own audit events. Its responsibilities are CRD schema, `olsconfig.yaml` generation for stdout audit and in-cluster trace export, and OTEL Collector operand reconciliation (`OtelCollectorReady`). See `reconciliation.md` and `templog.md`.
 
 ## Migration (breaking change)
 
@@ -64,8 +66,8 @@ The previous `spec.audit.logging` (`Enabled`/`Disabled`) and `spec.audit.otel` b
 | Previous | New |
 |---|---|
 | `spec.audit.logging: Enabled/Disabled` | `spec.ols.auditEventsEnabled: true/false` |
-| `spec.audit.otel.endpoint` | removed — operator-injected endpoint (currently disabled, OLS-3737) |
-| `spec.audit.otel.tlsMode: Insecure` | removed — TLS mode `Secure` when re-enabled (OLS-3737 Phase 3) |
+| `spec.audit.otel.endpoint` | removed — service always exports to in-cluster collector |
+| `spec.audit.otel.tlsMode: Insecure` | removed — service always uses `Secure` to collector |
 | (none) | `spec.audit.tracingEndpoint` — external trace export via collector |
 
 Existing CRs with the old `spec.audit` shape must be rewritten manually before upgrade. There is no conversion webhook.

.ai/spec/what/templog.md

@@ -4,14 +4,14 @@ Implementation details for the lightspeed-operator's role in the templog / OTEL
 
 ## Architecture
 
-The OTEL Collector is the in-cluster telemetry hub. Its reconciliation is currently **disabled** (OLS-3737) until e2e tests prove the pipeline works; it will not be deployed on new installs. Configuration is split between **service behavior** (stdout audit events) and **collector behavior** (Postgres storage, trace forwarding).
+The OTEL Collector is the in-cluster telemetry hub. It is **always deployed** when Lightspeed is installed. Configuration is split between **service behavior** (stdout audit events, trace export to collector) and **collector behavior** (Postgres storage, trace forwarding).
 
 ```text
 lightspeed-service
   ├─ stdout JSON audit events     ← spec.ols.auditEventsEnabled
-  └─ OTLP traces (gRPC :4317)   ← disabled (OLS-3737); no-op tracer fallback
+  └─ OTLP traces (gRPC :4317)   ← always → lightspeed-otel-collector Service
 
-OTEL Collector (disabled — OLS-3737; re-enable in Phase 3)
+OTEL Collector (always deployed)
   ├─ logs pipeline → Postgres     ← spec.audit.logging (*bool, default true)
   │     (only service.name=lightspeed-agentic-sandbox)
   ├─ postgres_admin HTTPS :8080   ← always (templog cleanup / GET for agentic-operator)
@@ -64,10 +64,12 @@ The operator generates service audit config independently of `spec.audit`:
 | olsconfig.yaml | Source |
 |----------------|--------|
 | `audit.logging` | `spec.ols.auditEventsEnabled` (default Enabled) |
+| `audit.otel.endpoint` | Always `lightspeed-otel-collector.<ns>.svc:4317` |
+| `audit.otel.tls_mode` | Always `Secure` (OTLP/gRPC with TLS) |
 
-**OLS-3737**: OTEL endpoint injection (`audit.otel.endpoint`, `audit.otel.tls_mode`) is **disabled** until e2e tests prove the collector pipeline works. The service falls back to a no-op tracer when the otel section is absent. Re-enablement tracked in OLS-3737 Phase 3.
+The operator mounts the OpenShift service-ca bundle into the app-server at `/etc/certs/otel-collector-ca/service-ca.crt`, adds it to `extra_ca` in `olsconfig.yaml`, and sets `OTEL_EXPORTER_OTLP_CERTIFICATE` to that path (required for OTLP/gRPC; `extra_ca` alone is not used by the exporter). See `tls.md`.
 
-The operator still mounts the OpenShift service-ca bundle into the app-server at `/etc/certs/otel-collector-ca/service-ca.crt`, adds it to `extra_ca` in `olsconfig.yaml`, and sets `OTEL_EXPORTER_OTLP_CERTIFICATE` to that path. These mounts are retained for Phase 3 re-enablement. See `tls.md`.
+Service continues to use the existing gRPC OTLP trace exporter (`opentelemetry.exporter.otlp.proto.grpc`).
 
 ## Operator image flag ([OLS-3509](https://redhat.atlassian.net/browse/OLS-3509))
 
@@ -80,15 +82,13 @@ The operator still mounts the OpenShift service-ca bundle into the app-server at
 
 The Postgres bootstrap script creates only `quota` and `conversation_cache` schemas. It does **not** create the `templogs` schema or tables.
 
-When the collector is enabled, it creates and manages the `templogs` schema, `logs` table, and indexes via the `postgres_admin` extension at collector startup (`postgres_admin` is always enabled for clients). `spec.audit.logging` only controls whether new OTLP logs are exported into that schema. The operator never drops this schema. The `logs` table uses `agentic_run_id` (AgenticRun UID, normalized to 32-char hex) and `phase` (audit phase name) as the primary query dimensions, with a composite index on `(agentic_run_id, phase)`.
+The OTEL Collector always creates and manages the `templogs` schema, `logs` table, and indexes via the `postgres_admin` extension at collector startup (`postgres_admin` is always enabled for clients). `spec.audit.logging` only controls whether new OTLP logs are exported into that schema. The operator never drops this schema. The `logs` table uses `agentic_run_id` (AgenticRun UID, normalized to 32-char hex) and `phase` (audit phase name) as the primary query dimensions, with a composite index on `(agentic_run_id, phase)`.
 
 See `postgres.md` for Postgres bootstrap scope and `templog.md` (lightspeed-service repo) for table DDL semantics.
 
 ## Collector Operand ([OLS-3510](https://redhat.atlassian.net/browse/OLS-3510), [OLS-3513](https://redhat.atlassian.net/browse/OLS-3513), [OLS-3656](https://redhat.atlassian.net/browse/OLS-3656))
 
-> **OLS-3737**: Collector reconciliation is **disabled** — the rules below describe the target architecture when re-enabled in Phase 3. The collector code is commented out, not deleted.
-
-1. Deploy a single-replica Collector Deployment. Service exposes OTLP gRPC `:4317`, OTLP HTTP `:4318`, `postgres_admin` HTTPS `:8080`, and HTTPS Prometheus metrics `:8888`. Health check listens on `:13133` (pod-local; not on the Service).
+1. **Always** deploy a single-replica Collector Deployment. Service exposes OTLP gRPC `:4317`, OTLP HTTP `:4318`, `postgres_admin` HTTPS `:8080`, and HTTPS Prometheus metrics `:8888`. Health check listens on `:13133` (pod-local; not on the Service).
 2. Image from `GetOtelCollectorImage()`; pod scheduling from `spec.ols.deployment.otelCollector`.
 3. ConfigMap pipelines driven by `spec.audit`:
    - `logging` true/absent → logs pipeline with `routing/logs` connector and `postgresexporter`; only OTLP logs where `service.name == "lightspeed-agentic-sandbox"` are stored in Postgres; unmatched logs go to `logs/unmatched` → `nop`
@@ -116,7 +116,7 @@ Agentic-operator reads OTLP/admin endpoints from `lightspeed-agentic-configurati
 
 ## Constraints
 
-1. Collector is a single replica (when enabled).
+1. Collector is always a single replica.
 2. Collector container image is operator-managed via `--otel-collector-image` (not user-supplied in CR).
 3. `templogs` schema is created by the OTEL Collector, not Postgres bootstrap; the operator never drops it.
 4. Only sandbox audit logs (`service.name=lightspeed-agentic-sandbox`) are routed to Postgres.

config/default/deployment-patch.yaml

@@ -16,6 +16,9 @@
 - op: add
   path: /spec/template/spec/containers/0/args/-
   value: --openshift-mcp-server-image=__REPLACE_OPENSHIFT_MCP_SERVER__
+- op: add
+  path: /spec/template/spec/containers/0/args/-
+  value: --otel-collector-image=__REPLACE_LIGHTSPEED_OTEL_COLLECTOR__
 - op: add
   path: /spec/template/spec/containers/0/args/-
   value: --rhokp-image=__REPLACE_RHOKP__

hack/deploy-jaeger-tls.sh

@@ -0,0 +1,235 @@
+#!/usr/bin/env bash
+# Deploy Jaeger all-in-one with TLS on OpenShift using service-ca certificates.
+# The OTEL Collector's otlp/tracing exporter can then send traces securely.
+#
+# Usage: hack/deploy-jaeger-tls.sh [NAMESPACE]
+#   NAMESPACE defaults to "observability"
+#
+# To remove: hack/deploy-jaeger-tls.sh --delete [NAMESPACE]
+
+set -euo pipefail
+
+NAMESPACE="${2:-${1:-observability}}"
+JAEGER_IMAGE="quay.io/jaegertracing/jaeger:latest"
+
+if [[ "${1:-}" == "--delete" ]]; then
+  echo "Removing Jaeger from namespace ${NAMESPACE}..."
+  oc delete route jaeger-query -n "${NAMESPACE}" --ignore-not-found
+  oc delete service jaeger-query jaeger-otlp-grpc -n "${NAMESPACE}" --ignore-not-found
+  oc delete deployment jaeger -n "${NAMESPACE}" --ignore-not-found
+  oc delete configmap jaeger-config -n "${NAMESPACE}" --ignore-not-found
+  oc delete serviceaccount jaeger -n "${NAMESPACE}" --ignore-not-found
+  echo "Done. Namespace ${NAMESPACE} left in place."
+  exit 0
+fi
+
+echo "Deploying Jaeger all-in-one with TLS in namespace: ${NAMESPACE}"
+
+# Create namespace if it doesn't exist
+oc get namespace "${NAMESPACE}" &>/dev/null || oc create namespace "${NAMESPACE}"
+
+# ServiceAccount
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: jaeger
+  namespace: ${NAMESPACE}
+EOF
+
+# Service with service-ca annotation — OpenShift will auto-generate TLS certs
+# in a Secret named jaeger-otlp-tls, signed by the cluster's service-ca.
+oc apply -f - <<EOF
+apiVersion: v1
+kind: Service
+metadata:
+  name: jaeger-otlp-grpc
+  namespace: ${NAMESPACE}
+  annotations:
+    service.beta.openshift.io/serving-cert-secret-name: jaeger-otlp-tls
+  labels:
+    app: jaeger
+spec:
+  selector:
+    app: jaeger
+  ports:
+    - name: otlp-grpc
+      port: 4317
+      targetPort: 4317
+      protocol: TCP
+EOF
+
+# Jaeger query UI service
+oc apply -f - <<EOF
+apiVersion: v1
+kind: Service
+metadata:
+  name: jaeger-query
+  namespace: ${NAMESPACE}
+  labels:
+    app: jaeger
+spec:
+  selector:
+    app: jaeger
+  ports:
+    - name: query-http
+      port: 16686
+      targetPort: 16686
+      protocol: TCP
+EOF
+
+# Wait for the service-ca to generate the TLS secret
+echo "Waiting for service-ca to issue TLS certificate..."
+for i in $(seq 1 30); do
+  if oc get secret jaeger-otlp-tls -n "${NAMESPACE}" &>/dev/null; then
+    echo "TLS secret jaeger-otlp-tls is ready."
+    break
+  fi
+  if [[ $i -eq 30 ]]; then
+    echo "ERROR: Timed out waiting for jaeger-otlp-tls secret. Is the service-ca operator running?"
+    exit 1
+  fi
+  sleep 2
+done
+
+# Jaeger v2 config — uses OTEL Collector format internally
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: jaeger-config
+  namespace: ${NAMESPACE}
+  labels:
+    app: jaeger
+data:
+  config.yaml: |
+    receivers:
+      otlp:
+        protocols:
+          grpc:
+            endpoint: "0.0.0.0:4317"
+            tls:
+              cert_file: /etc/tls/tls.crt
+              key_file: /etc/tls/tls.key
+    exporters:
+      jaeger_storage_exporter:
+        trace_storage: memstore
+    extensions:
+      jaeger_storage:
+        backends:
+          memstore:
+            memory:
+              max_traces: 100000
+      jaeger_query:
+        http:
+          endpoint: "0.0.0.0:16686"
+        grpc:
+          endpoint: "0.0.0.0:16685"
+        storage:
+          traces: memstore
+    service:
+      extensions: [jaeger_storage, jaeger_query]
+      pipelines:
+        traces:
+          receivers: [otlp]
+          exporters: [jaeger_storage_exporter]
+EOF
+
+# Deployment — Jaeger v2 with TLS on the OTLP gRPC collector endpoint
+oc apply -f - <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: jaeger
+  namespace: ${NAMESPACE}
+  labels:
+    app: jaeger
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: jaeger
+  template:
+    metadata:
+      labels:
+        app: jaeger
+    spec:
+      serviceAccountName: jaeger
+      containers:
+        - name: jaeger
+          image: ${JAEGER_IMAGE}
+          args:
+            - --config=/etc/jaeger/config.yaml
+          ports:
+            - name: otlp-grpc
+              containerPort: 4317
+            - name: query-http
+              containerPort: 16686
+          volumeMounts:
+            - name: tls-certs
+              mountPath: /etc/tls
+              readOnly: true
+            - name: jaeger-config
+              mountPath: /etc/jaeger
+              readOnly: true
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 16686
+            initialDelaySeconds: 5
+            periodSeconds: 10
+      volumes:
+        - name: tls-certs
+          secret:
+            secretName: jaeger-otlp-tls
+        - name: jaeger-config
+          configMap:
+            name: jaeger-config
+EOF
+
+# Route for the Jaeger UI
+oc apply -f - <<EOF
+apiVersion: route.openshift.io/v1
+kind: Route
+metadata:
+  name: jaeger-query
+  namespace: ${NAMESPACE}
+  labels:
+    app: jaeger
+spec:
+  to:
+    kind: Service
+    name: jaeger-query
+  port:
+    targetPort: query-http
+  tls:
+    termination: edge
+    insecureEdgeTerminationPolicy: Redirect
+EOF
+
+# Wait for rollout
+echo "Waiting for Jaeger deployment to be ready..."
+oc rollout status deployment/jaeger -n "${NAMESPACE}" --timeout=120s
+
+# Print connection info
+ROUTE_HOST=$(oc get route jaeger-query -n "${NAMESPACE}" -o jsonpath='{.spec.host}')
+
+cat <<DONE
+
+========================================
+  Jaeger deployed successfully!
+========================================
+
+OTLP gRPC endpoint (for OLSConfig CR):
+  tracingEndpoint: "jaeger-otlp-grpc.${NAMESPACE}.svc.cluster.local:4317"
+
+Jaeger UI:
+  https://${ROUTE_HOST}
+
+Example OLSConfig patch:
+  oc patch olsconfig cluster --type=merge -p '{"spec":{"audit":{"tracingEndpoint":"jaeger-otlp-grpc.${NAMESPACE}.svc.cluster.local:4317"}}}'
+
+To remove:
+  hack/deploy-jaeger-tls.sh --delete ${NAMESPACE}
+
+DONE

internal/controller/agenticintegration/assets.go

@@ -114,17 +114,14 @@ func GenerateAgenticConfigurationConfigMap(r reconciler.Reconciler, cr *olsv1alp
 	}
 
 	ns := r.GetNamespace()
-	// OLS-3737: OTEL endpoint data disabled until e2e tests prove the
-	// collector pipeline works. Re-enable in OLS-3737 Phase 3.
-	// otelHost := fmt.Sprintf("%s.%s.svc", utils.OtelCollectorServiceName, ns)
+	otelHost := fmt.Sprintf("%s.%s.svc", utils.OtelCollectorServiceName, ns)
 
 	data := map[string]string{
-		utils.AgenticConfigurationSandboxModeKey:    string(SandboxModeFromCR(cr)),
-		utils.AgenticConfigurationSandboxPodSpecKey: string(podSpecJSON),
-		// OLS-3737: OTEL keys disabled until collector pipeline is proven.
-		// utils.AgenticConfigurationOtelCollectorEndpointKey: fmt.Sprintf("%s:%d", otelHost, utils.OtelCollectorGRPCPort),
-		// utils.AgenticConfigurationOtelAdminEndpointKey:     fmt.Sprintf("https://%s:%d", otelHost, utils.OtelCollectorAdminPort),
-		// utils.AgenticConfigurationOtelCASecretKey:          utils.AgenticOtelCASecretName,
+		utils.AgenticConfigurationSandboxModeKey:           string(SandboxModeFromCR(cr)),
+		utils.AgenticConfigurationSandboxPodSpecKey:        string(podSpecJSON),
+		utils.AgenticConfigurationOtelCollectorEndpointKey: fmt.Sprintf("%s:%d", otelHost, utils.OtelCollectorGRPCPort),
+		utils.AgenticConfigurationOtelAdminEndpointKey:     fmt.Sprintf("https://%s:%d", otelHost, utils.OtelCollectorAdminPort),
+		utils.AgenticConfigurationOtelCASecretKey:          utils.AgenticOtelCASecretName,
 	}
 	if utils.BoolDeref(cr.Spec.OLSConfig.IntrospectionEnabled, true) {
 		data[utils.AgenticConfigurationMCPEndpointKey] = utils.OpenShiftMCPServerServiceURL(ns)