docs: explain Envoy upstream Host port workaround

docs/en/solutions/How_to_Preserve_a_Non_Default_Backend_Port_in_the_Upstream_Host_Header_with_Envoy_Gateway_on_ACP.md

@@ -0,0 +1,247 @@
+---
+kind:
+  - Solution
+products:
+  - Alauda Container Platform
+ProductsVersion:
+  - 4.3.x and later
+tags:
+  - LB
+---
+
+# How to Preserve a Non-Default Backend Port in the Upstream Host Header with Envoy Gateway on ACP
+
+## Issue
+
+An Envoy Gateway route forwards traffic to an HTTPS backend that listens on a non-default port:
+
+```yaml
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: Backend
+metadata:
+  name: external-model
+  namespace: model-serving
+spec:
+  type: Endpoints
+  endpoints:
+    - fqdn:
+        hostname: model.example.com
+        port: 7448
+```
+
+The route also uses backend hostname rewriting, either directly or through a resource generated by Envoy AI Gateway:
+
+```yaml
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: HTTPRouteFilter
+metadata:
+  name: rewrite-host-to-backend
+  namespace: model-serving
+spec:
+  urlRewrite:
+    hostname:
+      type: Backend
+```
+
+Envoy connects to backend port `7448`, but the upstream HTTP `Host` header or HTTP/2 `:authority` contains only the hostname:
+
+```text
+model.example.com
+```
+
+If the upstream virtual host requires the non-default port, it expects:
+
+```text
+model.example.com:7448
+```
+
+The upstream may reject the request with an error such as:
+
+```text
+403 Host forbidden model.example.com:443
+```
+
+This error does not prove that Envoy connected to TCP port `443`. The backend socket port and HTTP authority are independent values.
+
+## Applicable Envoy Gateway Version
+
+The exact `EnvoyPatchPolicy` configuration in this document was validated with:
+
+```text
+Envoy Gateway: v1.8.0
+Envoy Proxy:   v1.38.0
+```
+
+`EnvoyPatchPolicy` modifies generated xDS configuration and is an unstable advanced API. For Envoy Gateway v1.7.x, v1.9.x, or another version, inspect that version's generated RouteConfiguration and adjust the resource name and route indexes before applying the policy.
+
+## Root Cause
+
+The Backend endpoint port and the upstream HTTP authority are configured independently:
+
+| Value | Example | Result |
+| --- | --- | --- |
+| Backend socket address | `model.example.com:7448` | Envoy connects to port `7448` |
+| HTTP `Host` or `:authority` | `model.example.com` | Used by the upstream virtual host |
+| TLS SNI | `model.example.com` | Contains a hostname only, never a port |
+
+`HTTPRouteFilter` with `hostname.type: Backend` is translated to:
+
+```yaml
+auto_host_rewrite: true
+```
+
+Envoy uses the endpoint hostname for automatic host rewriting but does not append the endpoint port. Therefore, a strict upstream virtual host does not receive `:7448` in the authority.
+
+## Resolution
+
+### Option 1: Make Upstream Host Matching Port-Independent
+
+If the upstream gateway can be changed, configure it to ignore the port during virtual-host matching or accept both authorities:
+
+```text
+model.example.com
+model.example.com:7448
+```
+
+This is the preferred long-term solution because it does not depend on generated xDS structure.
+
+### Option 2: Use EnvoyPatchPolicy
+
+Use this workaround when the upstream must receive exactly:
+
+```text
+Host: model.example.com:7448
+```
+
+#### 1. Enable EnvoyPatchPolicy
+
+Merge the following field into the existing `EnvoyGatewayCtl`. Do not replace its other configuration.
+
+Comments in the example use these labels:
+
+- `CHANGE`: replace the value for the target environment.
+- `KEEP`: copy the field and value unchanged.
+- `MAY CHANGE`: keep the field name, but determine the value from the generated route configuration.
+
+```yaml
+apiVersion: envoy-gateway.alauda.io/v1 # KEEP
+kind: EnvoyGatewayCtl # KEEP
+metadata:
+  name: model-gateway-instance # CHANGE: EnvoyGatewayCtl name
+  namespace: envoy-gateway-system # CHANGE: EnvoyGatewayCtl namespace
+spec:
+  config:
+    envoyGateway:
+      extensionApis:
+        enableBackend: true # KEEP when Backend resources are used
+        enableEnvoyPatchPolicy: true # KEEP
+```
+
+Do not directly edit the ConfigMap or Deployment generated by `EnvoyGatewayCtl`.
+
+#### 2. Identify the Values to Replace
+
+The PatchPolicy targets the Gateway and modifies the generated Envoy RouteConfiguration. Replace the following example values:
+
+| Example value | Meaning | Action |
+| --- | --- | --- |
+| `model-serving` | Gateway namespace | `CHANGE` |
+| `model-gateway` | Gateway name | `CHANGE` |
+| `http` | Gateway listener name | `CHANGE` |
+| `model-serving/model-gateway/http` | RouteConfiguration name: `<gateway-namespace>/<gateway-name>/<listener-name>` | `CHANGE` |
+| First `0` in the path | Virtual-host array index | `MAY CHANGE` |
+| Second `0` in the path | Route array index containing the affected HTTPRoute | `MAY CHANGE` |
+| `model.example.com:7448` | Host or authority required by the backend | `CHANGE` |
+
+The `targetRef` must remain a `Gateway`. The HTTPRoute name is not placed directly in `targetRef`; it determines which generated route index must be patched.
+
+If the Gateway contains one virtual host and one route, the indexes are commonly `0` and `0`. For Gateways with multiple routes, use the Envoy configuration dump to locate the route containing `auto_host_rewrite: true` and use its indexes.
+
+#### 3. Apply the PatchPolicy
+
+```yaml
+apiVersion: gateway.envoyproxy.io/v1alpha1 # KEEP
+kind: EnvoyPatchPolicy # KEEP
+metadata:
+  name: upstream-authority-with-port # CHANGE: any valid resource name
+  namespace: model-serving # CHANGE: must be the Gateway namespace
+spec:
+  targetRef:
+    group: gateway.networking.k8s.io # KEEP
+    kind: Gateway # KEEP
+    name: model-gateway # CHANGE: Gateway name
+  type: JSONPatch # KEEP
+  jsonPatches:
+    - type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration # KEEP
+      name: model-serving/model-gateway/http # CHANGE: namespace/Gateway/listener
+      operation:
+        op: remove # KEEP
+        path: /virtual_hosts/0/routes/0/route/auto_host_rewrite # MAY CHANGE: only the two indexes
+    - type: type.googleapis.com/envoy.config.route.v3.RouteConfiguration # KEEP
+      name: model-serving/model-gateway/http # CHANGE: same value as above
+      operation:
+        op: add # KEEP
+        path: /virtual_hosts/0/routes/0/route/host_rewrite_literal # MAY CHANGE: same two indexes as above
+        value: model.example.com:7448 # CHANGE: required backend authority
+```
+
+Both patch operations are required:
+
+- The first operation removes `auto_host_rewrite`.
+- The second operation adds `host_rewrite_literal` with the required port.
+
+Apply the policy:
+
+```bash
+kubectl apply -f upstream-authority-with-port.yaml
+```
+
+Confirm that the policy reports:
+
+```text
+Accepted=True
+Programmed=True
+```
+
+## Verification
+
+Send a request through Envoy Gateway and inspect the raw Host header at the backend. The backend must receive:
+
+```text
+Host: model.example.com:7448
+```
+
+For an NGINX backend, log `$http_host` rather than `$host` because `$http_host` preserves the raw Host header including the port.
+
+The validation environment returned:
+
+```text
+http_host="model.example.com:7448"
+```
+
+This confirms that the PatchPolicy changed the upstream Host or authority to the expected hostname and port.
+
+## Rollback
+
+Delete only the PatchPolicy:
+
+```bash
+kubectl delete envoypatchpolicy upstream-authority-with-port -n model-serving
+```
+
+Envoy Gateway will regenerate the route with its original `auto_host_rewrite` behavior. The Gateway, HTTPRoute, Backend, and Envoy Gateway instance do not need to be deleted.
+
+## Limitations
+
+- The exact patch in this document is validated for Envoy Gateway v1.8.0.
+- Route indexes can change after adding, removing, or reordering HTTPRoutes or after upgrading Envoy Gateway.
+- Recheck `Accepted` and `Programmed` after changing Gateway or route resources.
+- Prefer port-independent upstream virtual-host matching when that configuration is under your control.
+
+## References
+
+- [ACP Envoy Gateway Operator: Advanced Config Via EnvoyGatewayCtl](https://docs-dev.alauda.cn/container_platform/main/networking/operators/envoy_gateway_operator#envoygatewayctl)
+- [Envoy Gateway v1.8: Envoy Patch Policy](https://gateway.envoyproxy.io/v1.8/tasks/extensibility/envoy-patch-policy/)
+- [Envoy issue #26022: auto_host_rewrite port number loss](https://github.com/envoyproxy/envoy/issues/26022)
+- [Envoy Gateway issue #8823: Support for Port in auto_host_rewrite](https://github.com/envoyproxy/gateway/issues/8823)
+- [Envoy AI Gateway issue #2500: Generated Backend host rewrite drops non-default port](https://github.com/envoyproxy/ai-gateway/issues/2500)