The Problem

You have workloads running on OpenShift. You want zero-trust networking — every connection mutually authenticated, encrypted, and authorized by identity rather than network location. The traditional answer is a service mesh with sidecar injection, but that means an Envoy container in every pod, increased memory usage, pod restarts for injection, and a meaningful blast radius when the mesh misbehaves.

OSSM v3 (OpenShift Service Mesh v3, based on Istio 1.28) introduces ambient mode — a sidecar-free mesh architecture that gives you mTLS everywhere without touching your application pods.

This post walks through how ambient mesh works, what it gives you, and what you need to watch out for. The demo repo includes an interactive console that demonstrates all of this live against an OpenShift cluster.

Ambient Mode vs. Sidecars

In traditional sidecar mode, every pod gets an Envoy container injected. Full L7 processing on every connection, but at a cost:

	Sidecar	Ambient
Proxy	Per-pod Envoy	Per-node ztunnel (Rust)
Resource overhead	High (memory per pod)	Low (shared per node)
Pod modification	Yes (injection)	No
L7 policy	Always available	Opt-in (waypoint proxy)
Restart required	Yes (for injection/removal)	No
Protocol	Full Envoy L7	L4 always, L7 opt-in

The key insight: most connections only need L4 mTLS. You don’t need to parse HTTP headers to encrypt a connection. Ambient mode separates L4 (always on, per-node) from L7 (opt-in, per-namespace), so you pay for L7 processing only where you actually need it.

How ztunnel Works

ztunnel is a Rust-based, per-node DaemonSet that handles L4 mTLS for all enrolled pods. Here’s what happens when a pod starts in an ambient-enrolled namespace:

1. istio-cni (a chained CNI plugin, also a DaemonSet) detects the new pod
2. istio-cni enters the pod’s network namespace and injects iptables rules:
   • Inbound TCP → redirect to ztunnel port 15006
   • Outbound TCP → redirect to ztunnel port 15001
   • Kubelet probes are exempted via SNAT address 169.254.7.127
3. ztunnel creates listening sockets inside the pod’s network namespace
4. ztunnel requests a SPIFFE certificate from istiod for this pod’s identity
5. All traffic now flows: app → iptables → ztunnel → HBONE mTLS → remote ztunnel → destination app

The application serves plain HTTP. ztunnel handles encryption transparently. The plain HTTP segment only exists between ztunnel and the app inside the same network namespace — like two processes on localhost. It never crosses a network boundary unencrypted.

HBONE: The Transport Protocol

HBONE (HTTP-Based Overlay Network Encapsulation) is the wire protocol for ambient mesh. It’s HTTP/2 CONNECT over mTLS on port 15008. HTTP/2 was chosen for multiplexing (multiple pods share one tunnel), metadata in the handshake, and built-in flow control. Once the CONNECT handshake completes, it’s a raw byte pipe — L7 for setup, L4 for data.

SPIFFE Workload Identity

Every workload gets a SPIFFE Verifiable Identity Document (SVID) — an X.509 certificate with a URI in the Subject Alternative Name:

spiffe://cluster.local/ns/<namespace>/sa/<service-account>

These certificates are:

• Issued by istiod (which validates the pod’s identity via Kubernetes TokenReview)
• Automatically rotated every 24 hours
• Held by ztunnel, not the application — the app never touches TLS

This is the foundation for identity-based policy. Instead of “allow traffic from 10.128.2.15” (which tells you nothing about what’s running there), you get “allow traffic from spiffe://cluster.local/ns/my-app/sa/frontend.”

Enrolling a Namespace

Two labels. That’s it.

apiVersion: v1
kind: Namespace
metadata:
  name: my-workloads
  labels:
    istio.io/dataplane-mode: ambient
    istio.io/use-waypoint: mesh-waypoint  # only if you want L7

No pod restarts. No sidecar injection. Existing pods are enrolled immediately — istio-cni detects the label change and injects iptables rules into running pods’ network namespaces.

Zero Trust with PeerAuthentication

With pods enrolled, you can enforce STRICT mTLS:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: strict-mtls
spec:
  mtls:
    mode: STRICT

In STRICT mode, every inbound connection must present a valid SPIFFE certificate. Plain HTTP is rejected outright. You can prove this with three escalating tests:

1. Through the mesh gateway with the CA cert → 200 OK (mutual TLS established)
2. Without the CA cert → TLS handshake failure (can’t verify the server)
3. Bypass the gateway, curl the pod directly → connection refused (STRICT rejects non-mTLS)

Every connection is logged with the source SPIFFE identity, timestamp, and bytes transferred. You get a cryptographic audit trail for free.

A Note on PeerAuthentication Port Exclusions

If you’re running Prometheus monitoring, you’ll need port-level PERMISSIVE mode for metrics scraping:

spec:
  mtls:
    mode: STRICT
  portLevelMtls:
    8080:
      mode: PERMISSIVE

Prometheus typically doesn’t present mesh certificates, so its scrape requests need to be allowed through without mTLS on the metrics port.

L7 Authorization with Waypoint Proxies

ztunnel gives you L4 mTLS — connection-level encryption and identity. But if you need to make decisions based on HTTP paths, methods, or headers, you need a waypoint proxy.

A waypoint is an Envoy instance deployed per-namespace via a Gateway CR:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mesh-waypoint
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE

With a waypoint deployed, you can write AuthorizationPolicies that reference HTTP attributes:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: model-access
spec:
  targetRefs:
    - kind: Service
      name: iris-onnx-mesh
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/models/sa/mesh-gateway-istio"
              - "cluster.local/ns/models/sa/mesh-waypoint"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v2/models/*/infer"]

This restricts the model to accept inference requests only from the mesh gateway, only via POST, only on the inference path. A test pod trying to reach the model directly gets a 403.

Important detail: traffic through a waypoint involves two HBONE mTLS hops (source → waypoint, waypoint → destination). The destination’s ztunnel sees the waypoint’s SPIFFE identity as the source. So the waypoint itself must be in the ALLOW list — otherwise it can establish the first hop but the second hop gets denied.

NetworkPolicy Is Dead (in the Mesh)

This is a critical point that’s easy to miss. Once a namespace is ambient-enrolled, NetworkPolicy can’t distinguish source workloads anymore. All mesh traffic arrives from ztunnel’s IP address, not the originating pod’s IP. NetworkPolicy operates on packet headers (source IP, port), so every mesh connection looks like it’s coming from the same source.

AuthorizationPolicy replaces NetworkPolicy in mesh-enrolled namespaces. It reads identity from the mTLS certificate’s SPIFFE SAN, not from packet headers. This is more secure (cryptographic identity vs. spoofable IP) and more expressive (identity + HTTP attributes vs. IP + port).

Traffic Management Through the Waypoint

The waypoint isn’t just for authorization. It’s a full Envoy proxy, so you get L7 traffic management:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: httpbin-resilience
spec:
  hosts:
    - httpbin
  http:
    - timeout: 5s
      retries:
        attempts: 3
        retryOn: 5xx,reset,connect-failure
      route:
        - destination:
            host: httpbin

Call a service that takes 10 seconds to respond, and the waypoint enforces the 5-second timeout — the client gets a 504 instead of waiting. The app doesn’t need to implement timeout logic.

What the Waypoint Can (and Can’t) See

A common question: if the waypoint proxy is processing my L7 traffic, can it read request bodies? The answer is nuanced:

• Headers: Yes, the waypoint inspects and can route/filter on HTTP headers
• Body: The body transits through the proxy’s memory buffers, but the waypoint doesn’t parse, inspect, or log body content by default. It’s like a mail carrier who can read the envelope but doesn’t open the letter.
• This is by design. Authorization decisions are made on headers, method, and path — not body content. Body inspection would add latency and memory pressure.

The Architectural Reality

Ambient mesh is not magic. Here’s what it actually is:

• ztunnel is a per-node transparent proxy that adds mTLS to every TCP connection
• Waypoint proxies are opt-in Envoy instances for L7 policy
• istiod is the control plane that distributes config and signs certificates
• istio-cni is the glue that redirects traffic via iptables

ztunnel uses custom, ambient-specific xDS resource types (Address and Authorization) rather than standard Envoy resource types. These are roughly 10x more efficient in size, allocations, and CPU time — important when you’re running one instance per node handling traffic for all pods.

The protocol agnosticism at L4 is real. ztunnel tunnels raw TCP bytes in HBONE. Non-HTTP protocols (Kafka binary protocol, gRPC, database connections) flow through unchanged. If Kafka also uses TLS, you get double encryption (Kafka TLS inside HBONE mTLS), but AES-NI hardware acceleration makes this negligible.

Platform Gotchas

A few things I ran into deploying this:

OVN-Kubernetes routingViaHost: true is required. Without this, kubelet health probes fail for all ambient-enrolled pods. ztunnel exempts probes via SNAT to 169.254.7.127, and the return path needs host routing to work.

Headed services for KServe. If you’re running RHOAI, set rawDeploymentServiceConfig: Headed in the DataScienceCluster CR. Headless services bypass kube-proxy port mapping, breaking the port translation that ztunnel relies on.

GatewayClass race condition. The Kuadrant/OSSM operator checks for GatewayClasses on startup and caches the result. If the GatewayClass isn’t created before the operator starts, it permanently reports MissingDependency. The fix is deleting the namespace entirely and letting it recreate.

What’s Next

In a future post, I’ll cover the custom CA chain architecture (cert-manager → intermediate CA → SPIFFE certs) and multi-cluster trust federation — sharing a root CA across clusters so workloads in different clusters can mutually authenticate. The demo repo includes an interactive console that walks through all of this live if you want to try it yourself.