What We’re Building

Imagine you have a simple REST API. You want API key authentication, rate limiting, and automatic failover across two clusters in different clouds — and you don’t want to touch a single line of application code to get any of it.

That’s what Red Hat Connectivity Link (RHCL) gives you. Built on the upstream Kuadrant project, RHCL attaches API management policies directly to standard Kubernetes Gateway API resources. Auth, rate limiting, and DNS-based traffic management all live at the infrastructure layer.

In this walkthrough, I’ll show how to:

1. Deploy a sample API behind a Kubernetes Gateway
2. Add API key auth and rate limiting with zero app changes
3. Distribute the entire stack across two clusters with ACM
4. Set up DNS-based traffic management across clouds
5. Visualize it all with a live dashboard

I’ll also be honest about where the edges are — particularly around multi-cloud DNS failover, where the behavior is more nuanced than the architecture diagrams suggest.

The full demo repo is at ultraJeffOrg/connectivity-link-apicurio-demo.

Prerequisites

• Two OpenShift clusters (this demo uses blue on Azure/centralus and aws-ai on AWS/us-east-2)
• Red Hat Advanced Cluster Management (ACM) 2.17 with both clusters registered as managed spokes
• An AWS Route53 hosted zone for your DNS domain
• cert-manager installed (RHCL depends on it for TLS)
• oc and kubectl CLI access to the hub cluster

The Sample API

The Incident API is a minimal Node.js/Express app running on UBI 9. It has a handful of CRUD endpoints and an in-memory data store — nothing special on purpose. The point is that this is a plain HTTP service with no auth middleware, no rate limiting, no security logic whatsoever.

GET    /api/incidents          # list all
GET    /api/incidents/:id      # get one
POST   /api/incidents          # create
PATCH  /api/incidents/:id      # update
GET    /healthz                # health check

This is key to the pattern: the application stays simple, and the platform handles the rest.

Step 1: Install RHCL

RHCL ships as an operator. When you install it, it brings along three sub-operators automatically: Authorino (auth), Limitador (rate limiting), and the DNS Operator. As of RHCL 1.3+ on OCP 4.19+, it does NOT require OpenShift Service Mesh or Sail.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: rhcl-operator
  namespace: kuadrant-system
spec:
  channel: stable
  name: rhcl-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

After the operator is ready, activate it with a Kuadrant CR:

apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
  namespace: kuadrant-system

A gotcha worth knowing: the Kuadrant operator checks for available GatewayClasses on startup and caches the result. If your openshift-default GatewayClass isn’t created before the operator starts, it will permanently report MissingDependency even after the GatewayClass appears. The fix is to delete the kuadrant-system namespace entirely and let it recreate. This is a race condition, not a config error — sequence matters.

Step 2: Set Up the Gateway and Route

We use the standard Kubernetes Gateway API — no proprietary CRDs for traffic routing:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: incident-api-gateway
  labels:
    kuadrant.io/gateway: "true"
spec:
  gatewayClassName: openshift-default
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      hostname: incidents.example.com
      allowedRoutes:
        namespaces:
          from: Same

Note the kuadrant.io/gateway: "true" label — RHCL needs this to know which Gateways to manage. Then an HTTPRoute to wire traffic to the backend:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: incident-api-route
spec:
  parentRefs:
    - name: incident-api-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
        - path:
            type: Exact
            value: /healthz
      backendRefs:
        - name: incident-api
          port: 3000

Nothing RHCL-specific so far — this is vanilla Gateway API.

Step 3: Add API Key Authentication

Here’s where RHCL starts earning its keep. An AuthPolicy attaches to the HTTPRoute via targetRef and enforces API key authentication:

apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: incident-api-auth
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: incident-api-route
  rules:
    defaults:
      when:
        - predicate: request.path != '/healthz'
      authentication:
        api-key:
          apiKey:
            selector:
              matchLabels:
                app.kubernetes.io/component: api-key
            allNamespaces: true
          credentials:
            customHeader:
              name: x-api-key

The policy says: for all requests (except /healthz), look for an x-api-key header and validate it against Kubernetes Secrets that have the label app.kubernetes.io/component: api-key. The /healthz exclusion matters — without it, the DNSPolicy health checks will fail auth and your DNS records will never go healthy.

Note allNamespaces: true — the default is false, and if your API key Secret isn’t in the exact same namespace as the AuthPolicy, authentication will silently fail. This one cost me some debugging time.

The API keys are just Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: api-key-demo
  labels:
    app.kubernetes.io/component: api-key
  annotations:
    api-consumer: demo-client
type: Opaque
stringData:
  api_key: demo-key-12345

No code changes. No SDK. No middleware. The gateway handles it.

Step 4: Add Rate Limiting

A RateLimitPolicy attaches to the Gateway (not the HTTPRoute) and sets a global rate limit:

apiVersion: kuadrant.io/v1
kind: RateLimitPolicy
metadata:
  name: incident-api-ratelimit
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: incident-api-gateway
  defaults:
    limits:
      global:
        rates:
          - limit: 50
            window: 10s

50 requests per 10-second window. Any request beyond that gets a 429 Too Many Requests. The application never sees the rejected requests.

One thing to watch out for: don’t use source.address as a rate limit counter. It includes ephemeral ports, so every request looks unique and the rate limit never triggers. No counters at all = global rate limit, which is what you want here.

Also, the RateLimitPolicy must target the Gateway, not the HTTPRoute. Each spoke runs its own Limitador instance, so rate limits are per-cluster — there’s no cross-cluster coordination.

At this point you have a fully managed API: authenticated, rate-limited, routed through a standard Gateway, and the app code hasn’t changed at all.

Step 5: Go Multi-Cluster with ACM

This is where it gets interesting. We want the exact same stack — app, gateway, policies — deployed to two spoke clusters in different clouds. ACM’s governance framework handles this.

The approach uses two ACM Policy resources on the hub:

Policy 1: Operators — Ensures each spoke has cert-manager, the RHCL operator, the GatewayClass, and a Kuadrant CR. It uses extraDependencies to sequence things properly — the Kuadrant CR isn’t created until the operator subscription reports AtLatestKnown:

extraDependencies:
  - apiVersion: policy.open-cluster-management.io/v1
    kind: ConfigurationPolicy
    name: incident-api-rhcl-subscription
    compliance: Compliant

This is critical. Without extraDependencies, ACM will try to create the Kuadrant CR before the operator is ready, and it’ll fail silently or trigger the GatewayClass race condition I mentioned earlier.

Policy 2: Application + Config — Depends on Policy 1 being Compliant, then pushes the namespace, Deployment, Service, Gateway, HTTPRoute, and all RHCL policies to each spoke.

A Placement resource targets the right clusters:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: incident-api-placement
spec:
  clusterSets:
    - global
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: name
              operator: In
              values: [blue, aws-ai]

An Important Detail About musthave

ACM’s ConfigurationPolicy uses musthave compliance by default, which means it merges fields rather than replacing the full object. If you had a field in a previous version of your policy that you’ve since removed, ACM won’t remove it from the spoke — it’ll persist as a stale field. This matters for iterating on RHCL policies during development.

Also, don’t enforce spec.replicas in your ConfigurationPolicy if you want the dashboard to scale deployments up and down for failover testing. ACM will fight you and reset the replicas.

Secure Secret Distribution

Route53 AWS credentials never appear in Git. ACM hub-templates handle this:

aws_access_key_id: '{{hub fromSecret "incident-api-policies"
  "aws-route53-credentials" "aws_access_key_id" hub}}'

At apply time, ACM pulls the credential from a Secret on the hub and injects it into the spoke resources. The secret exists only on the hub cluster.

Step 6: DNS-Based Traffic Management

This is the centerpiece. A DNSPolicy attaches to the Gateway on each spoke and registers it with Route53:

apiVersion: kuadrant.io/v1
kind: DNSPolicy
metadata:
  name: incident-api-dns
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: incident-api-gateway
  providerRefs:
    - name: aws-route53-credentials
  loadBalancing:
    weight: 120
    geo: US
  healthCheck:
    endpoint: /healthz
    port: 80
    protocol: HTTP
    interval: 20s
    failureThreshold: 2

Both spokes share the same hostname. Each spoke’s DNSPolicy registers its Gateway address with Route53 and sets up health checks against /healthz. Under the hood, RHCL creates a CNAME chain in Route53: the hostname points to a klb. (Kuadrant load balancer) prefix, which points to a geo record, which points to per-cluster weighted endpoint records. Each spoke coordinates via TXT ownership records so they don’t step on each other’s DNS entries.

The A Record vs. CNAME Problem

Here’s where multi-cloud gets messy. Different cloud providers expose Gateway addresses differently:

• Azure gives you an IP address — which becomes an A record in DNS
• AWS gives you a hostname (an ELB DNS name) — which becomes a CNAME record

This isn’t something you can control. Azure’s load balancer natively provides IPs (ipMode: VIP). AWS’s load balancer natively provides hostnames. You can’t force either to behave like the other.

RHCL’s DNSPolicy has two modes: simple and loadBalancing. Simple mode gives you true failover — when health checks fail, the unhealthy record is actually removed from DNS. But simple mode can’t handle a mix of A records and CNAME records for the same hostname. You need loadBalancing mode for that.

The catch: loadBalancing mode intentionally does NOT remove unhealthy records. When a health check fails, it sets the Route53 weight to 0 to deprioritize the unhealthy endpoint, but the record stays in DNS. From the RHCL docs:

When a record has been published using the load balancing options (GEO and Weighting) via DNSPolicy, a failing health check will not remove the endpoint record.

This is by design to avoid NXDOMAIN — the DNS response you get when a domain doesn’t exist at all. If both spokes happened to be temporarily unhealthy and RHCL removed both records, clients would get NXDOMAIN errors. That’s worse than routing to an unhealthy endpoint, because many clients don’t handle DNS failures gracefully. At least with a 503, the client can retry.

So what does this mean in practice? In loadBalancing mode, you get “prefer healthy” routing, not true failover. Route53 will deprioritize the unhealthy spoke by dropping its weight to 0, but it can still route traffic there — especially during weight transitions or when both endpoints have issues. And there’s a validFor reconciliation interval (default 15 minutes) that controls how often RHCL re-evaluates DNS records, which adds latency to any weight changes.

Getting True Failover

If you need hard failover where unhealthy endpoints are actually removed from DNS, you need simple mode — which means both spokes need to publish the same record type. You have a few options:

1. Use the same cloud provider for both spokes. Two clusters on Azure would both get A records. Two on AWS would both get CNAMEs. Either way, simple mode works.
2. Use on-prem or bare-metal spokes. Clusters that aren’t behind a cloud load balancer will get direct IPs. Two on-prem OpenShift clusters with simple mode gives you real failover.
3. Accept loadBalancing mode. For many use cases, weight-based deprioritization is good enough. Traffic will strongly prefer the healthy spoke, even if the cutover isn’t absolute.
4. Build a weight controller. You could write a simple controller on each spoke that watches Deployment readiness and patches the DNSPolicy weight to 0 when pods are down. This fills the gap that RHCL’s health checks in loadBalancing mode don’t actually change Route53 weights. If you go this route, make sure to remove the weight field from ACM’s ConfigurationPolicy so ACM doesn’t overwrite your controller’s changes.

This is the kind of thing you discover when you actually run a multi-cloud setup, not just diagram one. The tradeoff (weight-based deprioritization vs. NXDOMAIN risk) is a reasonable one. But it’s worth understanding before you promise “automatic failover” to stakeholders.

The Dashboard

The demo includes a live dashboard that runs on the ACM hub cluster and gives you a real-time view of the entire multi-cluster setup. It uses ACM’s ManagedClusterView API to read resources from spoke clusters and ManagedClusterAction to mutate them — all without direct kubeconfig access to the spokes.

For each spoke, the dashboard fires three parallel ManagedClusterView reads — Deployment (replica counts), Gateway (addresses), and DNSPolicy (health/enforcement status) — and maps everything together. For AWS spokes, it resolves ELB hostnames to actual IPs so you can see which IP belongs to which cluster.

The dashboard shows:

• Cluster health cards — real-time replica counts, gateway addresses, resolved IPs, and DNS health/enforcement status for each spoke, with pulsing red banners when a cluster is down
• DNS resolution panel — live DNS lookups showing which IPs the hostname resolves to and which cluster they map to
• Failover controls — take a cluster down (scales the deployment to 0 via ManagedClusterAction) or bring it back up, then watch the DNS weights shift
• Rate limit testing — a “Blast 60 Requests” button that fires 60 concurrent requests and shows how many get 200 OK vs 429 Too Many Requests
• Auth testing — send requests with and without an API key to see the AuthPolicy accept or reject

Every proxied request is annotated with the DNS resolution — you can see “DNS resolved to 1.2.3.4 → blue (Azure)” on each response, so you always know which cluster handled the traffic.

The ManagedClusterView/Action pattern is worth studying even outside this demo. Views are fire-and-forget: create, poll for result, delete. The dashboard cleans up orphaned views on startup in case of a previous crash. It’s a clean way to build multi-cluster operational UIs on top of ACM without needing kubeconfigs for every spoke.

Architectural Reality Check

It’s worth stepping back and being clear about what RHCL is and isn’t in a multi-cluster context. RHCL is not a coordinated multi-cluster system — it’s independent gateways that happen to share a DNS name. Each spoke runs its own Authorino, its own Limitador, and manages its own DNS records independently. ACM distributes the configuration. Route53 splits the traffic. RHCL does local policy enforcement and DNS automation.

This means:

• Rate limits are per-cluster, not global. If your limit is 50 req/10s and you have two spokes, the effective limit is 100 req/10s total.
• Auth is per-cluster. Each spoke has its own Authorino instance validating against its own API key Secrets. They’re identical because ACM pushed the same config, but they’re not coordinated.
• There’s no hub-as-gateway. Traffic goes directly to spoke clusters via DNS. The hub doesn’t proxy anything.

None of this is a problem if you understand it. It’s a simple, decoupled architecture. But if you’re expecting a single control plane that coordinates policy across clusters in real time, that’s not what this is.

Why This Pattern Matters

Gateway API as the control plane. RHCL policies attach to standard Gateway and HTTPRoute resources via targetRef — the Gateway API’s Policy Attachment pattern. There are no proprietary ingress controllers or custom routing CRDs. If you swap your gateway implementation, the policies still work.

Zero-app-change API management. Application teams ship plain HTTP services. The platform adds authentication, rate limiting, and traffic management declaratively. This is how platform engineering should work — the platform provides capabilities, not requirements.

DNS-level multi-cluster resilience — with caveats. The traffic management operates at the DNS layer, which is beautifully simple: no service mesh federation, no VPN tunnels, no shared control plane. But multi-cloud DNS has real edge cases around record types and failover semantics. Understanding the difference between simple mode (true failover, same record types required) and loadBalancing mode (mixed record types, weight-based deprioritization) is essential before you design your topology.

What’s Next

In future posts, I’ll dig into the Kustomize patterns used to structure these manifests (base/overlay/component) and the broader GitOps workflow for managing multi-cluster deployments with ACM. If you want to try this yourself, the full demo is at ultraJeffOrg/connectivity-link-apicurio-demo.