%%Taming the Cross-AZ Tax:%% Optimizing Kubernetes Traffic on AWS EKS

Now, AWS doesn't publish exact numbers on this, but talk to any cloud architect and they'll tell you the same thing: Multi-AZ is the absolute must for production. It is just how things are built today.

But here is the catch. If you are running microservices that constantly talk to each other, this 'cross-AZ tax' can completely hijack your monthly AWS bill. On a busy EKS cluster, you could easily be looking at a four [or even five-figure] line item every single month. And the crazy part? You are paying that massive bill simply because Kubernetes, by default, has no opinion on where your bytes go.

This post is about giving it one.

What you're paying for

When you deploy workloads on AWS, traffic that stays inside the same Availability Zone is free. The moment a packet crosses an AZ boundary in the same region, AWS starts the meter: $0.01 per gigabyte in each direction. The charge is symmetric: both sides pay, on their respective ENIs. TCP, UDP, gRPC, HTTPS: same rate, every byte.

Consider a 60-node EKS cluster across three AZs running ~150 microservices behind an AWS Network Loadbalancer (NLB). Internal traffic sits around 1.5 TB/day, roughly 67% of it cross-zone. That works out to ~30 TB/month of cross-AZ traffic × $0.02/GB ≈ $600/month, or $7,200/year.

Notice the tradeoff buried in the setup: the only reason you pay this is because you spread the workload across three AZs in the first place. That spread is the entire point of high availability: if one zone fails, the other two carry the traffic. The bill in front of you is the literal cost of being able to lose a zone. The question for the rest of this post isn't whether to keep that property. Single-zone clusters are a worse answer for almost every production workload. The question is how to keep the HA property without paying the full price.

[.radek][.poznamka]NOTE
‍You can review it by yourself on the AWS Cost and Usage Report (CUR), cross-AZ traffic lands as the [.span]{Region}-DataTransfer-Regional-Bytes[.span] UsageType (for example, [.span]EUC1-DataTransfer-Regional-Bytes[.span] in [.span]eu-central-1[.span]) under [.span]EC2-Other[.span], with API operations [.span]InterZone-In[.span] and [.span]InterZone-Out[.span]. The CUR reports the bytes but not the pods responsible. Pod-level attribution is covered in the next section.[.poznamka][.radek]

Seeing the bytes

Before you can fix the bill, you have to know which pods are generating it, and that is the gap the CUR leaves open.

Raw VPC Flow Logs alone do not get you there either. They show IP-to-IP byte counts, but in a busy cluster pod IPs are recycled constantly: rolling deploys, HPA scale-outs, Karpenter consolidation. By the time you run the query, the IP in the flow record has likely belonged to several different pods since. Attribution ends up landing on whichever pod currently holds the IP, which is rarely the one that actually moved the bytes.

A few tools help close this gap, each with its own tradeoffs:

• aws-samples/amazon-eks-inter-az-traffic-visibility is the lightest-weight path. A Lambda periodically snapshots the pod-IP-to-pod-name mapping into S3, and Athena joins those snapshots against the Flow Logs by timestamp.
• Cilium Hubble is a strong option if you already run Cilium. You get live, pod-aware flow visibility, so IP churn does not break attribution. The tradeoff: inter-AZ accounting is still on you. You will need to enrich each flow with pod-to-node-to-zone metadata.
• EKS Container Network Observability is the recently added AWS managed option, announced in November 2025. It produces an EKS console view of network flows classified as Intra-AZ, Inter-AZ, EC2→S3, and EC2→DynamoDB, without your own enrichment pipeline. It requires EKS 1.35+. We have not tested it yet, so treat it as an upgrade-path option rather than today's default.

Once attribution is in place, the next question is where the top entries actually come from. In an EKS cluster, almost every cross-AZ byte falls into one of three buckets: pod-to-pod service calls, DNS lookups, and inbound traffic through your load balancer. Each one rides on whichever routing dataplane your cluster runs, and the dataplane decides whether topology hints are honoured at all.

[.radek][.poznamka]NOTE
Topology hints are zone labels ([.span]hints.forZones[.span]) that Kubernetes writes onto a Service's EndpointSlices, marking which zone each endpoint should serve. The dataplane reads them to keep traffic in-zone — but only if it honors hints at all.[.poznamka][.radek]

Two upstream Kubernetes features can write topology hints onto Services — Topology Aware Routing (TAR) and [.span]spec.trafficDistribution[.span], and we cover each in its own section later on. For now, treat them just as named placeholders. Pin the dataplane down first, then walk the flows.

How EKS routes traffic

In EKS the dataplane is what actually rewrites a Service ClusterIP connection to a specific pod IP. It also decides whether [.span]hints.forZones[.span] on the EndpointSlice gets consulted. Pick the dataplane your cluster runs:

• A) kube-proxy alone. EKS default. [.span]kube-proxy[.span] programs iptables, IPVS rules or nftables, and the kernel rewrites ClusterIP connections to a random EndpointSlice entry. It reads [.span]hints.forZones[.span] automatically when present.
• B) kube-proxy + Cilium in chaining mode. AWS VPC CNI (or whichever CNI you use) still handles pod IPAM and pod-to-pod connectivity; Cilium chains on top to provide NetworkPolicy, Hubble observability, and optionally encryption. [.span]kube-proxy[.span] continues to do Service routing, so it reads [.span]hints.forZones[.span] natively. Service-routing behavior is identical to A.
• C) Cilium kube-proxy-replacement (KPR). Cilium's eBPF load-balancer replaces [.span]kube-proxy[.span] entirely. Faster, more observable in Hubble, but it ignores EndpointSlice hints unless [.span]loadBalancer.serviceTopology=true[.span] is set in the Cilium Helm values. Without that flag, your cluster silently has no topology-aware routing. For the rest of this post, "Cilium KPR" means Cilium KPR with [.span]loadBalancer.serviceTopology=true[.span].

As we've mentioned before three patterns dominate the cross-AZ bill on most EKS clusters: pod-to-pod Service calls, DNS lookups, and inbound traffic through a load balancer.

Flow 1: Pod-to-pod via ClusterIP

Let's have an example where a pod connects to [.span]orders.production.svc.cluster.local:8080[.span]. CoreDNS returns a ClusterIP. The dataplane picks one endpoint at random from the Service's EndpointSlice and rewrites the destination. With backends spread across three AZs and no further tuning, that is roughly a two-thirds chance the packet lands in a different zone. Real world services rarely have perfectly even pod distribution.

Here is the key mechanism for everything that follows: the dataplane honors a zonal hint if one is present on the EndpointSlice, but it does not create that hint. Hints are written by the EndpointSlice controller in [.span]kube-controller-manager[.span], triggered by one of the two features mentioned above: [.span]topologyMode: Auto[.span] (TAR heuristic) or [.span]spec.trafficDistribution[.span] (newer, declarative replacement). In short, the controller produces hints and the dataplane consumes them.

Flow 2: Service DNS resolution

Every [.span]*.svc.cluster.local[.span] lookup is a (UDP/TCP):53 packet to a CoreDNS pod, routed through the [.span]kube-dns[.span] Service's ClusterIP. Same dataplane, same routing logic, same default ~67% cross-AZ as Flow 1. Per-query bytes are tiny. But a noisy application doing service discovery on every outbound request can issue 10,000+ DNS lookups per second.

For most teams the answer here is one line: deploy NodeLocal DNS Cache. It is a DaemonSet from [.span]kubernetes-sigs/node-local-dns[.span] that intercepts cluster DNS via a link-local IP (usually set to [.span]169.254.20.10[.span]) and serves answers from a local cache on every node.

Hit rates stay high because most cluster DNS queries are for Service names, while Service ClusterIPs are long-lived, so cached answers remain valid. A pod that resolves [.span]orders.production.svc.cluster.local[.span] once does not need to ask again for the rest of the answer's TTL (default 30 seconds for CoreDNS, configurable down to 5 if you are worried about endpoint churn). Across thousands of pods making the same handful of lookups, the local cache absorbs almost everything. Misses that fall through to CoreDNS are a tiny fraction of the original volume.

[.radek][.poznamka]NOTE
‍One gotcha worth checking: NodeLocal DNS Cache only intercepts traffic if a pod's [.span]/etc/resolv.conf[.span] actually points at the link-local IP, which depends on cluster configuration. After deploying, spot-check a sample pod with [.span]kubectl exec <pod> -- cat /etc/resolv.conf[.span]. If the nameserver is not [.span]169.254.20.10[.span], the cache is running but nothing is using it.[.poznamka][.radek]

All three dataplanes sit underneath NodeLocal DNS Cache identically: with one caveat for Cilium KPR, which needs a Local Redirect Policy to actually intercept the cluster DNS IP, since [.span]kube-proxy[.span] is no longer there to install the iptables rules.

Flow 3: Inbound: client → NLB → ingress controller → backend Service

Three hops, governed by completely different mechanisms.

Hop 1 - client picks an NLB node. A client (in-cluster pod or external caller) resolves the NLB's DNS name. The answer contains one IP per AZ the NLB has a node in, and which IP the client actually receives is governed by the NLB's [.span]dns_record.client_routing_policy[.span] attribute:

• [.span]availability_zone_affinity[.span]: DNS returns only the same-zone NLB IP when one exists. This ensures the traffic originating from a zone stays within the same zone when routed by NLB.
• [.span]partial_availability_zone_affinity[.span]: same-zone IP with a weighted preference; cross-AZ IPs are still returned some of the time.
• [.span]any_availability_zone[.span] (the default): any zone, with no preference, so the client→NLB connection is cross-AZ ~67% of the time.

If the client is a pod inside the cluster, the DNS lookup itself is Flow 2: without zonal hints on the [.span]kube-dns[.span] Service (or NodeLocal DNS Cache in place), the DNS query goes cross-AZ on its way to CoreDNS before the NLB IP is even returned. That can stack two cross-AZ hops on one connection setup, and each hop incurs data transfer cost.

Hop 2 - NLB to the ingress controller pod. The AWS Load Balancer Controller supports two NLB target types:

The EKS Best Practices guide is direct: in [.span]ip[.span] mode there are no data-transfer charges between the LB and the pod. Switching to [.span]ip[.span] mode is the single biggest cost improvement you can make on this flow.

Separately from target type, AWS NLBs have a setting called [.span]load_balancing.cross_zone.enabled[.span]. When enabled, the NLB spreads incoming traffic across targets in all zones: better availability if a whole zone fails, but a high probability of crossing zones at the NLB layer regardless of pod-side routing. When disabled (the default for [.span]ip[.span]-mode NLBs; verify on the live console), the NLB sends connections only to targets in the client's entry AZ, so backend pod-to-AZ affinity matters.

The end-to-end in-zone posture needs both NLB settings set:

• [.span]dns_record.client_routing_policy = availability_zone_affinity[.span]
• [.span]load_balancing.cross_zone.enabled = false[.span].

Setting only the first means the client lands on the same-zone NLB node, but the NLB can still forward cross-zone to a remote backend. Setting only the second means the NLB respects zone affinity on egress, but the client may have arrived from another zone in the first place. Cost-sensitive ingress posture: both NLB attributes configured, ip target type, topology spread plus [.span]trafficDistribution[.span] downstream.

Hop 3 - ingress controller to the backend Service. This is Flow 1 again. The ingress controller opens a connection to a backend Service ClusterIP and the dataplane picks an endpoint. All depends on the ingress controller - nginx-ingress and Traefik can be configured to bypass [.span]kube-proxy[.span] and load-balance directly over EndpointSlices.

Locality vs Reliability

Three flows, three dataplanes, one running theme: at every hop, Kubernetes will send your bytes across an AZ because by default it has no opinion about where they should go. The nuclear option we have not covered yet, [.span]internalTrafficPolicy: Local[.span], buys locality at the cost of reliability by forcing traffic to stay on the Node. Set it aggressively if you want to move your costs from the data-transfer bill to your on-call's team budget.

The good news: the right primitives now exist to be locality-preferring, and able to fall back safely when needed.

Topology Aware Routing: and why it has been superseded

TAR is the answer most clusters have already tried. It has been the "official" recommendation since Kubernetes 1.21 (originally as Topology Aware Hints, renamed TAR in 1.27), and it is one annotation away on any Service. It is also no longer the recommended path for new Services.

You enable TAR by setting [.span]service.kubernetes.io/topology-mode: Auto[.span] on a Service. The EndpointSlice controller allocates endpoints to zones proportionally based on allocatable CPU per zone [not pod count, not request rate, just CPU] and writes [.span]endpoints[*].hints.forZones[.span]. From there, the dataplane story from Flow 1 takes over.

That CPU-based proportional allocation has a hard cutoff. The controller only emits hints if, in every zone, the expected load per endpoint stays within about 20% of the ideal share (the [.span]overloadThreshold = 0.2 constant[.span] in [.span]topologycache.go[.span]). If that bound cannot be met, it disables hints and falls back to cluster-wide routing.

The controller has six safeguards that suppress hints when triggered. When any of them fires, the controller silently falls back to cluster-wide random routing:

1. Endpoint count below zone count. A Deployment with 2 replicas across 3 AZs is inherently unbalanceable, so TAR refuses to try.
2. Proportional allocation cannot keep every zone under the overload threshold. This usually happens when allocatable CPU is uneven across zones.
3. One or more nodes missing the [.span]topology.kubernetes.io/zone[.span] label. Usually a misconfigured node-bootstrap, or a self-managed node group that skipped the AWS cloud-controller integration.
4. Per-zone allocatable CPU unreadable. TAR can't read allocatable CPU by zone during node-registration transitions, so it suppresses hints.
5. Nodes ready in only one zone. A real failure mode during partial outages or aggressive Karpenter consolidation.
6. Consumer-side hint gap. If the EndpointSlice controller is mid-update and not every slice has hints yet, [.span]kube-proxy[.span] falls back to cluster-wide routing for the whole Service to avoid an asymmetric state.

Three of these: (1), (2), and (5), are tripped by ordinary Cluster Autoscaler or Karpenter activity all the time. AWS itself calls this out in the EKS Best Practices doc: hints may go unassigned when capacity fluctuates across zones, including with EC2 Spot, because TAR does not detect interruptions in real time.

Three pieces of evidence that TAR is the wrong answer in 2026:

1. The Buoyant experiment. A controlled test disabled the warehouse-service pods in one zone. The order-service pods in that same zone, which TAR had pinned to those warehouse endpoints, dropped to 0% success rate. TAR refused to fall back cross-zone. The failure mode at small scale is not more cross-AZ traffic; it is dropped traffic, hidden until a zone actually degrades. (Source.)
2. AWS's own caveat. Spot and Karpenter interactions are called out by name in the canonical AWS document.
3. The 1.33 GA-scope reduction. When TAR's underlying API (the [.span]hints[.span] field on EndpointSlice) graduated to GA in Kubernetes 1.33, the heuristic itself was left at beta. SIG-Network shipped the plumbing as stable and signalled clearly that the heuristic was not. (KEP-2433.)

TAR is not broken; it does exactly what its algorithm says. The algorithm was built for specific scenarios, not for the real-world challenges that most clusters experience. The replacement is simpler, more predictable, and explicitly designed for the failure mode TAR cannot handle: cross-zone fallback.

Traffic Distribution: the replacement

Thankfully, while TAR is a heuristic with surprising failure modes, [.span]spec.trafficDistribution[.span] is a simpler mechanism with predictable behaviour. No overload thresholds, no per-zone CPU math, no mid-flight safeguards. It went GA in Kubernetes 1.33 (KEP-4444).

Set [.span]spec.trafficDistribution: PreferSameZone[.span] on a Service and the EndpointSlice controller writes [.span]hints.forZones[.span] on each endpoint so that every endpoint hints for its own zone. The dataplane reads the hint and picks an in-zone endpoint when one exists. Simple as that - the whole algorithm.

The reliability win is the part most teams underestimate. Per KEP-4444, when [.span]PreferClose[.span] (the deprecated alias for [.span]PreferSameZone[.span]) is configured and no endpoints exist in the client's zone, traffic routes to other zones rather than dropping. Automatic fallback, no extra configuration. The Buoyant 0%-success-rate experiment cannot happen here. You pay $0.02/GB for the fallback traffic; you do not pay with downtime. That makes [.span]trafficDistribution[.span] a strict improvement on reliability, separately from the cost story.

apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app: orders
  ports:
    - port: 8080
  trafficDistribution: PreferSameZone   # GA in 1.33; aliased by the deprecated PreferClose
  # internalTrafficPolicy: Cluster      # default — do NOT set to Local with trafficDistribution

[.radek][.poznamka]NOTE 
`internalTrafficPolicy: Local` overrides `trafficDistribution`. Local is strict node-local routing with no fallback. If you set both, Local wins and you lose trafficDistribution's fallback. Do not combine them.[.poznamka][.radek]

[.span]spec.trafficDistribution[.span] is necessary, but not sufficient on its own. The moment you pin traffic in-zone, the burden of balancing shifts to you: it becomes the cluster operator's job to keep the workload evenly spread across AZs. Once each zone's clients are served only by that zone's pods, pod distribution is load distribution — a zone with half the replicas of its neighbors takes roughly double the per-pod load, hot-spotting those backends even while cluster-wide CPU looks fine. Skew that the random cross-AZ default used to cover over now lands directly on whichever zone is short.

It also only routes in-zone when a [.span]Ready[.span] in-zone endpoint exists; whenever the in-zone set goes empty, traffic falls back cross-AZ. That can happen two ways — pods aren't evenly distributed across zones, or the autoscaler can't provision a new node in the right zone when one is needed. Each maps to an operator-side responsibility:

1. Even zonal pod distribution via [.span]topologySpreadConstraints[.span] ([.span]maxSkew: 1[.span], [.span]whenUnsatisfiable: DoNotSchedule[.span], [.span]topologyKey: topology.kubernetes.io/zone[.span]). The strict [.span]DoNotSchedule[.span] is what makes the spread real. The alternative, [.span]ScheduleAnyway[.span], is advisory, and the scheduler will ignore it under load. In simple terms, [.span]maxSkew: 1[.span] means the most-loaded and least-loaded zones can differ by at most one pod. In 3 AZs, 5 replicas can land as 2-2-1 and 8 replicas as 3-3-2. Pods go Pending only when the next placement would break that rule or the required zone has no eligible capacity.
2. Fast, zone-targeted node provisioning. This is where your choice of autoscaler stops being incidental. The legacy Cluster Autoscaler + AWS Auto Scaling Group combination cannot decide which zone a new node lands in; it scales the ASG and lets EC2's AZ-rebalancer pick. When [.span]topologySpreadConstraints[.span] blocks a pod from scheduling in zones B and C, Cluster Autoscaler will keep scaling an ASG while AWS keeps placing nodes in B or C, and the pod stays Pending for minutes. Karpenter does not have this problem: it reads the pending pod's topology constraints, asks the EC2 Fleet API for capacity in exactly the zone needed, and a node shows up there in 30–60 seconds. If you are serious about [.span]trafficDistribution[.span], you should be serious about Karpenter. They are the same architectural bet.

trafficDistribution gets you to steady state. Scale-up keeps you there; scale-down can break it.

The HPA scale-down gap

A typical workload gets scaled out during peak hours and scaled back in when traffic drops. A Deployment running 9 replicas at noon may shrink to 6 at midnight via HPA, and that scale-down is where another gap opens.

[.span]topologySpreadConstraints[.span] only applies at scheduling time: it tells the scheduler where to place new pods. It does not influence which pod the ReplicaSet deletes when the replica count drops. The ReplicaSet controller's built-in ordering has no zone awareness of its own. Across zones, the choice is effectively arbitrary.

The gap in concrete terms

Three AZs, one Deployment, peak state of 9 replicas spread 3-3-3 across [.span]eu-central-1a/b/c[.span]. HPA scales the Deployment down to 6 as load drops. The ReplicaSet has to delete 3 pods, but its built-in ordering does not know about zones.

A plausible bad outcome: the ReplicaSet deletes the 3 youngest pods, which happen to all be in [.span]eu-central-1a[.span] (because that's the zone Karpenter most recently added capacity to during the scale-up). The cluster lands at 0-3-3. Until the next scale-up event re-populates zone A, every request from a client in eu-central-1a falls back to cross-AZ. The zone went empty during a routine off-peak scale-down.

Even without that worst case, random deletion typically leaves you skewed, 1-2-3 instead of 2-2-2 is common. The under-represented zone is more vulnerable to fallback under load and runs more risk of being emptied entirely by the next scale-down.

Descheduler does not fix it

The instinct is to reach for [.span]kubernetes-sigs/descheduler[.span] with the [.span]RemovePodsViolatingTopologySpreadConstraint[.span] profile. But descheduler is reactive. It evicts pods that are already violating spread. The pods are recreated, the next scale-down may break the spread again, and the cycle restarts. Each pass through that cycle adds cross-AZ traffic in the window between violation and rebalance.

What you actually need is to influence the ordering of pods that get deleted before the ReplicaSet picks one.

Pod-deletion-cost controller

That is what we built. lablabs/pod-deletion-cost-controller. A Kubernetes operator that watches Deployments annotated with [.span]pod-deletion-cost.lablabs.io/enabled: "true"[.span] and writes per-pod values into the standard [.span]controller.kubernetes.io/pod-deletion-cost[.span] annotation.

[.radek][.poznamka]NOTE
‍The [.span]controller.kubernetes.io/pod-deletion-cost[.span] annotation, beta and on-by-default since Kubernetes 1.22 (KEP-2255), lives on pods and lets you express a preference for which pods inside a ReplicaSet are deleted first when the replica count drops. Pods with lower cost go first. Nothing outside of the ReplicaSet controller's scale-down path reads this annotation, but that path is exactly where the gap is.[.poznamka][.radek]

The algorithm is keyed on [.span]topology.kubernetes.io/zone[.span] - within each zone, pods get descending cost values (first pod = [.span]MaxInt32[.span], second = [.span]MaxInt32 - 1[.span], and so on). Each zone allocates independently. When the ReplicaSet scales down, the lowest-cost pods go first, and because each zone has assigned its own descending sequence, the deletion lands one pod per zone proportionally.

Practical example: from 3-3-3 down to 6 replicas, the ReplicaSet picks the lowest-cost pod from each zone and lands cleanly at 2-2-2 instead of an arbitrary skew like 0-3-3 or 1-2-3. Topology spread is preserved across the scale-down without Descheduler's reactive churn, and [.span]trafficDistribution[.span] keeps doing in-zone routing without fallback windows.

To use it you just need one annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  annotations:
    pod-deletion-cost.lablabs.io/enabled: "true"
spec:
  replicas: 6
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule

A few caveats before adopting:

• It manages an upstream-supported annotation and nothing else — no eviction triggering, no Karpenter integration. The ReplicaSet controller does the rest.
• Now it has only one built in algorithm for this particular use case but we made it modular for different use cases as well.
• It addresses HPA scale-down, manual replica reduction, and Deployment rollouts.
• v0.1.0, so still early. Feedback welcome.

Why this matters to a platform team

The cross-AZ line item rarely shows up in cost reviews because it has no service name attached to it — it is spread across every microservice, every DNS lookup, every NLB hop. By the time it lands on a CTO's spreadsheet, the bill has usually been accumulating for years. And the fix is not a vendor purchase: a Service annotation, a Deployment annotation, and an autoscaler choice the team has probably already made. The longest piece of work is measuring, because nothing else gets prioritized until the bill has a number against a service name.

The cost story is the lead, but not the only payoff. In-zone routing shortens the request path  [shorter physical distance, fewer ENIs to traverse, lower tail latency on chatty service-to-service calls] and [.span]trafficDistribution[.span] is a strict reliability upgrade over TAR, with automatic cross-zone fallback when an in-zone endpoint set goes empty. The same set of changes lowers the bill, shortens the path, and tightens the failure modes.

Those choices still have to be made deliberately, and remade as the cluster changes. That is the design center of our LARA platform: the HA-safe, locality-preferring choice as the paved path, inherited by default rather than rediscovered cluster by cluster. Karpenter, NodeLocal DNS Cache, cost-tuned NLBs, and the pod-deletion-cost-controller come wired in, with zone-aware Service routing ready to switch on per workload. Cost-optimal and highly available is the baseline, not a tradeoff to engineer toward.

Practical remediation on AWS EKS

To bring cross-AZ traffic down on a production EKS cluster, apply these steps in order. Earlier steps unblock the ones below them, and none of them require a new vendor.

1. Gain visibility. Set up pod-level attribution before touching anything else. Raw VPC Flow Logs are not enough on a churning cluster. Pick one of the options from Seeing the bytes and confirm that the same Services keep showing up as your worst offenders week over week.
2. Tune your NLBs for in-zone traffic. Switch to ip target mode via [.span]service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip[.span] to eliminate the NodePort hop, and set [.span]dns_record.client_routing_policy=availability_zone_affinity[.span] via [.span]service.beta.kubernetes.io/aws-load-balancer-attributes[.span] to pin the client→NLB connection to the same zone. Add [.span]load_balancing.cross_zone.enabled=false[.span] on the same annotation only after backend zonal spread is in place, otherwise the NLB can find itself with no in-zone backends.
3. Deploy NodeLocal DNS Cache. DaemonSet from [.span]kubernetes-sigs/dns[.span]. Cache hits collapse cluster-DNS cross-AZ traffic to near zero. Cheap, well-tested, no application changes — and a prerequisite for the rest of the work if your DNS volume dominates the bill.
4. Apply [.span]spec.trafficDistribution: PreferSameZone[.span] on hot Services, paired with [.span]topologySpreadConstraints[.span] ([.span]maxSkew: 1[.span], [.span]whenUnsatisfiable: DoNotSchedule[.span], [.span]topologyKey: topology.kubernetes.io/zone[.span]) on the backing Deployments, with Karpenter underneath. Without zone-targeted node provisioning, topology-spread leaves pods Pending every time the cluster scales. These three pieces are one architectural bet, not three independent decisions.
5. Check the Cilium KPR flag. If you run Cilium kube-proxy-replacement, set [.span]loadBalancer.serviceTopology: true[.span] in your Helm values. Without it, step 4 silently no-ops on Cilium-routed traffic.
6. Even out HPA scale-down. If daily peak/off-peak cycles or rollouts are leaving your zones uneven, opt the affected Deployments into lablabs/pod-deletion-cost-controller. One annotation per Deployment; step 1's observability tells you whether this is your problem.
7. Monitor and iterate. Roll changes out one Service at a time and watch the CUR [.span]InterZone-*[.span] lines together with your normal SLO dashboards. The bill should drop on the same Services you flipped: if it does not, the most common causes are step 5 (Cilium KPR without the flag), Pending pods from a too-strict [.span]maxSkew[.span] on awkward replica counts, or a noisy-neighbor Service that was not on the top-N list when you started.

What not to do: do not turn on Topology Aware Routing ([.span]service.kubernetes.io/topology-mode: Auto[.span]) on a new Service in 2026. Upstream left the heuristic at beta when the [.span]hints[.span] API field went GA in 1.33, for a reason. [.span]trafficDistribution[.span] is the same idea, simpler, and reliability-safe. If you already have TAR enabled, the migration is a one-line swap.

Now you can see the bytes, and stop sending the ones you do not have to.

References

1. AWS EKS Best Practices — Cost Optimization, Networking - the canonical AWS document on EKS inter-AZ data transfer.
2. Amazon EKS — Enhanced Container Network Observability (Nov 2025 announcement) - managed Intra-AZ / Inter-AZ flow visibility, EKS 1.35+.
3. aws-samples/amazon-eks-inter-az-traffic-visibility - Lambda + Athena pod-IP-to-pod-name attribution pattern for VPC Flow Logs.
4. KEP-2433 — Topology Aware Routing - the original heuristic and the [.span]hints[.span] API field that graduated to GA in 1.33.
5. KEP-4444 — Service Traffic Distribution - [.span]spec.trafficDistribution[.span], GA in Kubernetes 1.33.
6. KEP-2255 — Pod Deletion Cost - the [.span]controller.kubernetes.io/pod-deletion-cost[.span] annotation, beta and on-by-default since 1.22.
7. Buoyant — "The trouble with Topology Aware Routing" - the controlled experiment showing TAR's 0%-success failure mode.
8. Kubernetes — Using NodeLocal DNSCache in Kubernetes clusters - DaemonSet, link-local interception, and cache mechanics.
9. Cilium — Local Redirect Policy - required for NodeLocal DNS Cache interception under Cilium kube-proxy-replacement.
10. lablabs/pod-deletion-cost-controller - Labyrinth Labs operator that writes per-zone descending [.span]pod-deletion-cost[.span] values to preserve topology spread across HPA scale-down.

‍

‍