My guess is the network routing is happening in the software defined network AWS creates below the traditional networking layers.

Sometimes it's tempting to think AWS' services are just like traditional appliances you'd stick in the middle of a network path but in reality they're all virtual and run on top of their software defined network in something they call hyperplane.

That means traditional intuition about how packets flow, like needing to be in the path to preserve the IP, may not apply.

It seems like in this case they need to preserve the static public IP that was used on the NLB for the incoming request across the whole transaction but maybe aren't doing that.

> they need to preserve the static public IP that was used on the NLB for the incoming request

Even if they did that, to determine which outgoing public IP the packet should go to, they would need to guess based on segment numbers and ack offsets. This would still be a guessing game

Oh for sure. Distributed routing like this is complex!

My main point is we have no idea what AWS' SDN looks like under the covers. Traditional rules about routing may not apply and each node in the path may have a lot more information about traffic than a traditional router would have.

You can run NLB in ip mode instead of instance mode and I don’t think it will have this issue. In ip mode your server sees the IP address of the NLB instead of the client.

There is no simple way to do this when using Kubernetes. NLB provisioned via Kubernetes will use instance mode, and you cannot change that, and aws-alb-ingress-controller doesn't support NLBs.

Weirdly, provisioning NLB via Kubernetes supports `aws-load-balancer-cross-zone-load-balancing-enabled` annotation, even though this is quite broken behaviour, as per the article.