Constellates

Cloud NAT & VPC Networking

6 min read

VPC (Virtual Private Cloud)

A VPC is your private network inside GCP. Think of it as your own isolated section of the cloud where your VMs, containers, and services talk to each other using private IPs. Traffic within a VPC stays private — the internet can’t reach in unless you explicitly allow it.

A Shared VPC lets one “host” project own the network, while other “service” projects (like your GKE clusters) attach to it. This way, one team controls the network, and other teams just use it.

CIDR Notation

CIDR (Classless Inter-Domain Routing) is how you write IP ranges. 10.36.80.0/20 means “the first 20 bits are the network prefix, the remaining 12 bits are for hosts.” That gives 2^12 = 4,096 addresses (10.36.80.0 through 10.36.95.255).

Common sizes: /16 = 65K IPs, /20 = 4K IPs, /24 = 256 IPs, /32 = 1 IP.

Subnets

A VPC is divided into subnets — IP ranges (CIDRs) assigned to a region. Each subnet lives in one region (e.g., us-east4). VMs and pods get their IPs from the subnet they’re in.

Subnets matter for NAT because NAT rules can target specific subnets. If you want different NAT behavior for different workloads, you put those workloads on different subnets.

Cloud NAT

Cloud NAT lets VMs with only private IPs access the internet without exposing them publicly. It sits at the edge of your VPC and translates private IPs to public IPs for outbound traffic.

How it works

Pod (private IP: 10.0.1.5)
  → leaves the VPC
  → Cloud NAT rewrites source to a public IP (e.g., 35.199.0.71)
  → request reaches GitHub
  → GitHub sees 35.199.0.71, not 10.0.1.5
  → response comes back to 35.199.0.71
  → Cloud NAT translates back to 10.0.1.5
  → Pod receives the response

NAT IPs and Ports

Each public NAT IP has ~64K usable ports. Every outbound connection from a VM uses one port. So one IP can handle ~64K concurrent connections.

When you have multiple IPs, Cloud NAT distributes VMs across them. Each VM gets a reservation of ports (controlled by min_ports_per_vm and max_ports_per_vm).

Port Allocation Settings

  • min_ports_per_vm — ports reserved upfront per VM, even if idle. Lower = more VMs can share an IP. Higher = guaranteed headroom for bursts but fewer VMs per IP.
  • max_ports_per_vm — cap on how many ports a single VM can grab. Prevents one bursty VM from eating an entire IP.
  • Dynamic port allocation — VMs start at min and scale up to max as needed. There’s a small lag when scaling up.
  • tcp_established_idle_timeout / tcp_time_wait_timeout — how long ports stay reserved after connections close. Lower = ports recycle faster.

NAT Rules

NAT rules let you route traffic to different IPs based on the destination. Each rule matches a destination IP range and assigns specific NAT IPs.

Example: if GitHub’s servers resolve to 3 different IP ranges, you can create 3 rules, each with different NAT IPs. This controls which of your public IPs are used for which destinations.

The key learning: Cloud NAT does not evenly distribute traffic across IPs within the same rule. The allocation algorithm is not publicly documented, and in practice, traffic tends to concentrate on a few IPs. The fix: split into one IP per rule so there’s no choice to make — each destination range gets exactly one IP.

Monitoring

  • nat/allocated_ports — ports currently reserved per VM per IP. Shows distribution.
  • nat/dropped_sent_packets_count with reason OUT_OF_RESOURCES — packets dropped because no ports available.
  • get-nat-mapping-info — live snapshot of which VMs have ports on which IPs. Not historical, just the current moment.

Cloud NAT operates at Layer 3/4 (IP and TCP). It cannot see HTTP status codes, URLs, or headers. If someone returns a 401, Cloud NAT has no idea — it just forwards the TCP packets.

GKE Secondary IP Ranges (Pod Ranges)

A GKE subnet has a primary IP range (for node IPs) and one or more secondary IP ranges (for pod IPs). When you create a node pool, you assign it a secondary range via pod_range — all pods on that pool get IPs from that range.

Multiple secondary ranges can live on the same subnet, and multiple node pools can share the same secondary range — each pod just gets a unique IP from the pool. The only reason to use separate ranges is for network-level isolation (e.g., routing different ranges through different NAT gateways).

This matters for NAT because Cloud NAT can target specific secondary ranges using LIST_OF_SECONDARY_IP_RANGES, giving you per-pool egress control without needing separate subnets.

GKE Node Pools and NAT

See Kubernetes Concepts for details on Node Pools, nodeSelector, and Tolerations.

Different node pools can use different pod ranges (secondary IP ranges). Cloud NAT can route different secondary ranges through different NAT gateways. By steering workloads to specific node pools, you control which egress IPs they use.

Isolating Traffic with a Dedicated NAT

To route a subset of workloads through a specific egress IP:

  1. Add a new secondary IP range to the existing subnet
  2. Create a new Cloud NAT with its own static IP, targeting only that range
  3. Modify the existing NAT to exclude the new range — switch from ALL_IP_RANGES to an explicit LIST_OF_SECONDARY_IP_RANGES
  4. Create a node pool with pod_range pointing to the new range, plus a taint for scheduling control
  5. Configure workloads to tolerate the taint

Key constraint: Cloud NAT does not allow two gateways on the same router to cover the same IP range. When carving out a dedicated range, you must explicitly enumerate all other ranges in the existing NAT.

GKE SNAT Masquerade Breaks Secondary Range NAT

Gotcha: GKE masquerades pod IPs → node IPs (iptables SNAT) before packets reach Cloud NAT. So a dedicated NAT targeting a secondary pod range never sees traffic — Cloud NAT sees the node IP (primary range) and routes through the main NAT instead.

Fix: disable default SNAT on the cluster:

default_snat_status {
  disabled = true
}

Updates in-place, no cluster recreation. Pod IPs are preserved → Cloud NAT sees the actual secondary range → routes to the correct gateway. Safe for CI/CD clusters where all external traffic goes through Cloud NAT.

ip-masq-agent is an alternative (ConfigMap + DaemonSet with nonMasqueradeCIDRs: ["0.0.0.0/0"]) but the cluster flag is simpler.

GKE multi-subnet clusters (1.30.3+) aren’t a fix either — you can’t control which subnet a node pool uses. GKE auto-selects based on IP availability.

Graph View

Backlinks