When VPC Peering Stops Scaling: Moving to Transit Gateway

Introduction

When I started at my current job, we were running self-managed, bare-metal Kubernetes clusters on kOps, partway through a migration to EKS. That migration was also our chance to redesign the networking. Under kOps, a handful of clusters shared a VPC, roughly one VPC for every three or four clusters, so around 80 clusters came out to a manageable number of VPCs. For EKS we wanted a tighter boundary: one VPC per cluster, so a cluster’s blast radius stopped at its own network. That gave us the isolation we wanted, and a new problem we hadn’t planned for. Eighty clusters now meant close to eighty VPCs, and VPC peering doesn’t stay manageable at that size.

The mesh problem

VPC peering connections are point to point. There’s no such thing as transitive peering, so if VPC A peers with B and B peers with C, A still can’t reach C unless you peer A and C directly too. That means the number of connections you need grows with the number of pairs of VPCs, not the number of VPCs.

With 4 VPCs that’s 6 peering connections. With 10 VPCs it’s 45. Worse, that growth lands on you one VPC at a time: every new VPC needs its own peering connection to every VPC that’s already there.

Add a fifth VPC to a set of four and that’s four new peering connections, each with its own route table entries on both sides and its own security group rules, not one. Multiply that by eighty clusters and you stop reasoning about the mesh as a diagram at all, you just grep route tables when something can’t connect.

Where Transit Gateway fits

A Transit Gateway is a regional router that VPCs attach to instead of attaching to each other. Each VPC needs exactly one attachment, and traffic between any two VPCs routes through the gateway.

Adding a VPC now means one attachment and one route, not one new connection per existing VPC. The route tables live on the gateway, in one place, instead of scattered across every VPC that needs to reach every other one.

The attachment itself is a small piece of Terraform:

hcl
resource "aws_ec2_transit_gateway" "main" {
  description = "Cluster VPC connectivity"
}

resource "aws_ec2_transit_gateway_vpc_attachment" "cluster" {
  transit_gateway_id = aws_ec2_transit_gateway.main.id
  vpc_id             = var.vpc_id
  subnet_ids         = var.private_subnet_ids
}

Every cluster VPC gets one of these attachments, and a route pointing at the transit gateway for the CIDR ranges of the other VPCs. That’s the whole change, per VPC, from here on.

The same problem for datastores and Lambdas

Cluster VPCs peering with each other isn’t the only traffic pushing on this. Datastores usually end up isolated into their own VPC too, an RDS instance behind its own subnets and security groups, and every cluster that reads from it needs a path in. A Lambda function calling a service running inside a cluster needs the same kind of path, just from the other direction.

Neither of those is a new cluster added to the mesh, but each one is a new participant, and peering treats it the same way it treats a cluster VPC: one more VPC, one more connection to every VPC it needs to reach. A shared RDS VPC that every one of 80 cluster VPCs reads from picks up all 80 of those connections itself. Its route table and its security groups end up tracking every consumer that was ever added.

Put that RDS VPC on the Transit Gateway instead and it’s one more attachment, same as a cluster. A Lambda VPC is the same story. Reachability now comes down to Transit Gateway route tables , not which VPCs happen to have a peering connection to the datastore: an attachment only gets a route to it if it’s associated with a route table that carries one.

hcl
resource "aws_ec2_transit_gateway_route_table" "datastores" {
  transit_gateway_id = aws_ec2_transit_gateway.main.id
}

resource "aws_ec2_transit_gateway_route_table_association" "rds" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.rds.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.datastores.id
}

resource "aws_ec2_transit_gateway_route_table_association" "cluster" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.cluster.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.datastores.id
}

Only attachments associated with datastores get a route to the RDS VPC. Everything else on the gateway can still reach the cluster VPCs directly, just not the datastore behind them.

This is also where the Lambda cross-region PrivateLink setup I wrote about earlier stops applying. That one existed because the Lambda needed to cross a region boundary we deliberately don’t route through. A Lambda in the same region calling a cluster service doesn’t need a one-off endpoint at all once its VPC is already an attachment on the gateway everything else uses. It just needs a route, the same as any other consumer.

What Transit Gateway costs

VPC peering connections themselves are free. Data crossing a peering connection is billed like any other cross-AZ traffic, $0.01 per GB, same region, in each direction. Transit Gateway bills differently: $0.05 per attachment per hour, whether or not it’s carrying anything, plus $0.02 per GB processed, twice the peering rate (us-east-1 figures, other regions vary slightly).

Take the 80 cluster VPCs from the redesign, plus the RDS VPC and the Lambda VPC from the last section, moving a combined 20 TB in a month:

text
VPC peering, full mesh
  cluster to cluster   80 × 79 / 2           = 3,160 connections
  RDS to each cluster  80 × 1                =    80 connections
  Lambda to its cluster                      =     1 connection
  connections total                          = 3,241

  data transfer  20,000 GB × $0.01/GB        = $200/mo

Transit Gateway
  attachments    82 × $0.05 × 24h × 30d      = $2,952/mo
  data transfer  20,000 GB × $0.02/GB        =   $400/mo
  total                                      = $3,352/mo

That extra $3,152 a month buys 82 attachments to one gateway instead of 3,241 pairwise connections nobody could keep straight, the RDS VPC and the Lambda VPC included.

Conclusion

Peering doesn’t fail loudly. Nothing breaks when you add one more VPC past the point where the mesh stopped scaling, you just accumulate more manual route table work than any one person can track correctly. The other cost never shows up on a bill: try explaining a 3,241-connection peering topology to an engineer who joined last week, versus pointing at one gateway and saying everything goes through here. If I were redesigning that networking again knowing we’d end up with eighty VPCs, I’d go straight to a Transit Gateway instead of peering.