In M12, we released the multi-edge feature in VMC on AWS. It is possible to oversaturate the NSX-T edge router under very heavy loads. If that happens, one way to resolve it is to deploy an additional edge pair to spread the load.
Traffic is routed to a specific edge node based on configuring traffic groups. Traffic groups are built using source subnets – your traffic is directed to a specific edge based on the IP address given to the VM in VMC on AWS.
Given the scenario:
VM-A = 10.10.10.10
VM-B = 10.10.10.20
VM-C = 10.10.10.30
We could create a traffic group for 10.10.10.10/32, 10.10.10.20/32, 10.10.10.30/32, creating dedicated edges for each VM. Multi-edge does get expensive as you burn 2 entire hosts worth of capacity just for the edge – one for primary and one for failover.
When you deploy the traffic groups as shown above, the route table in the connected VPC will show the following:
10.10.10.10/32 => ENI1
10.10.10.20/32 => ENI2
10.10.10.30/32 => ENI3
This configuration does nothing to balance the load sourced from EC2 instances in the Connected VPC, meaning if we have 8 EC2 application servers that need to write to VM-A, they will all hit the same ENI – ENI1.
Any EC2 application server in the Connected VPC destined for VM-A will always use ENI1
Any EC2 application server in the Connected VPC destined for VM-B will always use ENI2
Any EC2 application server in the Connected VPC destined for VM-C will always use ENI3