
VPA and HPA Together: Salesforce's Rules for 1,400 EKS Clusters
Cloud Platform EngineeringSalesforce runs the Vertical Pod Autoscaler and Horizontal Pod Autoscaler on the same workloads across more than 1,400 Amazon EKS clusters, and says the combination drove more than $5 million in cumulative infrastructure savings. Most teams use one or the other. For platform teams sitting on low-utilization fleets, the rules that kept the two from fighting each other are the useful part.
The Framework
Salesforce’s platform team describes the setup in an AWS Containers blog post. Individual clusters reach 2,500 nodes and about 1,000 workloads. At around 100 clusters, static CPU and memory settings worked; as the fleet grew, they caused throttling, terminated pods, and constant manual tuning.
The organizing idea is to match each workload to one of three patterns by how it scales: VPA alone for workloads that grow by resource size (leader-follower workloads, EKS add-ons), HPA alone for stateless services and webhook servers, or both together. The team also moved from Cluster Autoscaler to Karpenter. The rules below are what made the combined pattern safe.
1. Scale on Independent Metrics
VPA and HPA can share a workload only if they never react to the same signal. Salesforce’s example is the Istio control plane: HPA adds istiod replicas to hold roughly 500 connected proxies per pod, while VPA resizes each pod’s CPU and memory, because every replica generates the same configuration and extra replicas don’t help that part of the job.
2. Start in Off Mode
New workloads get VPA in Off (read-only) mode first, so engineers can review recommendations without automatic pod restarts in production.
3. Let VPA Manage Requests Only
In testing, letting VPA adjust limits from request-to-limit ratios caused throttling or pod termination during spikes, because VPA couldn’t raise limits fast enough. The fix is controlledValues: "RequestsOnly", leaving limits as defined in the workload spec.
4. Set Generous Limits
Salesforce recommends scale-testing to find peak usage, then setting limits to at least 4 times the maximum observed usage. VPA, which works from an 8-day decaying histogram, is for steady-state optimization, not traffic spikes.
5. Throttle the Updater
The team runs the VPA updater every 30 minutes rather than continuously, with an eviction rate limit, Pod Disruption Budgets, and minimum replica thresholds to prevent restart loops.
6. Cap Recommendations With maxAllowed
Without maxAllowed, VPA can recommend more CPU or memory than any node can allocate, leaving critical pods stuck in Pending indefinitely.
Where It Doesn’t Apply
This is a vendor-hosted case study, and the $5 million figure is cumulative with no baseline published. It also assumes serious supporting tooling: Spinnaker and Terraform rollouts, staged bake times, feature-toggle rollback to static configs, and Grafana dashboards for eviction and restart rates. Teams without that safety net should stay in Off mode longer. Stack Archive’s look at Datadog’s approach to Kubernetes autoscaling at scale is worth comparing, and the Kubernetes VPA feature docs cover the in-place resize mode Salesforce plans to adopt next.
The So What
Platform teams with over-provisioned EKS fleets should stop treating VPA and HPA as either-or. Pick one workload with two separable scaling dimensions, run VPA in Off mode beside its existing HPA, set RequestsOnly and maxAllowed before switching it on, and measure the savings before expanding the pattern.
Source: AWS Containers Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →