The situation
The client was a Series B FinTech running two LLM workloads on Azure AKS: a document analysis pipeline processing loan applications (batch, latency-tolerant) and a real-time assistant for their operations team (interactive, latency-sensitive). Both ran on Standard_NC24ads_A100_v4 nodes (A100 80GB).
Their problem wasn't that GPU was expensive — it was that they had no visibility into why it was expensive. The Azure bill showed one number. No breakdown by model, by team, by time of day. The document pipeline and the interactive assistant shared the same node pool, making it impossible to optimize either without understanding which was the actual cost driver.
On top of that: nodes ran 24/7. The document pipeline processed loans during business hours (8am–6pm IST). The assistant was used 9am–7pm. From 7pm to 8am — roughly 13 hours — the GPU nodes were idle, accumulating cost with zero utilization.
The actual problem: not high GPU costs, but unobserved GPU costs. You can't optimize what you can't measure. Step one was always going to be attribution before automation.
Step 1 — cost attribution with OpenCost
Before touching a single node pool, we installed OpenCost into the cluster and labeled workloads by team and model. The default Kubernetes cost allocation by namespace wasn't granular enough — they had all inference workloads in one namespace.
The fix was straightforward: add app and team labels to every Deployment, configure OpenCost to aggregate by those labels, and expose the output via the OpenCost Prometheus exporter into their existing Grafana stack.
# Added to every inference Deployment metadata: labels: app: "loan-doc-analysis" # or "ops-assistant" team: "ml-platform" cost-center: "lending-ops"
Within 48 hours of turning on OpenCost, the answer was clear: the document pipeline was consuming 71% of GPU spend despite being lower-priority batch work. The interactive assistant — the one actually customer-facing — was running on 29% of the budget. That inversion drove the entire optimization strategy.
Step 2 — split the node pools
The two workloads had completely different requirements, so they needed separate node pools:
- Batch pool (document pipeline): latency-tolerant, can tolerate interruption, runs during business hours only → spot instances, scale-to-zero at night
- Interactive pool (operations assistant): latency-sensitive, needs guaranteed capacity, ~10 hour window → on-demand reserved, minimum 1 node always warm
Splitting them in Terraform took about half a day. The key was setting node_taints and using Kubernetes nodeSelector to ensure each workload landed on its intended pool.
# Terraform — batch spot node pool resource "azurerm_kubernetes_cluster_node_pool" "batch_gpu" { name = "batchgpu" kubernetes_cluster_id = azurerm_kubernetes_cluster.main.id vm_size = "Standard_NC24ads_A100_v4" priority = "Spot" eviction_policy = "Delete" spot_max_price = -1 # pay up to on-demand price min_count = 0 max_count = 4 enable_auto_scaling = true node_taints = ["kubernetes.azure.com/scalesetpriority=spot:NoSchedule"] node_labels = { "workload-type" = "batch" } }
Spot A100 nodes on Azure run at roughly 60-70% discount vs on-demand during off-peak. For batch workloads that don't have hard latency requirements, this is essentially free money.
Step 3 — KEDA scale-to-zero for the batch pool
Even with spot pricing, idle nodes cost money. The batch pipeline ran 8am–6pm on business days. Outside those hours, the nodes sat at 0 utilization but still billed.
We used KEDA with an Azure Storage Queue scaler — the pipeline drops jobs onto a queue, and KEDA scales the Deployment based on queue depth. When the queue is empty, KEDA scales to zero replicas, and the cluster autoscaler removes the now-empty node.
apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: loan-doc-analysis-scaler namespace: inference spec: scaleTargetRef: name: loan-doc-analysis minReplicaCount: 0 # full scale-to-zero maxReplicaCount: 8 cooldownPeriod: 300 # 5 min before scaling down triggers: - type: azure-queue metadata: queueName: loan-doc-jobs queueLength: "2" # 1 replica per 2 queued jobs connectionFromEnv: AZURE_STORAGE_CONNECTION_STRING
The cold start penalty for vLLM loading a 13B model onto an A100 is roughly 45-90 seconds. For batch processing, that's completely acceptable. For interactive workloads it isn't — which is why the interactive pool kept a minimum of 1 warm node.
Result: the batch pool now runs zero nodes from roughly 6pm to 8am on weekdays, and all weekend. That's about 14 hours/day + 48 weekend hours = ~118 idle hours eliminated per week.
Step 4 — vLLM tuning on the interactive pool
The interactive assistant was using the default vLLM configuration. Two quick wins here:
Continuous batching was already on (vLLM default since 0.2.x), but max_num_seqs was set conservatively. With the operations assistant having ~20 concurrent users peak, we raised it from 32 to 64 — better GPU utilization per request without extra hardware.
GPU memory utilization was set to the default 0.90. Profiling showed the model was only using ~55GB of the 80GB A100. We dropped gpu_memory_utilization to 0.75, which freed headroom and reduced OOM-related pod restarts that were silently causing retries and inflating token usage.
# vLLM deployment args — after tuning args: - "--model" - "mistralai/Mistral-7B-Instruct-v0.3" - "--max-num-seqs" - "64" - "--gpu-memory-utilization" - "0.75" - "--max-model-len" - "8192" # trimmed from 32768 — actual p99 context was 4K - "--enable-prefix-caching" # big win for repeated system prompts
The --enable-prefix-caching flag was the biggest surprise. The operations assistant had a long system prompt (~800 tokens) that was identical on every request. With prefix caching on, that token computation was reused across requests — measurable reduction in GPU compute per request, which means better throughput on the same hardware.
Step 5 — reserved capacity for the baseline
The interactive pool kept 1 node warm at all times. On-demand A100 pricing on Azure runs ~$3.40/hr for the NC24ads_A100_v4. Over a year, that's ~$30K for one always-on node.
With usage patterns confirmed (the pool had been stable for 4+ months), we committed to a 1-year reserved instance for that node. Azure reserved pricing for A100 is roughly 35-40% cheaper than on-demand. That saved another ~$10-12K/year on just the baseline node.
The numbers
All figures are approximate. Client details are anonymized per NDA. Numbers have been reviewed and confirmed as directionally accurate.
| Change | Monthly saving |
|---|---|
| Batch pool → spot instances (60% discount on compute) | ~$1,800 |
| KEDA scale-to-zero (eliminated ~118 idle GPU hours/week) | ~$1,200 |
| Reserved instance on interactive baseline node | ~$900 |
| vLLM prefix caching (fewer tokens computed, lower GPU hours) | ~$100 |
| Total monthly saving | ~$4,000 |
From $13K/month to $9K/month. $48K/year. Done in two months, with three engineers involved (one on the ML side for model config, one on the infra side, and me coordinating the architecture).
What surprised us
Attribution was the hardest part, not the optimization. The actual KEDA YAML, spot node pool config, and vLLM flags took maybe 3 days combined. Getting clean cost attribution in OpenCost — properly labeled namespaces, correct Prometheus aggregation, Grafana dashboards the engineering team would actually look at — took 2 weeks. Most companies skip this step and go straight to optimization, which means they can't tell if their changes worked.
The batch workload was the cost driver, not the interactive one. Everyone assumed the always-on interactive pool was expensive. It wasn't — it was well-utilized during business hours. The batch pipeline was the problem because it had no scale-to-zero and was running on expensive on-demand nodes for work that had zero latency requirements. The attribution step was what revealed this. Without it, we might have optimized the wrong thing entirely.
Cold start on vLLM is a non-issue for batch. The team was worried about KEDA scaling to zero because they'd heard vLLM was slow to start. Turns out 60-90 seconds cold start is irrelevant when your batch jobs take 3-10 minutes each. This mental block had been stopping them from implementing scale-to-zero for months.
What we didn't do
We did not quantize the models, switch to smaller models, or implement any changes that could affect output quality. Every cost saving came from infrastructure decisions: where workloads run, when they run, and how they're priced. The models themselves were untouched.
We also didn't implement multi-tenant vLLM model sharing (running multiple models on the same GPU via time-slicing or MPS). That's a valid next step if cost pressure continues, but the savings profile is complicated and the operational risk wasn't justified at their current scale.
The monitoring setup that made it stick
The final piece was making the savings visible on an ongoing basis. We built a Grafana dashboard with four panels:
- GPU cost by workload (OpenCost, daily, 30-day rolling)
- Node pool utilization vs idle time (DCGM metrics)
- KEDA scaling events (when did we scale to zero, how long was cold start)
- Month-over-month cost trend vs baseline
The ML team now owns the GPU cost dashboard. When a new model experiment goes to prod, they can see its cost attribution within 24 hours. That cultural change — ML engineers thinking about cost at deployment time, not after the bill arrives — is probably worth more than the $48K/year in the long run.
Could this apply to your setup?
The combination that worked here — OpenCost attribution + split node pools + KEDA scale-to-zero + spot for batch — applies to any AKS or EKS cluster running LLM inference at scale. The specific numbers depend on your workload mix, instance types, and usage patterns.
If you're spending more than $5K/month on GPU inference and you don't have per-model cost attribution, you almost certainly have the same problem: you don't know where the money is going, so you can't optimize it.
The first step is always the same: get attribution in place before you touch anything else.
Want a free GPU cost audit for your cluster?
We'll review your current setup — node pools, vLLM config, idle patterns — and give you a written report with the top 3 savings levers. No commitment.
Book a Free 15-Min Call →