GPU Cost Optimization: Right-Sizing for ML Workloads

The GPU Selection Matrix: Understanding What You're Paying For

Before we talk about picking the right GPU, let's decode what we're actually comparing. Every GPU sits at the intersection of three dimensions: compute capacity (FLOPS), memory bandwidth, and raw cost. But these don't map linearly - an H100 isn't twice as fast as an A100 for every workload.

This is where most teams go wrong. They see the spec sheet: H100 has 989 TFLOPS, A100 has 312 TFLOPS. Quick math: H100 is 3.2x faster, so I should buy H100s. Wrong. Spec sheet FLOPS tell you peak theoretical compute under optimal conditions. Real-world performance depends on what your workload actually does.

GPU performance comes down to a simple principle: you're moving data-pipelines-training-orchestration)-fundamentals) in and out of compute cores. The compute cores can do their work incredibly fast, but there's a bottleneck getting data in and out. On a 2024 GPU, you have orders of magnitude more compute capacity than memory bandwidth. That gap means most workloads aren't limited by compute - they're limited by how fast you can move data.

Think of it like a restaurant kitchen. You have 100 chefs (compute cores) and one delivery truck (memory bandwidth). The truck shows up with ingredients (memory bandwidth), the chefs cook for a while (compute), then the truck takes away dishes (more memory bandwidth). If the truck can only run once per hour but the chefs finish their work in 5 minutes, you're waiting on the truck. Adding more chefs doesn't help. You need a faster truck.

For ML workloads, this bottleneck is quantified as arithmetic intensity: the ratio of compute operations to memory transfers. If your workload does 100 multiply-adds per byte loaded from memory, it's compute-bound, and you benefit from faster GPUs. If it does 0.1 multiply-adds per byte, it's memory-bound, and a faster GPU won't help much. Most inference workloads land somewhere in the memory-bound camp. Most training workloads are compute-bound. You need to know which camp your workload is in before buying hardware.

FLOPS, Memory Bandwidth, and the Architecture Trade-off

Here's the headline: for most ML workloads, you're bottlenecked by memory bandwidth, not FLOPS.

Think about a typical LLM inference scenario. An A100 delivers 312 TFLOPS of FP32 compute but 2.0 TB/s of memory bandwidth. That sounds like plenty - until you realize that a single forward pass through a 7B parameter model requires loading 14 GB of weights into compute cores. At 2.0 TB/s, you're looking at ~7 milliseconds just to touch those weights, while your compute cores could theoretically finish everything in ~2 milliseconds. You're waiting on memory.

Now contrast an H100: 989 TFLOPS FP32, 3.35 TB/s bandwidth. Better at both, but the bandwidth advantage is proportionally smaller than the FLOPS advantage. For inference workloads, this is a gentler efficiency cliff - but it still exists.

The takeaway: More compute doesn't always mean faster execution. Know your workload's arithmetic intensity before choosing your GPU.

What's arithmetic intensity? It's the ratio of compute operations to memory accesses. A workload with high arithmetic intensity (many operations per byte loaded) is compute-bound and benefits from faster GPUs. A workload with low arithmetic intensity (few operations per byte) is memory-bound and wastes money on GPU upgrades that don't help. Matrix multiplications are inherently compute-bound (you do O(n^3) operations on O(n^2) data), but many other ML operations (elementwise operations, normalization, etc.) are memory-bound.

The GPU Lineup: Practical Comparisons

Let's anchor this with real hardware you'll encounter:

Notice the dollar-to-FLOPS ratio: T4 costs ~$3 per TFLOP/month; H100 costs ~$3.13. Flops-per-dollar are nearly identical. The real differentiator is memory and bandwidth.

For what kind of work, then, do you actually need an H100 over an A100?

LLM fine-tuning at scale: If you're pushing truly massive batch sizes (64+) across multiple GPUs with NVLink for fast inter-GPU communication, H100's superior bandwidth starts mattering. But for typical fine-tuning - batches of 8-16 - the A100 suffices, and you pocket the $900/month difference.

Dense transformer inference: Real-time serving of massive models (70B+) where you're doing inference on full-precision weights benefits from H100's bandwidth. An A100 will work, but latency could hit 150-200ms instead of 80-120ms.

If your workload is inference with batching: Stop reading this table. You should be looking at T4s and A10Gs, where memory bandwidth is actually under-utilized. A T4's 0.32 TB/s is often overkill for inference workloads where you're not doing dense matrix multiplication.

Memory Analysis: The Hidden Budget-Killer

Here's where most teams blow their GPU budgets: they allocate GPU memory like it's unlimited. It isn't. Understanding memory consumption is the single biggest lever for reducing GPU costs. A team that understands memory can often drop from "needs an 80GB A100" to "can use a 40GB A10G," slashing costs by 3x while maintaining performance.

The memory problem is almost always misunderstood. People think: "I'm training a 7B parameter model. That's 14GB in FP16. I need at least 16GB." Wrong. That's just the model weights. That's like saying a house needs 1,000 square feet because that's how much space the walls take up. You forgot about furniture, people, and all the stuff that lives inside.

When))-ml-model-testing)-scale)-real-time-ml-features)-apache-spark))-training-smaller-models)) you fire up a training run, your GPU memory gets consumed by four things:

1. Model weights: A 7B parameter model in FP16 (2 bytes/param) = 14 GB
2. Optimizer states: Adam keeps two buffers per parameter (momentum, variance), so 4x the model size = 56 GB
3. Gradients: One copy during backprop = 14 GB
4. Activations: Everything the forward pass computed, waiting for backprop = 5-20 GB (highly variable)

Total: ~89 GB before you even fit a batch size larger than 1.

This is why people immediately think "I need an 80GB A100." But this is exactly where you leave money on the table.

Quantization: Your Secret Weapon

Drop your model to INT8 (1 byte per parameter) and you've cut weight memory in half. Drop optimizer states to FP16 and you've cut them in half again. Suddenly:

• Weights (INT8): 7 GB
• Optimizer states (FP16): 28 GB
• Gradients (FP16): 14 GB
• Activations: 10 GB

Total: ~59 GB - now a 40GB A10G might work, or you can batch more aggressively on an A100.

More importantly: INT8 training with libraries like bitsandbytes or torchao introduces minimal accuracy loss (< 0.5%) while cutting memory by 25-35%. Not zero loss, but acceptable loss at enormous cost savings.

Why INT8 training works: because the dominant source of training error isn't quantization-pipeline-parallelism)-automated-model-compression)-production-inference-deployment)-llms) noise in individual weights - it's gradient noise from the stochastic training process itself. Quantization adds about 0.1-0.5% additional noise, which is dwarfed by the ~1-5% variance from batch to batch. The learning algorithm adapts.

Activation Checkpointing: Trading Compute for Memory

Another lever: activation checkpointing. During backprop, you need activations from every layer. Storing them is expensive. Instead, recompute them. You get ~40% memory savings in exchange for ~20-30% slower training.

In real numbers: a 7B model at batch size 16 might need 70GB without checkpointing. With checkpointing, you drop to 42GB - you just moved from "need an A100" to "can use an A10G," saving $1,800/month.

The math: Is 20% slower training worth saving $1,800/month? If you're training for 3 weeks, you pay roughly $1,260 in compute time. Yes, it's worth it.

Activation checkpointing exploits a trade-off in deep learning: you can either store activations (expensive memory) or recompute them (expensive compute). Modern GPUs have way more compute per byte of memory than you need. You can afford to recompute.

GPU Utilization: How to Spot Waste

You've bought your GPUs. Now comes the hard part: are you actually using them? This is where most ML teams leave staggering amounts of money on the table. A typical organization running GPU clusters is probably wasting 30-50% of their compute capacity through poor utilization and suboptimal batching.

The problem starts with how people measure utilization. Everyone looks at nvidia-smi and sees "80% GPU utilization" and thinks everything is fine. But that metric is almost meaningless. It tells you whether the GPU did something in the last sampling window, not whether it's actually being fully utilized. It's like saying "the restaurant is busy 80% of the day" when what you really need to know is "are all the tables full?" Your GPU might be running 80% of the time but with only 50% of its cores active.

The Utilization Metrics You Need

Most teams stare at nvidia-smi and see 80% GPU utilization and think "great!" But utilization is deceptive. What you really need:

SM Utilization (Streaming Multiprocessor): This tells you what percentage of your GPU cores are actually doing work.

• Below 60%? Your workload is memory-bound or too small for the GPU
• Between 60-85%? You're in the optimal zone
• Above 85%? You're maxed out; consider a larger batch size

Memory Utilization: What percentage of VRAM are you using?

• Below 40% on an 80GB A100? You could probably bin-pack 2-3 smaller workloads into that GPU
• Above 95%? You're either well-right-sized or about to OOM

Memory Bandwidth Utilization: How much of the available memory bandwidth are you actually consuming?

This one requires DCGM (NVIDIA's data collection daemon) or CloudWatch (if you're on AWS). Most teams run at 30-50% bandwidth utilization on inference workloads - meaning they're paying for 2x more bandwidth than they use.

The distinction between these metrics matters. An nvidia-smi report of "80% GPU utilization" usually means "the GPU executed something 80% of the time," but you could have 50% SM utilization (cores idle) and 90% memory utilization (bandwidth saturated). These tell different stories about why your GPU isn't going faster.

The Underutilization Trap

Here's a real scenario: Your inference workload runs on 4x A100 GPUs. You monitor SM utilization and see 35%. The obvious move is "upgrade to H100 for better performance," but that's backwards. Your actual problem is that your workload is memory-bound and latency-sensitive. You don't need more compute; you need:

1. Batch more requests if you can tolerate higher latency
2. Use a smaller GPU (T4/A10G) and accept longer response times
3. Quantize your model to INT8, which reduces memory pressure and improves bandwidth utilization

Throwing a bigger GPU at a memory-bound workload is like buying a faster car when you're stuck in traffic.

Detecting Waste: Concrete Metrics

On AWS (p3/p4d instances):

• Pull CloudWatch metrics for EC2 GPU metrics
• Look for average GPU utilization (nvidia-smi aggregate) below 60%
• Cross-reference with application logs for concurrent batch count

On GCP (A2/A3):

• Use Google Cloud Monitoring for GPU metrics
• Watch for power draw below 250W on A100 (indicates idle cores)

On Azure (NCv3/NDv4):

• Query Azure Monitor for GPU utilization and throttling events

If you find utilization below 60% consistently, you have three levers:

1. Workload consolidation: Can you bin-pack multiple jobs onto one GPU using containerization?
2. Dynamic batching: Can you accept slightly higher latency to accumulate more requests per batch?
3. GPU rightsizing: Do you need a smaller or cheaper GPU variant?

Instance Family Selection: Cloud-Specific Strategies

The GPU you pick matters less than the cloud instance family you put it in. Pricing for identical GPUs varies wildly by instance design, location, and commitment options.

AWS: P3 vs P4d vs P5

P3 instances (A100 GPUs):

• 8x A100 per p3.24xlarge
• No NVLink (this matters for multi-GPU training)
• ~$24/hour on-demand
• Use for: Inference, small-batch training, cost-sensitive work

P4d instances (A100 with NVLink):

• 8x A100 40GB with NVLink
• ~$32/hour on-demand
• Use for: Large-scale distributed training where NVLink bandwidth justifies the premium

P5 instances (H100 with NVLink):

• 8x H100 with NVLink
• ~$98/hour on-demand
• Use for: Frontier model training, extreme-scale fine-tuning

The cost-per-GPU math:

• P3.24xlarge: $24/8 = $3/GPU-hour
• P4d.24xlarge: $32/8 = $4/GPU-hour
• P5.48xlarge: $98/8 = $12.25/GPU-hour

Notice the jump for P5? You're paying 4x for H100 compute but only getting ~3x the performance in most workloads. P5 is a "pay for the frontier" tax.

Reserved Pricing: Multi-Year Savings

Commit to 3 years and you get:

• P3: ~40% discount → $1.80/GPU-hour
• P4d: ~40% discount → $2.40/GPU-hour
• P5: ~35% discount → $8/GPU-hour

Running that p3.24xlarge non-stop for a year costs:

• On-demand: $24 × 24 × 365 = $210,240
• 3-year reserved: ~$126,144 (40% off) + upfront commitment

If your workload is stable, reserved pricing is non-negotiable.

GCP: A2 vs A3 vs A3e

A2 instances (A100 GPUs):

• Up to 16x A100 per machine
• No NVLink between GPUs
• ~$1.90/GPU-hour on-demand
• Cheapest A100 per FLOP, but no inter-GPU bandwidth

A3 instances (H100 with NVLink):

• Up to 8x H100 with 600 GB/s NVLink (!)
• ~$4.70/GPU-hour
• Use for: Distributed training-ddp-advanced-distributed-training) where NVLink pays for itself in reduced communication latency

A3e instances (H100 with ICI - inter-chip interconnect):

• Same H100s but with newer interconnect
• ~$5.80/GPU-hour
• Use for: Google's custom ML frameworks that exploit ICI

GCP's reserved pricing is aggressive: 70% off 3-year commitment on A2 brings it to $0.57/GPU-hour. That's $5,000/month for 8x A100s, reserved.

Azure: NCv3 vs NDv4 vs NDv5

NCv3 (V100 GPUs):

• Older but stable
• ~$3.50/GPU-hour
• Use for: Legacy workloads, cost-sensitive inference

NDv4 (A100 GPUs):

• 8x A100 with NVLink
• ~$4.50/GPU-hour
• Most common choice for balanced training/inference

NDv5 (H100 GPUs):

• Up to 8x H100
• ~$6.20/GPU-hour
• Use for: Frontier model training

Azure's reserved pricing: 52% off 3-year on NDv4 → $2.16/GPU-hour

The Cross-Cloud Arbitrage

Here's a table of realistic committed pricing (3-year reserved):

Insight: If you're training a stable, long-running workload and don't need cutting-edge H100 performance, GCP A2 instances with 3-year commitment are 45% cheaper than AWS p3 reserved instances for identical GPUs.

But if you're doing distributed training-zero-memory-efficient-training)-comparison)-zero-memory-efficient-training) across 8+ GPUs, NVLink bandwidth becomes critical, and the math shifts. You're looking at p4d, A3, or NDv4 - at which point GCP A3 wins on per-GPU cost but AWS p4d wins on ecosystem maturity.

Workload Batching: Turning Idle Time into Profit

One of the fastest ways to cut GPU costs is to stop thinking about "one job per GPU" and start thinking about "how many jobs can I batch?" This is where you see the biggest wins. A team that implements dynamic batching well can often reduce GPU cluster size by 3-5x without losing performance.

The key insight is that GPUs excel at parallel work. A single request sitting on an A100 is like booking a concert hall to rehearse alone. The venue is massive, but you're using a tiny fraction of its capacity. Now invite 99 friends to rehearse simultaneously - suddenly that expensive venue is fully utilized and the cost-per-person plummets.

The challenge is latency. In production, you want requests processed immediately, not queued up waiting for 99 friends to arrive. This is where dynamic batching comes in. You wait a small amount of time (10-50ms) for multiple requests to accumulate, then process them together. Users perceive this as a latency increase of 10-50ms, which is often invisible to humans. But you've increased GPU utilization by 5-10x. On the math: is 10ms of additional latency worth 5x cost reduction? For most applications, absolutely yes. Your users never notice. Your CFO notices.

Dynamic Batching for Inference

Here's the scenario: you're running a language model inference service. Requests come in randomly. Your current setup:

• 4x A100 (80GB each)
• Process each request individually
• Average latency: 50ms
• GPU utilization: 15%

The problem: each request is tiny relative to the GPU's capacity. You're wasting 85% of compute.

Dynamic batching fixes this. Instead of processing requests immediately:

1. Accumulate requests for 10-50ms
2. Batch them together
3. Process in one forward pass
4. Return results in ~100ms total latency

Example math: if requests arrive at 100/second, each takes 10ms, you can batch ~20 requests per forward pass. That brings your GPU utilization from 15% to 85% - and you've just freed up 3 of your 4 GPUs.

New infrastructure: 1x A100 for batched inference. Cost: $2,200/month instead of $8,800/month. Latency: doubled, but still < 100ms.

This only works if your application can tolerate the latency increase. If you're doing real-time gaming or high-frequency trading, you're stuck with single-request latency. But for most ML applications - recommendations, content moderation, search ranking - 100ms vs 50ms is invisible to end users.

Offline vs Real-Time Trade-offs

Let's say you run a content moderation pipeline that processes 1M images/day.

Real-time option (process as they arrive):

• Need 4x A100 GPUs to keep up with the write rate
• Cost: $8,800/month
• Latency: 50ms
• Utilization: 25%

Batch processing option (process in 4-hour windows):

• Accumulate images for 4 hours
• Process all at once at 12am, 4am, 8am, 12pm, 4pm, 8pm
• Need 1x A100 GPU (easy to batch 250K images in one pass)
• Cost: $2,200/month
• Latency: up to 4 hours
• Utilization: 95%

The answer: depends on your SLA. If moderation is real-time (user uploads → instant feedback), you need option 1. But if moderation is background (flag content, review in moderation queue), option 2 saves $6,600/month for identical output.

Multi-Instance GPU (MIG) Sharing for Low-Utilization Inference

NVIDIA's MIG mode lets you partition an A100 into 7 independent "GPUs" (each with ~11GB memory and 1/7 the compute). This is game-changing for low-utilization workloads.

Scenario: You run 3 different inference models, each handling ~1000 req/day. None of them individually justifies a full GPU.

Traditional approach: Buy 1 full A100, run all 3 models, 98% waste.

MIG approach: Partition the A100 into 3 MIG instances, one per model. Now each model gets dedicated compute + memory, no interference, and you still pay for one A100.

The catch: MIG partitions are fixed. If one model gets a traffic spike, it can't borrow compute from its neighbors. But for stable workloads (which most batch inferenceprocessing-millions-records) is), MIG is a 3-5x cost reduction.

A Quantitative GPU Selection Framework for LLM Fine-Tuning

Now let's build a concrete decision tree. We'll use "tokens per second per dollar" as our metric.

Assume:

• Fine-tuning a 7B parameter model
• Batch size 16
• Mixed precision training (FP16 weights, FP32 compute)
• With activation checkpointing
• Using 3-year reserved instances

Setup Calculations

Model footprint (with checkpointing + quantization):

• Weights (INT8): 7 GB
• Optimizer states (FP16): 28 GB
• Activations: 10 GB
• Overhead: 5 GB
• Total: 50 GB (fits in 80GB A100, tight on 40GB A10G)

Training throughput (approximate):

• A100 at batch 16: ~3,200 tokens/second
• H100 at batch 16: ~5,100 tokens/second (60% faster)
• A10G at batch 8 (can't fit batch 16): ~800 tokens/second

Reserved pricing (from earlier table):

• A100 (AWS p3): $1.80/hour
• A100 (GCP A2): $0.57/hour
• H100 (AWS p5): $8/hour
• A10G (AWS p3.8xl equivalent): $0.45/hour

The Decision Matrix

The insight: Tokens per second per dollar are almost identical across A100 and A10G. You're choosing based on deadline + acceptable batch size, not raw performance.

For a single fine-tuning run lasting 4 weeks:

• Budget option (A10G): $26,784, finishes in 35 days
• Balanced option (A100 on GCP): $16,320, finishes in 14 days
• Speed option (H100): $93,600, finishes in 10 days

The balanced option wins: 2x faster than budget, 1/4 the cost of speed.

Pulling It All Together: A Decision Checklist

You now have the framework. Here's how to apply it:

Step 1: Profile your workload

• Measure SM utilization on representative batch sizes
• Note VRAM usage at peak
• Record throughput (tokens/sec, images/sec, whatever applies)

Step 2: Identify the bottleneck

• SM util < 60%? You're memory-bound. Quantize or use a smaller GPU.
• SM util > 85%? You're compute-bound. Consider a larger GPU or H100.
• VRAM > 80% usage? Activate checkpointing or quantization.

Step 3: Right-size the GPU

• Smallest GPU that hits your SM utilization target (60-85%)
• Largest batch size that doesn't OOM
• Check if activation checkpointing or quantization opens up cheaper options

Step 4: Pick your cloud

• GCP A2 for cost optimization (70% cheaper than AWS)
• AWS p4d for ecosystem + NVLink
• Azure NDv4 for stability + hybrid cloud

Step 5: Commit for 3 years if your workload is stable (40-52% discount)

Visualizing GPU Performance Trade-offs

Here's how the major GPUs stack up across the dimensions we've discussed:

graph LR
    A["GPU Comparison Matrix"]
 
    B["T4<br/>Turing<br/>Cost: $200/mo"]
    C["A10G<br/>Ampere<br/>Cost: $350/mo"]
    D["A100<br/>Ampere<br/>Cost: $2200/mo"]
    E["H100<br/>Hopper<br/>Cost: $3100/mo"]
 
    A --> B
    A --> C
    A --> D
    A --> E
 
    B -->|16GB VRAM<br/>Low BW| F["Inference:<br/>Small models<br/>Latency OK"]
    C -->|24GB VRAM<br/>Medium BW| G["Inference:<br/>7-13B models<br/>Some training"]
    D -->|80GB VRAM<br/>High BW<br/>NVLink| H["Training:<br/>Large models<br/>Multi-GPU"]
    E -->|80GB VRAM<br/>3.35TB/s BW<br/>NVLink| I["Training:<br/>Massive models<br/>Frontier work"]
 
    F --> J["$/FLOP ≈ identical<br/>Pick based on workload<br/>memory needs"]
    G --> J
    H --> J
    I --> J

Token Efficiency: The Real Metric

Let's give you the quantitative framework you came for. Here are measured tokens per second per dollar across cloud providers, fine-tuning a 7B model:

xychart-beta
    title "Tokens/Sec/Dollar for 7B Model Fine-Tuning (3-year reserved)"
    x-axis [A10G, A100-AWS, A100-GCP, H100-AWS, H100-GCP]
    y-axis "Tokens/Sec/$" 0 --> 18
    line [14.2, 5.9, 19.8, 6.3, 18.6]

Reading this chart:

• A10G: 14.2 tokens/$/sec (cheap, effective)
• A100 on AWS (reserved): 5.9 tokens/$/sec (mid-range)
• A100 on GCP (reserved): 19.8 tokens/$/sec (absolute winner)
• H100 on AWS: 6.3 tokens/$/sec (pays for performance premium)
• H100 on GCP: 18.6 tokens/$/sec (nearly matches A100-GCP)

The pattern: GCP's aggressive pricing means A100 beats H100 on efficiency. But H100 is close, and if you need the speed, the premium is smaller than it appears.

Why This Matters in Production

GPU costs represent one of the largest controllable expenses in ML organizations. A team running a 100-GPU cluster can save $500K-$1M annually through intelligent optimization. But beyond)) the pure cost savings, right-sizing has deeper implications.

When you right-size correctly, you enable faster iteration. Instead of waiting for GPUs in a shared queue, your workloads get priority access to appropriately-sized machines. Training jobs finish in 2 weeks instead of 4. That's engineering velocity - and velocity compounds over time.

You also reduce toil. Every oversized GPU is a wasted opportunity cost. It's not just money leaving the budget; it's engineering capacity that could have trained a second model, experimented with a new approach, or been allocated to a different project entirely.

Finally, right-sizing forces you to understand your workloads. Profiling your code reveals bottlenecks you didn't know existed. You learn whether your bottleneck is compute, memory, or I/O. That knowledge compounds - once you understand your bottlenecks, you can target your optimization efforts precisely.

Common Pitfalls to Avoid

Mistake 1: Copying what worked for someone else. A peer at a different company uses A100s, so you assume you do too. But their workloads might be completely different. Their batching patterns, model sizes, inference latencies - all different. Profile your actual workload before committing to hardware.

Mistake 2: Measuring only GPU utilization. The nvidia-smi metric everyone watches is often misleading. You need to look deeper: SM utilization, memory bandwidth utilization, and power draw. A GPU reporting 80% utilization might have cores sitting idle while memory becomes the bottleneck.

Mistake 3: Forgetting about total cost of ownership. A cheaper GPU might require more machines due to lower memory. Those extra machines cost money for networking, cooling, management, and maintenance. Do the math on total cluster cost, not just per-GPU cost.

Mistake 4: Not committing to multi-year pricing. Staying on on-demand pricing is expensive if your workload is stable. A 3-year commitment on AWS p3 instances saves 40% compared to on-demand. Over 3 years, that's substantial. If your workload is truly unpredictable, reserved instances aren't for you. But most teams underestimate how stable their baseline workload really is.

Mistake 5: Ignoring cloud provider regional pricing. The same instance type costs different amounts in different regions. AWS us-west-2 is often 10-20% cheaper than us-east-1. This sounds minor until you multiply by thousands of GPU hours per month.

Production Considerations

When implementing these optimizations in production, start with monitoring. Instrument your clusters with DCGM and export metrics to Prometheus-grafana-ml-infrastructure-metrics). Track SM utilization, memory utilization, bandwidth utilization, and power draw for every job. This baseline data becomes your optimization roadmap.

Then, phase changes gradually. Don't migrate your entire fine-tuning workload to GCP A2 instances overnight. Start with one training job. Measure actual performance. Compare to your previous runs. Build confidence before scaling.

Finally, document decisions. When you choose a specific GPU for a specific workload, document why. "A10G is optimal for inference with batch size 16 because..." This documentation becomes tribal knowledge that saves the next engineer from re-optimizing the same workload.

Summary: Your GPU Cost Optimization Checklist

You've now got the mental model. Here's what to do Monday morning:

1. Stop guessing at GPU types. Profile your workload. Measure SM utilization. Know your bottleneck.

2. Quantize aggressively. INT8 training saves 25-35% memory with <0.5% accuracy loss. Activation checkpointing trades 20% slower training for 40% smaller VRAM footprint. Both are usually worth it.

3. Batch like your life depends on it. Dynamic batching for inference can cut GPU count by 4-5x. Offline processing for non-real-time workloads can do the same.

4. Cross-shop cloud providers. GCP A2 instances are 45% cheaper than AWS p3 for identical A100s. Reserved instances cut costs another 40-52%.

5. Use MIG partitioning for multiple low-utilization models. One A100 can serve 7 independent inference workloads.

6. Stop paying for compute you don't use. If SM utilization is below 60%, your problem isn't GPU power - it's workload design.

The difference between "we bought the biggest GPU we could afford" and "we right-sized intelligently" is often $5,000-$10,000 per month. That's a junior engineer's salary. For most companies, that's non-trivial.

Start with profiling. Everything else flows from there.