Introduction
Getting a large language model to produce impressive outputs in a Jupyter notebook is the easy part. Serving that same model to thousands of concurrent users with sub-second latency, while keeping GPU costs from spiraling, is where most engineering teams hit a wall. LLM inference optimization is the discipline that closes this gap, transforming a promising prototype into a cost-efficient production service. The seven techniques covered here represent the most impactful levers available today for inference latency reduction, each with concrete tradeoffs that determine when and where to apply them. The difference between a well-optimized and a naively deployed inference pipeline can easily be 5x to 10x in throughput and cost efficiency.
Reducing Computational Overhead at the Model Level
The first category of optimization targets the model itself, reducing the raw computational cost of each forward pass. These techniques modify how weights are stored, how attention is computed, and how decode steps are generated. They deliver the largest per-request latency improvements and typically require minimal changes to your serving infrastructure.
Technique 1: Weight Quantization
Quantization compresses model weights from 16-bit or 32-bit floating point to lower precision formats like INT8 or INT4, slashing memory bandwidth requirements and enabling faster matrix multiplications on modern GPUs. A 70B parameter model that requires 140 GB of VRAM in FP16 fits comfortably on a single A100 (80 GB) when quantized to 4-bit precision. The practical impact is substantial: INT8 quantization typically yields a 1.5x to 2x speedup in token generation with less than 1% degradation in benchmark accuracy, while INT4 pushes closer to 3x at the cost of more noticeable quality tradeoffs on reasoning-heavy tasks.
GPTQ: Post-training quantization method that uses calibration data to minimize quantization error per layer, ideal for offline model preparation
AWQ (Activation-aware Weight Quantization): Preserves salient weight channels based on activation patterns, often outperforming GPTQ on quality at equal bit-width
bitsandbytes NF4: Provides on-the-fly 4-bit quantization with double quantization, useful for fine-tuning workflows but slightly slower than static methods at inference time
SmoothQuant: Migrates quantization difficulty from activations to weights via mathematically equivalent transforms, enabling effective INT8 quantization for both weights and activations
Technique 2: KV-Cache Optimization
During autoregressive generation, transformer models store key-value pairs from all previous tokens, so they do not need to be recomputed at each decode step. This KV-cache grows linearly with sequence length and batch size, quickly becoming the dominant memory bottleneck. For a 70B model serving 32 concurrent requests at 4096 tokens each, the KV-cache alone can consume over 40 GB of VRAM. Techniques like PagedAttention (used in vLLM) treat KV-cache memory like virtual memory pages, eliminating fragmentation and enabling near-optimal utilization. Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce cache size by sharing key-value heads across attention groups, cutting memory by 4x to 8x with minimal accuracy loss. If you are running long-context workloads, KV-cache management is likely your single biggest optimization opportunity for GPU inference.
Scaling Throughput and Accelerating Decode
Once the per-request cost is reduced, the next set of techniques focuses on how requests are batched, scheduled, and decoded. These optimizations operate at the serving layer, maximizing GPU utilization across many simultaneous requests rather than optimizing a single forward pass in isolation.
Technique 3: Continuous Batching
Static batching, where the server waits for a fixed batch of requests before processing them together, wastes GPU cycles whenever requests in the batch have different output lengths. A request that finishes in 20 tokens sits idle while the batch waits for a 500-token response to complete. Continuous batching (also called iteration-level batching or in-flight batching) solves this by inserting new requests into the batch at every decode iteration, as soon as a slot opens up. The result is dramatically higher inference throughput scaling without increasing individual request latency.
In benchmarks published by the vLLM team, continuous batching delivered 2x to 4x higher throughput compared to static batching under identical hardware conditions. This technique is now the default in production-grade serving frameworks and should be considered table stakes for any deployment handling more than a handful of concurrent users.
Technique 4: Speculative Decoding
Autoregressive generation is inherently sequential: each token depends on the previous one, which means the GPU spends most of its time in memory-bound operations during the decode phase. Speculative decoding breaks this bottleneck by using a small, fast draft model to generate several candidate tokens in parallel, then verifying them in a single forward pass of the larger target model. Tokens that match are accepted instantly; tokens that diverge trigger regeneration only from the first mismatch.
When the draft model has a high acceptance rate (typically 70-90% for well-matched model pairs), speculative decoding can reduce decode latency by 2x to 3x without any change in output quality. The key tradeoff is finding a draft model that is fast enough to provide a net speedup while being accurate enough to maintain a high acceptance rate. Medusa and EAGLE are self-drafting variants that add lightweight prediction heads to the target model itself, eliminating the need for a separate draft model entirely.
Framework Selection and System-Level Tuning
The final three techniques shift focus from algorithmic improvements to the runtime environment and deployment architecture. Choosing the right serving framework and applying system-level optimizations often delivers the last 30-50% of latency reduction that separates a good deployment from a great one.
Technique 5: Optimized Serving Frameworks
The choice between serving frameworks has a measurable impact on latency and throughput. vLLM has emerged as the de facto open-source standard for LLM serving, combining PagedAttention, continuous batching, and tensor parallelism into a single package that works out of the box. NVIDIA's TensorRT-LLM takes a different approach: it compiles the model into a highly optimized engine using NVIDIA-specific CUDA kernels and fusion patterns, extracting maximum performance from NVIDIA hardware at the cost of longer setup times and reduced flexibility.
For US tech teams evaluating TensorRT vs ONNX Runtime, the decision often comes down to hardware constraints and operational requirements. TensorRT-LLM consistently wins on raw throughput for NVIDIA GPUs, typically by 20-40% over vLLM for batch inference workloads. ONNX Runtime offers broader hardware compatibility and is the stronger choice when targeting mixed GPU/CPU environments or cloud-agnostic deployments. Meanwhile, vLLM vs Ray Serve comparisons reveal that vLLM excels at single-model serving while Ray Serve provides superior orchestration for multi-model pipelines and complex routing logic.
Technique 6: Kernel Fusion and Graph Compilation
Individual operations in a transformer (matrix multiplications, layer norms, activation functions, residual additions) each require separate GPU kernel launches by default. Every launch carries overhead: memory reads, writes, and synchronization. Kernel fusion combines multiple sequential operations into a single kernel, reducing this overhead dramatically. FlashAttention fuses the entire attention computation (Q*K, softmax, V multiplication) into a single IO-aware kernel, cutting attention latency by 2x to 4x while also reducing memory usage from quadratic to linear in sequence length.
Graph compilation tools like torch. compile (PyTorch 2.x) and XLA analyze the full computation graph to identify fusion opportunities automatically. These optimizations compound with quantization inference techniques: a fused INT8 attention kernel is faster than either optimization applied independently. For teams building production inference pipelines, enabling FlashAttention and basic graph compilation should be the first steps, as they require minimal code changes and deliver consistent improvements.
Technique 7: Request Routing and Load Balancing
No amount of per-GPU optimization matters if incoming requests are not distributed efficiently across your serving fleet. Naive round-robin load balancing ignores the fact that different requests have vastly different computational costs: a 50-token completion costs a fraction of a 2000-token generation. Intelligent request routing assigns requests based on current GPU utilization, queue depth, and estimated generation length. Prefix caching, where system prompts shared across requests are cached and reused, can eliminate redundant computation for RAG pipeline workloads where every request shares the same lengthy context window.
At the infrastructure level, autoscaling policies should be driven by token throughput metrics (tokens per second per GPU) rather than simple CPU or memory utilization, which lag behind actual inference load. NinjaStudio.ai has covered inference cost breakdowns by provider, and the data consistently shows that teams with intelligent routing and autoscaling spend 30-50% less on compute compared to static fleet configurations.
Conclusion
These seven techniques form a layered optimization strategy: start with quantization and KV-cache management to reduce per-request cost, implement continuous batching and speculative decoding to maximize GPU utilization, then select the right framework and apply system-level tuning to extract every remaining millisecond. The techniques compound, meaning applying all seven in a well-tuned inference pipeline can yield 10x or greater improvements over a default deployment. The key is measuring before and after each change, using standardized benchmarks at realistic concurrency levels rather than single-request latency tests that mask production behavior.
Explore NinjaStudio.ai's LLM technical deep dives for further guidance on building production-grade AI inference acceleration workflows.
Frequently Asked Questions (FAQs)
How to optimize LLM inference?
Apply a combination of weight quantization, KV-cache management, continuous batching, speculative decoding, and framework-level tuning (vLLM or TensorRT-LLM) to reduce latency and increase throughput at each layer of the serving stack.
What are inference optimization techniques?
They include quantization, KV-cache optimization, continuous batching, speculative decoding, kernel fusion, graph compilation, and intelligent request routing, each targeting a different bottleneck in the model serving pipeline.
How does quantization improve inference?
Quantization reduces weight precision from 16-bit to 8-bit or 4-bit, cutting memory bandwidth requirements and enabling faster matrix operations, which typically delivers 1.5x to 3x speedups with minimal accuracy loss.
How does batching improve inference?
Continuous batching inserts new requests into processing slots at every decode iteration, keeping the GPU fully utilized instead of wasting cycles on completed requests within a static batch.
Which is better, TensorRT or ONNX Runtime for production?
TensorRT-LLM delivers 20-40% higher throughput on NVIDIA GPUs through hardware-specific optimizations, while ONNX Runtime offers broader hardware compatibility and is better suited for mixed or cloud-agnostic deployment environments.