Fix vLLM CUDA Out of Memory at Startup: KV Cache + Docker

Two Different OOM Errors, Two Different Fixes

You downloaded the model. The weights fit on the GPU. vLLM starts loading. Then, before the first prompt reaches the server, the engine dies.

Depending on your vLLM version and where initialization failed, the useful part of the traceback may look like this:

ValueError: No available memory for the cache blocks.
Try increasing `gpu_memory_utilization` when initializing the engine.

Or you may see the more familiar:

torch.OutOfMemoryError: CUDA out of memory

These errors look similar in logs. They are not fixed the same way. If you blindly lower gpu_memory_utilization from 0.90 to 0.80, you may fix a CUDA OOM while making a KV cache capacity error worse. If you blindly raise it, you can do the opposite. The right fix depends on which part of vLLM's startup memory calculation failed.

Current vLLM code explicitly recommends increasing gpu_memory_utilization for KV cache errors and either increasing utilization or decreasing max_model_len when the requested sequence length will not fit. (vLLM GitHub)

This article walks through that calculation, the Docker-specific checks worth doing, and the vLLM settings that actually matter for production deployment. For broader production AI architecture patterns, see secure enterprise RAG architecture.

Why the Widely Copied Fix Is Partly Wrong

A common Stack Overflow recommendation looks like this:

LLM(
    model=model_id,
    gpu_memory_utilization=0.80,
    block_size=8,
    swap_space=4,
    enforce_eager=True,
    disable_custom_all_reduce=True
)

Several parts of that configuration deserve correction.

gpu_memory_utilization=0.80 can help if vLLM is attempting to use too much VRAM and CUDA itself runs out during profiling. It is not a universal fix. If vLLM is telling you the KV cache has no available memory, lowering utilization gives vLLM an even smaller memory budget. Current vLLM documentation lists the default as 0.92, not 0.90 as older tutorials claim. (vLLM Docs)

block_size=8 is not a generic OOM remedy. Current vLLM documentation does not define a universal static block-size default. The value is resolved according to platform and cache configuration. vLLM sizes the KV cache primarily from its available memory budget. Reducing block granularity does not reduce total KV-cache memory. (vLLM Docs)

swap_space=4 is deprecated and ignored in current vLLM's LLM constructor. Modern vLLM exposes separate mechanisms for model-weight CPU offloading and KV-cache offloading instead. (vLLM Docs)

disable_custom_all_reduce=True disables vLLM's custom all-reduce kernel and falls back to NCCL. That can be useful for compatibility issues, but current documentation does not describe it as an OOM-control mechanism. (vLLM Docs)

Do not tune five unrelated flags because one Stack Overflow answer happened to start successfully with them. Diagnose which memory boundary failed first.

What vLLM Does With Your GPU During Startup

Stop thinking about VRAM as a single number. If nvidia-smi says 24564 MiB total, that does not mean vLLM has 24 GB available for the KV cache. The GPU must accommodate several consumers:

+-----------------------------------+
|         Total GPU VRAM            |
|  (e.g. 24564 MiB = ~24 GB)        |
+-----------------------------------+
|  Model weights                    |
|  (e.g. 8B params x 2 bytes = 16GB)|
+-----------------------------------+
|  Runtime / non-PyTorch allocs     |
+-----------------------------------+
|  Peak activation memory           |
+-----------------------------------+
|  CUDA graph memory                |
+-----------------------------------+
|  KV cache                         |
|  (what remains after profiling)   |
+-----------------------------------+
|  Safety / unused headroom         |
+-----------------------------------+

vLLM's GPU worker performs a profiling run because it cannot know all of those values from model parameter count alone. The engine loads the model, executes a profiling pass with dummy inputs to estimate non-KV peak memory, then determines how much remains for the KV cache. Recent vLLM versions also account for estimated CUDA graph memory in this calculation. (vLLM Docs)

The startup profiling sequence looks like this:

vLLM Startup Memory Profiling
         |
         v
  Load model weights onto GPU
         |
         v
  Run profiling pass with dummy inputs
  (measures peak activation memory)
         |
         v
  Estimate CUDA graph memory
  (v0.21.0+ accounts for this)
         |
         v
  Calculate remaining budget:
  gpu_memory_utilization x total VRAM
    - weights - activations - graphs
    = available KV cache memory
         |
         v
  Allocate KV cache blocks
         |
    +----+----+
    |         |
   OK        FAIL
    |         |
    v         v
 Server     "No available memory
 starts     for the cache blocks"

Conceptually, the calculation is: requested vLLM memory budget minus model, activation, runtime, and CUDA graph memory equals available KV-cache memory. This mental model is more useful than treating GPU memory as a single pool.

The Diagnostic Table

The distinction between failure patterns will save more time than any magic 0.80 setting:

The Two Failure Paths

The most important distinction in vLLM startup OOM is which memory boundary failed. These two failure paths require opposite fixes in some cases:

Failure A: CUDA allocation OOM          Failure B: KV cache capacity error
         |                                       |
         v                                       |
  "torch.OutOfMemoryError:              "No available memory for
   CUDA out of memory"                  the cache blocks"
         |                                       |
         v                                       v
  GPU ran out of physical              vLLM budget too small for
  memory during allocation             required KV cache capacity
         |                                       |
         v                                       v
  FIX: Lower utilization               FIX: Increase utilization
  (give vLLM smaller target)          (if real VRAM is available)
  OR: Free GPU from other procs        OR: Reduce max_model_len
  OR: Reduce max_model_len             OR: Use FP8 KV cache
  OR: Disable CUDA graphs              OR: Tensor parallelism
         |                                       |
         v                                       v
  More headroom = safer               More budget = more cache
  but less cache capacity             but less safety headroom

Lowering gpu_memory_utilization helps Failure A but worsens Failure B. Raising it helps Failure B but worsens Failure A. Read the error message before touching any setting.

Step 1: Verify Docker GPU Access

A Docker container does not automatically give vLLM a private pool of VRAM. The NVIDIA Container Toolkit's --gpus and NVIDIA_VISIBLE_DEVICES controls determine which GPUs are visible. (NVIDIA Docs)

Start on the host:

nvidia-smi

Then check inside your vLLM container:

docker exec -it <vllm-container> nvidia-smi

If the server has not started yet, validate GPU access independently:

docker run --rm --gpus all \
  nvidia/cuda:12.8.0-base-ubuntu22.04 \
  nvidia-smi

What matters is the memory picture immediately before vLLM starts. If another Python process, notebook, or neighboring container owns several gigabytes of VRAM, changing vLLM's internal cache settings does not reclaim that memory. For production container deployment patterns, see building internal AI applications with Streamlit, RAG, and MCP.

Step 2: Confirm Model Weights Fit

Paged KV caching cannot rescue a model whose weights consume all available VRAM. A rough unquantized weight calculation: parameter count times bytes per parameter. An 8-billion-parameter model in BF16 is approximately 16 GB of weights. That is before runtime memory, activations, CUDA graphs, and KV cache.

If model weights leave almost no GPU headroom, tuning block_size is not the fix. Your options: use a smaller model, use an appropriate quantized model, use tensor parallelism across multiple GPUs, or offload model weights to CPU. vLLM's memory-conservation guide recommends quantization and tensor parallelism for this class of problem. (vLLM Docs)

Step 3: Reduce max_model_len Before Chasing Obscure Flags

Modern open-weight models advertise context windows of tens or hundreds of thousands of tokens. That does not mean your deployment needs to reserve enough KV-cache capacity to serve that entire context. If your application only expects 4K prompts, launching with a much larger maximum imposes memory requirements you do not need.

vllm serve "$MODEL" \
  --max-model-len 4096 \
  --max-num-seqs 1

vLLM explicitly recommends reducing max_model_len and max_num_seqs to reduce memory consumption. (vLLM Docs) The production value should come from your workload requirements, not from whichever number makes the server boot.

Step 4: Tune gpu_memory_utilization Based on the Error

For a true CUDA OOM, a conservative diagnostic run:

vllm serve "$MODEL" \
  --gpu-memory-utilization 0.80 \
  --max-model-len 4096 \
  --max-num-seqs 1

If that starts successfully, the previous configuration was operating too close to the device's usable memory boundary. You can then increase utilization gradually while monitoring startup logs.

But if the result is No available memory for the cache blocks, the smaller memory target left too little room for the cache. Do not keep marching utilization downward. You are shrinking vLLM's cache budget. Either reduce the requested context or, if nvidia-smi confirms genuine unused VRAM, allow vLLM a larger fraction. The error text matters. Read it.

Step 5: Use enforce_eager for CUDA Graph Memory Issues

vLLM uses CUDA graphs for performance, and graphs consume GPU memory. The project's memory-conservation documentation explicitly recommends enforce_eager=True to disable CUDA graph capture completely. (vLLM Docs)

vllm serve "$MODEL" \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.80 \
  --enforce-eager

If eager execution starts but the normal configuration does not, CUDA graph memory is part of the boundary you are crossing. Treat eager mode as a diagnostic and capacity lever, not an automatic production default. Disabling graphs can reduce inference performance. For runtime verification of production AI systems, see llmverify.

Step 6: Consider FP8 KV Cache for KV Capacity Bottlenecks

vLLM supports quantized KV cache formats including FP8 on supported hardware. FP8 KV-cache quantization can significantly reduce the cache memory footprint, allowing more tokens or greater concurrency. (vLLM Docs)

vllm serve "$MODEL" \
  --kv-cache-dtype fp8

Quantization strategy and scale calibration affect accuracy. vLLM's documentation recommends dataset-based calibration when appropriate. Use KV-cache quantization because you have measured a KV-capacity bottleneck and tested model quality, not simply because an OOM appeared.

The Production Tuning Order

When vLLM will not start, change things in this order:

That order deliberately favors explanations over superstition.

A Production Debugging Decision Tree

vLLM fails during startup
          |
          v
Is the error a real CUDA allocation OOM?
          |
     +----+----+
     |         |
    YES        NO
     |         |
     v         v
Check       Does vLLM say
nvidia-smi  KV cache is too small?
     |         |
Free VRAM     +----+----+
     |        YES        NO
Reduce         |          |
memory         v          v
target     Lower        Inspect exact
context    max_model_len failure stage
batch          |
graphs     If physical
     |     VRAM exists,
     |     consider higher
     |     utilization
     v
Does model weight footprint fit?
     |
 +---+---+
 |       |
NO      YES
 |       |
 v       v
Smaller/   Profile and
quantized  tune workload
model,
TP,
offload

What Not to Deploy

Do not build production logic that catches a CUDA OOM and instantiates another vLLM engine with increasingly arbitrary settings:

try:
    llm = LLM(gpu_memory_utilization=0.80)
except RuntimeError:
    llm = LLM(gpu_memory_utilization=0.70)

That pattern hides the distinction between OOM classes and makes deployment behavior dependent on runtime failure. A server that silently falls back to 0.70 because 0.80 failed today will behave differently tomorrow when a neighboring container frees 2 GB of VRAM. The 0.70 configuration was never validated. It was just the next number down.

Use explicit configuration validated before the service receives traffic:

import os
from vllm import LLM

MODEL_ID = os.environ["MODEL_ID"]

llm = LLM(
    model=MODEL_ID,
    gpu_memory_utilization=float(
        os.getenv("VLLM_GPU_MEMORY_UTILIZATION", "0.80")
    ),
    max_model_len=int(
        os.getenv("VLLM_MAX_MODEL_LEN", "4096")
    ),
    max_num_seqs=int(
        os.getenv("VLLM_MAX_NUM_SEQS", "1")
    ),
    enforce_eager=os.getenv(
        "VLLM_ENFORCE_EAGER", "false"
    ).lower() == "true",
)

The values should come from capacity testing for your model and hardware. A server should not silently decide that because 80% failed today, 70% is suddenly the correct production configuration. For governance of production AI deployments, see how to secure and govern AI systems.

Startup OOM vs Runtime KV Cache Pressure

Do not confuse startup OOM with runtime KV-cache pressure. At runtime, concurrent requests and long contexts consume KV-cache capacity. vLLM may preempt requests and recompute when cache pressure becomes high rather than crashing. The performance-tuning guide recommends increasing available KV cache, decreasing maximum batch parameters, or distributing the model across more GPUs when preemption becomes frequent. (vLLM Docs)

A server that cannot initialize at all is a different problem. The distinction matters when searching logs because both involve KV cache and memory.

Final Checklist: vLLM OOM Before the First Request

• Confirm the exact vLLM version
• Run nvidia-smi on the host
• Run nvidia-smi inside the container
• Check for other GPU processes
• Confirm model weights realistically fit
• Set max_model_len to the application's real requirement
• Reduce max_num_seqs while diagnosing
• Identify CUDA OOM vs KV-cache-capacity error
• Tune gpu_memory_utilization in the correct direction
• Test enforce_eager if graph memory is implicated
• Consider quantization if weights are the problem
• Consider FP8 KV cache if KV capacity is the problem
• Consider tensor parallelism for multi-GPU systems
• Consider CPU/KV offloading only with measured tradeoffs
• Load-test the final configuration after startup succeeds

The one setting that does not belong at the top of that list is block_size. And changing gpu_memory_utilization from 0.90 to 0.80 does not instantly fix vLLM. Sometimes it does. Sometimes the correct move is exactly the opposite. The error message tells you which problem you actually have.

If you are deploying self-hosted LLM infrastructure and need an independent systems advisor to audit your production readiness before traffic hits, schedule a strategic evaluation.

FAQ

How do I fix vLLM CUDA out of memory during startup? First check actual GPU usage with nvidia-smi. Then identify whether CUDA itself failed an allocation or vLLM calculated insufficient KV-cache capacity. For true CUDA OOMs, reducing gpu_memory_utilization, max_model_len, or max_num_seqs can help. For insufficient KV cache errors, reducing max_model_len or increasing the vLLM memory budget when physical VRAM is available is usually more appropriate.

Should I set gpu_memory_utilization from 0.90 to 0.80? Not automatically. Lowering it provides additional GPU headroom and may solve actual CUDA allocation OOMs. But it also reduces the memory budget available to vLLM's KV cache. If the error says the KV cache is too small, lowering utilization may make the problem worse.

What is the default gpu_memory_utilization in vLLM? It is version dependent. Many older vLLM releases documented a default of 0.90, while current vLLM documentation lists 0.92. Always check the documentation matching your installed version.

Does lowering block_size fix a vLLM OOM? Not reliably. Current vLLM does not use one universal static block-size default, and total KV-cache capacity is primarily driven by available memory. Block size controls cache organization, not total memory.

Does enforce_eager reduce vLLM memory? It disables CUDA graph capture, which consumes additional GPU memory. This can reduce memory requirements, but the amount saved is model- and hardware-dependent and may come with a performance cost.

Can FP8 KV cache reduce vLLM memory? Yes, on supported hardware. vLLM documents FP8 KV-cache quantization as a way to significantly reduce KV-cache memory and support more tokens or concurrency. Accuracy and calibration should be tested before production use.

Does Docker cause vLLM CUDA OOM errors? Not inherently. Docker and the NVIDIA Container Toolkit control which GPUs are exposed to the container. You still need to inspect actual VRAM consumption and other processes using the same device.

What is PagedAttention in vLLM? PagedAttention is the memory-management approach introduced with vLLM for handling the KV cache in blocks, inspired by virtual-memory paging. It reduces KV-cache waste and enables efficient sharing and batching. It cannot make a configuration fit when the model, runtime, and required KV cache exceed physical GPU capacity. (arXiv)