Ollama vs llama.cpp vs vLLM: The Abstraction Tax Comes for Your GPU

A hands-on comparison of Ollama, llama.cpp, and vLLM for local inference, pricing out the abstraction tax in tokens per second, VRAM, and setup friction.

GPU server running local LLM inference with Ollama, llama.cpp, and vLLM

Ollama vs llama.cpp vs vLLM: The Abstraction Tax Comes for Your GPU

Nolan Lawson's post on the AI cost of JavaScript frameworks sat near the top of HN this week, and the argument underneath it isn't really about React. It's about abstraction taxes. A framework that renders beautifully but forces an LLM to emit 40% more tokens to accomplish the same task is not free — you just moved the cost from the developer's brain to the inference bill.

I read that post while staring at nvidia-smi on my homelab box, watching a 14B model chug through a RAG query at 22 tokens/sec, and felt a very specific kind of recognition.

Because we do the exact same thing with local inference. We pick Ollama because the ollama run experience is lovely, then act surprised when a 4-way concurrent load test gets queued instead of batched. We pick vLLM because it has PagedAttention in the README, then discover it wants 24GB of VRAM before it will even say hello to a 7B model.

Same tax, different currency. Here, you pay in tokens per second, watts, and the number of evenings you spend reading GitHub issues.

Let me price it out.

The lineage matters, because it explains the trade-offs

llama.cpp is the bottom of the stack. Georgi Gerganov's C/C++ implementation of the LLaMA architecture, built originally so a 7B model could run on a MacBook. It's CPU-first by design, with GPU offload via -ngl, and it supports basically every accelerator that isn't actively hostile: Metal, CUDA, ROCm, Vulkan, SYCL. It defines the GGUF format. It ships llama-server, which is a genuinely good OpenAI-compatible HTTP server with continuous batching, slot management, and a /slots endpoint you can poll to see who's hogging the GPU.

Ollama is a Go daemon that wraps llama.cpp (with a growing number of alternative runners — most notably for Apple Silicon) and adds the things a C++ project is bad at: a model registry, a Modelfile format, sensible defaults, automatic model loading and unloading, and a systemd unit that actually works. It exposes both its native /api/chat and an OpenAI-compatible /v1 surface. If llama.cpp is a hand-tuned engine, Ollama is the car around it, including the cupholders.

vLLM is not a wrapper around anything. It's a Python serving engine built from scratch around PagedAttention — the idea that KV cache should be managed like virtual memory pages rather than one contiguous allocation per sequence. That single design decision is why vLLM can hold 10–30x the concurrent sequences of a naive implementation. It's GPU-first, tensor-parallel aware, and designed for the thing you do when you're serving an actual product: many requests, at once, with predictable latency.

The tax is visible in the lineage. Ollama is an abstraction over llama.cpp. llama.cpp is an abstraction over ggml. Every layer buys convenience and charges throughput.

The comparison, honestly

llama.cpp Ollama vLLM
Language C/C++ Go (+ llama.cpp) Python
Primary target CPUs, Apple Silicon, single-GPU Desktops, laptops, small servers Multi-GPU, datacenter
Model format GGUF GGUF HF safetensors (AWQ/GPTQ/FP8 quants)
Quantization breadth Excellent (Q4_K_M, IQ quants, etc.) Same as llama.cpp Narrower but GPU-native
Continuous batching Yes (llama-server) Yes, but tunable via env vars Yes, that's the whole point
PagedAttention No No Yes
Tensor parallelism Limited (--split-mode) No Yes (--tensor-parallel-size)
GPU memory preflight Rarely fails Rarely fails Fails loudly, often
OpenAI-compatible API Yes Yes Yes
Setup friction Medium (build flags) Lowest Highest
Good at 1 user Great Great Fine, wasteful
Good at 32 users Mediocre Mediocre Excellent

The row that matters most is the last one. Everything above it is arguing about the shape of the cupholder.

What actually changed recently

Three things, roughly in order of how much they affected my setup.

KV cache quantization stopped being scary. llama.cpp's --cache-type-k q8_0 --cache-type-v q8_0 cuts KV memory roughly in half with a quality hit I cannot detect on a 14B model. Ollama exposes the same thing via OLLAMA_KV_CACHE_TYPE. Two years ago this was a fork-and-pray situation. Now it's a flag, and it's the difference between an 8K context and a 32K context on a 24GB card.

Flash Attention became a first-class flag (-fa / --flash-attn) rather than something you enabled and then debugged for a weekend. Combined with the KV cache work, the practical ceiling for single-GPU inference moved meaningfully.

vLLM's V1 engine rewrite cut scheduler overhead substantially and made prefix caching far more aggressive. If you're serving a RAG workload where every request shares the same 2,000-token system prompt and retrieved context, prefix caching is not a nice-to-have. It's a 5-10x throughput multiplier on the prefill phase.

And one tangent worth naming: Postgres 19 is late because AI-assisted tooling is finding more bugs than expected. That's the second-order cost nobody models. More people running more inference means more edge cases surfacing in the serving layer too — which is an argument for picking the stack with a decade of bug reports behind it over the one with the best benchmark slide.

A concrete setup, with numbers you should not trust

Here's what I actually run. Two configs, same box, same model, different jobs.

For interactive use on the workstation — one person, long context, occasional tool calls:

llama-server \
  -m ./models/Qwen3-14B-Q4_K_M.gguf \
  -ngl 99 \
  -c 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -fa \
  --parallel 2 \
  --host 0.0.0.0 --port 8080

--parallel 2 gives me two slots, so a background RAG job doesn't block my editor's autocomplete. -ngl 99 offloads every layer. This is the config I recommend to anyone who asks, because it's boring and it works.

For the batch jobs — nightly document embedding, eval runs, anything with concurrency — I switch the same box to vLLM:

vllm serve Qwen/Qwen3-14B-AWQ \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --port 8000

Note that vLLM wants the HF repo, not a GGUF file. That's the biggest friction point for people coming from Ollama: your model library doesn't transfer. AWQ quants are the closest analogue to Q4_K_M and they're GPU-native, which is why vLLM doesn't bother with GGUF.

To find out which one you actually need, run a concurrency test instead of a single-shot benchmark. This is the test that changes people's minds:

seq 1 16 | xargs -P 16 -I{} curl -s -o /dev/null -w "%{time_total}\n" \
  http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-14B-AWQ","messages":[
       {"role":"system","content":"You are a terse assistant."},
       {"role":"user","content":"Explain TCP slow start in 3 sentences."}],
       "max_tokens":128}'

Run that against :8080 and :8000. On my hardware, llama.cpp with --parallel 2 serializes into roughly four waves and the worst request lands around 30 seconds. vLLM finishes all 16 in under 6, because it's batching them into a single forward pass instead of pretending the queue doesn't exist. Single-request latency is nearly identical between the two. The difference is entirely in what happens when the second request shows up.

That's the number to measure. Not tokens/sec on a benchmark with one user, which is the AI equivalent of measuring a web framework by how fast it renders a hello world.

How I actually decide

If it's one person, one machine, and the model needs to fit in whatever VRAM you have: llama.cpp, wrapped in Ollama if you want the registry and the service management. The GGUF ecosystem is the reason. Nothing else lets you try a dozen quant levels in an evening.

If you're serving RAG pipelines with local models where every request shares a fat system prompt, or you have more than a handful of concurrent users, or you want tensor parallelism across two cards: vLLM. The prefix caching alone justifies it.

If you want to try models without thinking about any of this: Ollama. Just know that you're paying a tax and that OLLAMA_NUM_PARALLEL and OLLAMA_KV_CACHE_TYPE are the two env vars that determine whether your tax is 10% or 300%.

Personal caveat: I've flip-flopped on this three times. I ran vLLM as my daily driver for six months and went back to llama.cpp because the VRAM preflight failures were eating my evenings. vLLM is fantastic at what it does and hostile to being run casually. If you're not actually serving concurrent traffic, the PagedAttention tax isn't worth paying.

What to do this week

  1. Measure your actual concurrency. If you've never run the xargs -P 16 test above, you don't know which tool you need. Most homelabbers discover they need vLLM exactly zero times, and everyone running a shared service discovers they needed it six months ago.
  2. Flip on KV cache quantization. --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp, OLLAMA_KV_CACHE_TYPE=q8_0 in Ollama. It's free context length. If you can detect the quality difference on a blind eval, turn it off and email me.
  3. If you're on cloud GPUs, revisit the math. The egress era changed the break-even point for local inference significantly. A justified $2,000 GPU pays for itself faster than it did a year ago, and a serving engine that gets 3x the throughput out of that GPU changes the date again.
  4. Don't migrate your model library for a benchmark. The GGUF-to-safetensors conversion is real work. Run the concurrency test on the tool you already have, and only move if the numbers say the tax is bigger than the migration.

The JS ecosystem spent fifteen years learning that the framework you pick has a cost that shows up somewhere else. Local inference is on the same curve, just compressed into about eighteen months. Pick the layer you're willing to pay for.

Disclosure: some links below are affiliate links. If you buy through them I may earn a commission at no extra cost to you. I only list tools I actually run.

ToolWhy it is here
Amazon - SSDs & NVMeBoot drives and storage for your homelabVisit
Amazon - Mini PCsLow-power always-on nodes (ProDesk, EliteDesk, N100 boxes)Visit
Amazon - PC ComponentsCPUs, RAM and cases for a buildVisit
Amazon - GPUsConsumer GPUs for local inference (16GB VRAM is the sweet spot)Visit
Amazon - RAM & MemoryMore RAM is the cheapest upgrade for VMs and containersVisit