Qwen3.5-9B on an RTX 4060: Ollama vs. llama.cpp

A local Qwen3.5-9B benchmark on an RTX 4060 laptop, with measured throughput from Ollama and llama.cpp.

I tested Qwen3.5-9B on a Lenovo LOQ 15IRX9. It has an NVIDIA RTX 4060 Laptop GPU, 8 GB of video memory, and 24 GB of system memory.

The model ran fully in video memory with both Ollama and llama.cpp. Ollama generated 23.77 tokens per second. The llama.cpp CUDA benchmark generated 37.73 tokens per second.

Result

The llama.cpp result was 59% higher in this test. The tools use different benchmark paths, so treat the result as an observed difference on this machine, not as a universal engine comparison.

Test setup

Item	Value
Laptop	Lenovo LOQ 15IRX9
GPU	NVIDIA RTX 4060 Laptop GPU with 8 GB of video memory
System memory	24 GB
Operating system	Windows 11
Model	Qwen3.5-9B-Q4_K_M
Model file	5.63 GB GGUF with 4-bit quantization
Engines	Ollama and llama.cpp with CUDA

I selected the 9-billion-parameter model because its quantized file fits on an 8 GB GPU. This test asks a practical question: Can a common gaming laptop run the model fast enough for interactive work?

Ollama result

Ollama provides a simple local setup and an OpenAI-compatible API at localhost:11434.

ollama pull qwen3.5:9b
ollama run qwen3.5:9b --verbose

Ollama reported:

eval rate:         23.77 tokens/s
prompt eval rate:  35.53 tokens/s
VRAM usage:        6126MB / 8188MB
GPU utilization:   55%

The eval rate measures output generation. The prompt eval rate measures input processing. I also ran nvidia-smi during inference and confirmed that Ollama did not offload model layers to the CPU.

At 23.77 tokens per second, the model responded fast enough for interactive use. The response was not instant, but I did not have to wait long between generated phrases.

llama.cpp CUDA result

llama.cpp provides lower-level control over the model runtime. On Windows, I used the prebuilt CUDA package.

To set it up, I:

Downloaded the prebuilt CUDA binary from the llama.cpp releases page.
Copied the required CUDA runtime dynamic-link libraries into the same folder.
Ran --list-devices to confirm that llama.cpp detected the GPU.
Ran llama-bench with the model fully assigned to the GPU.

.\llama-bench.exe `
  -m "C:\models\Qwen3.5-9B-Q4_K_M.gguf" `
  -ngl 99 `
  -p 512 `
  -n 128

llama.cpp reported:

backend: CUDA | ngl: 99
pp512:   1828.82 +/- 68.91 t/s
tg128:     37.73 +/- 1.65 t/s

pp512 is prompt processing for a 512-token input. tg128 is generation for 128 output tokens. The output result was 37.73 tokens per second.

Compare the observed output rates

Engine	Reported output rate	Difference from Ollama
Ollama	23.77 tokens/s	Baseline
llama.cpp CUDA benchmark	37.73 tokens/s	+59%

The model, quantization, and GPU were the same. The engines and benchmark harnesses were different. This result shows that runtime choice changed the measured throughput on my laptop. It does not prove that llama.cpp is 59% faster for every prompt, configuration, or device.

For a stricter comparison, I would run the same prompts, sampling settings, context length, output length, and number of repetitions through both servers. I would then report median end-to-end latency as well as engine throughput.

GPU behavior under load

During llama.cpp inference, nvidia-smi reported:

GPU-Util:  95%
Pwr:       79W / 80W
Temp:      70C
VRAM:      5913MiB used
Process:   llama-server.exe (CUDA compute)

The GPU ran near its 80-watt power limit. It used 95% of its compute capacity at the time of the sample. The model remained in video memory.

One sample does not describe sustained thermal behavior. A longer test should record clock speed, temperature, power, and throughput over time.

Test with a coding agent

I connected the llama.cpp server to OpenCode through its OpenAI-compatible API.

The model handled file reads, basic retrieval, direct tool calls, and short code edits. The 37-token-per-second output rate felt comfortable for these tasks.

The limits appeared on complex, multi-step work. The model reasoned less reliably than frontier hosted models, and large-context tasks competed with the 8 GB memory limit.

This leaves a useful middle ground. A local model can handle routine work when the task is narrow and the result is easy to verify. Hard reasoning still benefits from a stronger hosted model.

Practical routing policy

I would route tasks by difficulty and data sensitivity:

Use a local model for routine, repetitive, or privacy-sensitive work.
Use a frontier model for difficult reasoning and open-ended synthesis.
Validate tool arguments and code changes from either model.

Local inference has no per-token API charge, works without a network connection, and keeps model input on the machine by default. Tools, telemetry, and integrations can still send data elsewhere, so the complete system determines privacy.

Reproduce the test

The required tools and model are available from these sources:

Qwen3.5-9B GGUF model. Use the Q4_K_M file for this configuration.
Ollama
llama.cpp releases
OpenCode. Install it with npm i -g opencode-ai, then configure its OpenAI-compatible endpoint as localhost:8081/v1.

I ran this test on Windows 11 in March 2026. The hardware was a Lenovo LOQ 15IRX9 with an RTX 4060 Laptop GPU.