A local model can fit in VRAM and still fail before loading when the runtime does not support its file architecture.
A model file can fit in VRAM and still fail before the GPU uses it. The runtime must first parse the file and implement its architecture. Memory placement comes later.
I hit this boundary while trying to start a Qwen3.5 GGUF in FreeToken on an 8 GB RTX 4060. The app showed a RAM estimate, so I checked the engine settings before changing the model. The VRAM budget was set to 95%, or about 7.6 GiB, and the MoE mode was offload.
The model entered the loading state. It then stopped with:
ValueError: GGUF architecture 'qwen35' is not supported (known: ['gemma4'])
That message changed the diagnosis. The model had not run out of memory. The loader did not support the architecture recorded in the GGUF file.
memory placement and architecture are different problems
MoE offload solves a memory placement problem. FreeToken documents this mode as keeping experts in host RAM while caching expert slots on the GPU. When an expert is not cached, the runtime streams it over PCIe.
That path can reduce the VRAM needed for an MoE model. It cannot add support for a new GGUF architecture. A higher VRAM budget cannot change the list of architectures that the loader knows how to parse.
FreeToken's model documentation makes the boundary explicit. The runtime loads HF safetensors checkpoints directly and supports native GGUF for Gemma-4. Its supported Qwen3.5 MoE entries are HF checkpoints, not generic Qwen3.5 GGUF files. The open architecture issue reports the same qwen35 error, and the roadmap lists GGUF support across model architectures as future work.
The remedy is therefore different from an out-of-memory fix. Use a checkpoint format that the runtime supports, or use a runtime that implements the architecture in the GGUF file. Changing the VRAM percentage is not enough.
use a compatibility-first preflight
When a local model does not start, check the layers in this order:
- Identify the file format, such as GGUF or HF safetensors.
- Check the architecture recorded in the file against the runtime's supported model list.
- Estimate model weights, context memory, and working memory against available VRAM.
- Select full GPU placement, offload, or CPU execution based on the memory result.
This order prevents a loader error from looking like a hardware error. It also makes logs easier to read. qwen35 is not supported points to the format or backend. An allocation failure points to memory capacity or fragmentation. Those failures need different fixes.
a working benchmark does not prove every runtime works
The same model family can run in one engine and fail in another. My Qwen3.5 benchmark measured full GPU inference through Ollama and llama.cpp on the same class of 8 GB RTX 4060 hardware. That result proves those runtimes handled that model file. It does not prove that another loader supports the same architecture.
Local inference has two separate questions:
- Can this runtime read and execute this model format and architecture?
- After it can, does the selected placement fit the device?
Answer the first question before tuning the second. It is a small change to the troubleshooting order, but it avoids spending time on settings that cannot affect the failure.