Separating GGUF Weights, Tokenizer, and Config in vLLM

When vLLM loads GGUF, treating quantized weights, tokenizer, and model configuration as one input makes startup errors harder to trace.

They may live in the same Hugging Face repository, but they provide different inputs to vLLM.

When a GGUF repository lacks config.json, specifying only --tokenizer does not resolve the configuration error.

Check GGUF support first

GGUF is a file format for storing quantized model weights.

Quantization can reduce weight size, so inference may require less memory than loading the original weights unchanged.

Using GGUF does not mean that every vLLM feature is available.

The official vLLM GGUF documentation describes support as experimental and warns about compatibility with other features.

GGUF support also involves vllm-gguf-plugin, so match the installation procedure to the vLLM version in use.

Put the three inputs in one table

Input	Main role	vLLM example
GGUF weights	Quantized model parameters	owner/model-GGUF:Q4_K_M
Tokenizer	Tokenization, vocabulary, and chat template	--tokenizer owner/base-model
Hugging Face config	Architecture, layer count, and hidden size	--hf-config-path owner/base-model

GGUF weights provide the parameters used for inference.

The tokenizer converts input text into tokens and builds the chat format when required.

The Hugging Face configuration determines how the weights are interpreted as a model architecture.

The three inputs are related, but do not need to point to the same repository or file.

Basic launch example

When the GGUF repository and original-model repository are separate, specify them like this:

uv pip install vllm-gguf-plugin

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \
  --tokenizer Qwen/Qwen3-0.6B \
  --hf-config-path Qwen/Qwen3-0.6B

unsloth/Qwen3-0.6B-GGUF:Q4_K_M selects the quantized weights.

--tokenizer Qwen/Qwen3-0.6B selects the tokenizer and chat settings from the original model.

--hf-config-path Qwen/Qwen3-0.6B directs model-configuration lookup to the original model.

Replace repository names, quantization method, and plugin requirements according to the distributor and vLLM version.

Isolate a config.json error

An error such as the following means that the remote GGUF repository did not provide a Hugging Face-format configuration:

Could not find config.json for remote GGUF model repo.

Checking only whether --tokenizer is present is insufficient.

--tokenizer specifies the tokenizer source.

It does not specify the source for config.json.

Specify the configuration source with --hf-config-path.

vllm serve owner/model-GGUF:Q4_K_M \
  --tokenizer owner/base-model \
  --hf-config-path owner/base-model

When the target model cannot be converted automatically, also check the documented option of preparing a Hugging Face-compatible configuration file.

HF_HOME is a cache location, not a source

HF_HOME changes where Hugging Face-related caches are stored.

export HF_HOME="$HOME/src/llm-server/models/huggingface"

This changes where downloaded files are saved.

It does not choose the repository for the model, tokenizer, or configuration.

Those sources are selected by the model argument, --tokenizer, --hf-config-path, and other options.

When config.json is missing, inspect the GGUF and original-model arguments before inspecting the cache location.

Obtain the tokenizer from the original model

The vLLM documentation guides users toward the original model’s tokenizer rather than converting one from GGUF.

GGUF tokenizer conversion can take time and may be unstable for some models.

After confirming that the original and quantized models are compatible, use:

vllm serve ./Qwen3-0.6B-Q4_K_M.gguf \
  --tokenizer Qwen/Qwen3-0.6B

Even for a local GGUF file, the tokenizer can come from a separate Hugging Face repository.

Checks before starting

1. Confirm that GGUF contains quantized weights for the target model
2. Confirm compatible vLLM and vllm-gguf-plugin versions
3. Confirm the original tokenizer and chat template
4. Confirm that the original model has the required config.json
5. Add --hf-config-path when necessary
6. Start with a short input and confirm the model name and tokenizer
7. Confirm GPU support for the quantization method

Reducing memory with GGUF does not automatically resolve tokenizer, model-configuration, or GPU-kernel compatibility.

Separating weight selection from correct model interpretation makes startup failures easier to investigate.

References

• GGUF | vLLM
• Quantization | vLLM