What models are currently good for running coding tasks? I just ran Qwen3-14B-Q6_K.gguf with llama.cpp on my card with 16GB of vram (+32GB ddr4), but I get really close to filling the entire vram on a single short conversation, so I am looking for some (smaller) alternatives to test.

I might throw OpenCode container in the mix next, if that is relevant information.

spoiler 
podman run --rm --replace --pull=newer \
  --name llama \
  -p 8080:8080 \
  -v ./llama_models:/models:Z \
  --device /dev/dri/card1:/dev/dri/card1 \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  ghcr.io/ggml-org/llama.cpp:full-vulkan \
  --server \
  -m /models/Qwen3-14B-Q6_K.gguf \
  -ngl 99 \
  -fa on \
  -c 16384 \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95 \
  --jinja \
  --host 0.0.0.0 --port 8080
  • Svinhufvud@sopuli.xyzOPEnglish
    3·
    17 days ago

    Thanks! Why the --cpu-moe flag though? And can you share your parameter section (the after --server part), if you’re running llama?

    • Avid Amoeba@lemmy.caEnglish
      2·
      17 days ago

      I use model preset file but that just contains the equivalent of command line options. Here’s what I have for Qwen:

      [Qwen3.6-35B-A3B-MTP-Q4_K_XL]                                                                                                                
m = /models/Qwen3.6-35B-A3B-MTP/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf                                                                              
mmproj = /models/Qwen3.6-35B-A3B-MTP/mmproj-BF16.gguf                                                                                        
spec-type = draft-mtp                                                                                                                        
spec-draft-n-max = 2                                                                                                                         
chat-template-kwargs = {"preserve_thinking": true}                                                                                           
temp = 1.0                                                                                                                                   
top-p = 0.95                                                                                                                                 
top-k = 20                                                                                                                                   
min-p = 0.0                                                                                                                                  
presence-penalty = 1.5                                                                                                                       
repeat-penalty = 1.0

      I don’t use cpu-moe because I have enough VRAM for the whole model. If you have 16GB VRAM, you add cpu-moe which makes llama.cpp put only the active layers on the GPU. Keeps the rest of them in system RAM. Then it swaps them around as needed. The result is lower but still decent speed. On my hw, this model does 90-100tps when fully in VRAM. When using cpu-moe, I think it falls to 40-50tps, if I remember correctly.