Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
So - I setup that model according to the docs and gave it this prompt:
Write me a highly optimized n-queens solver in go. It should take advantage of parallelism (what little there is) and output only the solution and how long it took.
After 10 minutes it gave me code that didn’t compile.
It took another 3 mins to fix the compile error and the output is not correct.
As I said - LLMs on 8Gig VRAM just aren’t worth it.
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
Since implementation of the --fit parameter and its relatives, and --fit on becoming the default, llama.cpp intelligently decides what to offload. For me, it made --n-cpu-moe obsolete.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.
You can control how much context should be fitted with --fit-ctx and how much space the algorithm should leave unallocated (even on a per-GPU basis) with --fit-target.
I’ve tried a few times but with only 8gig of vram it’s simply not worth it.
Have you tried qwen3.5-9b? It’s pretty solid for its size.
Yeah, it’s “good for its size” but it’s just too flaky for me to use for any significant coding.
Yeah, I wouldn’t use it for coding. It’s a bit dumb unfortunately.
How much CPU RAM do you have?
64G. But CPU inference is painfully slow.
Not anymore. Not with hybrid offloading, where the GPU handles dense tensors and the CPU only runs the sparse MoEs. I’m running a 300B model on a single 3090, and its faster than I can read.
You just need to use the right framework, and the right model.
I’d suggest trying ik_llama.cpp and a MoE like one of these: https://huggingface.co/models?other=ik_llama.cpp&sort=modified&search=35B
And speculative decoding like DFlash or MTP (which you can also get specific models for).
EDIT: Wrong link.
I’ll check that out - speed isn’t my biggest issue so much as coding performance… The qwen 3.5 model I was using can write code, but it’s… Meh? Like sometimes it doesn’t even compile.
I did try tweaking llama.cpp to do some cpu offloading and it does seem to allow for much larger contexts at a modest performance loss. I’ll check out larger models.
Try qwen3.6-35b-a3b with a lightweight harness like pi.dev
Having it be able to run commands and try to compile or run the code and see the output helps especially on the “doesn’t compile” part of things
Yeah - I’ve been playing around more with the Qwen3-Coder-30B-A3B-Instruct MoE model and it’s still quite… Meh. I’ve been using llama.cpp and I’ve tried a bunch of tuning. It works and performs well enough (15t/s) but the output is just garbage. I can do some simple coding but I’m finding I’m fighting with it more than if I just wrote the code myself. Maybe I just have standards that are too high. Claude Opus 3.7 is just in an entirely different league…
When you run it, do you use unsloth’s recommended settings for coding?
https://unsloth.ai/docs/models/qwen3.6
Also have preserve thinking on, it helps it stay consistent in multi turn work.
Which model version you’re using can also affect results, usually unsloth’s ones are good.
With all that said, it’s of course a small model so it’s not a super coder. The 27b is better (I’d guess 25-35% better), but of course still a small model so…
So it’ll maybe not be good enough still, but should give it the chance to let it do the best it can :)
So - I setup that model according to the docs and gave it this prompt:
Write me a highly optimized n-queens solver in go. It should take advantage of parallelism (what little there is) and output only the solution and how long it took.After 10 minutes it gave me code that didn’t compile.
It took another 3 mins to fix the compile error and the output is not correct.
As I said - LLMs on 8Gig VRAM just aren’t worth it.
CPU offloading is too slow unless you use a hybrid MoE model, with the --n-cpu-moe parameter, specifically.
This only offloads “sparse” parts of the model to the CPU, which take up a lot of RAM but are very compute-lite to run. In practice, thats most of the size of modern MoE LLMs.
Since implementation of the
--fitparameter and its relatives, and--fit onbecoming the default, llama.cpp intelligently decides what to offload. For me, it made--n-cpu-moeobsolete.Mostly, yeah.
Sometimes it’s better to “cut it close,” with (for instance) a 27B model that’s nearly OOMing your VRAM fully offloaded, but you know will be fine in regular use without too many programs open.
In my case, with MiMo 2.5, it fills both my CPU and GPU RAM rather completely, so it’s best to set a static value so I don’t swap CPU RAM, and don’t OOM on the GPU either.
You can control how much context should be fitted with
--fit-ctxand how much space the algorithm should leave unallocated (even on a per-GPU basis) with--fit-target.