Hey guys,
What’s currently the best LLM for low-VRAM machines with only 6 GB VRAM? I’ve got 32GB RAM as well.
I’m experimenting a little with SillyTavern and I’m curious which model gets the most out of my setup. Should be multilingual and suitable for “casual chatting”.
I know I will probably not get very far with this, but I’m still interested in how far we’ve already come.
(Using KoboldCPP if that matters).
~sp3ctre
You must log in or # to comment.
I’m running gemma-4-e4b on my 8GB machine. I’ll drop down to e2b on CPU. It’s probably the best you’ll get. 140 languages, vision, decent at agentic work. Not great at code.
My setup is a laptop with 8 GB vram and 16 gb ram.
I have been using ministral 3b (fast) and 14b (slower but somewhat smarter/capable) via ollama. They work remarkably well considering how small they are.
I have been using it as a text translator, summarizer and assistant for discussing more basic things, including integrating it in pycharm using the ollama assist plugin as a coding assistant.
For autocomplete in pycharm I have to use llama 3.1 8b, since ministral cannot do autocomplete (?).
I can recommend ministral, Mistral are really great at creating small distilled models that have a lot of bang for the parameters they have.
On my MacBook Air m2, I’m currently using Qwen 3.5 4b with 8 bit quantisation, and even at its maximum context length, multiple web search RAGs, and the model being built for vision and reasoning, it only ever hits 4.3gb of memory tops.
I run it though LM Studio, so paired with the fact it’s a Mac, your mileage may vary in terms of how much memory it uses, but it does have [from my experience] an output quality a bit over ChatGPT 4o, and is actually really solid for research purposes if that’s what you’re looking for.
I’m running Qwen 3.5 4B Q4 on 16GB RAM. Yup, no VRAM haha. 5-6tk/s. Llama.cpp
I mean, do you need it to be fast? You could probably run a pretty decent 20b model if you are okay with the speed of offloading.
Doesn’t necessarily need to be very fast, but I don’t plan to wait a minute for one simple sentence as well :)
Is that possible without tinkering too much?
There’s been a few videos on Youtube lately discussing using a particular Qwen model that lets you load only particular expert sections at a time onto the GPU and the rest in RAM. This one was the first I watched (https://www.youtube.com/watch?v=8F_5pdcD3HY), I haven’t tried it, but it makes sense on why it would work.
I have a Qwen3.6-35b-a3b model running on a dated desktop machine with 4GB VRAM.
I use 8-bit-quant, but also have 48GB normal RAM.
Delivers ~7tk/s, which is already totally usable for most things.
Tried it on my recent Core-i7 company laptop with 8GB VRAM and got 20tk/s.
Oh, and I am also using KoboldCPP (on a Linux foundation).I’ll try my luck and download Qwen3.6-35B-A3B-GGUF. Thanks!
I have been using qwen-3.5-9b as a general purpose LLM that I can still load up while gaming (on a 16GB card). I never have issues if I’m under 10ish gigs of VRAM usage for the game, so I imagine it should work for your use case.
I’ve been generally happy with the results on everyday reasoning tasks and programming questions.
