[Feature]: GPU tier disabled on 32 GB RAM + 4 GB VRAM; trunk placement could still help

Problem

So i have a computer with 32GB RAM and an Nvidia 3050ti (4GB VRAM)

i tried the simpliest way:

./c/coli chat --model colibri-models/qwen36-35b/ --gpu auto --auto-tier
# ...
[qtier] cap=182 != n_experts=256 -> tier disabled (needs full RAM residency)

the model works but in RAM only.
I wont pretend i understand everything, but here the caps not at 256 block the usage of my GPU

Here the test forcing my GPU

systemd-run --user --scope -p MemoryMax=26G -p MemorySwapMax=0   env COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=1.5 COLI_TIMERS=1   ./c/coli chat --model colibri-models/qwen36-35b/ --gpu 0 --cap 256 --vram 3
...
[qtier] dev 0: 3.6 GB free, allowance 3.0 GB
[place] auto: dev 0 holds 1708.0 MB of trunk (lmhead yes, 30 dnproj layers), 1.33 GB left for experts
[qtier] dev 0: budget 1.33 GB for experts (~755 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)
[gpu] MoE experts -> CUDA VRAM tier
  ⠧ loading qwen3.6-colibri… 9s[place] probe: one [8192 x 2048] int8 GEMV takes 0.381 ms on CUDA dev 0, 1.438 ms on the CPU -> trunk to VRAM
[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)
  ⠇ loading qwen3.6-colibri… 9s[dnp] 30 DeltaNet-Projektionen auf GPU (0.70 GB VRAM)
  ⠏ loading qwen3.6-colibri… 9s[dense] 70 trunk matrices on GPU (dnout/attnproj/shexp, 0.49 GB VRAM)
[qwen36] int4 packed weights detected — unpacking to int8 in slot
  ⠧ loading qwen3.6-colibri… 17sComplété                 systemd-run --user --scope -p MemoryMax=26G -p MemorySwapMax=0 env COLI_CUDA=1 COLI_GPUS=0 CUDA_EXPERT_GB=1.5 COLI_TIMERS=1 ./c/coli chat --model colibri-models/qwen36-35b/ --gpu 0 --cap 256 --vram 3

As you see, and from what i saw when monitoring my gpu, it is used the right way, and there is enough space to add some more.
The issue here is that when i force cap to 256 (required for GPU usage), everything is loaded in my ram at the start and [[lmh] lm_head [248320 x 2048] int8 resident on CUDA dev 0 (0.47 GB)] with all the space, it just outpassed the limit i set and stoped.

Proposed solution

My request is the have the possibility to use the VRAM of my GPU even when cap isn't at 256 (everything loaded in the cpu if i understand).
For a little computer as the one i have, and as may others, this could be a really good add to this project.

Alternatives considered

No response

Scope and compatibility

No response

qwen36: the CUDA tier at any RAM size, and the trunk whatever the exp…

Merge pull request #2098 from JustVugg/fix/qwen36-tier-any-cap

Tested #2098 on the hardware from this issue (i5-10500H, 32 GB, GTX 1650 Max-Q 4 GB) at fa40b23e on dev: the tier now starts below full residency (RAM holds 225 of 256 experts a layer, 1076 experts in VRAM), with no OOM, and the probe correctly keeps the trunk on the CPU. So the fix works here, and this can probably be closed.

One catch: on this card the tier is still slower than --gpu none (decode 9.2 vs 10.9 tok/s, prefill 14 vs 46 tok/s), and its output is not reproducible at temperature 0. Full numbers and commands are in #2211.

Hi,
I also tried the fix, for me it works around 15% better with the gpu.

Here i made some data if you're interested (same prompt each time, for results around the same thing)

COLI_CUDA=1 COLI_TIMERS=1   ./c/coli web --model colibri-models/qwen36-35b/ --gpu 0 --vram 3 --no-browser
COLI_CUDA=1 COLI_TIMERS=1   ./c/coli web --model colibri-models/qwen36-35b/ --gpu none --no-browser

No GPU, test 1, (no correct timer here)
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[timers] window: moe 342 ms/token (last 32)
[timers] window: moe 244 ms/token (last 32)
[timers] window: moe 243 ms/token (last 32)
[timers] window: moe 197 ms/token (last 32)
[timers] window: moe 161 ms/token (last 32)
[timers] window: moe 132 ms/token (last 32)
[timers] window: moe 137 ms/token (last 32)
[timers] window: moe 177 ms/token (last 32)
[timers] window: moe 173 ms/token (last 32)
[timers] window: moe 178 ms/token (last 32)
[timers] window: moe 148 ms/token (last 32)
[timers] window: moe 164 ms/token (last 32)
[timers] window: moe 169 ms/token (last 32)
[timers] window: moe 163 ms/token (last 32)
[timers] window: moe 159 ms/token (last 32)
[timers] window: moe 177 ms/token (last 32)
[timers] window: moe 164 ms/token (last 32)
[timers] window: moe 153 ms/token (last 32)
[timers] window: moe 145 ms/token (last 32)
[timers] window: moe 176 ms/token (last 32)
[qwen36 spec gate] turn 1: T(1) 278.8 ms, rows 2:1.49, rho 0.75, acceptance by position 0.18 0.50 0.50 0.50 0.50
[qwen36 lookup] turn 1: 1.00 tokens/forward (678 forwards per 679 tokens) | acceptance 11.1% (1/9 drafts in 9 verifies) | gate on, 97 declined, 9 probes

With GPU, test 1, +3min10
[timers] window: moe 309 ms/token (last 32)
[timers] window: moe 218 ms/token (last 32)
[timers] window: moe 140 ms/token (last 32)
[timers] window: moe 148 ms/token (last 32)
[timers] window: moe 117 ms/token (last 32)
[timers] window: moe 104 ms/token (last 32)
[timers] window: moe 107 ms/token (last 32)
[timers] window: moe 96 ms/token (last 32)
[timers] window: moe 144 ms/token (last 32)
[timers] window: moe 146 ms/token (last 32)
[timers] window: moe 122 ms/token (last 32)
[timers] window: moe 116 ms/token (last 32)
[timers] window: moe 134 ms/token (last 32)
[timers] window: moe 124 ms/token (last 32)
[timers] window: moe 126 ms/token (last 32)
[timers] window: moe 124 ms/token (last 32)
[timers] window: moe 109 ms/token (last 32)
[timers] window: moe 109 ms/token (last 32)
[timers] window: moe 118 ms/token (last 32)
[timers] window: moe 110 ms/token (last 32)
[timers] window: moe 105 ms/token (last 32)

No GPU, test 2, +3min40
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[timers] window: moe 310 ms/token (last 32)
[timers] window: moe 228 ms/token (last 32)
[timers] window: moe 225 ms/token (last 32)
[timers] window: moe 185 ms/token (last 32)
[timers] window: moe 176 ms/token (last 32)
[timers] window: moe 148 ms/token (last 32)
[timers] window: moe 152 ms/token (last 32)
[timers] window: moe 207 ms/token (last 32)
[timers] window: moe 196 ms/token (last 32)
[timers] window: moe 172 ms/token (last 32)
[timers] window: moe 187 ms/token (last 32)
[timers] window: moe 173 ms/token (last 32)
[timers] window: moe 173 ms/token (last 32)
[timers] window: moe 160 ms/token (last 32)
[timers] window: moe 165 ms/token (last 32)
[timers] window: moe 186 ms/token (last 32)
[timers] window: moe 183 ms/token (last 32)
[timers] window: moe 171 ms/token (last 32)
[timers] window: moe 166 ms/token (last 32)
[timers] window: moe 178 ms/token (last 32)
[qwen36 spec gate] turn 1: T(1) 266.7 ms, rows 2:1.66, rho 0.79, acceptance by position 0.18 0.50 0.50 0.50 0.50
[qwen36 lookup] turn 1: 1.00 tokens/forward (678 forwards per 679 tokens) | acceptance 11.1% (1/9 drafts in 9 verifies) | gate on, 97 declined, 9 probes

With GPU, test 2, +3min
[timers] window: moe 322 ms/token (last 32)
[timers] window: moe 224 ms/token (last 32)
[timers] window: moe 142 ms/token (last 32)
[timers] window: moe 138 ms/token (last 32)
[timers] window: moe 131 ms/token (last 32)
[timers] window: moe 123 ms/token (last 32)
[timers] window: moe 103 ms/token (last 32)
[timers] window: moe 113 ms/token (last 32)
[timers] window: moe 127 ms/token (last 32)
[timers] window: moe 148 ms/token (last 32)
[timers] window: moe 133 ms/token (last 32)
[timers] window: moe 122 ms/token (last 32)
[timers] window: moe 130 ms/token (last 32)
[timers] window: moe 124 ms/token (last 32)
[timers] window: moe 123 ms/token (last 32)
[timers] window: moe 117 ms/token (last 32)
[timers] window: moe 103 ms/token (last 32)
[timers] window: moe 123 ms/token (last 32)
[timers] window: moe 111 ms/token (last 32)
[timers] window: moe 121 ms/token (last 32)
[timers] window: moe 101 ms/token (last 32)

No GPU, test 3, +3min40
[qwen36] int4 packed weights detected — kept int4, repacked planar for expert_ffn.h
[timers] window: moe 317 ms/token (last 32)
[timers] window: moe 236 ms/token (last 32)
[timers] window: moe 234 ms/token (last 32)
[timers] window: moe 183 ms/token (last 32)
[timers] window: moe 168 ms/token (last 32)
[timers] window: moe 147 ms/token (last 32)
[timers] window: moe 143 ms/token (last 32)
[timers] window: moe 195 ms/token (last 32)
[timers] window: moe 203 ms/token (last 32)
[timers] window: moe 184 ms/token (last 32)
[timers] window: moe 184 ms/token (last 32)
[timers] window: moe 183 ms/token (last 32)
[timers] window: moe 189 ms/token (last 32)
[timers] window: moe 156 ms/token (last 32)
[timers] window: moe 166 ms/token (last 32)
[timers] window: moe 190 ms/token (last 32)
[timers] window: moe 191 ms/token (last 32)
[timers] window: moe 176 ms/token (last 32)
[timers] window: moe 168 ms/token (last 32)
[timers] window: moe 189 ms/token (last 32)
[qwen36 spec gate] turn 1: T(1) 304.6 ms, rows 2:1.45, rho 0.74, acceptance by position 0.18 0.50 0.50 0.50 0.50
[qwen36 lookup] turn 1: 1.00 tokens/forward (678 forwards per 679 tokens) | acceptance 11.1% (1/9 drafts in 9 verifies) | gate on, 97 declined, 9 probes

With GPU, test 3, +3min10
[timers] window: moe 328 ms/token (last 32)
[timers] window: moe 189 ms/token (last 32)
[timers] window: moe 153 ms/token (last 32)
[timers] window: moe 151 ms/token (last 32)
[timers] window: moe 125 ms/token (last 32)
[timers] window: moe 113 ms/token (last 32)
[timers] window: moe 124 ms/token (last 32)
[timers] window: moe 149 ms/token (last 32)
[timers] window: moe 133 ms/token (last 32)
[timers] window: moe 128 ms/token (last 32)
[timers] window: moe 137 ms/token (last 32)
[timers] window: moe 134 ms/token (last 32)
[timers] window: moe 112 ms/token (last 32)
[timers] window: moe 126 ms/token (last 32)
[timers] window: moe 118 ms/token (last 32)
[timers] window: moe 103 ms/token (last 32)
[timers] window: moe 123 ms/token (last 32)
[timers] window: moe 134 ms/token (last 32)
[timers] window: moe 127 ms/token (last 32)
[timers] window: moe 128 ms/token (last 32)

Thanks for testing! I reran with your settings on my machine (i5-10500H, 32 GB, GTX 1650 Max-Q), same session, median of 3 warm runs:

	decode	prefill (1281 tok)	experts in VRAM
`--gpu none`	9.6 tok/s	35 tok/s	–
`COLI_CUDA=1 --gpu 0 --vram 3` (yours)	8.0 tok/s	14 tok/s	1702
`--gpu auto --auto-tier --vram 3`	8.1 tok/s	14 tok/s	1702

So more VRAM (3 GB vs the 1.9 GB auto allowance) doesn't flip it here: the tier is still ~15 % slower on decode, and ~2.5× slower on prefill. My guess is the CPU makes the difference: your runs look like ~3.5 tok/s end to end, while this CPU does ~10 tok/s on its own, so the copies to the GPU cost more than they save. Which CPU are you on? Could you also post end-to-end tok/s, not only the moe timer, and a long-prompt prefill? That would help tell where the tier pays off.

Here the stats
(and a question, how do you get these detailed stats?)

--gpu auto --vram 3
--gpu none
CPU: i7-12700H

With GPU, test 1 (did not start on real warm)
666 tok · 184s · 3.62 tok/s
(little + in answer)

No GPU, test 1
679 tok · 183s · 3.71 tok/s

With GPU, test 2
649 tok · 142s · 4.56 tok/s
(little +)

No GPU, test 2
679 tok · 193s · 3.52 tok/s

With GPU, test 3
694 tok · 155s · 4.49 tok/s
(little +)

No GPU, test 3
679 tok · 189s · 3.58 tok/s

Sorry for the late reply. The numbers in my tables come from a small client against the OpenAI-compatible API (coli serve), not from engine logs: it streams the response, takes prefill = prompt tokens / time to first token, and decode = generated tokens / time after the first token, at temperature: 0 with thinking off, after a warm-up request. Script below; run it against coli serve --model ... --port 8001, one server per setting.

One thing in your numbers: 3.6 tok/s CPU-only on an i7-12700H looks low, since this i5-10500H does ~11 tok/s decode on the same model. If the 3.6 is total tokens / total time it includes the prefill and any thinking tokens; how much RAM does that machine have, and is thinking on?

bench.py

"""Repeat the #2211 client: warm-up, 2 rounds of 3 decode prompts, one long prefill prompt."""
import json, statistics, sys, time, urllib.request

URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8001/v1/chat/completions"

def req(prompt, max_tokens):
    body = json.dumps({"model": "qwen3.6-colibri", "messages": [{"role": "user", "content": prompt}],
                       "max_tokens": max_tokens, "temperature": 0, "stream": True,
                       "stream_options": {"include_usage": True},
                       "chat_template_kwargs": {"enable_thinking": False},
                       "enable_thinking": False}).encode()
    r = urllib.request.Request(URL, body, {"Content-Type": "application/json"})
    t0 = time.time(); first = None; n = 0; usage = None; text = []
    with urllib.request.urlopen(r, timeout=3600) as resp:
        for line in resp:
            line = line.decode().strip()
            if not line.startswith("data:") or line == "data: [DONE]":
                continue
            d = json.loads(line[5:])
            if d.get("usage"):
                usage = d["usage"]
            for c in d.get("choices", []):
                piece = (c.get("delta") or {}).get("content")
                if piece:
                    if first is None:
                        first = time.time()
                    n += 1; text.append(piece)
    t1 = time.time()
    ntok = usage["completion_tokens"] if usage else n
    return {"ttft": first - t0, "decode": (ntok - 1) / (t1 - first) if ntok > 1 else 0,
            "prompt_tokens": usage["prompt_tokens"] if usage else None, "text": "".join(text)}

PROMPTS = ["Escribe un párrafo sobre qué es una GPU y para qué sirve.",
           "Escribe en Python la criba de Eratóstenes con comentarios.",
           "Dame 5 consejos para dar mantenimiento preventivo a una laptop."]

PARA = ("El mantenimiento preventivo de equipos de cómputo en una empresa de servicios técnicos "
        "incluye la limpieza interna de ventiladores y disipadores, el reemplazo de pasta térmica, "
        "la revisión del estado de los discos, la actualización de controladores y del sistema "
        "operativo, la verificación de respaldos y la inspección de cables, fuentes y baterías. ")
LONG = (PARA * 18) + "\n\nResume el texto anterior en una sola frase."

req("Hola", 30)  # warm-up
rounds = []
for rnd in range(2):
    rounds.append([req(p, 150)["decode"] for p in PROMPTS])
pre = [req(f"Documento {i + 1} de 3.\n\n" + LONG, 128) for i in range(3)]  # distinct start: no prefix-cache hit
r2 = rounds[1]
print(json.dumps({"decode_round2": [round(x, 2) for x in r2], "decode_median": round(statistics.median(r2), 2),
                  "prefill_prompt_tokens": pre[0]["prompt_tokens"],
                  "prefill_tok_s": [round(x["prompt_tokens"] / x["ttft"], 1) for x in pre],
                  "prefill_decode_tok_s": [round(x["decode"], 2) for x in pre]}))

Navigation Menu

Repository navigation

[Feature]: GPU tier disabled on 32 GB RAM + 4 GB VRAM; trunk placement could still help #2090

Problem

Proposed solution

Alternatives considered

Scope and compatibility

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[Feature]: GPU tier disabled on 32 GB RAM + 4 GB VRAM; trunk placement could still help #2090

Description

Problem

Proposed solution

Alternatives considered

Scope and compatibility

Activity

javieratcservicios commented on Oct 18, 2026

Yva1n commented on Oct 18, 2026

javieratcservicios commented on Oct 18, 2026

Yva1n commented on Oct 20, 2026

javieratcservicios commented on Oct 21, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions