[LLM] models update
This commit is contained in:
Coja
@@ -60,9 +60,11 @@ starves the desktop and freezes the PC (reboot). Hard-won rules:
|
||||
| `Qwen3.5-9B-UD-Q6_K_XL` | 32.1 | 32k | vision, MTP, reasoning | small Qwen, quick tasks |
|
||||
| `Qwen3-Coder-30B-Instruct-UD-Q3_K_XL` | **~30** clean / 27.0 evening @moe12 | 32k | — | **main agent coder** — won the 2026-08-06 quant race; moe12 verified evening 08-07 + clean 08-08 |
|
||||
| `GLM-4.7-Flash-UD-Q4_K_XL` | **21.5** @moe22 | 24k | reasoning | quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08 |
|
||||
| `GLM-4.7-Flash-UD-Q3_K_XL` | **26.0** @moe12 (2.0 daytime baseline; ~28 clean) | 24k | reasoning | **fast coder** — re-added 09-21 because Q4's 21.5 dragged in agent loops; 3-bit, evening-safe at moe12 (13.4 used / 3.8 free @2.0); Q4 stays for hard reviews |
|
||||
| `Qwen3-Coder-Next-UD-IQ3_XXS` | 16.0 | **128k** | 80B-A3B | long-session coder (128k ctx) |
|
||||
| `Qwen3.8-27B-UD-IQ3_XXS` | 14.5 (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~30). 2.5 GB free @1.6 baseline @24k (✓ 09-10); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
|
||||
| `Qwen3-Embedding-0.6B` | 33 (CPU) | 8k | CPU-only, `/v1/embeddings` | RAG/search embedder (not a chat model) |
|
||||
| `Ornith-1.5-35B-A3B-IQ3_XXS` | **32.1** @moe16 | 24k | MTP, vision, thinking | **agentic coder** (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0 |
|
||||
|
||||
Expected run-to-run spread: MTP models swing ±15% with draft **acceptance rate** (content-
|
||||
dependent); CPU-heavy presets (Coder-Next, embedder) dip under daytime CPU contention.
|
||||
@@ -72,6 +74,11 @@ Treat clean night runs as the reference; don't retune on daytime deltas.
|
||||
|
||||
- **Agent coding loops** (edit/test cycles): `Qwen3-Coder-30B` — best speed/quality balance at 32k.
|
||||
- **Hard code, reviews, tricky bugs**: `GLM-4.7-Flash` — strongest 30B-class coder, slightly slower.
|
||||
- **GLM at agent-loop speed**: `GLM-4.7-Flash-UD-Q3_K_XL` — same model, 3-bit, ~28 t/s; pick it
|
||||
when the Q4's 21 t/s drags, keep Q4 for the review that has to be right.
|
||||
- **Agentic coding with reasoning** (⚗ new, judge over the next weeks): `Ornith-1.5-35B-A3B` —
|
||||
Qwen3.5-MoE derivative, 32 t/s *with* MTP and a thinking preamble (~4 s TTFT), vision. If its
|
||||
launch numbers hold in practice it displaces GLM/Coder-30B for agent loops.
|
||||
- **Marathon sessions / huge conversation history**: `Qwen3-Coder-Next` — 128k ctx at only
|
||||
6 GB VRAM (hybrid attention). Caveat: ~33 t/s prefill means it's for *growing* sessions
|
||||
(`cache-reuse` makes turns incremental), **not** for cold-dumping 100k tokens.
|
||||
@@ -131,6 +138,8 @@ the router, but chats/presets saved against the old id need re-picking.
|
||||
2026-08-16: `Qwen3.8-27B-UD-IQ3_XXS` (⚗ candidate) added to all six client files.
|
||||
2026-09-10: kept as ⚠ clean-desktop specialty after the gate bench finally ran on the v3 gguf
|
||||
(14.5 t/s MTP / 10.9 base, 2.8 GB free @1.5); server ctx restored to 24576 = client ctx, no client edits.
|
||||
2026-09-21: `GLM-4.7-Flash-UD-Q3_K_XL` (⚡ fast coder) added to all six client files next to the Q4.
|
||||
2026-09-21: `Ornith-1.5-35B-A3B-IQ3_XXS` (agentic coder, thinking + vision) added to all six client files after passing its gate (32.1 t/s @moe16).
|
||||
|
||||
## Tuning cheat-sheet
|
||||
|
||||
|
||||
@@ -455,14 +455,73 @@ top-p = 0.95
|
||||
top-k = 64
|
||||
sleep-idle-seconds = 30
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Ornith (Qwen3.5-MoE derivative)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[Ornith-1.5-35B-A3B-IQ3_XXS]
|
||||
# ✓ KEPT 2026-09-21 (gate passed: 32.1 t/s MTP, 13.8 used / 3.3 free @moe16) — Ornith-1.5-35B-A3B (ornith-ai / DeepReinforce, MIT,
|
||||
# released ~2026-08-24): Qwen3.5-MoE architecture (qwen35moe: 40 layers, hybrid GDN 3:1, 256
|
||||
# experts / 8 active + shared, ~3B active, vision, n_ctx_train 262144), continued-pretrained and
|
||||
# RL'd with a self-improvement loop. Card claims it beats Qwen3.6-35B-A3B on every coding/agentic
|
||||
# bench (Terminal-Bench 2.1 67.8 vs 52.5, SWE-bench Verified 79 vs 73.4). Same shape as the daily
|
||||
# driver → same recipe: MTP draft + experts on CPU. Role if it passes: AGENTIC CODER (the only
|
||||
# coder here with MTP — Coder-30B 30 t/s and GLM Q3 26 t/s have none) and/or daily driver.
|
||||
# MTP head VERIFIED in the gguf header 09-21 (nextn_predict_layers = 1, block_count 41 = 40+1) —
|
||||
# present in bartowski's plain quants and the official Q4_K_M alike; no "-MTP" repo needed.
|
||||
# Thinking is ON by default (<think> block; card sampling 0.6/0.95/20). Kept ON for the coder
|
||||
# role; an instruct alias = same file + `reasoning-budget = 0` exactly like the daily driver.
|
||||
# VRAM: file is +1.27 GB vs the daily driver's 14.07 → start at moe20 (≈ Qwen@moe16 footprint:
|
||||
# predicted ~13.9 used / ~3.3 free @2.0 baseline). Until downloaded, sweep with `-x` this id.
|
||||
# GATE: MTP decode ≥ ~30 t/s AND free ≥ 2.5 → keep; step moe 20→16 while free ≥ 2.5; then wire
|
||||
# the six client files + README row. Anything less → retire + delete (the Qwen stays).
|
||||
# ✓ LOADED + BENCHED 2026-09-21 17:31 @moe20 (2.0 baseline): 27.9 t/s / 52 pp / 12.4 used / 4.8
|
||||
# free → footprint only 10.4 GB (predicted 11.9). MTP draft active (spec column). Speed gate
|
||||
# not met yet but 2.3 GB of headroom is unused → step moe 20→16 (= daily driver; the Qwen went
|
||||
# 19.8→35.9 t/s over moe 24→16). Predicted @moe16: ~13.8 used / ~3.4 free @2.0.
|
||||
# ✓ GATE PASSED 17:34 @moe16 (2.0 baseline): 32.1 t/s / 74 pp / TTFT 3.9 s (thinking preamble) /
|
||||
# 13.8 used / 3.3 free — prediction exact. Both gate halves met → wired into the six client
|
||||
# files + README 09-21. KEEP moe16 (= daily-driver envelope; moe14 ≈ +1-2 t/s for the last
|
||||
# 0.8 GB of margin — not on an evening box). Draft acceptance not recorded yet (grep the log).
|
||||
# Role: agentic coder with reasoning — the only coder here with MTP (Coder-30B 30, GLM Q3 26).
|
||||
# Instruct alias (reasoning-budget = 0, same file) still optional — add if agent loops want
|
||||
# the ~4 s thinking TTFT gone.
|
||||
# ⬇ DOWNLOAD from bartowski/Ornith-1.5-35B-A3B-GGUF (imatrix; the official repo has only Q4_K_M+
|
||||
# ≥ 21.7 GB, unsloth has no 1.5 quants yet):
|
||||
# Ornith-1.5-35B-A3B-IQ3_XXS.gguf 15,340,447,392 B · sha256 8918ccb9ee29abe3875efec0c3e86f0e35ef0b8a03e3f0e1c422869859a518d0
|
||||
# mmproj-Ornith-1.5-35B-A3B-f16.gguf 899,283,296 B · sha256 815c9a671991aceeef46ca1aa103b64499db8cfadf21c83005a20ed839d53c7c
|
||||
model = /home/anon/software/models/Ornith-1.5-35B-A3B-IQ3_XXS.gguf
|
||||
mmproj = /home/anon/software/models/mmproj-Ornith-1.5-35B-A3B-f16.gguf
|
||||
spec-type = draft-mtp
|
||||
spec-draft-n-max = 2
|
||||
ctx-size = 24576 # = the daily driver; n_ctx_train 262144 — raise only after a bench
|
||||
n-gpu-layers = 99
|
||||
n-cpu-moe = 16 # ✓ KEEP (09-21: 32.1 t/s, 13.8 used / 3.3 free @2.0; moe20 was 27.9 /
|
||||
# 12.4 / 4.8). = daily-driver envelope; moe14 not worth the margin.
|
||||
threads = 6
|
||||
flash-attn = on
|
||||
cache-type-k = q8_0
|
||||
cache-type-v = q4_0 # mixed KV is fine on this arch (daily driver runs the same)
|
||||
batch-size = 2048
|
||||
ubatch-size = 512
|
||||
cache-reuse = 256 # inert under mmproj ("not supported by multimodal") — kept for parity
|
||||
defrag-thold = 0.1
|
||||
temp = 0.6 # model card, general use: 0.6 / 0.95 / 20 (1.0 only for bench reproduction)
|
||||
top-p = 0.95
|
||||
top-k = 20
|
||||
min-p = 0
|
||||
jinja = on
|
||||
sleep-idle-seconds = 120
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# GLM
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[GLM-4.7-Flash-UD-Q4_K_XL]
|
||||
# 30B-A3B MoE, primary coder. Upgraded UD-Q3_K_XL (~14 GB) → UD-Q4_K_XL (17.5 GB, unsloth's
|
||||
# recommended quant) 2026-07-16, benched clean same day → the old GLM-4.7-Flash-UD-Q3_K_XL.gguf
|
||||
# can be deleted.
|
||||
# 30B-A3B MoE, QUALITY coder. Upgraded UD-Q3_K_XL (~14 GB) → UD-Q4_K_XL (17.5 GB, unsloth's
|
||||
# recommended quant) 2026-07-16, benched clean same day. 2026-09-21: the Q3 quant is BACK as the
|
||||
# separate fast-coder preset [GLM-4.7-Flash-UD-Q3_K_XL] below (21.5 t/s here was too slow for
|
||||
# daily agent use) — keep BOTH files; this one stays for hard reviews.
|
||||
# Tuning trajectory: Q3@moe9 = 28.2 t/s → Q4@moe24 = 17.9 (measured 2026-07-16) → now moe18,
|
||||
# lower stepwise toward 16 while bench keeps free ≥2.5 GB.
|
||||
model = /home/anon/software/models/GLM-4.7-Flash-UD-Q4_K_XL.gguf
|
||||
@@ -508,6 +567,48 @@ reasoning-preserve = true # ⚗ option ACCEPTED by the build (loaded + ra
|
||||
# next days; remove if reasoning bleed ever pollutes final answers.
|
||||
sleep-idle-seconds = 60
|
||||
|
||||
[GLM-4.7-Flash-UD-Q3_K_XL]
|
||||
# ⚡ FAST CODER — same GLM-4.7-Flash, 3-bit. RE-ADDED 2026-09-21: the user found the Q4 preset's
|
||||
# 21.5 t/s (22 expert layers in RAM) too slow for agent loops; this quant fit far more of the
|
||||
# experts on the GPU and was the fastest coder this box ever ran. HISTORY (07-16, pre-KV-fix
|
||||
# era): Q3@moe9 = 28.2-29.7 t/s, @moe12 = 28.1, @moe24 = 18.9 (9.5 used / 7.6 free @16k).
|
||||
# Quality: UD 3-bit — "coders want ≥4-bit" (README) is why Q4 exists; pick Q3 for speed, Q4 for
|
||||
# hard reviews. Same deepseek2/MLA arch → same-type KV (q8/q8) is mandatory here too.
|
||||
# GATE: keep if bench ≥ ~27 t/s AND free ≥ 2.5 GB @ a ≤1.9 baseline. Start at moe12 (safe),
|
||||
# step to moe9 if free allows (~+0.75 GB per 3 layers on GPU, +~1 t/s).
|
||||
# ✓ BENCHED 2026-09-21 16:29 (2.0 DAYTIME baseline): 26.0 t/s / 46 pp / 13.4 used / 3.8 free →
|
||||
# footprint 11.4 GB. Speed = July's 28.1 minus the usual busy-desktop tax (clean-night
|
||||
# reference still to record); VRAM passes with room. +21% over the Q4's 21.5.
|
||||
# Re-run 16:34 (2.1 baseline): 25.9 t/s / 65 pp / 13.5 used / 3.7 free — reproducible; the
|
||||
# first run's 46 pp was CPU contention (prefill now matches the Q4's ~60).
|
||||
# DECISION: KEEP moe12, do NOT step to moe9. This preset's job is EVENING agent loops, so the
|
||||
# evening envelope rules: 11.4 + 3.3 evening baseline = 14.7 used / 2.5 free = exactly the
|
||||
# target; moe9 would be ~15.4 used on a busy evening = the coder's old death zone, for ~+1 t/s.
|
||||
# Want more speed on a QUIET desktop? A same-file alias at moe6 (~+1.5 GB, maybe ~29-30) is the
|
||||
# gpt-oss-low pattern — not added; ask.
|
||||
# ⬇ DOWNLOAD unsloth/GLM-4.7-Flash-GGUF (repo unchanged since 2026-02-12):
|
||||
# GLM-4.7-Flash-UD-Q3_K_XL.gguf — 13,782,004,064 B (13.78 GB)
|
||||
# sha256 582728c0f6931a1cf734d5a7cf60ec73522fb79811e004cbc12476dd69b033bf
|
||||
model = /home/anon/software/models/GLM-4.7-Flash-UD-Q3_K_XL.gguf
|
||||
ctx-size = 24576 # = clients (MLA KV is compact; n_ctx_train 202752)
|
||||
n-gpu-layers = 99
|
||||
n-cpu-moe = 12 # ✓ KEEP (09-21: 26.0 t/s, 13.4 used / 3.8 free @2.0 daytime). Evening-
|
||||
# safe (~14.7 used @3.3). moe9 = +~1 t/s for ~15.4 evening → no.
|
||||
threads = 6
|
||||
flash-attn = on
|
||||
cache-type-k = q8_0
|
||||
cache-type-v = q8_0 # MLA: mixed K/V types are rejected at context creation (see Q4 preset)
|
||||
batch-size = 2048
|
||||
ubatch-size = 512
|
||||
cache-reuse = 256
|
||||
defrag-thold = 0.1
|
||||
temp = 0.6 # mirrors the Q4 preset (unsloth: 0.7/top-p 1.0 tools, 1.0/0.95 general)
|
||||
top-p = 0.95
|
||||
min-p = 0.01
|
||||
jinja = on
|
||||
reasoning-preserve = true # mirrors the Q4 preset — same behaviour flag, same review caveat
|
||||
sleep-idle-seconds = 60
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Utility & candidates
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user