[Sync] LLM clients → Qwen3.6-35B-A3B-Thinking default, Ornith preset, fl GPU monitor abbrs
Re-export of private HEAD 73b4a65: default-model flip + 8k thinking budget in the aichat/opencode/pi client configs, the Ornith-1.5-35B-A3B preset and its gate bench in fl/.config/llamacpp, and the fl fish abbrs `vram` / `vramx` with the new gpumon.sh (sysfs VRAM/GTT monitor).
This commit is contained in:
1 parent
2fa113cd15
commit
93b4f695ce
15 files changed
+340
-18
No files matched your search
@@ -4,7 +4,7 @@ llama.cpp in **router mode** serving ~10 model presets from one 16 GB GPU.
|
||||
Last full tune: **2026-07-16** (decode numbers measured then via `bench.py`, unless a preset comment in `config.ini` says otherwise).
|
||||
Last full clean sweep: **2026-08-08 03:22** — all 11 presets loaded and ran green (baseline 2.0 GB) after the week's coder-race/KV-fix/gemma-sidecar round.
|
||||
|
||||
- **Box**: AMD RX 7600 XT 16 GB (~288 GB/s; backend = Arch `llama.cpp-vulkan` / RADV, **not** ROCm —
|
||||
- **Box**: AMD RX 7600 XT 16 GB (~288 GB/s; backend = Arch `llama-cpp` 0.5.0 + `ggml-vulkan` / RADV = llama.cpp **b11146** since 2026-10-02, **not** ROCm —
|
||||
`rocm-smi` is only the VRAM monitor) + Ryzen 5600X (6 cores) + 48 GB RAM.
|
||||
⚠ The GPU **also drives the display** — see [VRAM safety](#vram-safety-the-golden-rules).
|
||||
- **Endpoints**: LAN `http://$SERVER:11343/v1` · remote `https://<own-domain>/api/v1`
|
||||
@@ -35,6 +35,14 @@ symlinks, so on the server `dotsync` (pull; re-stows only if files were added/re
|
||||
`sudo systemctl restart llama.service` — the router reads presets once at startup.
|
||||
After ANY preset change: restart + `./bench.py -m <changed-ids>` and check the guards.
|
||||
|
||||
**Upgrading llama.cpp itself** (2026-10-02, b10615 → b11146): Arch replaced `llama.cpp-vulkan`
|
||||
with a unified `llama-cpp` package on top of the system `ggml`, whose backends are split out.
|
||||
`sudo pacman -Syu ggml llama-cpp ggml-vulkan` — name **`ggml`** explicitly (it conflicts with
|
||||
the old `ggml-cpu` without replacing it, and the removal prompt defaults to N), and **`ggml-vulkan`**
|
||||
explicitly (only an optdepend — without it llama-server silently runs on the CPU backend).
|
||||
Then restart, and `llama-server --list-devices` must show `Vulkan0`. `whisper-cpp` moves to the
|
||||
same `ggml` (CPU backends are inside the base package) — restart its unit too.
|
||||
|
||||
## VRAM safety (the golden rules)
|
||||
|
||||
The amdgpu driver does not OOM cleanly — an overcommitted model spills into GTT (system RAM),
|
||||
@@ -62,7 +70,7 @@ starves the desktop and freezes the PC (reboot). Hard-won rules:
|
||||
| `GLM-4.7-Flash-UD-Q4_K_XL` | **21.5** @moe22 | 24k | reasoning | quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08 |
|
||||
| `GLM-4.7-Flash-UD-Q3_K_XL` | **26.0** @moe12 (2.0 daytime baseline; ~28 clean) | 24k | reasoning | **fast coder** — re-added 09-21 because Q4's 21.5 dragged in agent loops; 3-bit, evening-safe at moe12 (13.4 used / 3.8 free @2.0); Q4 stays for hard reviews |
|
||||
| `Qwen3-Coder-Next-UD-IQ3_XXS` | 16.0 | **128k** | 80B-A3B | long-session coder (128k ctx) |
|
||||
| `Qwen3.8-27B-UD-IQ3_XXS` | 14.5 (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~30). 2.5 GB free @1.6 baseline @24k (✓ 09-10); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
|
||||
| `Qwen3.8-27B-UD-IQ3_XXS` | **15.5** (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner @ **xhigh** | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~35). 2.5 GB free @1.7 baseline @24k (✓ 10-02 on b11146, `-n 512`, acceptance 0.84). The *felt* slowness is the **xhigh thinking preamble** (17.5 s before the first answer token on a trivial prompt; `reasoning_effort=low` measured 6.45 s at the same decode on 10-02 — siblings benched then removed by decision, the kwarg is one line if ever wanted); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
|
||||
| `Qwen3-Embedding-0.6B` | 33 (CPU) | 8k | CPU-only, `/v1/embeddings` | RAG/search embedder (not a chat model) |
|
||||
| `Ornith-1.5-35B-A3B-IQ3_XXS` | **32.1** @moe16 | 24k | MTP, vision, thinking | **agentic coder** (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0 |
|
||||
|
||||
@@ -144,6 +152,7 @@ the router, but chats/presets saved against the old id need re-picking.
|
||||
2026-09-21: `GLM-4.7-Flash-UD-Q3_K_XL` (⚡ fast coder) added to all six client files next to the Q4.
|
||||
2026-09-21: `Ornith-1.5-35B-A3B-IQ3_XXS` (agentic coder, thinking + vision) added to all six client files after passing its gate (32.1 t/s @moe16).
|
||||
2026-09-21: default model → `Qwen3.6-35B-A3B-Thinking` everywhere (opencode global + plan/build/auto, pi defaultModel + thinking level medium, aichat common/lw/wm) — gemma-4-26B refused routine sysadmin work. Thinking preset gets `reasoning-budget 8192`. wm aichat caught up (GLM Q3, Ornith).
|
||||
2026-10-12: Ornith in pi: "Response was truncated before completion." = `finish_reason: length` — the `<think>` block counts against pi's 4096 `max_tokens` and the preset has no reasoning-budget. Fixed client-side: pi sends `thinking_budget_tokens` per thinking level on Ornith (`compat.thinkingTokenBudgetField` + `settings.json > thinkingBudgets`, common + lw). The preset deliberately keeps NO CLI reasoning-budget — it would override the per-request field (see the preset comment).
|
||||
|
||||
## Tuning cheat-sheet
|
||||
|
||||
@@ -183,8 +192,17 @@ verifies a clean card with rocm-smi before every load, records a ⚠ row and ski
|
||||
generation when a model lands too tight (free < 1.5 GB or GTT ballooning).
|
||||
Results: `bench-results.md` (latest) + `bench-history.md` (append-only; diff runs there).
|
||||
|
||||
**Reading TTFT**: bench.py stamps it on the first non-empty `content` delta, so for thinking
|
||||
models TTFT = prompt eval **plus the whole reasoning preamble**, and `0.00` means the answer never
|
||||
started within `-n` tokens (thinking ate the budget). It is the column that compares
|
||||
reasoning-effort siblings; for stable MTP decode numbers use `-n 512`.
|
||||
|
||||
## Watch list
|
||||
|
||||
- **GLM MTP in llama.cpp** — the single biggest pending win (primary coder ~22 → ~30+).
|
||||
- **DFlash2 for Qwen3.8-27B** — `z-lab/Qwen3.8-27B-DFlash2-GGUF` drafter, upstream 1.81× vs the
|
||||
MTP head's 1.33× here (b11146 has the support). Blockers: #27408 (spec + mmproj → image requests
|
||||
500, so only as a text-only sibling preset) and #28158 (Vulkan OOB draft token, open). Trial =
|
||||
`spec-type draft-dflash`, `spec-draft-n-max 3`, no mmproj; n-max 7 does not fit the VRAM envelope.
|
||||
- **Coder-Next 256k**: if the load log's `n_ctx_train` says 262144, ctx can likely double
|
||||
again for ~1 GB (KV measured nearly flat).
|
||||
Reference in new issue
Block a user