[Sync] LLM clients → Qwen3.6-35B-A3B-Thinking default, Ornith preset, fl GPU monitor abbrs

Re-export of private HEAD 73b4a65: default-model flip + 8k thinking budget
in the aichat/opencode/pi client configs, the Ornith-1.5-35B-A3B preset and
its gate bench in fl/.config/llamacpp, and the fl fish abbrs `vram` /
`vramx` with the new gpumon.sh (sysfs VRAM/GTT monitor).
This commit is contained in:
Coja committed 2026-10-16 14:11:09 +02:00
1 parent 2fa113cd15
commit 93b4f695ce
15 files changed
+340 -18

No files matched your search

+20 -2
View File
@@ -4,7 +4,7 @@ llama.cpp in **router mode** serving ~10 model presets from one 16 GB GPU.
Last full tune: **2026-07-16** (decode numbers measured then via `bench.py`, unless a preset comment in `config.ini` says otherwise).
Last full clean sweep: **2026-08-08 03:22** — all 11 presets loaded and ran green (baseline 2.0 GB) after the week's coder-race/KV-fix/gemma-sidecar round.
- **Box**: AMD RX 7600 XT 16 GB (~288 GB/s; backend = Arch `llama.cpp-vulkan` / RADV, **not** ROCm —
- **Box**: AMD RX 7600 XT 16 GB (~288 GB/s; backend = Arch `llama-cpp` 0.5.0 + `ggml-vulkan` / RADV = llama.cpp **b11146** since 2026-10-02, **not** ROCm —
`rocm-smi` is only the VRAM monitor) + Ryzen 5600X (6 cores) + 48 GB RAM.
⚠ The GPU **also drives the display** — see [VRAM safety](#vram-safety-the-golden-rules).
- **Endpoints**: LAN `http://$SERVER:11343/v1` · remote `https://<own-domain>/api/v1`
@@ -35,6 +35,14 @@ symlinks, so on the server `dotsync` (pull; re-stows only if files were added/re
`sudo systemctl restart llama.service` — the router reads presets once at startup.
After ANY preset change: restart + `./bench.py -m <changed-ids>` and check the guards.
**Upgrading llama.cpp itself** (2026-10-02, b10615 → b11146): Arch replaced `llama.cpp-vulkan`
with a unified `llama-cpp` package on top of the system `ggml`, whose backends are split out.
`sudo pacman -Syu ggml llama-cpp ggml-vulkan` — name **`ggml`** explicitly (it conflicts with
the old `ggml-cpu` without replacing it, and the removal prompt defaults to N), and **`ggml-vulkan`**
explicitly (only an optdepend — without it llama-server silently runs on the CPU backend).
Then restart, and `llama-server --list-devices` must show `Vulkan0`. `whisper-cpp` moves to the
same `ggml` (CPU backends are inside the base package) — restart its unit too.
## VRAM safety (the golden rules)
The amdgpu driver does not OOM cleanly — an overcommitted model spills into GTT (system RAM),
@@ -62,7 +70,7 @@ starves the desktop and freezes the PC (reboot). Hard-won rules:
| `GLM-4.7-Flash-UD-Q4_K_XL` | **21.5** @moe22 | 24k | reasoning | quality coder (opencode subagents); KV fix + moe22 ✓ verified clean 08-08 |
| `GLM-4.7-Flash-UD-Q3_K_XL` | **26.0** @moe12 (2.0 daytime baseline; ~28 clean) | 24k | reasoning | **fast coder** — re-added 09-21 because Q4's 21.5 dragged in agent loops; 3-bit, evening-safe at moe12 (13.4 used / 3.8 free @2.0); Q4 stays for hard reviews |
| `Qwen3-Coder-Next-UD-IQ3_XXS` | 16.0 | **128k** | 80B-A3B | long-session coder (128k ctx) |
| `Qwen3.8-27B-UD-IQ3_XXS` | 14.5 (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~30). 2.5 GB free @1.6 baseline @24k (✓ 09-10); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
| `Qwen3.8-27B-UD-IQ3_XXS` | **15.5** (10.9 w/o MTP) | 24k | vision, MTP, hybrid reasoner @ **xhigh** | ⚠ **clean-desktop specialty** (kept 09-10 over the ≥22 gate) — dense-27B quality, slow: its Gated-DeltaNet layers are Vulkan-bound (prefill ~35). 2.5 GB free @1.7 baseline @24k (✓ 10-02 on b11146, `-n 512`, acceptance 0.84). The *felt* slowness is the **xhigh thinking preamble** (17.5 s before the first answer token on a trivial prompt; `reasoning_effort=low` measured 6.45 s at the same decode on 10-02 — siblings benched then removed by decision, the kwarg is one line if ever wanted); **never on a busy evening desktop** (~1 GB free = GTT freeze) |
| `Qwen3-Embedding-0.6B` | 33 (CPU) | 8k | CPU-only, `/v1/embeddings` | RAG/search embedder (not a chat model) |
| `Ornith-1.5-35B-A3B-IQ3_XXS` | **32.1** @moe16 | 24k | MTP, vision, thinking | **agentic coder** (added 09-21) — Qwen3.5-MoE derivative claiming +15 Terminal-Bench / +5.6 SWE-bench over Qwen3.6-35B; the only coder with MTP; 13.8 used / 3.3 free @2.0 |
@@ -144,6 +152,7 @@ the router, but chats/presets saved against the old id need re-picking.
2026-09-21: `GLM-4.7-Flash-UD-Q3_K_XL` (⚡ fast coder) added to all six client files next to the Q4.
2026-09-21: `Ornith-1.5-35B-A3B-IQ3_XXS` (agentic coder, thinking + vision) added to all six client files after passing its gate (32.1 t/s @moe16).
2026-09-21: default model → `Qwen3.6-35B-A3B-Thinking` everywhere (opencode global + plan/build/auto, pi defaultModel + thinking level medium, aichat common/lw/wm) — gemma-4-26B refused routine sysadmin work. Thinking preset gets `reasoning-budget 8192`. wm aichat caught up (GLM Q3, Ornith).
2026-10-12: Ornith in pi: "Response was truncated before completion." = `finish_reason: length` — the `<think>` block counts against pi's 4096 `max_tokens` and the preset has no reasoning-budget. Fixed client-side: pi sends `thinking_budget_tokens` per thinking level on Ornith (`compat.thinkingTokenBudgetField` + `settings.json > thinkingBudgets`, common + lw). The preset deliberately keeps NO CLI reasoning-budget — it would override the per-request field (see the preset comment).
## Tuning cheat-sheet
@@ -183,8 +192,17 @@ verifies a clean card with rocm-smi before every load, records a ⚠ row and ski
generation when a model lands too tight (free < 1.5 GB or GTT ballooning).
Results: `bench-results.md` (latest) + `bench-history.md` (append-only; diff runs there).
**Reading TTFT**: bench.py stamps it on the first non-empty `content` delta, so for thinking
models TTFT = prompt eval **plus the whole reasoning preamble**, and `0.00` means the answer never
started within `-n` tokens (thinking ate the budget). It is the column that compares
reasoning-effort siblings; for stable MTP decode numbers use `-n 512`.
## Watch list
- **GLM MTP in llama.cpp** — the single biggest pending win (primary coder ~22 → ~30+).
- **DFlash2 for Qwen3.8-27B** — `z-lab/Qwen3.8-27B-DFlash2-GGUF` drafter, upstream 1.81× vs the
MTP head's 1.33× here (b11146 has the support). Blockers: #27408 (spec + mmproj → image requests
500, so only as a text-only sibling preset) and #28158 (Vulkan OOB draft token, open). Trial =
`spec-type draft-dflash`, `spec-draft-n-max 3`, no mmproj; n-max 7 does not fit the VRAM envelope.
- **Coder-Next 256k**: if the load log's `n_ctx_train` says 262144, ctx can likely double
again for ~1 GB (KV measured nearly flat).