[Sync] LLM clients → Qwen3.6-35B-A3B-Thinking default, Ornith preset, fl GPU monitor abbrs
Re-export of private HEAD 73b4a65: default-model flip + 8k thinking budget in the aichat/opencode/pi client configs, the Ornith-1.5-35B-A3B preset and its gate bench in fl/.config/llamacpp, and the fl fish abbrs `vram` / `vramx` with the new gpumon.sh (sysfs VRAM/GTT monitor).
This commit is contained in:
1 parent
2fa113cd15
commit
93b4f695ce
15 files changed
+340
-18
No files matched your search
@@ -299,6 +299,7 @@ sleep-idle-seconds = 60
|
||||
# (157/195, mean len 2.6) — so MTP IS engaging; at 2.6 tok/step that is ~175 ms per step ≈
|
||||
# 3–4× a plain forward pass. NOTE the box runs the Arch llama.cpp-vulkan package (RADV,
|
||||
# device Vulkan0) — NOT HIP/ROCm (this header said "ROCm" until 09-10); rocm-smi = monitoring only.
|
||||
# [10-02: that package is gone — now Arch `llama-cpp` 0.5.0 (b11146) + `ggml-vulkan`, see RE-PROBE below]
|
||||
# Suspects: the 48 GatedDeltaNet layers on the Vulkan backend (upstream #20354: missing/slow
|
||||
# GDN shader → CPU fallback) and/or recurrent-state checkpoint/restore per speculative step
|
||||
# (~150 MiB for this model; reported net-negative on hybrid 27Bs). The hybrid 35B-A3B does
|
||||
@@ -327,6 +328,57 @@ sleep-idle-seconds = 60
|
||||
# buffer buys nothing), MTP stays (+33% for 1.5 GB). ✓ CONFIRMED @24576 2026-09-10 03:51
|
||||
# (1.6 baseline): 15.1 t/s / 31 pp / 14.6 used / 2.5 free — footprint 13.0, exactly as
|
||||
# predicted; free = the 2.5 target at a normal idle, ~2.8 pristine, ~0.8 on a 3.3 evening.
|
||||
# ✦ RE-PROBE 2026-10-02 03:11 on llama.cpp b11146 = Arch `llama-cpp 0.5.0` + `ggml-vulkan 0.25.3`
|
||||
# (the `llama.cpp-vulkan b10615` package is gone; `ggml` conflicts with the old `ggml-cpu` and
|
||||
# does not replace it → name it: `sudo pacman -Syu ggml llama-cpp ggml-vulkan`, answer y twice;
|
||||
# `ggml-vulkan` is only an optdepend — without it the server silently runs on CPU).
|
||||
# 4.0 DIRTY baseline: 14.8 t/s / 18 pp / 14.4 used / 2.8 free. Decode IDENTICAL to 09-10
|
||||
# (14.5–15.1) → the build moves nothing on speed, as predicted. VRAM columns INVALID this run:
|
||||
# 14.4 used at a 4.0 baseline = model "added" 10.4 GB < its own 10.93 GB file — amdgpu evicted
|
||||
# the desktop's allocations to GTT to make room (the 08-16 masking effect, 3 GB this time);
|
||||
# footprint stays 13.0 from the clean 09-10 runs. TTFT 0.00 — decoded in the 03:34 entry
|
||||
# below (NOT a bench quirk, as this line first claimed).
|
||||
# Re-run 03:16 @1.8 normal-idle baseline: 13.8 t/s / 33 pp / 14.7 used / 2.4 free → footprint
|
||||
# 12.9, VRAM half exactly as on 09-10 (2.5 @1.6); 13.8 is inside the ±15% MTP acceptance
|
||||
# swing (14.5–15.1 before). b11146 confirmed NEUTRAL on both speed and VRAM for this preset.
|
||||
# Third run 03:19 @1.7: 12.5 t/s / 38 pp / TTFT 17.1 / 14.7 used / 2.4 free. Three back-to-back
|
||||
# runs → decode BAND 12.5–14.8 with VRAM identical each time; prefill 18→33→38 = Vulkan
|
||||
# pipeline warm-up on the first load after the restart, decode drift = acceptance at temp 1.0
|
||||
# over 256 tokens (±15% documented) and possibly GPU heat after 3 consecutive loads.
|
||||
# ✦ DEFINITIVE 2026-10-02 03:34, -n 512 @1.7 baseline: 15.5 t/s / 38 pp / TTFT 20.6 / 14.7 used /
|
||||
# 2.5 free. Server log: eval 442 tok @ 15.50 t/s (64.5 ms/tok), draft acceptance 0.842
|
||||
# (278/330), mean len 2.68, graphs reused 163 → MTP = 1.42× over the 10.9 base. The 256-token
|
||||
# runs under-read (short window + acceptance noise). REFERENCE: 15.5 (10.9 w/o MTP).
|
||||
# TTFT DECODED (bench.py stamps ttft on the first non-empty `content` delta; reasoning deltas
|
||||
# do not count): TTFT = prompt eval + the whole THINKING preamble. 17.1 s / 20.6 s ≈ 230-300
|
||||
# tokens of xhigh thinking on the bench's trivial prompt; "0.00" on the 256-token runs =
|
||||
# thinking consumed the entire budget and no answer was ever produced. The GGUF's embedded
|
||||
# template (header read 10-02) defaults reasoning_effort to XHIGH — heaviest of
|
||||
# xhigh/medium/low — and this preset passes nothing, so every answer pays it. That, not the
|
||||
# 15 t/s, is why it feels slow. → sibling presets benched 03:49 (next entry), then removed.
|
||||
# ✦ SIBLING BENCH 2026-10-02 03:49, -n 512 @1.9: xhigh 16.2 t/s / TTFT 17.5 s / 14.9 used / 2.2
|
||||
# free; -low 15.9 t/s / TTFT 6.45 s / 15.0 / 2.1 → reasoning_effort=low cuts the preamble 63%
|
||||
# at identical decode. -medium SKIPPED by the guard (15.7 used / 1.4 free at load) = VRAM
|
||||
# drain lag after two back-to-back loads, not a medium cost (same weights). → USER DECISION
|
||||
# 10-02: both siblings REMOVED from the config. The lever remains one line on THIS preset —
|
||||
# chat-template-kwargs = {"reasoning_effort":"low"} (or "medium") — or per request via
|
||||
# chat_template_kwargs; xhigh stays the default here.
|
||||
# What b11146 DOES carry for this preset: #27812 (merged 08-28) fixes the Vulkan graph
|
||||
# optimizer reordering nodes that alias Qwen3.8's recurrent state — on b10615 that meant
|
||||
# possible wrong tokens at temp 0 and acceptance numbers "that mean nothing"; everything from
|
||||
# 10-02 on is from a correct build. Also DFlash2 support (#27342, 08-27). NOT in b11146:
|
||||
# RDNA3 int8 coopmat prefill kernels (#27952, 09-24 — K-quant/q8 tensors only, not IQ3_XXS
|
||||
# blocks anyway) and the GDN tune (#29476, 09-29 — an Intel fix, AMD measured flat).
|
||||
# CORRECTION to the 09-10 suspects: the Vulkan GDN shader has existed since 2026-03
|
||||
# (#20334 → #20662 → #24581); there is no CPU fallback — 10.9 base IS the RDNA3 kernel
|
||||
# ceiling. The remaining lever is speculative acceptance LENGTH, not the kernel:
|
||||
# z-lab/Qwen3.8-27B-DFlash2-GGUF (Q4_K_M 1.14 GB drafter, mrope-fixed 08-24) — upstream
|
||||
# 10.4 → 18.9 t/s (1.81×, AL ≈ 5) on this model vs MTP's 1.33× here (AL 2.6). Costs: the
|
||||
# drafter + ~150 MiB recurrent rollback plane PER draft token → spec-draft-n-max 3–4 fits
|
||||
# the MTP envelope, the default 7 does not. DFlash + mmproj = open #27408 (image requests
|
||||
# stall/500) → any trial is a separate TEXT-ONLY preset; this one keeps MTP + vision.
|
||||
# Open on Vulkan for MTP and DFlash alike: #28158 — intermittent draft token id 248320
|
||||
# (= n_vocab) → HTTP 500 "Invalid input batch", mostly after very long prompts. If seen: that.
|
||||
# ⚠ Needs a CURRENT llama.cpp build: first "qwen35" file HERE with ssm.*/hybrid-attn keys
|
||||
# (the 9B is regular-attention qwen35). If the load fails with unknown-architecture /
|
||||
# missing-tensor errors, update llama.cpp on fl before touching this preset.
|
||||
@@ -493,6 +545,13 @@ sleep-idle-seconds = 30
|
||||
# Role: agentic coder with reasoning — the only coder here with MTP (Coder-30B 30, GLM Q3 26).
|
||||
# Instruct alias (reasoning-budget = 0, same file) still optional — add if agent loops want
|
||||
# the ~4 s thinking TTFT gone.
|
||||
# ⚠ NO reasoning-budget LINE IN THIS PRESET, ON PURPOSE (2026-10-12): pi caps the think block
|
||||
# PER REQUEST (`thinking_budget_tokens`, from pi settings.json > thinkingBudgets, tested on
|
||||
# b11429: budget 40 → trace cut at ~40 tokens, answer still arrives, finish_reason stop) and
|
||||
# llama.cpp IGNORES that field whenever the preset sets a CLI --reasoning-budget. Adding one
|
||||
# here "like the Thinking preset" silently disables the pi knob and brings back pi's
|
||||
# "Response was truncated before completion." (= finish_reason length: think block + answer
|
||||
# > the client's 4096 max_tokens). An instruct alias (budget 0) in a SEPARATE section is fine.
|
||||
# ⬇ DOWNLOAD from bartowski/Ornith-1.5-35B-A3B-GGUF (imatrix; the official repo has only Q4_K_M+
|
||||
# ≥ 21.7 GB, unsloth has no 1.5 quants yet):
|
||||
# Ornith-1.5-35B-A3B-IQ3_XXS.gguf 15,340,447,392 B · sha256 8918ccb9ee29abe3875efec0c3e86f0e35ef0b8a03e3f0e1c422869859a518d0
|
||||
|
||||
Reference in new issue
Block a user