Tuning local LLM inference on an AMD Strix Halo box. The settings below are worth copying. The more useful part is the record of what sounded right and wasn't — each took one command to disprove, and one would have cost weeks. That includes this project's own benchmark, which spent two weeks reporting a tie that its scoring rule had manufactured.
Result
--model Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 99 -fa on --spec-type draft-mtp --spec-draft-n-max 3 +79% generation -b 2048 -ub 256 +29% prefill, vs the usual 1024 (dense; a MoE can invert it) -ctk q8_0 -ctv q8_0 free, halves the KV cache -c 262144 full context costs 3%
| reading | value | note |
|---|---|---|
| generation | 20.3 t/s | ~38 t/s on code, where drafts land better |
| prefill | 167 t/s | a 44k prompt waits ~4.5 min before word one |
| weights | 16.7 GB | the smallest of the five models benchmarked |
| quality | withdrawn | the benchmark was measuring itself |
Signature finding
Before each experiment I wrote down what I expected. Hollow marker: the prediction. Solid marker: what the hardware actually did. The dashed span between them is the error.
Generation speed · higher is better →
Generation is memory-bound, so a smaller model should read fewer bytes and run faster. It doesn't: unpacking the more compressed formats costs more arithmetic than the bandwidth it saves. Speed runs backwards from file size. Below roughly Q4_K you spend compute to save bandwidth you were never short of.
Measured
Generation speed, tokens/sec · higher is better →
A built-in draft head guesses the next few tokens; the model verifies them in one pass. Correct guesses are free — but depth 5 is slower than switching speculation off entirely, because every rejected token still costs a full verification pass that yields nothing.
Prefill speed, tokens/sec · higher is better → · smaller -ub wins
The accented bar is not the longest one, deliberately. 128 measured fastest,
by 0.9% — one run each, so this sweep cannot actually separate it from 256. The curve has gone
flat by then: every config at 256 or below lands between 167.4 and 169.0. We ship
-ub 256 as the knee of the curve, where essentially all of the gain has been
collected, rather than chase a 1.6 t/s edge that a single run can't establish. If you want that
edge, measure 128 against 256 on your own box with repeats — don't take it from this chart.
-ub 256 is 29% faster than the 1024 commonly recommended — including by an
earlier version of this repo, which measured it on a different model architecture.
-b makes no measurable difference at matched -ub. Re-running the 512
case in a separate session landed within 0.1%, so this is the flag and not thermal drift.
qwen3next, solo,
build b11003) runs the other way: prefill of a 4096-token prompt at depth 32768 goes
297.8 (-ub 256) → 356.2 (512) → 401.3 (1024) → 339.2 (2048)
t/s — so -ub 1024 is +34.8% on that model, and 2048 regresses.
Generation speed is flat to four values across the whole 8× range (44.0/44.0/44.2/44.1),
re-confirming that batch size does not move tg on this box. Neither number is withdrawn; they are
both correct, for different architectures. The shipped default stays 256 and the coder model
carries a per-model override. The lesson is the one this page already gave — sweep it
— and now it applies per model class, not just per GPU.Generation speed, tokens/sec · higher is better · baseline 20.06
| change | t/s | verdict |
|---|---|---|
| -ctk/-ctv q8_0 | 20.23 | free — and halves the KV cache |
| -c 262144 | 19.67 | full context costs 3% |
| -ctk/-ctv q4_0 | 18.99 | −5%, no reason to go below q8_0 |
| draft-mtp,ngram-mod | 17.97 | stacking speculators loses |
| -fa off | 17.62 | −14%, keep flash attention on |
Refuted
Each was a sound inference from real evidence. Each took a single command to disprove.
| the idea | why it was reasonable | what happened |
|---|---|---|
| rewrite the serial kernel | the code really is token-by-token, and upstream fixed it for NVIDIA only | 1.1% of prefill |
| attention collapses at depth | a 3.3× cliff in the profiler | that data was decode |
| smaller quant is faster | dense model, memory-bound | backwards |
| bigger batches are efficient | fewer, larger operations | 29% slower |
| less reasoning saves time | fewer thinking tokens | 74% slower |
test-backend-ops perf -o <OP> costs about a minute and needs no downtime. It
refuted a multi-week plan here in one command. Finding that a kernel is inefficient proves only
that the kernel is inefficient — not that it matters.Attribution
Share of total prefill time · composition, not a score
Reading a 44k-token prompt costs about 2.45 PFLOP, and the weight matmuls measure 10.83 TFLOPS — close to this hardware's practical ceiling. Roughly half of prefill is irreducible arithmetic. That's why the kernel rewrite above was worth 1.1%, not the weeks it would have taken.
Quality · withdrawn
Four models spanning 16.7 GB to 89 GB all scored 70/70 on the private coding suite and 27–29 of 29 on tool calling. That is not four tied models. That is a benchmark with no information in it — and chasing why turned up something worse.
Both suites ran at temperature 0, picked for reproducibility. For thinking models it is measurably the wrong choice. Auditing the saved transcripts found that every truncated turn was a repetition loop that emitted no answer at all — 9 of 9, across all four models, one line repeated up to 439 times until the token budget ran out.
It stayed invisible because the scorer excluded truncated turns — a rule meant to stop a harness limit being scored as a coding failure. It was quietly rescuing the turns where the model produced nothing. Re-scoring the same transcripts without that rescue:
Raw last-turn score, against what was published · higher is better
| model | task | raw | reported |
|---|---|---|---|
| Qwen3.5-122B-A10B | token_budget | 0/20 | 70/70 |
| Qwen3.5-122B-A10B | window_merge | 0/18 | 70/70 |
| Qwen3.8-27B | window_merge | 3/18 | 70/70 |
| Laguna-S-2.1 | window_merge | no code | 70/70 |
| Ornith-1.0-35B | quant_pick | 15/18 | 70/70 |
The replacement
Three new multi-turn tasks, 89 hidden tests, where the tests probe what each spec
implies rather than what it states: a token bucket that stays correct under clock skew
and sub-millisecond refill, semver range semantics including prerelease exclusion, and a SQL
WHERE evaluator in full three-valued logic.
The first model through it — Ornith-1.0-35B, which scores 70/70 on the easy tier — reached 49/89. This page said 16/89 for a day, and that number was mostly our own bug.
Hidden tests passed · higher is better
| task | turn 1 | corrected | as first published |
|---|---|---|---|
| hard_ratelimit | 10/10 | 16/25 | 16/25 |
| hard_semver | 11/11 | 11/27 | 0/27 |
| hard_where | 14/14 | 22/37 | 0/37 |
Range class", so models replied with only the new code. The harness
overwrites the file wholesale, so a partial reply loses the earlier turns and fails on import —
which was anticipated and flagged, except the check asked whether the entry symbol appeared
anywhere in the text. A fragment that merely calls it contains the string. Five
turns were graded as complete solutions scoring zero.
Reproduce it
Nothing to compile. The first script downloads a prebuilt llama.cpp release; the second fetches a model and verifies its byte count; the third serves it as an OpenAI-compatible endpoint that Claude Code, Codex and any OpenAI SDK can point straight at.
git clone https://github.com/lazarevtill/strix-halo-llm cd strix-halo-llm .\scripts\windows\fetch-llamacpp.ps1 -Build b10431 1 · the engine .\scripts\windows\fetch-models.ps1 -Only qwen38 2 · a model, 16.7 GB .\scripts\windows\run-solo.ps1 3 · serve it on :8080
Linux and macOS have ported equivalents under scripts/linux/ and
scripts/macos/. They are honest drafts — written here, never run on their own
platform — so start with --dry-run, which prints the exact invocation and launches
nothing.
| platform | backend | state |
|---|---|---|
| Windows 11 | Vulkan | measured — every number here |
| Linux | Vulkan | ported, unproven |
| macOS · Apple silicon | Metal | ported, unproven |
-ub 256 is the biggest win here
(+29% prefill on a dense model) and the most architecture-specific: it works because a
256-row tile fits gfx1151's 32 KB of shared memory. It does not even hold across architectures on
this GPU — a MoE measured +34.8% at -ub 1024. Sweep it rather than
trusting this number, per GPU and per model class.