Repository navigation
Misc. bug: moe-cache in auto causes self-disabling #397
Description
Name and Version
I just updated from 0fc2202 (#339 merge commit) to HEAD (6e09de1) and ran into an issue with moe-cache.
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
llama-server, llama-cli
Command line
Problem description & steps to reproduce
Previously, I loaded Qwen3.6-35B-A3B using --cpu-moe and moe-cache=auto (expert layers are fused gate+up tensors, so named blk.N.ffn_gate_up_exps.weight and blk.N.ffn_down_exps.weight) and it all worked great.
Since updating, I noticed it would fail to enable moe-cache with
[37577] 0.00.744.812 I llama_model_has_cacheable_moe_weights: MoE cache disabled (no cacheable expert tensors found; largest expert slab=1152 KiB, minimum=512 KiB)
After tracking it down and putting some logging in llama_model_has_cacheable_moe_weights, I discovered that in auto it's failing on the ggml_backend_buffer_is_host check:
llama-cpp-turboquant/src/llama-context.cpp
Line 968 in 6e09de1
failing because those tensors got repacked and placed in the
CPU_REPACK buffer.
Loading with moe-cache=on works (because it disables repacking entirely.
Doesn't that defeat the purpose of auto though? It has effectively turned auto into off for me. I was of the understanding that the repacking part was moe-cache aware, and only repack tensors if it was compatible with caching. As it is now, it effectively does experts -> repack -> can't use moe-cache now -> moe-cache disabled.
First Bad Commit
Unsure, but it's related to the moe-cache reworking that was done recently.
Relevant log output
Logs
NOTE: additional logging added by me, snipped to relevant portions[37577] 0.00.027.494 I cmn common_param: device_info:
[37577] 0.00.142.799 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 4070 (11870 MiB, 8244 MiB free)
[37577] 0.00.230.208 I cmn common_param: - CUDA1 : NVIDIA GeForce GTX 1080 Ti (11165 MiB, 11009 MiB free)
[37577] 0.00.230.213 I cmn common_param: - BLAS : BLIS (0 MiB, 0 MiB free)
[37577] 0.00.230.224 I cmn common_param: - CPU : AMD Ryzen 9 5900X 12-Core Processor (64195 MiB, 64195 MiB free)
...
[37577] 0.00.688.223 D create_tensor: loading tensor blk.0.ffn_down_exps.weight
[37577] 0.00.688.256 D tensor blk.0.ffn_down_exps.weight (144 MiB iq4_nl) buffer type overridden to CPU_REPACK
[37577] 0.00.688.268 D create_tensor: loading tensor blk.0.ffn_gate_up_exps.weight
[37577] 0.00.688.280 D tensor blk.0.ffn_gate_up_exps.weight (288 MiB iq4_nl) buffer type overridden to CPU_REPACK
...
[37577] 0.00.694.898 D done_getting_tensors: tensor 'blk.0.ffn_down_exps.weight' (iq4_nl) (and 80 others) cannot be used with preferred buffer type CUDA0, using CPU_REPACK instead
...
[37577] 0.00.743.378 I llama_model_has_cacheable_moe_weights: MoE cache invalid tensor skipped: name=blk.0.ffn_down_exps.weight, buffer_is_host=0
...
[37577] 0.00.744.812 I llama_model_has_cacheable_moe_weights: MoE cache disabled (no cacheable expert tensors found; largest expert slab=1152 KiB, minimum=512 KiB)
[37577] 0.00.744.813 I sched_reserve: MoE cache requested=auto resolved=off
Activity