llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp
2.85T2 sourcer/LocalLLaMA
r/LocalLLaMAoriginal source ↗
Source record
Published by r/LocalLLaMA (T2 source). The original is at https://www.reddit.com/r/LocalLLaMA/comments/1x03xkc/llama_add_a_gpu_cache_for_moe_experts_kept_in/.
Pipeline notes
The summary and note below are generated by the signal pipeline — they are Beyond Desk’s reading, not quotations from the source.
SummaryPull request to llama.cpp introducing a GPU cache for Mixture of Experts expert weights held in host memory, reducing expert lookup latency during local inference.
Why it mattersDirectly relevant to local LLM runners using MoE architectures; GPU-side expert caching lowers the VRAM barrier for running large sparse models.
Cited by
No citations on record.