Source record

Published by r/LocalLLaMA (T2 source). The original is at https://www.reddit.com/r/LocalLLaMA/comments/1xc55z3/2bit_copy_in_vram_exact_residual_in_system_ram_as/.

Pipeline notes

The summary and note below are generated by the signal pipeline — they are Beyond Desk’s reading, not quotations from the source.

A speculative decoding scheme for large language models that exceed available GPU memory: a 2-bit quantized copy of the model lives in VRAM while the exact residual weights stay in system RAM and serve as a verifier during draft-and-validate decoding.

Concrete VRAM/RAM split that may let users run models larger than their GPU, with exact weights retained as a check rather than discarded.

Cited by

No citations on record.