Cache mode for draft models to save VRAM. Unlike the model's cache_mode, this does not accept a k_bits,v_bits pair.
Declarations
Type
one of "FP16", "Q8", "Q6", "Q4"Default
"FP16"Example
"Q8"
option
Cache mode for draft models to save VRAM. Unlike the model's cache_mode, this does not accept a k_bits,v_bits pair.
one of "FP16", "Q8", "Q6", "Q4""FP16""Q8"