Transformers documentation
SpQR
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v5.22.2).
SpQR
Algoritma kuantisasi SpQR melibatkan struktur kuantisasi 3-bit grup bi-level bertile 16x16 dengan outlier yang jarang.
Untuk mengkuantisasi model dengan SpQR, lihat repositori Vahe1994/SpQR.
Muat model yang dikuantisasi SpQR dengan from_pretrained().
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
quantized_model = AutoModelForCausalLM.from_pretrained(
"elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf",
dtype=torch.half,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf")