Spaces:

CoderCowMoo
/

Llama-3-70b-Instruct-Orthoganalized

Runtime error

CoderCowMoo commited on May 5, 2024

Commit

5e96c36

verified ·

1 Parent(s): 332c1a1

Maybe auto device map

Files changed (1) hide show

app.py CHANGED Viewed

@@ -16,6 +16,7 @@ tokenizer = AutoTokenizer.from_pretrained(model_id)
 quantize_config = BaseQuantizeConfig(
         bits=4,
         group_size=128,
         desc_act=False
     )
 model = AutoGPTQForCausalLM.from_quantized(

 quantize_config = BaseQuantizeConfig(
         bits=4,
         group_size=128,
+        device_map="auto",
         desc_act=False
     )
 model = AutoGPTQForCausalLM.from_quantized(