I get the following error trying to save a quantized model. Can anyone help? Thanks. quantize model quantized_model = torch.quantization.quantize_dynamic( model, {torch.nn.Linear}, dtype=torch.qint8 ) quantized_model.save_pretrained(quantized_dir) AttributeError Trac…

I found the solution here [image] Pegasus Model Weights Compression/Pruning Models @lewtun I chose dynamic quantization approach. And I dont think from_pretrained support loading quantised models. I’d be happy if you can take a look. Can we hop on a google meet call and y…

How did this issue got resolved? Am still getting the same error.

Im getting the same error. Im trying to save a quantized model using the following line torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.float16). When running quantized_model.save_pretrained(path_dir), I’m running into the error AttributeError: ‘torch.dtype’ object has no …

I was actually able to resolve the error from the Pegasus Model Weights Compression/Pruning post above. Specifically, I was using Pytorch version 1.11.0 and HuggingFace version 4.20.1.

Error saving quantized model

Intermediate

omoekan June 30, 2022, 9:59pm 2

I found the solution here

Topic		Replies	Views
Model quantization Models	5	2720	February 15, 2023
Problem with pushing quantized model to hub 🤗Transformers	3	485	October 14, 2024
ValueError: The model is quantized with QuantizationMethod.QUANTO and is not serializable 🤗Transformers	1	417	May 20, 2024
Issues when trying to save quantized model locally Beginners	0	419	May 21, 2024
Exception in save_pretrained due to recent changes 🤗Transformers	0	211	June 23, 2024

Error saving quantized model

Related topics