can int4 model run on cpu directly?
#2
by
paradoxian
- opened
ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype