when i use torch.float16，i face this problem probability tensor contains either `inf`, `nan` or element < 0
#20
by
als-991011
- opened
self.model = Qwen2VLForConditionalGeneration.from_pretrained(
self.weight_path, torch_dtype=torch.float16, device_map=self.device_map
)
self.model = Qwen2VLForConditionalGeneration.from_pretrained(
self.weight_path,
torch_dtype=torch.float16,
attn_implementation="flash_attention_2", # add this can solve
device_map=self.device_map
)
I have the issue of generation fault like "!", "![]![]", etc. but attn_implementation="flash_attention_2" works for me.
Thanks
@als-991011