Omartificial-Intelligence-Space

/

Arabic-llama3.1-lora-FT

like 7

Hi, this is a lora model must be merged with the base model to be used.
If you need a model with all params merged in 16 bits , check here please : https://huggingface.co/Omartificial-Intelligence-Space/Arabic-llama3.1-16bit-FT

Hope I answer your query

Hi
thanks for sharing
But I meant params of the finetuning aka: lr, warmup, batch size, quantization (4bit or 8bit)...
Thanks in advance

The based model was loaded in 4 bits and then fine-tuned with the following params:

lora_alpha = 16,
lora_dropout = 0,
bias = "none",
learning_rate = 2e-4,
per_device_train_batch_size = 2,
gradient_accumulation_steps = 4,
warmup_steps = 5,

what was the rank for this model?

nvmsaw int eh config file "r": 16

16