llama-7b-SFT-qlora-eli5-wiki_DPO_ds_RM_top_2_1024_r_64_alpha_16

This model is a fine-tuned version of dhmeltzer/llama-7b-SFT_eli5_wiki65k_1024_r_64_alpha_16_merged on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6538
• Rewards/chosen: 0.1408
• Rewards/rejected: -0.0291
• Rewards/accuracies: 0.6248
• Rewards/margins: 0.1699
• Logps/rejected: -199.6676
• Logps/chosen: -203.9681
• Logits/rejected: 0.8159
• Logits/chosen: 0.8393

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.6828	0.2	37	0.6867	-0.3470	-0.4719	0.5792	0.1249	-201.8816	-206.4072	0.7977	0.8213
0.6666	0.41	74	0.6731	-0.1233	-0.2593	0.5855	0.1361	-200.8187	-205.2885	0.8159	0.8381
0.6713	0.61	111	0.6645	0.0492	-0.1110	0.6019	0.1602	-200.0772	-204.4260	0.8299	0.8526
0.6749	0.82	148	0.6593	0.2291	0.0917	0.5912	0.1374	-199.0636	-203.5266	0.8189	0.8414
0.6688	1.02	185	0.6538	0.1408	-0.0291	0.6248	0.1699	-199.6676	-203.9681	0.8159	0.8393
0.3721	1.23	222	0.6911	-0.3548	-0.6171	0.6007	0.2623	-202.6077	-206.4462	0.8193	0.8406
0.2845	1.43	259	0.6989	-0.3528	-0.5968	0.5984	0.2441	-202.5062	-206.4359	0.7886	0.8059
0.2646	1.64	296	0.6991	-0.4016	-0.6359	0.5880	0.2343	-202.7015	-206.6800	0.7696	0.7875
0.2263	1.84	333	0.7063	-0.4773	-0.7137	0.5925	0.2365	-203.0908	-207.0584	0.7653	0.7833

Framework versions

• Transformers 4.32.1
• Pytorch 2.0.1+cu118
• Datasets 2.14.4
• Tokenizers 0.13.3

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	44.03
ARC (25-shot)	54.1
HellaSwag (10-shot)	78.74
MMLU (5-shot)	45.44
TruthfulQA (0-shot)	43.4
Winogrande (5-shot)	73.64
GSM8K (5-shot)	4.55
DROP (3-shot)	8.35