Questions on ARC-C accuracy

#10
by damoict - opened

Meta claims the 0-shot ARC-C, the accuracy for "Llama 3 8B Instruct; Llama 3.1 8B Instruct; Llama 3 70B Instruct; Llama 3.1 70B Instruct; Llama 3.1 405B Instruct" 5 models is the following

Reasoning ARC-C 0 acc 82.4 83.4 94.4 94.8 96.9

but I downloaded their huggingface models and have never been able to reproduce this ARC-C with such a high accuracy, usually the ARC-challenge's accuracy is around 60% for those big models. Is there any suggestions to reproduce this results? Especially the 0-shot to achieve 94% accuracy on ARC-Challenge? It is quite unbelievable.

Screenshot 2024-07-23 at 8.15.42 PM.png

Sign up or log in to comment