Notes only for now, rework needs to be done.
Q2_K_S
Master :
PR : 2.30 GB (2.73 BPW) 2.14 GiB (2.73 BPW) PPL over 655 chunks for n_ctx=512 = 7.1215 +/- 0.04057
Q2_K
Master :
PR : 2.50 GB (2.97 BPW) 2.33 GiB (2.97 BPW) PPL over 655 chunks for n_ctx=512 = 6.7865 +/- 0.03827
Q2_K_L
PR : 2.73 GB (3.25 BPW) 2.55 GiB (3.25 BPW) PPL over 655 chunks for n_ctx=512 = 6.4599 +/- 0.03643
PR 2 : 2.78 GB (3.31 BPW) 2.59 GiB (3.31 BPW) PPL over 655 chunks for n_ctx=512 = 6.3916 +/- 0.03600
Q3_K_S
Master :
PR : 3.00 GB (3.56 BPW) 2.79 GiB (3.56 BPW) PPL over 655 chunks for n_ctx=512 = 6.1214 +/- 0.03426
Q3_K_M
Master :
PR : 3.23 GB (3.84 BPW) 3.01 GiB (3.84 BPW) PPL over 655 chunks for n_ctx=512 = 6.0295 +/- 0.03371
Q3_K_L
Master :
PR : 3.48 GB (4.13 BPW) 3.24 GiB (4.13 BPW) PPL over 655 chunks for n_ctx=512 = 5.9701 +/- 0.03337
Q3_K_XL
Master :
PR : 3.53 GB (4.19 BPW) 3.29 GiB (4.19 BPW) PPL over 655 chunks for n_ctx=512 = 5.9575 +/- 0.03329
PR 2 : 3.52 GB (4.17 BPW) 3.27 GiB (4.17 BPW) PPL over 655 chunks for n_ctx=512 = 5.9599 +/- 0.03331
PR 3 : 3.61 GB (4.29 BPW) 3.36 GiB (4.29 BPW) PPL over 655 chunks for n_ctx=512 = 5.9399 +/- 0.03320
IQ1_XS
PR : 1.58 GB (1.87 BPW) 1.47 GiB (1.87 BPW) PPL over 655 chunks for n_ctx=512 = 13.5558 +/- 0.08231
IQ1_S
Master :
PR : 1.63 GB (1.94 BPW) 1.52 GiB (1.94 BPW) PPL over 655 chunks for n_ctx=512 = 12.2136 +/- 0.07230
IQ1_M
Master :
PR : 1.72 GB (2.04 BPW) 1.60 GiB (2.04 BPW) PPL over 655 chunks for n_ctx=512 = 10.6855 +/- 0.06282
IQ1_XL
PR : 1.80 GB (2.13 BPW) 1.67 GiB (2.13 BPW) PPL over 655 chunks for n_ctx=512 = 9.1744 +/- 0.05364
PR 2 : 1.81 GB (2.15 BPW) 1.69 GiB (2.15 BPW) PPL over 655 chunks for n_ctx=512 = 9.0257 +/- 0.05266
IQ2_XXS
Master :
PR : 1.93 GB (2.29 BPW) 1.79 GiB (2.29 BPW) PPL over 655 chunks for n_ctx=512 = 7.9056 +/- 0.04535
IQ2_XS
Master :
PR : 2.09 GB (2.48 BPW) 1.94 GiB (2.48 BPW) PPL over 655 chunks for n_ctx=512 = 7.2907 +/- 0.04164
IQ2_S
Master :
PR : 2.18 GB (2.59 BPW) 2.03 GiB (2.59 BPW) PPL over 655 chunks for n_ctx=512 = 7.0517 +/- 0.04044
PR 2 : 2.20 GB (2.61 BPW) 2.05 GiB (2.61 BPW) PPL over 655 chunks for n_ctx=512 = 7.0115 +/- 0.04014
IQ2_M
Master :
PR : 2.37 GB (2.81 BPW) 2.20 GiB (2.81 BPW) PPL over 655 chunks for n_ctx=512 = 6.5630 +/- 0.03718
IQ2_XL
PR : 2.50 GB (2.97 BPW) 2.33 GiB (2.97 BPW) PPL over 655 chunks for n_ctx=512 = 6.4093 +/- 0.03615
PR 2 : 2.52 GB (2.99 BPW) 2.35 GiB (2.99 BPW) PPL over 655 chunks for n_ctx=512 = 6.3800 +/- 0.03599
IQ3_XXS
Master :
PR : 2.68 GB (3.19 BPW) 2.50 GiB (3.19 BPW) PPL over 655 chunks for n_ctx=512 = 6.2120 +/- 0.03494
IQ3_XS
Master :
PR : 2.88 GB (3.42 BPW) 2.68 GiB (3.42 BPW) PPL over 655 chunks for n_ctx=512 = 6.0963 +/- 0.03388
PR 2 : 2.89 GB (3.43 BPW) 2.69 GiB (3.43 BPW) PPL over 655 chunks for n_ctx=512 = 6.0918 +/- 0.03389
IQ3_S
Master :
PR : 3.03 GB (3.60 BPW) 2.83 GiB (3.60 BPW) PPL over 655 chunks for n_ctx=512 = 6.0106 +/- 0.03354
IQ3_M
Master :
PR : 3.17 GB (3.77 BPW) 2.96 GiB (3.77 BPW) PPL over 655 chunks for n_ctx=512 = 5.9766 +/- 0.03370
IQ3_XL
PR : 3.32 GB (3.94 BPW) 3.09 GiB (3.94 BPW) PPL over 655 chunks for n_ctx=512 = 5.9523 +/- 0.03345
IQ3_XXL
PR : 3.40 GB (4.03 BPW) 3.16 GiB (4.03 BPW) PPL over 655 chunks for n_ctx=512 = 5.9241 +/- 0.03322
PR 2 : 3.48 GB (4.13 BPW) 3.24 GiB (4.13 BPW) PPL over 655 chunks for n_ctx=512 = 5.9076 +/- 0.03308
IQ4_XS
Master : 3.62 GB (4.30 BPW) 3.37 GiB (4.30 BPW) PPL over 655 chunks for n_ctx=512 = 5.8784 +/- 0.03286
IQ4_XSR
PR : 3.97 GB (4.72 BPW) 3.70 GiB (4.72 BPW) PPL over 655 chunks for n_ctx=512 = 5.8579 +/- 0.03277
FP16
Master : PPL over 655 chunks for n_ctx=512 = 5.7977 +/- 0.03236