Language Modeling on SlimPajama (test)

0.5402PPL (CommonCrawl)

MIXChunk

Updated 4d ago

Evaluation Results

Method
MIXChunk 2024.02	0.5402	0.4867	0.6219	0.7443	0.582	0.5042	0.6531	0.5903	-
MIXChunk 2024.02	0.5429	0.495	0.6238	0.7665	0.5974	0.5001	0.6406	0.5952	-
UNIChunk 2024.02	0.5429	0.4888	0.6235	0.7483	0.5859	0.5065	0.6564	0.5932	-
UNIChunk 2024.02	0.5468	0.4984	0.6298	0.7709	0.6011	0.5033	0.6436	0.5991	-
BM25Chunk 2024.02	0.5489	0.4952	0.6391	0.7621	0.5919	0.5108	0.6599	0.6011	-
BM25Chunk 2024.02	0.5496	0.5021	0.6394	0.7782	0.6041	0.505	0.6452	0.6034	-
INTRADoc 2024.02	0.5506	0.4988	0.6443	0.7643	0.5936	0.5119	0.6597	0.6033	-
INTRADoc 2024.02	0.5507	0.5048	0.6426	0.7793	0.605	0.5062	0.6458	0.6049	-
INTRADoc 2024.02	8.994	13.173	6.073	5.01	7.894	10.701	4.705	8.079	-
BM25Chunk 2024.02	9.144	13.579	6.287	5.463	8.022	10.81	4.715	8.289	-
UNIChunk 2024.02	9.478	14.19	6.897	7.006	8.456	11.117	4.812	8.851	-
MIXChunk 2024.02	9.645	14.424	7.01	7.496	8.634	11.337	4.911	9.065	-
BM25Chunk 2024.02	11.418	13.677	6.237	4.585	7.623	11.253	5.059	8.55	-
INTRADoc 2024.02	11.631	13.197	6.084	4.252	7.535	11.13	5.03	8.41	-
UNIChunk 2024.02	11.805	13.65	6.546	5.518	7.839	11.353	5.106	8.831	-
MIXChunk 2024.02	13.284	13.884	6.811	5.531	8.051	11.623	5.203	9.172	-
Sheared Llama 2024.11	21.75	26	29.5	93.75	56.5	33.5	38.5	42.79	-
ReSheared 2024.11	30.5	30.25	23	89.25	47.5	32	81	47.64	-
Constant 2024.11	35	40	21.75	88	72.5	47.75	82	55.29	-
DRPruning 2024.11	44	51.5	33.5	94.75	90	48	86.5	64.04	-
ReSheared 2024.11	61.75	60.25	80.5	96	92	73	93.75	79.61	-
Sheared Llama 2024.11	81.25	89.5	89.25	95.5	82.75	96.5	90.5	89.32	-
DRPruning 2024.11	82.25	77.75	87.5	99	89.25	86.75	79.5	86	-
Transformer 2026.01	-	-	-	-	-	-	-	-	2.912
Transformer + Spelling Bee Embeddings 2026.01	-	-	-	-	-	-	-	-	2.897