NIAH

Benchmarks

Task Name	Dataset Name	SOTA Result
Information Retrieval	NIAH (test)	Average Score99.6	59
Long-context Retrieval	NIAH	Accuracy100	50
Retrieval	NIAH seq=4096	Accuracy100	44
Multi-needle retrieval	NIAH (M)	Accuracy (NIAH M)90.2	35
Needle-in-a-Haystack retrieval	NIAH-Single-3 256K context	Accuracy94	23
Needle-in-a-Haystack retrieval	NIAH-Single-3 128K context	Accuracy98.8	23
Needle-in-a-Haystack retrieval	NIAH-Single-3 64K context	Accuracy100	23
Needle-in-a-Haystack retrieval	NIAH-Single-2 32K context	Accuracy100	23
Needle-in-a-Haystack retrieval	NIAH-Single-1 256K context	Accuracy100	23
Needle-in-a-Haystack retrieval	NIAH-Single-1 128K context	Accuracy100	23
Needle-in-a-Haystack retrieval	NIAH-Single-1 64K context	Accuracy100	23
Long-context Retrieval	NIAH Single 3	Accuracy (1024)100	22
Long-context Retrieval	NIAH 128k	Single Score24.4	20
Long-context Retrieval	NIAH 64k	Single Score49.3	20
Long-context retrieval	NIAH multivalue	Speedup4.1	20
Needle-In-A-Haystack Retrieval	NIAH	NIAH Score100	14
Needle-in-a-Haystack retrieval	NIAH Single 64K context 2	Accuracy100	13
Long Context Retrieval	NIAH-Multi	Accuracy100	13
Long-context retrieval	NIAH-Single-3 128K context	Accuracy90.6	12
Long-context retrieval	NIAH-Single-3 64K context	Accuracy97	12
Long-context retrieval	NIAH Single 3 32K context	Accuracy98.6	12
Long-context retrieval	NIAH-Single-2 256K context	Accuracy93.4	12
Long-context retrieval	NIAH-Single-1 256K	Accuracy100	12
Long-context retrieval	NIAH-Single-1 128K	Accuracy100	12
Long-context retrieval	NIAH-Single-1 32K	Accuracy100	12

Showing 25 of 58 rows