systemone lab

Speed

Median time to answer one question on an Apple-silicon laptop with 64 GB, from runs with nothing else going on. Hosted services include network time, so they are not comparable with local models.

Speed against gain over guessing

Up and to the left is better. Labelled points are the ones nothing beats on both counts at once; hover any point for its name. Models scored on only some data sets are left out.

10 ms100 ms1 s10 s0+10+20+30+40+50time per answer (log scale)gain over guessing, ptsJev (hosted)Jev-Style 0.8BLaya multilingualTacet SonataJev (hosted): 101 ms per answer, +49 pts over guessing, runs on hostedruling 35B-A3B: 391 ms per answer, +47 pts over guessing, runs on Apple GPUWinnow E4B: 184 ms per answer, +47 pts over guessing, runs on Apple GPUJevK5 4B: 181 ms per answer, +44 pts over guessing, runs on Apple GPUSemIf 4B: 171 ms per answer, +43 pts over guessing, runs on Apple GPUJev-Style 0.8B: 39 ms per answer, +42 pts over guessing, runs on Apple GPUKev 9B · Ollaya: 3.2 s per answer, +42 pts over guessing, runs on CPUKev 4B · MLX: 131 ms per answer, +41 pts over guessing, runs on Apple GPUKev 4B · Ollaya: 1.8 s per answer, +41 pts over guessing, runs on CPUdecider 2B: 614 ms per answer, +40 pts over guessing, runs on CPULaya multilingual: 15 ms per answer, +37 pts over guessing, runs on Apple GPUcbjev multilingual: 20 ms per answer, +37 pts over guessing, runs on Apple GPULaya multilingual · fine-tuned: 15 ms per answer, +36 pts over guessing, runs on Apple GPUdecider 0.8B: 279 ms per answer, +36 pts over guessing, runs on CPULaya English · Ollaya: 32 ms per answer, +36 pts over guessing, runs on Apple GPUDecision 1.0 Eos: 391 ms per answer, +36 pts over guessing, runs on CPULaya multilingual · Ollaya: 13 ms per answer, +36 pts over guessing, runs on Apple GPULaya English: 35 ms per answer, +36 pts over guessing, runs on Apple GPUjeff (GLiFormer): 90 ms per answer, +35 pts over guessing, runs on Apple GPUNLI zero-shot (DeBERTa): 366 ms per answer, +35 pts over guessing, runs on CPUjevlike (trained per data set): 14 ms per answer, +30 pts over guessing, runs on Apple GPUKev 0.8B · Ollaya: 286 ms per answer, +29 pts over guessing, runs on CPUVon 1.1: 120 ms per answer, +24 pts over guessing, runs on CPUGLiClass instruct large: 123 ms per answer, +20 pts over guessing, runs on CPUTacet Sonata: 13 ms per answer, +8 pts over guessing, runs on Apple GPUCLM-8B: 114 ms per answer, +7 pts over guessing, runs on Apple GPU
Apple GPUCPUhosted service

Time per answer

ModelRuns on Per answerAvg gain
Tacet SonataApple GPU13 ms+8 pts
Laya multilingual · OllayaApple GPU13 ms+36 pts
jevlike (trained per data set)Apple GPU14 ms+30 pts
Laya multilingual · fine-tunedApple GPU15 ms+36 pts
Laya multilingualApple GPU15 ms+37 pts
cbjev multilingualApple GPU20 ms+37 pts
Laya English · OllayaApple GPU32 ms+36 pts
Laya EnglishApple GPU35 ms+36 pts
Jev-Style 0.8BApple GPU39 ms+42 pts
jeff (GLiFormer)Apple GPU90 ms+35 pts
Jev (hosted)hosted101 ms+49 pts
CLM-8BApple GPU114 ms+7 pts
Von 1.1CPU120 ms+24 pts
GLiClass instruct largeCPU123 ms+20 pts
Kev 4B · MLXApple GPU131 ms+41 pts
SemIf 4BApple GPU171 ms+43 pts
JevK5 4BApple GPU181 ms+44 pts
Winnow E4BApple GPU184 ms+47 pts
decider 0.8BCPU279 ms+36 pts
Kev 0.8B · OllayaCPU286 ms+29 pts
NLI zero-shot (DeBERTa)CPU366 ms+35 pts
ruling 35B-A3BApple GPU391 ms+47 pts
Decision 1.0 EosCPU391 ms+36 pts
decider 2BCPU614 ms+40 pts
Kev 4B · OllayaCPU1.8 s+41 pts
Kev 9B · OllayaCPU3.2 s+42 pts