[{"data":1,"prerenderedAt":297},["ShallowReactive",2],{"doc-\u002Fdocs\u002Flocal-benchmarks":3},{"id":4,"title":5,"body":6,"description":16,"extension":290,"meta":291,"navigation":292,"path":293,"seo":294,"stem":295,"__hash__":296},"content\u002Fdocs\u002Flocal-benchmarks.md","Latency and resource measurement",{"type":7,"value":8,"toc":285},"minimark",[9,13,17,20,27,30,41,46,61,89,96,106,110,117,201,204,211,214,217,221,229,232,235,275,281],[10,11,5],"h1",{"id":12},"latency-and-resource-measurement",[14,15,16],"p",{},"The benchmark runs the actual exported, merged checkpoint and the pinned Laya multilingual\nand Laya Vision checkpoints on one RX 7900 XTX (upstream) or the recorded\nRTX 4090 run (Linnaeus). Each engine receives\nthe same state and questions. Tokenizers, model sizes, and internal templates\ndiffer; this is an API workload comparison.",[14,18,19],{},"The local protocol uses BF16, eight CPU threads, three warm-up calls, and\n20 synchronized repetitions. End-to-end timing includes preprocessing, transfers,\nGPU execution, and response assembly. It excludes network and request queues.\nReport median, linearly interpolated p95, raw durations, decisions per second,\nand amortized latency per decision.",[14,21,22,26],{},[23,24,25],"code",{},"scripts\u002Fbenchmark.py"," uses one fixed suite: Laya Vision's one\u002Fthree-question\ntext and image cases, plus 1, 5, 10 and 50 distinct questions over a shared\nsupport state. Each reference uses its checkpoint's published preprocessing.",[14,28,29],{},"PyTorch allocated, reserved, and peak memory are recorded separately from board\nVRAM. Board VRAM includes the runtime and desktop. Power, utilization, temperature,\nand process RSS accompany timing samples. Shared-prefix execution does not imply\nconstant latency as batch size grows or predict performance under queued load.",[14,31,32,33,36,37,40],{},"The release workflow writes timing samples to ",[23,34,35],{},"runs\u002Fv1\u002Fbenchmarks\u002F"," and\nplotting data to ",[23,38,39],{},"runs\u002Fv1\u002Fmetrics\u002Flatency-samples.csv",". Quality, resource use,\nand latency are reported separately. Weight-preserving changes still require\nheld-out quality checks because BF16 arithmetic need not be bitwise identical.",[42,43,45],"h2",{"id":44},"jevbench","JevBench",[14,47,48,49,56,57,60],{},"The release workflow also evaluates the exported checkpoint with the official\n",[50,51,55],"a",{"href":52,"rel":53},"https:\u002F\u002Fgithub.com\u002Ffstandhartinger\u002Fjevbench\u002Ftree\u002Fv1.2.2",[54],"nofollow","JevBench v1.2.2 harness",",\npinned at ",[23,58,59],{},"e105a48f8cdb7f3babb3594424f73e5d7bdc97b9",". Its public tasks contain\n48 easy, 72 standard, and 111 hard decisions. The other 303 decisions in the\n534-item leaderboard are not distributed, including the entire judge tier.",[62,63,68],"pre",{"className":64,"code":65,"language":66,"meta":67,"style":67},"language-bash shiki shiki-themes github-light github-dark","uv run python scripts\u002Frun_jevbench.py\n","bash","",[23,69,70],{"__ignoreMap":67},[71,72,75,79,83,86],"span",{"class":73,"line":74},"line",1,[71,76,78],{"class":77},"sScJk","uv",[71,80,82],{"class":81},"sZZnC"," run",[71,84,85],{"class":81}," python",[71,87,88],{"class":81}," scripts\u002Frun_jevbench.py\n",[14,90,91,92,95],{},"Each task receives one serial ",[23,93,94],{},"Predictor.predict"," call with its original state,\ninstructions, criteria, and candidate order. Returned probabilities are native\nmodel distributions. The official runner stores requests, responses, failures,\nlatencies, and scores. Model loading is timed separately; the first decision is\nretained and reported. The adapter's 4,096-token inference limit applies: rejected inputs\ncount as wrong and are not silently truncated or removed from the denominator.\nThe benchmark never changes weights, temperatures, or the training recipe.",[14,97,98,101,102,105],{},[23,99,100],{},"runs\u002Fv1\u002Fjevbench\u002F"," contains raw evidence, a JSON summary, a Markdown report,\nand ",[23,103,104],{},"same-public-ids.csv",". Reference accuracy is recomputed from upstream's\npublished outcomes on the same 231 IDs. The report distinguishes this partial\nevaluation from the full leaderboard. It records raw local latency and the\nupstream assumed latency adjustment separately. Local execution has no provider\ntariff, so cost and the four-axis composite remain unmeasured.",[42,107,109],{"id":108},"laya-comparison-chart-protocols","Laya comparison-chart protocols",[14,111,112,113,116],{},"The official Laya research branch at ",[23,114,115],{},"28d43add7e47ce502489c9433310d55276c64e0f","\ncontains the builders for the application, multilingual, calibration, and latency\nmeasurements in its comparison chart. Run them on the exported Linnaeus checkpoint:",[62,118,120],{"className":64,"code":119,"language":66,"meta":67,"style":67},"git clone --branch research https:\u002F\u002Fgithub.com\u002FNandhaKishorM\u002Flaya.git .cache\u002Fupstream\u002Flaya-research\ngit -C .cache\u002Fupstream\u002Flaya-research checkout --detach 28d43add7e47ce502489c9433310d55276c64e0f\nuv sync --extra train\nuv run python scripts\u002Fprepare_laya_benchmark.py\nuv run python scripts\u002Frun_laya_benchmark.py\n",[23,121,122,143,163,177,189],{"__ignoreMap":67},[71,123,124,127,130,134,137,140],{"class":73,"line":74},[71,125,126],{"class":77},"git",[71,128,129],{"class":81}," clone",[71,131,133],{"class":132},"sj4cs"," --branch",[71,135,136],{"class":81}," research",[71,138,139],{"class":81}," https:\u002F\u002Fgithub.com\u002FNandhaKishorM\u002Flaya.git",[71,141,142],{"class":81}," .cache\u002Fupstream\u002Flaya-research\n",[71,144,146,148,151,154,157,160],{"class":73,"line":145},2,[71,147,126],{"class":77},[71,149,150],{"class":132}," -C",[71,152,153],{"class":81}," .cache\u002Fupstream\u002Flaya-research",[71,155,156],{"class":81}," checkout",[71,158,159],{"class":132}," --detach",[71,161,162],{"class":81}," 28d43add7e47ce502489c9433310d55276c64e0f\n",[71,164,166,168,171,174],{"class":73,"line":165},3,[71,167,78],{"class":77},[71,169,170],{"class":81}," sync",[71,172,173],{"class":132}," --extra",[71,175,176],{"class":81}," train\n",[71,178,180,182,184,186],{"class":73,"line":179},4,[71,181,78],{"class":77},[71,183,82],{"class":81},[71,185,85],{"class":81},[71,187,188],{"class":81}," scripts\u002Fprepare_laya_benchmark.py\n",[71,190,192,194,196,198],{"class":73,"line":191},5,[71,193,78],{"class":77},[71,195,82],{"class":81},[71,197,85],{"class":81},[71,199,200],{"class":81}," scripts\u002Frun_laya_benchmark.py\n",[14,202,203],{},"Preparation preserves the upstream builders, seed 13, candidate order, and\ndocument shortening, and records dataset revisions and file hashes. It requires\nall ten application suites, all fifty Colab suites, and all 51 MASSIVE languages\nto load successfully. The fixed questions are stored before model evaluation.",[14,205,206,207,210],{},"Results live in ",[23,208,209],{},"runs\u002Fv1\u002Flaya-chart\u002F",": raw logits and probabilities,\nper-suite CSV, all language scores, calibration diagnostics, option permutations,\nand end-to-end timing samples. Temperature refitting follows the upstream\nper-suite first-half\u002Fsecond-half protocol and never updates deployment temperatures.\nThe 51-language sweep uses 100 cases per language and 20 candidates; the Colab\nlanguage suites use 300 cases per language. These differ from the main mixture's\nfull-label-space evaluation.",[14,212,213],{},"Published Laya reference columns are historical measurements, not newly run\ncheckpoints. Their original dataset revisions and per-example hashes are absent,\nso reproducing the builders does not independently establish historical byte\nidentity. Calibration comparisons use the shared 49 suites because the upstream\nresult omits BANKING77. Typed-decisions soft accuracy is the inner product of\npredicted and teacher distributions, following this benchmark's definition.",[14,215,216],{},"Local latency used the RX 7900 XTX upstream; Linnaeus measurements ran on the RTX 4090. Mixed-language measurements\nuse the upstream English\u002FHindi request stream and one resident Linnaeus model; no\ncheckpoint routing or swapping is added. Local inference has a compute cost even\nwhen no API fee is paid.",[42,218,220],{"id":219},"model-card-figures","Model card figures",[14,222,223,224,228],{},"The ",[50,225,227],{"href":226},"figures\u002FREADME","comparison gallery"," contains the selected model's figures\nand chart values. To regenerate them, use the saved evaluation results below.",[14,230,231],{},"Use the saved results to render six comparisons: JevBench accuracy, application\naccuracy, multilingual accuracy, vision accuracy and calibration, local latency,\nand paired JevBench outcomes. The reference checkpoints are Laya multilingual\nand Laya Vision. The English Laya entry in JevBench's published results is a\ndifferent checkpoint and must not be relabeled as Laya multilingual.",[14,233,234],{},"Install the locked plotting group, then render the completed run. This group\ndoes not require model dependencies:",[62,236,238],{"className":64,"code":237,"language":66,"meta":67,"style":67},"uv sync --no-default-groups --group plot --no-install-project\nuv run python scripts\u002Fplot_model_card.py --run runs\u002Fv1\n",[23,239,240,258],{"__ignoreMap":67},[71,241,242,244,246,249,252,255],{"class":73,"line":74},[71,243,78],{"class":77},[71,245,170],{"class":81},[71,247,248],{"class":132}," --no-default-groups",[71,250,251],{"class":132}," --group",[71,253,254],{"class":81}," plot",[71,256,257],{"class":132}," --no-install-project\n",[71,259,260,262,264,266,269,272],{"class":73,"line":145},[71,261,78],{"class":77},[71,263,82],{"class":81},[71,265,85],{"class":81},[71,267,268],{"class":81}," scripts\u002Fplot_model_card.py",[71,270,271],{"class":132}," --run",[71,273,274],{"class":81}," runs\u002Fv1\n",[14,276,277,280],{},[23,278,279],{},"runs\u002Fv1\u002Ffigures\u002FREADME.md"," indexes the PNG, SVG and PDF figures. The directory\nalso contains a combined PDF, chart values in CSV, and a manifest of source\nchecksums. Captions identify the selected checkpoint, dataset coverage,\nmeasurement conditions, and whether references were measured locally or taken\nfrom published results. Model identity and the shared project version come from the\nselected checkpoint metadata.",[282,283,284],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}",{"title":67,"searchDepth":145,"depth":165,"links":286},[287,288,289],{"id":44,"depth":145,"text":45},{"id":108,"depth":145,"text":109},{"id":219,"depth":145,"text":220},"md",{},true,"\u002Fdocs\u002Flocal-benchmarks",{"title":5,"description":16},"docs\u002Flocal-benchmarks","5TFHeqceSWGgJoOIt3YAGu-b_ce4HQPBqNErxOcWG3c",1790560641965]