[{"data":1,"prerenderedAt":865},["ShallowReactive",2],{"doc-\u002Fdocs\u002Finference":3},{"id":4,"title":5,"body":6,"description":858,"extension":859,"meta":860,"navigation":143,"path":861,"seo":862,"stem":863,"__hash__":864},"content\u002Fdocs\u002Finference.md","Installation and inference",{"type":7,"value":8,"toc":852},"minimark",[9,13,30,93,112,117,120,127,150,153,172,176,179,303,306,387,413,427,431,438,477,480,484,511,617,650,653,677,680,821,848],[10,11,5],"h1",{"id":12},"installation-and-inference",[14,15,16,17,24,25,29],"p",{},"The Linnaeus runtime uses Python 3.12 and PyTorch 2.9.1 with CUDA 12.8 on an\nNVIDIA GPU (Ampere or newer for BF16). Upstream measured ROCm 6.4 on an\nAMD Radeon RX 7900 XTX; this release was trained on a single RTX 4090 24GB (OpenBayes). Install ",[18,19,23],"a",{"href":20,"rel":21},"https:\u002F\u002Fmise.jdx.dev\u002F",[22],"nofollow","mise"," (it provisions\nPython 3.12 and uv from ",[26,27,28],"code",{},"mise.toml","), then install from source:",[31,32,37],"pre",{"className":33,"code":34,"language":35,"meta":36,"style":36},"language-bash shiki shiki-themes github-light github-dark","git clone \u003Clinnaeus-remote>\ncd Linnaeus\nmise install\nuv sync\n","bash","",[26,38,39,66,76,84],{"__ignoreMap":36},[40,41,44,48,52,56,59,63],"span",{"class":42,"line":43},"line",1,[40,45,47],{"class":46},"sScJk","git",[40,49,51],{"class":50},"sZZnC"," clone",[40,53,55],{"class":54},"szBVR"," \u003C",[40,57,58],{"class":50},"linnaeus-remot",[40,60,62],{"class":61},"sVt8B","e",[40,64,65],{"class":54},">\n",[40,67,69,73],{"class":42,"line":68},2,[40,70,72],{"class":71},"sj4cs","cd",[40,74,75],{"class":50}," Linnaeus\n",[40,77,79,81],{"class":42,"line":78},3,[40,80,23],{"class":46},[40,82,83],{"class":50}," install\n",[40,85,87,90],{"class":42,"line":86},4,[40,88,89],{"class":46},"uv",[40,91,92],{"class":50}," sync\n",[14,94,95,96,99,100,103,104,107,108,111],{},"The lock file targets Python 3.12 on Linux (",[26,97,98],{},"required-environments","). The\n",[26,101,102],{},"tool.uv.sources"," entries in ",[26,105,106],{},"pyproject.toml"," bind PyTorch and torchvision to\nthe explicit CUDA 12.8 index; Triton resolves from PyPI.\nSee ",[18,109,110],{"href":110},"development"," for dependency groups, checks, and updating\nthe lock.",[113,114,116],"h2",{"id":115},"load-a-checkpoint","Load a checkpoint",[14,118,119],{},"Load the model by its Hugging Face repository ID. The loader downloads the\ndecision weights and the exact Qwen3.5-2B revision recorded in the checkpoint.\nDownloads use the Hugging Face cache and are reused by later calls. Both\nrepositories are public; downloading them does not require a Hugging Face login.",[14,121,122,123,126],{},"Run Python examples with ",[26,124,125],{},"uv run python"," from the repository root:",[31,128,132],{"className":129,"code":130,"language":131,"meta":36,"style":36},"language-python shiki shiki-themes github-light github-dark","from linnaeus.predictor import Predictor\n\nmodel = Predictor.from_checkpoint(\"pi-dal\u002FLinnaeus-0.1.0-2B\")\n","python",[26,133,134,139,145],{"__ignoreMap":36},[40,135,136],{"class":42,"line":43},[40,137,138],{},"from linnaeus.predictor import Predictor\n",[40,140,141],{"class":42,"line":68},[40,142,144],{"emptyLinePlaceholder":143},true,"\n",[40,146,147],{"class":42,"line":78},[40,148,149],{},"model = Predictor.from_checkpoint(\"pi-dal\u002FLinnaeus-0.1.0-2B\")\n",[14,151,152],{},"The loader verifies the checkpoint's weight checksum and base revision, merges\nLoRA, and applies its saved calibration temperatures. The checkpoint contains\nthe decision head as well as LoRA; load it through Linnaeus. It is not a standalone\nTransformers language model or a standard PEFT adapter export.",[14,154,155,156,159,160,163,164,167,168,171],{},"The same method accepts a local checkpoint directory. An optional ",[26,157,158],{},"revision","\npins a Hub commit, and ",[26,161,162],{},"base_model"," supplies an existing local base with its\nrecorded ",[26,165,166],{},"revision.txt",". ",[26,169,170],{},"model.metadata"," exposes the loaded model identity,\nweight checksum, and calibration. Weights are not bundled with the Python package.",[113,173,175],{"id":174},"ask-questions","Ask questions",[14,177,178],{},"Pass a shared state and a mapping of question names to question definitions:",[31,180,182],{"className":129,"code":181,"language":131,"meta":36,"style":36},"result = model.predict(\n    {\"message\": \"The replacement arrived, but it is broken too.\"},\n    {\n        \"route\": {\n            \"type\": \"choice\",\n            \"instructions\": \"Which team should handle this?\",\n            \"criteria\": {\"support\": \"Product issues\", \"billing\": \"Payment issues\"},\n        },\n        \"repeat_issue\": {\n            \"type\": \"noul\",\n            \"instructions\": \"Has the customer experienced this problem before?\",\n        },\n        \"urgency\": {\n            \"type\": \"score\",\n            \"instructions\": \"How urgently does this request need attention?\",\n            \"criteria\": [\"routine\", \"soon\", \"immediate\"],\n        },\n    },\n)\n\nanswers = result[\"answers\"]\n",[26,183,184,189,194,199,204,210,216,222,228,234,240,246,251,257,263,269,275,280,286,292,297],{"__ignoreMap":36},[40,185,186],{"class":42,"line":43},[40,187,188],{},"result = model.predict(\n",[40,190,191],{"class":42,"line":68},[40,192,193],{},"    {\"message\": \"The replacement arrived, but it is broken too.\"},\n",[40,195,196],{"class":42,"line":78},[40,197,198],{},"    {\n",[40,200,201],{"class":42,"line":86},[40,202,203],{},"        \"route\": {\n",[40,205,207],{"class":42,"line":206},5,[40,208,209],{},"            \"type\": \"choice\",\n",[40,211,213],{"class":42,"line":212},6,[40,214,215],{},"            \"instructions\": \"Which team should handle this?\",\n",[40,217,219],{"class":42,"line":218},7,[40,220,221],{},"            \"criteria\": {\"support\": \"Product issues\", \"billing\": \"Payment issues\"},\n",[40,223,225],{"class":42,"line":224},8,[40,226,227],{},"        },\n",[40,229,231],{"class":42,"line":230},9,[40,232,233],{},"        \"repeat_issue\": {\n",[40,235,237],{"class":42,"line":236},10,[40,238,239],{},"            \"type\": \"noul\",\n",[40,241,243],{"class":42,"line":242},11,[40,244,245],{},"            \"instructions\": \"Has the customer experienced this problem before?\",\n",[40,247,249],{"class":42,"line":248},12,[40,250,227],{},[40,252,254],{"class":42,"line":253},13,[40,255,256],{},"        \"urgency\": {\n",[40,258,260],{"class":42,"line":259},14,[40,261,262],{},"            \"type\": \"score\",\n",[40,264,266],{"class":42,"line":265},15,[40,267,268],{},"            \"instructions\": \"How urgently does this request need attention?\",\n",[40,270,272],{"class":42,"line":271},16,[40,273,274],{},"            \"criteria\": [\"routine\", \"soon\", \"immediate\"],\n",[40,276,278],{"class":42,"line":277},17,[40,279,227],{},[40,281,283],{"class":42,"line":282},18,[40,284,285],{},"    },\n",[40,287,289],{"class":42,"line":288},19,[40,290,291],{},")\n",[40,293,295],{"class":42,"line":294},20,[40,296,144],{"emptyLinePlaceholder":143},[40,298,300],{"class":42,"line":299},21,[40,301,302],{},"answers = result[\"answers\"]\n",[14,304,305],{},"Questions are independent: each sees the state and its own instructions and\ncandidates. A question cannot refer to another question's answer in the same call.",[307,308,309,325],"table",{},[310,311,312],"thead",{},[313,314,315,319,322],"tr",{},[316,317,318],"th",{},"Type",[316,320,321],{},"Criteria",[316,323,324],{},"Answer fields",[326,327,328,348,369],"tbody",{},[313,329,330,336,339],{},[331,332,333],"td",{},[26,334,335],{},"choice",[331,337,338],{},"Candidate list, or label-to-description mapping",[331,340,341,343,344,347],{},[26,342,335],{}," and ",[26,345,346],{},"probabilities",", keyed by candidate label",[313,349,350,355,364],{},[331,351,352],{},[26,353,354],{},"noul",[331,356,357,358,343,361],{},"Optional descriptions under ",[26,359,360],{},"false",[26,362,363],{},"true",[331,365,366,368],{},[26,367,354],{},", the estimated probability that the statement is true",[313,370,371,376,379],{},[331,372,373],{},[26,374,375],{},"score",[331,377,378],{},"Level descriptions in ascending order",[331,380,381,383,384,386],{},[26,382,375],{},", the expected zero-based level, and ",[26,385,346],{},", keyed by level index",[14,388,389,390,343,393,396,397,343,399,401,402,405,406,408,409,412],{},"Each answer also includes ",[26,391,392],{},"type",[26,394,395],{},"confidence",". For ",[26,398,335],{},[26,400,375],{},",\nconfidence is normalized inverse entropy, ",[26,403,404],{},"1 - H(p) \u002F log(K)","; for ",[26,407,354],{},", it is\n",[26,410,411],{},"max(p, 1 - p)",". This describes how concentrated the distribution is. It is not\nan empirical probability that the decision is correct.",[14,414,415,416,343,419,422,423,426],{},"The response includes ",[26,417,418],{},"model",[26,420,421],{},"usage"," alongside ",[26,424,425],{},"answers",". Usage records\ninput tokens and image count.",[113,428,430],{"id":429},"include-an-image","Include an image",[14,432,433,434,437],{},"Supply one decoded PIL image under the state's ",[26,435,436],{},"image"," key:",[31,439,441],{"className":129,"code":440,"language":131,"meta":36,"style":36},"from PIL import Image\n\nwith Image.open(\"parcel.jpg\") as image:\n    result = model.predict(\n        {\"image\": image.convert(\"RGB\")},\n        {\"damage\": {\"type\": \"noul\", \"instructions\": \"Is the packaging damaged?\"}},\n    )\n",[26,442,443,448,452,457,462,467,472],{"__ignoreMap":36},[40,444,445],{"class":42,"line":43},[40,446,447],{},"from PIL import Image\n",[40,449,450],{"class":42,"line":68},[40,451,144],{"emptyLinePlaceholder":143},[40,453,454],{"class":42,"line":78},[40,455,456],{},"with Image.open(\"parcel.jpg\") as image:\n",[40,458,459],{"class":42,"line":86},[40,460,461],{},"    result = model.predict(\n",[40,463,464],{"class":42,"line":206},[40,465,466],{},"        {\"image\": image.convert(\"RGB\")},\n",[40,468,469],{"class":42,"line":212},[40,470,471],{},"        {\"damage\": {\"type\": \"noul\", \"instructions\": \"Is the packaging damaged?\"}},\n",[40,473,474],{"class":42,"line":218},[40,475,476],{},"    )\n",[14,478,479],{},"Each question supports 2–128 candidates and a 4,096-token input budget,\nincluding state, instructions, candidates, and image tokens. Inputs that exceed\nthe budget are rejected; shorten the state or candidate descriptions and retry.\nImage features and the shared input prefix are reused where possible. More\nquestions still require more computation.",[113,481,483],{"id":482},"apple-silicon-mlx-on-device","Apple Silicon (MLX \u002F on-device)",[14,485,486,487,492,493,498,499,502,503,506,507,510],{},"The published MLX builds run the model natively on macOS and iOS via\n",[18,488,491],{"href":489,"rel":490},"https:\u002F\u002Fgithub.com\u002Fml-explore\u002Fmlx-lm",[22],"mlx-lm"," \u002F\n",[18,494,497],{"href":495,"rel":496},"https:\u002F\u002Fgithub.com\u002Fml-explore\u002Fmlx-swift-lm",[22],"mlx-swift-lm",".\nThe ",[26,500,501],{},"-MLX-*"," builds are text-only; the ",[26,504,505],{},"-MLX-VLM-*"," builds (converted via\nmlx-vlm) also keep the vision tower for ",[26,508,509],{},"state['image']"," decisions:",[307,512,513,529],{},[310,514,515],{},[313,516,517,520,523,526],{},[316,518,519],{},"Artifact",[316,521,522],{},"Size",[316,524,525],{},"JevBench v1.2.2 (231 tasks)",[316,527,528],{},"Use",[326,530,531,547,563,579,599],{},[313,532,533,538,541,544],{},[331,534,535],{},[26,536,537],{},"pi-dal\u002FLinnaeus-0.1.0-2B-merged",[331,539,540],{},"4.3 GB",[331,542,543],{},"71.0% (torch MPS)",[331,545,546],{},"Mac dev, conversion source",[313,548,549,554,557,560],{},[331,550,551],{},[26,552,553],{},"pi-dal\u002FLinnaeus-0.1.0-2B-MLX-8bit",[331,555,556],{},"1.9 GB",[331,558,559],{},"70.56%",[331,561,562],{},"Mac \u002F iPhone, text quality pick",[313,564,565,570,573,576],{},[331,566,567],{},[26,568,569],{},"pi-dal\u002FLinnaeus-0.1.0-2B-MLX-4bit",[331,571,572],{},"1.0 GB",[331,574,575],{},"67.53%",[331,577,578],{},"iPhone, text size pick",[313,580,581,586,589,596],{},[331,582,583],{},[26,584,585],{},"pi-dal\u002FLinnaeus-0.1.0-2B-MLX-VLM-8bit",[331,587,588],{},"2.5 GB",[331,590,591,592],{},"70.56% text + ",[593,594,595],"strong",{},"images",[331,597,598],{},"Mac \u002F iPhone multimodal",[313,600,601,606,609,614],{},[331,602,603],{},[26,604,605],{},"pi-dal\u002FLinnaeus-0.1.0-2B-MLX-VLM-4bit",[331,607,608],{},"1.6 GB",[331,610,611,612],{},"~67% text + ",[593,613,595],{},[331,615,616],{},"iPhone multimodal, size pick",[14,618,619,620,623,624,627,628,343,631,634,635,638,639,642,643,645,646,649],{},"The VLM builds are converted with ",[26,621,622],{},"mlx-vlm convert"," (vision tower stays\nbf16; only the language stack is quantized). A post-conversion step must\nwrite ",[593,625,626],{},"both"," ",[26,629,630],{},"quantization",[26,632,633],{},"quantization_config"," into ",[26,636,637],{},"config.json","\n({\"group_size\": 64, \"bits\": N, \"mode\": \"affine\"}) — mlx-vlm 0.7.1 omits\nthem, ",[26,640,641],{},"load_weights"," then rejects the quantized tensors, and Hugging Face\nmisclassifies the repo as a finetune instead of a quantization. Pass a\ndecoded PIL image via ",[26,644,509],{}," to the same ",[26,647,648],{},"MlxPredictor.predict()","\ncontract.",[14,651,652],{},"CUDA reference on the same tasks is 73.16%; the residual gap is the fla\nchunked delta-rule kernel vs. reference implementations, concentrated on\nhard-tier borderline tasks (easy\u002Fstandard are identical). Upstream's\npublished result on the same benchmark is 65.80%.",[31,654,656],{"className":129,"code":655,"language":131,"meta":36,"style":36},"from linnaeus.mlx_predictor import MlxPredictor\n\npredictor = MlxPredictor(\"runs\u002F2b\u002Fexports\u002Flinnaeus-2b-8bit\")\nresult = predictor.predict(state, questions)  # same contract as Predictor\n",[26,657,658,663,667,672],{"__ignoreMap":36},[40,659,660],{"class":42,"line":43},[40,661,662],{},"from linnaeus.mlx_predictor import MlxPredictor\n",[40,664,665],{"class":42,"line":68},[40,666,144],{"emptyLinePlaceholder":143},[40,668,669],{"class":42,"line":78},[40,670,671],{},"predictor = MlxPredictor(\"runs\u002F2b\u002Fexports\u002Flinnaeus-2b-8bit\")\n",[40,673,674],{"class":42,"line":86},[40,675,676],{},"result = predictor.predict(state, questions)  # same contract as Predictor\n",[14,678,679],{},"To reproduce the exports:",[31,681,683],{"className":33,"code":682,"language":35,"meta":36,"style":36},"python scripts\u002Fexport_merged_hf.py --base \u003CQwen3.5-2B snapshot> \\\n    --adapter runs\u002F2b\u002Fcheckpoint --out runs\u002F2b\u002Fexports\u002Fmerged-hf\n# text-only build\nuvx --from mlx-lm python -m mlx_lm convert \\\n    --hf-path runs\u002F2b\u002Fexports\u002Fmerged-hf \\\n    --mlx-path runs\u002F2b\u002Fexports\u002Flinnaeus-2b-8bit -q --q-bits 8\n# multimodal build (keeps vision tower)\npython -m mlx_vlm convert --hf-path runs\u002F2b\u002Fexports\u002Fmerged-hf \\\n    --mlx-path runs\u002F2b\u002Fexports\u002Flinnaeus-2b-vlm-8bit --quantize --q-bits 8\n",[26,684,685,712,726,732,757,767,784,789,807],{"__ignoreMap":36},[40,686,687,689,692,695,697,700,703,706,709],{"class":42,"line":43},[40,688,131],{"class":46},[40,690,691],{"class":50}," scripts\u002Fexport_merged_hf.py",[40,693,694],{"class":71}," --base",[40,696,55],{"class":54},[40,698,699],{"class":50},"Qwen3.5-2B",[40,701,702],{"class":50}," snapsho",[40,704,705],{"class":61},"t",[40,707,708],{"class":54},">",[40,710,711],{"class":71}," \\\n",[40,713,714,717,720,723],{"class":42,"line":68},[40,715,716],{"class":71},"    --adapter",[40,718,719],{"class":50}," runs\u002F2b\u002Fcheckpoint",[40,721,722],{"class":71}," --out",[40,724,725],{"class":50}," runs\u002F2b\u002Fexports\u002Fmerged-hf\n",[40,727,728],{"class":42,"line":78},[40,729,731],{"class":730},"sJ8bj","# text-only build\n",[40,733,734,737,740,743,746,749,752,755],{"class":42,"line":86},[40,735,736],{"class":46},"uvx",[40,738,739],{"class":71}," --from",[40,741,742],{"class":50}," mlx-lm",[40,744,745],{"class":50}," python",[40,747,748],{"class":71}," -m",[40,750,751],{"class":50}," mlx_lm",[40,753,754],{"class":50}," convert",[40,756,711],{"class":71},[40,758,759,762,765],{"class":42,"line":206},[40,760,761],{"class":71},"    --hf-path",[40,763,764],{"class":50}," runs\u002F2b\u002Fexports\u002Fmerged-hf",[40,766,711],{"class":71},[40,768,769,772,775,778,781],{"class":42,"line":212},[40,770,771],{"class":71},"    --mlx-path",[40,773,774],{"class":50}," runs\u002F2b\u002Fexports\u002Flinnaeus-2b-8bit",[40,776,777],{"class":71}," -q",[40,779,780],{"class":71}," --q-bits",[40,782,783],{"class":71}," 8\n",[40,785,786],{"class":42,"line":218},[40,787,788],{"class":730},"# multimodal build (keeps vision tower)\n",[40,790,791,793,795,798,800,803,805],{"class":42,"line":224},[40,792,131],{"class":46},[40,794,748],{"class":71},[40,796,797],{"class":50}," mlx_vlm",[40,799,754],{"class":50},[40,801,802],{"class":71}," --hf-path",[40,804,764],{"class":50},[40,806,711],{"class":71},[40,808,809,811,814,817,819],{"class":42,"line":230},[40,810,771],{"class":71},[40,812,813],{"class":50}," runs\u002F2b\u002Fexports\u002Flinnaeus-2b-vlm-8bit",[40,815,816],{"class":71}," --quantize",[40,818,780],{"class":71},[40,820,783],{"class":71},[14,822,823,824,827,828,831,832,835,836,839,840,843,844,847],{},"The merged checkpoint embeds the scalar decision head as an extra\n",[26,825,826],{},"embed_tokens","\u002F",[26,829,830],{},"lm_head"," row (",[26,833,834],{},"score_row_id","), so a stock LM forward is all a\nruntime needs: read ",[26,837,838],{},"logits[marker_pos, score_row_id]"," at each\n",[26,841,842],{},"\u003C|fim_suffix|>"," marker, apply the per-type temperature, softmax across a\nquestion's candidates. ",[26,845,846],{},"linnaeus-runtime.json"," carries that contract to\nnon-Python runtimes (e.g. a ~100-line Swift shim for iOS).",[849,850,851],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}",{"title":36,"searchDepth":68,"depth":78,"links":853},[854,855,856,857],{"id":115,"depth":68,"text":116},{"id":174,"depth":68,"text":175},{"id":429,"depth":68,"text":430},{"id":482,"depth":68,"text":483},"The Linnaeus runtime uses Python 3.12 and PyTorch 2.9.1 with CUDA 12.8 on an\nNVIDIA GPU (Ampere or newer for BF16). Upstream measured ROCm 6.4 on an\nAMD Radeon RX 7900 XTX; this release was trained on a single RTX 4090 24GB (OpenBayes). Install mise (it provisions\nPython 3.12 and uv from mise.toml), then install from source:","md",{},"\u002Fdocs\u002Finference",{"title":5,"description":858},"docs\u002Finference","Q90SqgxlntRfDW4iquz51ASITbTymXTu-WBfsAQUDjM",1790560641964]