[{"data":1,"prerenderedAt":522},["ShallowReactive",2],{"doc-\u002Fdocs\u002Fdata-and-evaluation":3},{"id":4,"title":5,"body":6,"description":16,"extension":515,"meta":516,"navigation":517,"path":518,"seo":519,"stem":520,"__hash__":521},"content\u002Fdocs\u002Fdata-and-evaluation.md","Data and evaluation",{"type":7,"value":8,"toc":502},"minimark",[9,13,17,22,190,193,198,201,204,218,221,224,228,245,248,252,255,275,278,282,285,293,297,437,440,444,458,461,481,484],[10,11,5],"h1",{"id":12},"data-and-evaluation",[14,15,16],"p",{},"Use public labels and programmatic scoring. The training mixture includes\npublic teacher-derived soft labels from typed-decisions; no new teacher calls\nor LLM judges are required. Deterministic conversion means reproducible examples\nand scores, not infallible annotations or known per-example true probabilities.",[18,19,21],"h2",{"id":20},"training-sources","Training sources",[23,24,25,41],"table",{},[26,27,28],"thead",{},[29,30,31,35,38],"tr",{},[32,33,34],"th",{},"Source",[32,36,37],{},"Decision tasks",[32,39,40],{},"Protocol",[42,43,44,66,84,107,130,142,154,167,179],"tbody",{},[29,45,46,56,63],{},[47,48,49],"td",{},[50,51,55],"a",{"href":52,"rel":53},"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002FAmazonScience\u002Fmassive",[54],"nofollow","MASSIVE 1.1",[47,57,58,59],{},"English\u002FChinese ",[60,61,62],"code",{},"choice",[47,64,65],{},"All 60 intents; fixed human-readable label mapping",[29,67,68,75,81],{},[47,69,70],{},[50,71,74],{"href":72,"rel":73},"https:\u002F\u002Fgithub.com\u002Fgoogle-research-datasets\u002Fboolean-questions",[54],"BoolQ",[47,76,77,78],{},"Text ",[60,79,80],{},"noul",[47,82,83],{},"Original passage, question, and Boolean answer",[29,85,86,93,104],{},[47,87,88],{},[50,89,92],{"href":90,"rel":91},"https:\u002F\u002Fcs.stanford.edu\u002Fpeople\u002Fjcjohns\u002Fclevr\u002F",[54],"CLEVR",[47,94,95,96,98,99,98,101],{},"Visual ",[60,97,62],{},", ",[60,100,80],{},[60,102,103],{},"score",[47,105,106],{},"Attributes, existence, and ordered counts",[29,108,109,116,124],{},[47,110,111],{},[50,112,115],{"href":113,"rel":114},"https:\u002F\u002Fgithub.com\u002Fgoogle-research-datasets\u002Fscreen_qa",[54],"ScreenQA",[47,117,118,119,121,122],{},"Screenshot ",[60,120,62],{}," and candidate-level ",[60,123,80],{},[47,125,126,127],{},"Original human annotations in ",[60,128,129],{},"answers_and_bboxes",[29,131,132,135,139],{},[47,133,134],{},"AG News \u002F emotion \u002F BANKING77",[47,136,77,137],{},[60,138,62],{},[47,140,141],{},"Full label vocabularies and official labeled test sets",[29,143,144,147,151],{},[47,145,146],{},"XNLI",[47,148,58,149],{},[60,150,62],{},[47,152,153],{},"Three labels; aligned translations and duplicate premises stay grouped",[29,155,156,159,164],{},[47,157,158],{},"A-OKVQA \u002F ScienceQA",[47,160,161,162],{},"Image ",[60,163,62],{},[47,165,166],{},"Four-choice A-OKVQA; image-only ScienceQA subset with official test retained",[29,168,169,172,176],{},[47,170,171],{},"VQAv2 yes\u002Fno",[47,173,161,174],{},[60,175,80],{},[47,177,178],{},"Soft answer votes; image-grouped split of labeled official validation",[29,180,181,184,187],{},[47,182,183],{},"typed-decisions",[47,185,186],{},"All three decision types",[47,188,189],{},"Public soft teacher distributions; state-grouped partitions",[14,191,192],{},"CLEVR scene graphs and programs verify labels but never enter model inputs.\nUse complete answer domains per question family rather than answer-dependent\ncandidate sampling. Count performance does not establish subjective scoring\nability. English visual benchmarks do not establish Chinese visual competence.",[194,195,197],"h3",{"id":196},"screenqa-conversion","ScreenQA conversion",[14,199,200],{},"Build candidate regions from the screenshot's Rico View Hierarchy using a fixed,\nanswer-independent rule. Present region IDs and geometry consistently; do not\nfeed annotator descriptions or ground-truth answers into the input.",[14,202,203],{},"Retain questions whose annotations agree on one answer element that maps\nunambiguously to a candidate. Record exclusion counts for multi-element answers,\ndisagreement, missing assets, and unmapped regions. Do not inject a target region\nto rescue an otherwise invalid example.",[205,206,207,213],"ul",{},[208,209,210,212],"li",{},[60,211,62],{},": select the matching region from the candidate set.",[208,214,215,217],{},[60,216,80],{},": determine whether a specified candidate is the answer region.",[14,219,220],{},"A negative candidate is not evidence that the answer is absent from the screen.\nPublish the negative-sampling rule and class balance; calibration is specific to\nthat distribution. Report these as derived decision tasks, not official ScreenQA\nquestion-answering scores. The inference contract requires candidate regions;\nautomatic UI detection is outside scope.",[14,222,223],{},"ScreenQA Short and ComplexQA are excluded from the data mix. The former\ncontains model-produced answers, and the latter uses model-generated questions\nand answers with human validation.",[18,225,227],{"id":226},"splits-and-reproducibility","Splits and reproducibility",[205,229,230,233,236,239,242],{},[208,231,232],{},"MASSIVE and ScreenQA retain official train\u002Fdev\u002Ftest partitions. Reserve a\ngroup-based calibration subset from train.",[208,234,235],{},"BoolQ and CLEVR use public labeled dev\u002Fval as frozen final evaluation sets,\nnot as official test results. Split train into fitting, development, and\ncalibration partitions.",[208,237,238],{},"Keep all questions from one image, all language variants of one source\nutterance, and questions sharing a passage in the same partition. Audit\nofficial splits for overlap and document any exclusions.",[208,240,241],{},"Store source revisions and hashes, sample IDs, split assignments, templates,\nfilters, candidate permutations, sampling weights, and seeds in manifests.",[208,243,244],{},"Pin preprocessing and metric implementations. Report cross-split duplicates\nand possible backbone pretraining contamination.",[14,246,247],{},"Select checkpoints and hyperparameters using development data only. Fit\ntemperatures on calibration data only. Freeze the recipe before final evaluation.\nSeeded runs improve repeatability but do not guarantee bitwise equivalence across\nGPU kernels or software versions.",[18,249,251],{"id":250},"evaluation","Evaluation",[14,253,254],{},"Report results by dataset, language, decision type, and candidate count:",[205,256,257,260,263,266,269,272],{},[208,258,259],{},"Accuracy and macro-F1 for decision quality.",[208,261,262],{},"NLL and explicitly normalized Brier score for distributions.",[208,264,265],{},"Ranked probability score and MAE for ordered counts.",[208,267,268],{},"Fixed-bin ECE and reliability diagrams for calibration.",[208,270,271],{},"Candidate-order sensitivity and correct-image\u002Fmissing-image\u002Fmismatched-image\nchecks for vision dependence; these are quality checks, not competing trainers.",[208,273,274],{},"End-to-end p50\u002Fp95 latency, throughput, and peak VRAM on the local GPU, including\npreprocessing and with image, sequence, candidate, and batch sizes disclosed.",[14,276,277],{},"Complete calibration, held-out evaluation, and benchmarks for seed 42 before\nreplication. Report variation only when multiple seeds have completed evaluation;\nfor one seed, leave sample standard deviation unmeasured. All probabilities\nmust be finite, in range, and normalized where appropriate. Report filtered\ncoverage alongside accuracy. Do not equate a hard target with a known conditional\nprobability, or a concentrated distribution with justified confidence.",[18,279,281],{"id":280},"release-assets-and-terms","Release assets and terms",[14,283,284],{},"Release the recipe, manifests, inference configuration, calibration parameters,\nmetrics, and model card alongside any weights. Keep raw data and checkpoints out\nof Git. Retain source attribution and audit data and image terms before release.",[14,286,287,288,292],{},"MASSIVE and CLEVR identify CC BY 4.0 terms; BoolQ identifies CC BY-SA 3.0 terms.\nScreenQA annotations identify CC BY 4.0, but Rico screenshot terms must also be\nchecked. Do not infer a weight license solely from an annotation or code license.\nScienceQA is included in the research mixture. Existing Laya Vision\nweights remain benchmark references. See ",[50,289,291],{"href":290},"upstream-alignment","upstream alignment","\nfor source protocols and comparison limits.",[18,294,296],{"id":295},"business-decision-sources","Business decision sources",[23,298,299,311],{},[26,300,301],{},[29,302,303,305,308],{},[32,304,34],{},[32,306,307],{},"Decision supervision",[32,309,310],{},"Evidence and limits",[42,312,313,331,348,366,384,402,420],{},[29,314,315,322,328],{},[47,316,317],{},[50,318,321],{"href":319,"rel":320},"https:\u002F\u002Fgithub.com\u002Famazon-science\u002Fesci-data",[54],"Amazon ESCI",[47,323,324,325,327],{},"Query\u002Fproduct ",[60,326,62],{},": exact, substitute, complement, irrelevant; English, Spanish and Japanese",[47,329,330],{},"Actual customer search queries with manual relevance judgments. Use explicit E\u002FS\u002FC\u002FI labels. The categories are not assumed to be an ordinal scale. Apache-2.0 project release.",[29,332,333,340,345],{},[47,334,335],{},[50,336,339],{"href":337,"rel":338},"https:\u002F\u002Fwww.microsoft.com\u002Fen-us\u002Fresearch\u002Fpublication\u002Fwikiqa-a-challenge-dataset-for-open-domain-question-answering\u002F",[54],"WikiQA",[47,341,342,343],{},"Query\u002Fpassage answerability ",[60,344,80],{},[47,346,347],{},"Real Bing queries, Wikipedia sentences and crowdsourced binary judgments. Explicit negative labels; never equate missing judgments with negatives. Microsoft Research Data License: research\u002Ftechnology development restrictions apply.",[29,349,350,357,363],{},[47,351,352],{},[50,353,356],{"href":354,"rel":355},"https:\u002F\u002Fsharc-data.github.io\u002Fdata.html",[54],"ShARC",[47,358,359,360,362],{},"Agent ",[60,361,62],{},": yes, no, irrelevant, ask for missing information",[47,364,365],{},"Public rule documents with crowdsourced scenarios\u002Fconversations. Real policy tasks, not a log of actual customer interactions. Evidence annotations and the target follow-up wording never enter model state. CC BY-SA 3.0.",[29,367,368,375,381],{},[47,369,370],{},[50,371,374],{"href":372,"rel":373},"https:\u002F\u002Fstanfordnlp.github.io\u002Fcontract-nli\u002F",[54],"ContractNLI",[47,376,377,378,380],{},"Full-contract ",[60,379,62],{},": entailment, contradiction, not mentioned",[47,382,383],{},"Human annotations on real contracts. Keep the entire document; exclude over-budget documents instead of selecting gold evidence spans. CC BY 4.0.",[29,385,386,393,399],{},[47,387,388],{},[50,389,392],{"href":390,"rel":391},"https:\u002F\u002Fspamassassin.apache.org\u002Fold\u002Fpubliccorpus\u002Freadme.html",[54],"SpamAssassin",[47,394,395,396,398],{},"Email spam ",[60,397,80],{},"; known ham supplies negative phishing examples",[47,400,401],{},"Public collected messages. Includes hard ham. Sender copyright remains with senders; the software's Apache license is not a blanket data license. This is a research corpus.",[29,403,404,411,417],{},[47,405,406],{},[50,407,410],{"href":408,"rel":409},"https:\u002F\u002Fmonkey.org\u002F~jose\u002Fphishing\u002FREADME.txt",[54],"Nazario phishing corpus",[47,412,413,414,416],{},"Positive phishing ",[60,415,80],{},", paired with separately identified ham",[47,418,419],{},"Hand-classified personal-inbox messages; 2023 for training\u002Fcalibration, 2024 development, 2025 test. CC BY 4.0. Positive\u002Fnegative source and age differ; report cross-corpus and temporal limitations.",[29,421,422,429,434],{},[47,423,424],{},[50,425,428],{"href":426,"rel":427},"https:\u002F\u002Farchive.ics.uci.edu\u002Fdataset\u002F228\u002Fsms+spam+collection",[54],"UCI SMS Spam Collection",[47,430,431,432],{},"Message spam ",[60,433,80],{},[47,435,436],{},"Public collected SMS with spam\u002Fham labels. Adds a short-message domain; it does not substitute for email evaluation. UCI release is CC BY 4.0.",[14,438,439],{},"The mixture includes research-use sources, including ScienceQA. Dataset annotations, raw documents and mixed trained\nweights have distinct rights; this document does not certify a commercial\ncheckpoint license.",[18,441,443],{"id":442},"conversion-and-isolation","Conversion and isolation",[14,445,446,449,450,453,454,457],{},[60,447,448],{},"scripts\u002Fprepare_data.py"," verifies every raw-file SHA-256 against\n",[60,451,452],{},"data\u002Fmanifests\u002Fenrichment-downloads.json"," before conversion. It creates the\nexisting ",[60,455,456],{},"{state, question, target}"," schema. One-hot targets come from source\nlabels. No model generates pseudo-labels, confidence scores or explanations.",[14,459,460],{},"Source rules are fixed in the converter:",[205,462,463,466,469,472,475,478],{},[208,464,465],{},"ESCI uses the official challenging-query subset. Select 450 training queries\nand at most 60 queries per held-out partition per locale using stable hashes.\nKeep whole queries, with official test priority. Products may occur under\nmultiple queries: this is unseen-query evaluation, not unseen-product evidence.",[208,467,468],{},"ShARC groups by source page and snippet. ContractNLI groups by full contract.\nWikiQA joins query and source-document aliases before resolving split conflicts.",[208,470,471],{},"Mail uses decoded bodies with the same plain-text\u002FHTML conversion for both\nclasses; labels, corpus names, spam-filter headers and acquisition timestamps\nare absent from state. URL\u002Fnumber-normalized body signatures group simple\ncampaign variants. Attachments and links are never executed or fetched.",[208,473,474],{},"Independent calibration comes from training groups. Original dev\u002Ftest labels\nare never moved into training. Conflicting group partitions keep the higher\npriority partition and exclude the others.",[208,476,477],{},"Frozen Laya\u002FJevBench inputs, the sampled training inputs, and its\nheld-out states are checked with punctuation-insensitive text hashes.\nChecks include Laya's 3,000-character body prefixes. Matching source groups are\nexcluded. This does not prove absence of semantic or paraphrase duplicates.",[208,479,480],{},"The fixed 2,048-token training budget is unchanged. Exclusions are counted.\nServing still accepts 4,096 tokens. Results on the eligible ContractNLI subset\nmust not be presented as scores on its complete official test set.",[14,482,483],{},"The training mixture combines 17 text\u002Fimage groups and nine business task\u002Flanguage\ngroups. Original dev\u002Fcalibration\u002Ftest rows retain their partitions. The existing uniform group\nsampler therefore allocates an expected 9\u002F26 of updates to business tasks and 17\u002F26\nto the other text\u002Fimage tasks. Its fixed 6,000-row training cap per group remains in effect.\nNo class rebalancing is hidden in calibration or evaluation. Phishing positives\nand WikiQA positives are minorities; report recall, macro-F1, PR-AUC and class\ncounts alongside accuracy, NLL, Brier and ECE.",[14,485,486,487,490,491,494,495,498,499,501],{},"The fixed 26-group mixture lives in ",[60,488,489],{},"data\u002Fprocessed\u002Fv1",". Its prepared partitions\ncontain 185,857 training, 64,843 development, 86,897 calibration and 180,031 test\nrows. The per-group training cap yields 143,238 eligible training rows.\n",[60,492,493],{},"manifest.json"," records counts, exclusions and hashes. The frozen benchmark text\nfingerprints in ",[60,496,497],{},"data\u002Fmanifests\u002Fevaluation-text-sha256.json"," are exclusion inputs,\nnever training examples. ",[60,500,448],{}," builds this mixture without\nrequiring a trained checkpoint or a previous training run.",{"title":503,"searchDepth":504,"depth":505,"links":506},"",2,3,[507,510,511,512,513,514],{"id":20,"depth":504,"text":21,"children":508},[509],{"id":196,"depth":505,"text":197},{"id":226,"depth":504,"text":227},{"id":250,"depth":504,"text":251},{"id":280,"depth":504,"text":281},{"id":295,"depth":504,"text":296},{"id":442,"depth":504,"text":443},"md",{},true,"\u002Fdocs\u002Fdata-and-evaluation",{"title":5,"description":16},"docs\u002Fdata-and-evaluation","WgB8znm6CmAcQYsDj-z2Z7q8oMho5QS7anM0bKDwRu4",1790560641963]