{"id":"arxiv/2608.07303","url":"https://arxiv.org/abs/2608.07303","tags":["cs.AI","cs.LG","math.ST"],"score":0,"title":"Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons","domain":"cs.AI","source":"arxiv","authors":["Guilin Zhang","Kai Zhao"],"reviews":0,"subject":"cs.AI","summary":"Comparisons between AutoML systems at short time budgets -- tens of seconds rather than hours -- are common in tool READMEs and workshop papers, and they are easy to get wrong. We report a case study in which a simple AutoML engine, Orcetra, appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% of them at a nominal 60-second budget and 78.4% of datasets against FLAML alone at 30 seconds. Both margins came from protocol defects that a results table cannot show. The search loop scored every candidate on the test split and reported the best, making the headline metric a…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-08-07","reactions":{}}