{"id":"arxiv/2606.07515","url":"https://arxiv.org/abs/2606.07515","tags":["cs.CL","cs.AI","cs.HC","math.PR"],"score":0,"title":"How reliable are LLMs when it comes to playing dice?","domain":"cs.CL","source":"arxiv","authors":["Luca Avena","Gianmarco Bet","Bernardo Busoni"],"reviews":0,"subject":"cs.CL","summary":"We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability problems. We constructed two datasets, respectively a set of standard exercises and a set of counterintuitive exercises, designed to trigger heuristic reasoning, and evaluated 8 state-of-the-art models, each tested with and without Chain-of-Thought prompting. Models achieve an average accuracy of 0.96 on standard problems but only 0.59 on counterintuitive ones. We further provide empirical evidence of token bias: performance drops by over 20% when…","category":"research","comments":0,"disputed":false,"division":"cs","mentions":0,"discussed":0,"published":"2026-06-05","reactions":{}}