New Taste-Bench benchmark shows AI agents fall short in long-horizon decisions, top score only 59.7%

Researchers from Microsoft and City University of Hong Kong call an agent's path-selection ability on long tasks 'taste' and built Taste-Bench around it. The benchmark has 502 questions, each giving the original task, the trajectory up to a branch point, and two candidate directions, drawn from thousands of scored coding and AI research trajectories such as SWE-bench Pro and METR's public MALT transcripts. Branches come in two types: divergences between two independent attempts at the same task, and detours from self-correction within a single run. The team tested 14 models including Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral, asking each question in both orders and scoring only when both answers were correct; random guessing scores 25%. GPT-5.6 Sol led at 59.7%, with GPT-5.5 at 59.5% and the rest notably lower. Average accuracy was 62.3% when the key clue already appeared in earlier steps, but fell to 21%—near random—when the clue only surfaced after substantial later work. Switching two models across three reasoning-effort levels barely changed accuracy, and models spent the most reasoning tokens on the hardest questions. The research also shows this judgment can be trained: distilling a teacher model's judgments into Qwen3.6-27B as a student raised accuracy on unseen tasks from 30% to 47.9%, and injecting the student's judgments as suggestions into task context lifted success on 41 held-out SWE-bench Pro tasks from 14.6% to 33.7%. For teams running long-horizon autonomous agents, this means adding external judgment signals at branch points beats adding inference compute. The benchmark and code are public.