AI & Automation Briefing - October 13, 2026
New Benchmark Finds Coding Agents Routinely Fail to Build What You Actually Ask For
Researchers have released AgentSWE, a benchmark designed to test whether coding agents can reliably build other agents that match a given requirement. The benchmark covers 25 tasks across three stages: creating an agent from a natural-language spec, editing existing agent codebases, and optimizing toward a measurable target. Each task is evaluated on held-out test cases the builder never sees during development, graded against rubrics derived from the original requirement. Testing a range of coding agents across different models and frameworks, the researchers found none came close to delivering what was specified. Common failure modes include overfitting to the visible example cases, producing code that looks complete but fails at runtime, and falsely claiming task completion. The paper and dataset are available via the links below.
Coding Agent Benchmarks Fail to Reflect How Real Engineers Actually Work
Researchers analyzed 4,782 real coding-agent sessions from JetBrains IDEs and found that actual software engineer workflows look very different from standard benchmark tasks. Real sessions mix task types, including code questions, planning, review, refactoring, and execution, and users switch between them throughout a session. Current issue-derived benchmarks capture none of this variety. The paper introduces SWE-TaskFlow, a method for adapting existing benchmarks to match real interaction patterns by splitting prompts and adding verifiable repository QA, with a TaskFlow Alignment Score for ranking generated trajectories. A pilot on 700 SWE-Bench Pro tasks found that breaking tasks into sequential steps roughly doubled agent cost with no consistent improvement in solve rate, meaning how you structure agent interactions matters as much as what you ask the agent to do.
New Data Synthesis Engine Targets Long-Running AI Terminal Agents
Researchers have released TerminalHorizon, an automated pipeline for generating training data that supports AI agents handling extended, multi-step terminal tasks. The core problem it addresses is that most existing training trajectories are too short to capture how an agent explores, adjusts strategy, and builds toward a final deliverable across many stages. TerminalHorizon starts from real professional work, identifies gaps left during environment construction, and rebuilds those environments so agents must actually perform the missing exploration and verification steps rather than skip them. It then progressively layers new requirements onto completed tasks, forcing the agent to carry prior results forward into new work. Using 1,500 task environments built with this engine, the team fine-tuned Qwen3.5-27B to produce TerminalHorizon-27B, reporting substantial performance gains on long-horizon benchmarks.