AI & Automation Briefing - September 22, 2026
New Benchmark Tests LLM Agents on Full Business Intelligence Workflows
Researchers have released BI-Bench, the first benchmark designed to measure how well LLMs handle end-to-end business intelligence tasks without human data preparation. The benchmark covers table identification, data transformation, join relationship building, and question answering, drawn from real-world BI projects and user dashboards. Frontier LLMs scored below 50% accuracy on BI-Bench. The team also built BI-Agent, a tool-augmented system that breaks BI workflows into structured subtasks and coordinates specialized data management methods across each stage. BI-Agent improved accuracy by up to 40 percentage points over base LLMs. A post-training framework using supervised fine-tuning and reinforcement learning on real BI project trajectories added further gains of up to 30 points. The results indicate that combining tool use with domain-specific training is necessary for reliable agentic performance on complex data workflows.
Amazon Blocks Meta's Muse Agent as Agentic Commerce Runs Into Platform Walls
Users of Meta's AI agent Muse began seeing error messages last weekend when attempting to purchase products on Amazon through the agent. Amazon's message cited its Conditions of Use and stated that access by an unauthorized AI agent is prohibited, effectively shutting Muse out of its platform. Amazon has its own foundation models and inference infrastructure, giving it competitive reasons to restrict a rival agent. There are also operational considerations: if Muse places a faulty order, Amazon bears the customer service burden. Muse has a relatively low hallucination rate among current AI models, but not low enough to eliminate that risk. This is a concrete early signal of platform gatekeeping shaping which agents can operate where, a dynamic that will directly affect how agentic automation workflows are designed. The block arrives as adoption data shows Muse is growing faster than ChatGPT did during its early mobile launch, making the access restrictions more significant for the broader agentic ecosystem.
Research Finds Automated Agent Harness Improvement Risks Overfitting, Proposes Fix
A new paper from researchers at Google and UNC Chapel Hill identifies a core problem with recursive self-improvement of AI agent harnesses: iterative automated optimization of prompts, control flow, tooling, memory, and context management tends to overfit to training tasks, producing strong in-distribution gains that shrink or disappear on new benchmarks. Their proposed method, RRSI (Regularized Recursive Self-Improvement of Agent Harnesses), addresses this by constraining how edits are proposed and selected during each improvement cycle. A time-decaying edit budget limits how many changes can be bundled in a single iteration, and the selection process filters out benchmark-specific, trivial, or costly changes in favor of reusable mechanisms. Tested across eight benchmarks covering coding, agentic workspace tasks, and engineering design, RRSI achieved up to 14.1 percentage points of improvement on in-distribution tasks and up to 4.7 points on out-of-distribution benchmarks. For anyone using automated prompt or workflow optimization in tools like n8n or Claude, this research is a concrete warning that gains measured only on familiar tasks may not hold when those workflows encounter new inputs or clients.