Inference Cost Near Zero: Berkeley Proposes Three Directions for Data Systems For/Of/By Agents

UC Berkeley BAIR notes that GPT-4 level inference costs have dropped from ~$30 per million tokens in early 2023 to under $1, with some providers below $0.10—a median 50x annual decrease. The authors argue that 'near-free intelligence' is coming, motivating three new research directions. First, 'For Agents': Agent data access patterns differ from humans or BI tools, generating many 'agentic speculation' requests. Experiments show that in multi-agent attempts on the same task, only 10-20% of subplans are unique, with 80-90% redundant. Systems should adopt multi-query optimization, approximate query processing (AQP), batch query interfaces, and proactive data delivery (e.g., latency estimates). Second, 'Of Agents': With thousands of agents executing long-term tasks, unstructured Markdown file memories won't scale. The paper proposes structured memory where each memory is tagged by module, language, framework, failure mode, etc., supporting parallel edits and rollback. Third, 'By Agents': Using cheap inference, agents can synthesize workload-specific data engines (e.g., Bespoke OLAP, GenDB, custom key-value stores) for a few dollars, discardable and rebuildable. However, incomplete specs may cause reward hacking, requiring auxiliary validator agents to auto-generate test cases. In summary, the boundary between agents and data systems will blur: agents may design their own runtime, and data systems evolve from passive query engines to intelligent, proactive, self-optimizing architectures.