Using DSPy to Evaluate and Improve Datasette Agent SQL System Prompt
Decision Brief
What changedA study uses DSPy to evaluate and improve Datasette Agent's SQL system prompt.
Why it mattersSchema with only table names causes the model to guess column names, leading to errors; prompt should include column names or relax constraints to reduce calls and errors.
Who should careAll AI builders
Affected stackClaude
Builder actionMonitor
Source confidenceMedium · Reliable media or first-hand reporting
Initiated by Datasette author, the study used Claude Fable 5 to install latest Datasette alpha, datasette-agent, and DSPy, testing with GPT 4.1 mini and nano. Results showed that when schema lists only table names and the prompt says “don't call describe_table if info already exists,” the model guesses column names (e.g., page_count, o.order_id), causing errors and retry loops. Improvements include adding column names to the schema or softening that suggestion.
Summary basis: official / RSS sourceUnless it says 'full article read', this summary is based only on publicly available content — it never pretends to have read restricted originals.
Sources
- Simon Willison：Blog
Hands-on notes on LLM tools, local models, and practical AI engineering.
- Simon Willison：Blog