Axiom SQL-Reflex v4
Axiom AI Studio (QwikZen)
Execution-aware, multi-agent Text-to-SQL: schema GraphRAG, a local LLM ensemble, safe execution and a semantic critic, running fully offline.
- Single-DB accuracy
- 55.6%
- 45 Spider questions, concert_singer
- Cross-DB accuracy
- 34%
- 100 unseen Spider dev questions, zero-shot
- External APIs
- 0
- Fully local inference
01
Problem
- Who
- Anyone asking questions of a real database in plain language.
- Why it mattered
- Most Text-to-SQL demos use toy tables, a single prompt and string-match scoring, so they hide whether the answer is actually correct.
02
Constraints
Infrastructure
Fully local: llama.cpp models, no paid or external APIs.
Technical
WRITE queries must never run destructively without checks.
03
Architecture
- Component
- Key decision (click)
Ground
- CartographerGraphRAG schema graph: FAISS + BM25 + k-hop joins
Generate
Verify
- Semantic criticRule checks + TinyLlama JSON verdicts
Converge
- Reflex orchestratorRetry with fresh context until confident
The Cartographer grounds the question in the real schema, the Architect proposes several SQL candidates, the verifier runs them safely, and the critic checks the result actually answers the question; the orchestrator retries until a confident answer or the budget runs out.
04
Key decisions
D1Execution-grounded evaluation, not string match
- Alternatives considered
- Exact-match SQL scoring
- Why
- Correctness means the right result on a real database.
- Trade-off
- Slower: ~25–36 s per question with the multi-agent loop.
D2A local multi-model ensemble with deduplicated candidates
- Alternatives considered
- One large hosted model
- Why
- Self-consistency across models, fully offline.
05
Build highlights
- 01
Safe WRITE pipeline
RBAC, schema-impact simulation, destructive-operation blocking, dry runs, an LLM audit of write intent and no-op update detection.
- 02
From "did it run" to "is it right"
A semantic critic checks metrics (COUNT, AVG, MAX), empty or nonsensical results, and whether the answer fits the question.
- 03
Honest benchmarking
On 100 mixed Spider dev questions: 34 correct, 32 wrong logic and 34 where the gold answer itself was invalid.
06
Results
55.6%
Accuracy, single schema
45 questions, ~24.7 s each
34%
Accuracy, unseen schemas
100 questions, ~36 s each
How this was measured: Execution-grounded evaluation on Spider (concert_singer and 100 mixed dev questions); no fine-tuning.
07
Lessons
What worked
- Grounding in real schema graphs stops the model hallucinating tables.
- Multi-agent retries recover from many first-attempt failures.
What didn't
- Cross-database generalisation stays hard for 6–7B local models.
- The small semantic critic is occasionally unreliable.
What I'd do differently
- Fine-tuning and larger local models to cut latency and lift accuracy.
08
