Overview
A desktop time-tracking application needed an in-app AI assistant that could answer natural-language questions about the user's own logged work — hours per activity, per period, comparisons against previous months — without that data ever leaving the machine. The privacy constraint ruled out cloud APIs entirely: the assistant runs on a locally-hosted, quantized open model. Because small local models fabricate confidently, the core design rule was strict: the model never answers a data question from memory — every data question triggers a tool call into the application's own database, and the reply must restate exactly the values the tool returned, never renamed, never mixed across categories.
Model Selection & Evaluation Loop
Candidate local models were compared with a scripted evaluation loop, not gut feel: a golden set of real question patterns (single-month lookups, previous-period comparisons, full-year aggregations) scored for precision and for fabrication. Transferable findings: streaming and tool-calling do not mix reliably in local runtimes, so tool rounds run non-streaming; low generation temperature is mandatory for factual answers; and a reasoning-tuned model in the ~8B class reliably chained multiple tool calls where a slightly smaller non-reasoning model confused periods and invented entries — while the next size class up would not fit the available memory at all. The promotion gate for any new model: a minimum precision threshold on the golden set with zero fabricated entities, measured against the incumbent model as the baseline. The selected model passed that gate; the evaluation loop stays in place permanently as the regression harness for any future model swap — no model change ships without beating the incumbent on the same golden set.
Anti-Fabrication Guardrails
Anti-fabrication ran in three layers. System-prompt rules: a tool call is required for EVERY data question; never invent names or numbers; copy tool values exactly; today's date is injected so relative periods ("last month") resolve deterministically, with worked examples of how to fill tool arguments. Generation constraints: temperature capped low — higher settings made small models invent clients and figures instead of calling the tool. Eval enforcement: the anti-fabrication check fails the entire evaluation run if any answer contains an entity or number absent from tool output. This is the same hallucination failure mode the AI Reliability line measures in RAG systems — here solved by grounding every answer in tool output instead of retrieved chunks.