Dashboard
—Eval scores
—These scores measure whether the Ask bot’s numbers match the database—not whether illness is rising or falling. They are answer-quality checks for reviewers. Low grounding scores mean a QA regression on the bot, not a clinical emergency.
Forecast backtest
—Error by forecast horizon (weeks ahead). Lower is better. Last-value is the product primary; models that lose are experimental.
How it works
Ask QA (grounding): a stale-table warning once included a day count (14) that never came from a tool result. Nightly answer-quality scoring then marked many answers as ungrounded (for example 16.7% vs a 100% baseline). The numbers in the surveillance tables were unchanged; the warning text had used a hardcoded day count instead of a value returned by the tool. Warnings are now number-free; scores here refresh on the next nightly after deploy.
Forecasts: backtests show complex models rarely beat repeating last week on these sparse weekly series—so outlooks lead with last-week persistence and label the rest experimental.
Live demo: the free hosted API sleeps when idle. Ask may take about a minute to wake; that is a hosting delay, not missing CDC data.
Recorded answers
—Real answers recorded during the eval run, replayed with the tool calls behind them.