fastdata
Guide

Knowledge base

The knowledge base is the main lever for answer quality. It is a folder of plain Markdown and YAML files, best kept in git and reviewed like code. The agent searches it the way a developer greps a repository: it reads INDEX.md first, then opens only the files it needs. There is no vector database to run.

Think of it as what you would tell a new analyst in their first week: which tables matter, what the business words mean, how metrics are calculated, and which mistakes to avoid.

Layout

knowledge/
  INDEX.md                         # start here
  glossary.md                      # business words → columns and filters
  rules.md                         # must / must not
  metrics/
    revenue.md                     # one agreed definition per metric
  examples/
    top-customers.yaml             # verified question → query pairs
  sources/
    <database>/
      _source.md                   # engine, dialect notes, freshness
      tables/
        <schema>.<table>.md        # one file per table

The location is set by knowledge: in fastdata.yaml (default ./knowledge, relative to the config file). The agent can read anything in this folder and nothing outside it.

Start with fastdata learn

You do not have to write the table files by hand. fastdata learn reads the schema through your connector and drafts the knowledge base with Claude:

fastdata learn                                   # uses the connector in fastdata.yaml, or asks for a URL
fastdata learn --url postgres://ro:pw@host/db    # set up the connector from a URL, then learn

What it does:

  1. Reads the schema with plain catalog queries (no model involved): tables, views, columns, types, keys, comments and row counts. For category-like text columns (status, country, …) it samples the most common values from up to 10,000 rows. Columns whose names suggest personal data or free text (email, name, phone, body, comment, …) are never sampled.
  2. Drafts descriptions with Claude: what one row of each table is, column notes, gotchas (status meanings, units, nullable timestamps, many-to-many joins), personal-data columns, a short overview, business areas and glossary terms. Claude gets no tools; it returns the text and fastdata writes the files.
  3. Writes sources/<name>/_source.md and one file per table in sources/<name>/tables/, and fills INDEX.md, glossary.md and rules.md if they are still the blank templates (an INDEX.md you have edited only gets a line for the new source).

Existing files are never overwritten unless you pass --force, so you can run it again after adding tables. Every drafted file starts with a comment saying so: review the drafts, fix what is wrong, and add what only people know (metric definitions, default filters, verified examples). Columns, types and keys come from the database; only descriptions come from Claude.

OptionMeaning
--url <url>postgres://…, mysql://…, sqlserver://… or a SQLite file. The URL is saved as DATABASE_URL in .env (added to .gitignore), and fastdata.yaml gets a connector that reads ${DATABASE_URL}.
--source <name>Folder name under sources/ (default: the database name).
--forceOverwrite files that already exist.
--no-samplesDo not sample column values (nothing but schema metadata reaches Claude).
--max-tables <n>Document at most this many tables (default 300).

Automatically: when the knowledge base documents no tables yet, fastdata init, fastdata and fastdata ask ask Let fastdata learn them from your database now? [Y/n] in a terminal, and ask for the database URL if there is no connector. A "no" is remembered (.fastdata/learn-declined); run fastdata learn any time.

Use a read-only database user, as for every connector. Schema details and sampled values are sent to Claude; use --no-samples if column values must not leave your network.

The files

INDEX.md (required)

The map. Keep it short: what the data is about, which sources exist, and where to look for which kind of question. A "question about → start with" table works well:

| Question about | Start with |
| --- | --- |
| revenue, MRR, churn | `metrics/mrr.md`, tables `subscriptions`, `accounts` |
| call volume, answer rate | `metrics/call-kpis.md`, table `calls` |

glossary.md

Maps the words people use to columns and filters. This is where most wrong answers are fixed.

## Active customer
`accounts.status = 'active'`. Trials (`status = 'trial'`) are not customers yet.

## Europe
Every `accounts.country` except `US`.

rules.md

Constraints the agent must respect: personal data, default filters, joins to avoid, date conventions.

## Personal data
- `users.email` is personal data. Do not select it unless the user asks for contact details.

## Default filters
- Revenue excludes orders with `status IN ('cancelled', 'refunded')`.

## Dates
- "Last month" means the previous full calendar month.

Rules guide the agent; they are not a security boundary. Hard limits (read-only access, which tables a role can see) belong in the database. See Security.

metrics/*.md

One file per metric with the agreed definition and reference SQL. The agent is told to reuse these rather than invent its own, so "revenue" is computed the same way every time.

# Revenue
Sum of `quantity × unit_price` over order lines, excluding cancelled and refunded orders.
`unit_price` already includes the discount.

```sql
SELECT SUM(oi.quantity * oi.unit_price) AS revenue
FROM order_items oi JOIN orders o ON o.id = oi.order_id
WHERE o.status NOT IN ('cancelled', 'refunded')
```

Common mistakes: using `products.unit_price` (list price) instead of `order_items.unit_price`.

examples/*.yaml

Verified question → query pairs. The agent uses them as patterns for similar questions, and the web chat shows their questions as starter buttons (the first six, alphabetically by file name).

question: Who are our top 10 customers by revenue?
query: |
  SELECT c.name, SUM(oi.quantity * oi.unit_price) AS revenue
  FROM customers c
  JOIN orders o ON o.customer_id = c.id
  JOIN order_items oi ON oi.order_id = o.id
  WHERE o.status NOT IN ('cancelled', 'refunded')
  GROUP BY c.name
  ORDER BY revenue DESC
  LIMIT 10
notes: Revenue as defined in metrics/revenue.md.
verified_by: hasan

question and query are required (fastdata doctor checks them); other fields are free-form.

sources/<database>/

_source.md describes the engine: dialect quirks (date functions, ILIKE support), how fresh the data is, who owns it.

tables/<schema>.<table>.md describes one table. Write what is not obvious from column names:

# orders
One row per checkout.

| Column | Type | Notes |
| --- | --- | --- |
| ordered_at | TEXT | UTC timestamp `YYYY-MM-DD HH:MM:SS` |
| status | TEXT | placed, shipped, delivered, cancelled, refunded |

Cancelled and refunded orders do not count as revenue.

Good things to note: units and currencies, time zones, the list of status values, soft deletes, which table is the source of truth, and joins that look right but are wrong.

How to grow it

  1. Start with INDEX.md, the 5–10 most-used tables, and the 3–5 metrics people ask about most.
  2. Ask the questions your team really asks. When an answer is wrong, read its Assumptions, Query and Based on, then fix the file that would have prevented the mistake.
  3. When an answer is right and the question is common, save it as an example.

Your database's own table and column comments help too: when the knowledge base does not cover something, the agent can look at the catalog (information_schema, pg_catalog, …) through the connector.

Complete examples