S
Airlines
AviationFlights, bookings, tickets and boarding passes from the Postgres Pro demo database.
- Tables
- 8
- Rows
- 2.29 M
- Right data
- 20 / 20
- Exact
- 13 / 20
fastdata-bench is an open text-to-SQL benchmark on three public PostgreSQL databases, from an 8-table airline to a 31-table hospital. It scores the data each answer returned, compared cell by cell with a gold result. It does not score the SQL text.
The tiers are set by the number of tables the agent has to understand, not by row count. All three databases are public, pinned to a version and checksum-verified, so anyone can rebuild the exact setup.
Flights, bookings, tickets and boarding passes from the Postgres Pro demo database.
A DVD rental business with films, actors, stores, customers, rentals and payments.
Hospital admissions, ICU stays, lab events, diagnoses and prescriptions for 100 patients.
Every database has 10 queries, each asked in two wordings, so each bar is 20 or so answers. Teal answers had the right rows and values, plus a column nobody asked for, such as a flight count next to an average delay.
The number on the right of each bar is right data / all answers. Hard questions did better than medium ones: 17 of 18 had the right data.
| Group | Exact match | Right data, extra columns | Wrong data | Right data / all |
|---|---|---|---|---|
| S · Airlines | 13 | 7 | 0 | 20 / 20 |
| M · Pagila | 16 | 1 | 3 | 17 / 20 |
| L · MIMIC-IV | 6 | 11 | 3 | 17 / 20 |
| Easy | 16 | 4 | 0 | 20 / 20 |
| Medium | 9 | 8 | 5 | 17 / 22 |
| Hard | 10 | 7 | 1 | 17 / 18 |
| All | 35 | 19 | 6 | 54 / 60 |
Nine of the 30 questions, three per difficulty level and one from each database (a few are shortened here). The agent saw only the question, connected as a read-only role, and wrote its own SQL.
Which cities are served by more than one airport? Return each city with its number of airports, sorted by the number of airports in descending order.
Exact in both wordingsWhich 10 countries have the most customers? Show the country and its number of customers, highest first; break ties alphabetically by country.
Exact in both wordingsHow many hospital admissions are there for each admission type? Show the admission type and the count, most frequent first.
Exact in both wordingsWhich films have no copies in any store's inventory? List their titles alphabetically.
Exact in both wordingsWhat is the in-hospital mortality rate for each insurance type? Show the insurance, the number of admissions, the number of admissions where the patient died in hospital and the mortality percentage rounded to 2 decimals, highest rate first.
Exact in both wordingsConsidering only flights with status 'Arrived', which 5 departure airports have the highest average departure delay? Show the airport code, airport name and the average delay in minutes, rounded to 2 decimal places.
Right data, added a flight countFor every day in July 2017, show the number of bookings made that day and the 7-day moving average of daily bookings (that day plus the six preceding days, which may fall in June), rounded to 2 decimals.
Exact in both wordingsFor each film category, which film was rented the most times? If several films tie for the top spot in a category, include all of them.
Exact in both wordingsOut of all hospital admissions, how many were followed by the same patient's next admission starting within 30 days after discharge? Return the totals and the readmission rate as a percentage.
Right data, added data-quality countsEach query is asked twice, worded differently, and both wordings must return the same result. The tie question above, asked the second way:
“Find the most frequently rented movie within every category, keeping all films that share the highest rental count in their category. Return category name, title and rental count, sorted by category and title.”
fastdata got 23 of the 30 queries to the same verdict in both wordings.
A system may write any query it likes. What counts is whether the result it showed its user matches the gold result.
Column names, column order and row order are ignored. Numbers are compared at the gold answer's precision, and dates and time zones are normalized.
The output score checks what the system returned. The SQL score re-runs its final query read-only. Both came out the same in this run: 35 exact, 54 lenient.
Strict requires exactly the gold columns. Lenient also accepts extra columns, as long as every gold column is there with the right values.
learnEach knowledge base was drafted by fastdata learn from the schema alone. It never saw the gold queries.The agent added context nobody asked for: an aircraft code next to the model name, the number of measurements behind an average heart rate, or a count of admissions with no discharge time. The lenient score accepts these.
On "top N" questions the agent fetched N + 2 rows so it could check for a tie at the cutoff. Its written answer listed exactly the top N, but the query it returned had the extra rows, so the result does not match.
Asked which drugs went to at least 80 patients, it followed the knowledge base and left out base solutions such as saline bags. It named the two entries it excluded in the answer.
The largest schema, the 31-table hospital database, scored lowest on exact matches but still had the right data in 17 of 20 answers. That is what we are working on next: keeping the returned result to the columns and rows the question asked for.
We published the benchmark so others can check our numbers and run their own systems on it. Come tell us what you think:
One command downloads and loads all three databases into PostgreSQL in Docker. The harness has an adapter interface, so you can plug in another system and compare.
$ git clone https://github.com/reegosoftware/fastdata-bench.git && cd fastdata-bench $ make setup # PostgreSQL 18 in Docker + the 3 databases $ make verify # table and row counts match the snapshots $ make check # the gold queries reproduce the committed answers $ make benchmark # ask all 60 questions, write the report