tomato/blog
· 5 min read · 5 parts

Databases in a tomato suite: migrations, resets and clean state

databasespostgresredisscylladbflakiness
Contents
Per language MigrationsPostgreSQLScyllaDB, CassandraRedisOther databases

A flaky suite is almost always a suite with state that leaks: one scenario leaves a row, a counter or a cache entry behind, and another scenario passes or fails because of it. Run the files in another order, or one scenario on its own, and the result changes.

tomato’s answer is one rule: every scenario starts from the same state. Before each scenario it empties every database it manages. There is no option to turn that off.

Without a reset
✓
A · the 4th order is refusedthe rate limit counter ends at 4
counter = 4 ⟶
✗
B · an order is acceptedgot 429: the counter A left behind

B passes alone and fails after A. Reorder the files, run one scenario, run in parallel: a different result each time.

With tomato's reset
✓
A · the 4th order is refusedthe rate limit counter ends at 4
reset ⟶ empty
✓
B · an order is acceptedthe counter starts at 0

Every scenario starts from the same state, so it passes or fails on its own, in any order.

A run, table by table

The app migrates the database once; before every scenario tomato empties its tables, except the migration history and the tables you exclude; each scenario then builds the state it needs.startscenario 1resetscenario 2… every scenarioorders-dbpostgres container · started by tomatoschema_migrations000_countries001_orders🔒 kept: historycountriesDE GermanyID IndonesiaNL Netherlands🔒 kept: excludeorders1 ada · DE2 bob · NL42 ada · ID✕ emptied, ids resettomato startsPostgres, then your app,which migrates it:tables, versions,reference data.once per run

Once per run, tomato starts the Postgres container and then your service, which runs its migrations as it does in production: tables, the migration history, reference data.

A scenario writes what it likes: through your service, or seeded directly in a Background.

Before the next scenario tomato empties every table in one TRUNCATE … RESTART IDENTITY, except the migration history (tomato knows the common tools' tables) and the tables you exclude.

The next scenario starts from exactly the state the first one did, and builds what it needs. No scenario can pass or fail because of another.

Three things happen, and only one of them per scenario:

WhenWhatWho
once, at startthe schema: tables, indexes, reference datayour service’s migrations (or a schema file)
before every scenarioevery table emptied, ids restartedtomato
in each scenariothe rows this scenario needsa Background, or your service

Where the schema comes from

The reset empties tables; it never drops them. So the schema is made once, and the best place is the way production does it:

  1. Your service migrates at startup. tomato starts the database, then your service, which runs its migrations. The migration code is tested by every run. Recommended.
  2. A schema file tomato runs. ScyllaDB’s schema: option, or an init script the database container runs (Postgres reads /docker-entrypoint-initdb.d). For services that don’t migrate themselves.
  3. A before_all hook. sql_file: against a resource. It runs after your service has started, so only for a service that doesn’t need the tables to boot.

Migrations shows each with real tools: golang-migrate, Flyway, Alembic, node-pg-migrate and dbmate, a start script, and init SQL.

Seeding: a scenario builds its own state

A scenario that needs data declares it, in the scenario or its Background, which runs after the reset:

Background:
  Given "orders-db" table "orders" has values:
    | id | email           | country | total |
    | 42 | ada@example.com | ID      | 700   |

Seed the ids you assert on. Ids restart from 1 after every reset in Postgres, but a scenario asserting id is "1" still depends on how many rows it created first; id is "42" after seeding 42 does not.

Keeping a table (when you must)

Some data shouldn’t be emptied:

  • Migration history. Empty it and the next start re-runs every migration. tomato keeps it automatically for golang-migrate, Rails and sqlx (schema_migrations), goose, Flyway and Liquibase.
  • Reference data a migration wrote: countries, currencies, roles. Without it, every scenario would have to seed it again.

Exclude those, and only those:

orders-db:
  type: postgres
  container: postgres
  options:
    exclude: [countries]

It’s an escape hatch, not a pattern:

  • Never exclude a table a scenario writes to. What one scenario adds, the next one sees: the flakiness the reset exists to prevent.
  • A migration tool tomato doesn’t know (Alembic, Django, Prisma, Knex, node-pg-migrate, sqlx…) needs its history table in exclude, or the history is emptied after the first scenario: the table.
  • Prefer a seed over an exclude. Data a scenario needs belongs in its Background, where a reader can see it.
Migrations in the app · a tool before it · init SQL Whether your service migrates itself, or a tool like Flyway, golang-migrate, Alembic or Prisma does it before … PostgreSQL postgres Every table emptied in one TRUNCATE … RESTART IDENTITY CASCADE before each scenario, migration history kept, … ScyllaDB, Cassandra scylladb tomato creates the keyspace, runs your CQL schema once, and truncates every table in the keyspace before each … Redis redis Redis holds the state that leaks most easily: counters, sessions, caches. tomato flushes it before every … Other databases hooks · exec MongoDB, MySQL, Elasticsearch: tomato has no resource for them, but the rule still holds. A before_scenario …

The checklist

  • The schema comes from your service’s own migrations.
  • Each scenario seeds what it needs, and asserts on what it seeded.
  • exclude holds the migration history (if tomato doesn’t already) and read-only reference data. Nothing a scenario writes.
  • A database tomato has no resource for gets a before_scenario hook that empties it.
  • Running any one scenario alone gives the same result as running the suite.