Configuration reference¶

Complete reference for tomato.yml configuration options.

File structure¶

version: 2              # Required: config version

settings:               # Test execution settings
  timeout: 5m
  fail_fast: false
  output: pretty
  containers:
    reuse: false        # keep containers between runs instead of starting fresh

app:                    # Application under test (optional)
  command: ./my-app
  port: 8080
  ready:
    type: http
    path: /health
  wait: 5s
  env:
    KEY: value

containers:             # Container definitions
  name:
    image: image:tag
    command: []         # Optional startup command (list or string)
    env: {}
    ports: []
    volumes: []
    depends_on: []
    wait_for: {}

resources:              # Resource/handler definitions
  name:
    type: http|http-server|grpc|grpc-server|postgres|scylladb|cassandra|redis|kafka|rabbitmq|s3|websocket|websocket-server|shell|aws|lifecycle|browser|pdf|network
    # aliases: postgresql, http-client, grpc-client, websocket-client, minio
    container: container_name
    options: {}

hooks:                  # Lifecycle hooks
  before_all: []
  after_all: []
  before_scenario: []
  after_scenario: []

features:               # Feature file settings
  paths:
    - ./features
  tags: ""

Settings¶

Field	Type	Default	Description
`timeout`	duration	`5m`	Global test timeout
`fail_fast`	bool	`false`	Stop on first failure
`output`	string	`pretty`	Output formats, comma-separated; see Reports
`profile`	bool	`true`	Measure the app and its containers per scenario; see Profile. Off when `parallel` is above 1

Reports¶

output takes one or more comma-separated formats. A format without a path prints to the console; format:path writes that format to a file. Use this to feed CI test reports and test-management tools.

settings:
  output: "pretty,junit:reports/tomato.xml,ctrf:reports/ctrf.json"

Format	Writes
`pretty`	Human-readable console output (default)
`progress`	One character per step
`junit`	JUnit XML, one `testsuite` per feature and one `testcase` per scenario
`cucumber`	Cucumber JSON, the input most Cucumber report tools import
`ctrf`	CTRF JSON, for the ctrf-io GitHub Actions and reporters: one test per scenario with its steps, the failed step's error, its screenshots as attachments and what it cost the app (the profile) in `extra`. File only
`tomato`	Structured events used by the GitHub Action's PR comment

Every run also keeps its events in .tomato/runs/<run>/events.log, whatever output says, so a report can be written for any past run:

tomato report                         # the last run, as CTRF JSON on stdout
tomato report 216b61f6 -f junit -o reports/junit.xml

tomato report writes ctrf or junit. Its JUnit carries more than the junit output does: the failed step and its error in <failure>, the steps in <system-out> with each screenshot as [[ATTACHMENT|path]], which Jenkins and GitLab show, and the scenario's profile as <properties>. A run that failed before its scenarios — a container that did not start, an app that never became ready — is one failed test named after where it failed, in both formats, so CI shows the run failed rather than zero tests. tomato ui exports the same files from its Report tab.

Tomato creates the report directories if they don't exist. When --format is passed on the command line (the GitHub Action passes --format tomato for PR comments), it replaces the console format but file outputs from output are still written.

The `tomato` event format¶

tomato writes one TOMATO_EVENT: line per event, followed by a JSON object with a type: feature_start, scenario_start, step_start, step_end, scenario_end, feature_end and summary. A step_end has the step's status, stepIndex, durationMs and, when it failed, its error text.

Before the first scenario the run says how it is coming up: a phase event as it enters containers, resources, app and tests; a container event with the container's name as it is starting, then ready or failed with durationMs; and an app event as the app is starting, listening (with its url), ready after its health check, or failed, with durationMs from when it was started. tomato ui plays these as the first two acts of a run's film.

A run that fails before its scenarios start writes a run_error instead: its phase (config, containers, resources or app), the error, and, when the app failed to start or its health check failed, the last lines it logged in logs. The GitHub Action shows it in the pull request comment and the job summary, so a run that never got to a scenario doesn't read as 0 passed.

{"type": "run_error", "status": "failed", "phase": "app",
 "error": "failed to start application: app not ready: app exited with exit status 1 300ms after it started, before it was ready",
 "logs": ["{\"level\":\"fatal\",\"message\":\"missing config\"}"]}

When a step failed because a value it compared did not match (stdout is:, response body is:, response json contains:, a Kafka, RabbitMQ or WebSocket message), its step_end also has a mismatch object with both sides, which tomato ui draws as a diff:

{"type": "step_end", "status": "failed", "error": "at status: expected created, got pending",
 "mismatch": {"what": "response JSON", "mode": "subset",
              "expected": "{\"status\": \"created\"}",
              "actual": "{\"id\": 1, \"status\": \"pending\"}"}}

Field	Meaning
`what`	What was compared, such as `stdout`, `body`, `message` or `response JSON`
`expected`	The expected value as text, after variables are replaced
`actual`	The actual value as text
`mode`	`text`: compared as text. `json`: compared as JSON, every key on both sides. `subset`: compared as JSON, keys only `actual` has are allowed

error is the same text with or without mismatch. The field is left out when expected and actual together are over 256 KB. The event format is not part of the stable surface.

In the tomato format, a step_end event carries screenshots when the step left images behind, as a list of {"resource": "alice-browser", "path": "…/after-apply.png"}: the browser resource's saves screenshot, and the page when a step failed. tomato ui reads it to put the picture on the step.

When the run is profiled, a step_end carries samples, the app's and its containers' readings since the last event that carried them ({"at", "app": {"cpu", "rss", "threads", "fds"}, "containers": {"postgres": {"cpu", "memory"}}}), and a scenario_end carries profile, what the scenario cost: its start, end and durationMs, the app's cpuMs, peakRss, rssStart and rssEnd, each container's cpuMs and peakMemory, the traffic each network resource saw, and the errors and warnings the app logged.

App configuration¶

Configure your application under test to run with test containers.

app:
  # Option 1: Run a command
  command: go run ./cmd/server
  workdir: ./

  # Option 2: Build from Dockerfile
  build:
    dockerfile: Dockerfile
    context: .

  # Connection settings
  port: 8080
  ready:
    type: http          # http, tcp, or exec
    path: /health       # For HTTP checks
    status: 200         # Expected status (default: 200)
    timeout: 30s        # Ready check timeout

  # Wait after ready check passes
  wait: 5s

  # Environment variables (supports container templates)
  env:
    DATABASE_URL: "postgres://test:test@{{.postgres.host}}:{{.postgres.port.5432}}/test"
    REDIS_URL: "redis://{{.redis.host}}:{{.redis.port.6379}}"

Port already in use

In command mode, tomato checks port before starting the app. If another process is already listening there (a local make run, a leftover test run), tomato fails instead of starting: otherwise the ready check would pass against the other process and the tests would hit the wrong app.

In command mode the app inherits tomato's environment, plus env. Some resources the app depends on while it starts are set up before it and add to that environment too: the aws resource points the app's AWS SDK at its STS and removes AWS_ACCESS_KEY_ID and friends inherited from your shell. The app's own env still wins over anything a resource sets.

The app runs in its own process group, and stopping it stops the whole group, so command: go run ./cmd/server (where go does not pass signals on) does not leave the server running and holding the port.

Template variables¶

In app.env, you can use templates to inject container addresses:

Template	Description
`{{.container_name.host}}`	Container hostname (`localhost` in command mode, the container's DNS name in container mode)
`{{.container_name.port.NNNN}}`	Host port that `NNNN`, the port inside the container, is mapped to
`{{.resource_name.url}}`	Base URL of an `http-server` resource

NNNN is required. A bare {{.container_name.port}} is left in the value verbatim, so the application receives the literal template text.

Example with PostgreSQL:

containers:
  postgres:
    image: postgres:15
    env:
      POSTGRES_USER: testuser
      POSTGRES_PASSWORD: testpass
      POSTGRES_DB: testdb
    ports:
      - "5432"

app:
  command: ./my-app
  env:
    # These are resolved at runtime when containers start
    DB_HOST: "{{.postgres.host}}"
    DB_PORT: "{{.postgres.port.5432}}"
    DATABASE_URL: "postgres://testuser:testpass@{{.postgres.host}}:{{.postgres.port.5432}}/testdb"

Note: The container field in resource definitions automatically handles host/port resolution - you don't need to specify connection strings manually for resources.

Containers¶

Define Docker containers for your test infrastructure.

containers:
  postgres:
    image: postgres:15
    env:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: test
    ports:
      - "5432"
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    depends_on:
      - other_container
    wait_for:
      type: port
      target: "5432"
      timeout: 30s

volumes are source:target[:mode]. A source path (./x, ../x, /x, ~/x) is bound into the container, relative paths resolved against the directory of tomato.yml; a bare name is a Docker volume. build (context, dockerfile) builds the image instead of pulling one, with context relative to tomato.yml too. The image is named tomato-<container> and kept, so the next run rebuilds from cache into the same image. Each container's output is written to .tomato/runs/<run>/container-<name>.log for as long as it runs, including when it fails to start.

Presets¶

A preset is a container tomato knows how to run. It fills in the image, env, ports and wait strategy; anything the entry sets itself (an env key, image, wait_for) wins.

containers:
  kafka:
    preset: kafka
    auth: aws_msk_iam   # optional: plaintext (default) or aws_msk_iam

Preset	What it runs
`chromium`	A headless Chromium for `browser` resources. Pages it opens on `localhost` reach the app on your machine, through a tunnel to each port the suite uses there. See Browser.
`verapdf`	A veraPDF validation service for the `pdf` resource's `conforms to` step: PDF/UA and PDF/A, with the rules a document breaks in the failure. See PDF.
`kafka`	A single-node KRaft broker, reachable from the host and from other containers. With `auth: aws_msk_iam`, a second listener speaks SASL `AWS_MSK_IAM` like MSK's IAM port. See Kafka.
`redis`	A Redis server. With `cluster: true` it runs in cluster mode, one node owning every slot; with `tls: true` it speaks TLS only, with a certificate from a CA made for the run. Together, Redis the way ElastiCache runs it with cluster mode and in-transit encryption. See Redis.

Resources tomato does not manage¶

tomato resets every resource before every scenario, and that is not configurable. A resource with no container: is a system tomato did not start, so resetting it destroys someone's real data. tomato refuses to load such a config:

resource "app-db" (postgres) has no container, so it points at a system tomato
does not manage — and tomato wipes every resource before every scenario,
which would destroy its data.

Give it a container: so tomato runs it, or, if it genuinely is a throwaway system, say so:

resources:
  app-db:
    type: postgres
    allow_destructive_reset: true   # yes, wipe it before every scenario
    options:
      host: scratch-db.internal

The acknowledgement stops the refusal, not the warning: tomato validate still flags it, and every run prints which unmanaged resources it is about to wipe. Only the stateful types are affected — postgres, redis, kafka, rabbitmq, s3, scylladb and their aliases. An http-client pointed at a remote URL holds no data, so it needs nothing.

Container naming and reuse¶

Every container and network tomato creates is named and labelled, so docker ps says whose they are:

tomato-<project>-<run>-<container>     e.g. tomato-9a987f0c-06e8b19f-postgres
tomato-<project>-<run>-net             the network they share

<project> is derived from the config file's path, so two projects on one Docker host never collide; <run> is the run id, so two runs of the same project can go at once. Labels tomato.managed, tomato.scope, tomato.run and tomato.role are set on all of them, which is enough to find leftovers:

docker rm -f $(docker ps -aq --filter label=tomato.managed=true)

Setting	Default	Meaning
`settings.containers.reuse`	`false`	Keep containers between runs instead of starting fresh ones

With reuse: true the run id drops out of the names, so a rerun finds what the last run left behind and skips container startup entirely. Resets still run before every scenario, so scenarios stay isolated from each other. What you give up: a container an earlier run corrupted is no longer thrown away, and two runs of the same project can no longer run at the same time. tomato leaves the containers running when the suite ends and prints the command to remove them.

Wait strategies¶

Type	Description	Fields
`port`	Wait for port to be ready	`target`
`log`	Wait for log message	`target` (regex)
`http`	Wait for HTTP endpoint	`path`, `target` (port), `method`
`exec`	Run command in container	`target` (command)

Resetting state¶

Resetting is configured per resource, not per container — see options.reset_strategy under Resources and on each resource's page. How often it runs is not configurable: every resource is reset before every scenario. A scenario that needs a starting state builds it in a Background, which runs after the reset.

The opt-outs v2.3.0 removed, settings.reset, a resource's reset and reset_strategy: none, are refused at load with this explanation. A config written for an earlier release fails to load rather than having a resource it used to leave alone wiped before every scenario.

Resources¶

Define handlers for test steps. The key is the name steps use to refer to the resource ("app-client" sends …), so it should say whose it is and what role it plays: orders-client, stripe-stub, orders-db, not api, mock or db. See Naming resources.

HTTP¶

resources:
  app-client:
    type: http
    base_url: http://localhost:8080
    options:
      timeout: 30s

Option	Default	Description
`timeout`	`30s`	Per-request timeout. For an event stream it bounds only the wait for the response headers; the stream stays open until the scenario closes it, and each receive step's `within` bounds its wait
`no_redirect`	`false`	Return the redirect response instead of following it
`port`	`8080`	Container port the base URL is built from. Only with `container`
`scheme`	`http`	Scheme for the base URL built from `container`
`health_path`	(none)	Path polled once at startup to check the service answers

Request headers are set by steps ("app-client" header "Authorization" is "..."), not here, and are cleared between scenarios.

PostgreSQL¶

resources:
  app-db:
    type: postgres
    container: postgres
    database: test
    options:
      user: test
      password: test
      # Reset every table in these schemas (if neither this nor tables is set,
      # every table in public)
      schemas:
        - app
        - "plugin_*"       # any schema whose name matches
      # Reset these tables too
      tables:
        - public.outbox
      # Never truncate these tables (added to the defaults below)
      exclude:
        - countries        # reference data seeded by a migration

Reset behavior¶

By default, PostgreSQL resources truncate all tables in the public schema before each scenario. You can control this behavior:

Option	Description
`schemas`	Truncate every table in these schemas instead of public. A name containing `*` is a pattern
`tables`	Tables to truncate. Alone, only these are truncated; with `schemas`, these are truncated as well
`exclude`	Extra tables to never truncate, on top of the defaults: `table` in any schema, or `schema.table`

schemas is for an app that keeps its tables in more than one schema, a schema per module or per plugin, so the suite doesn't have to list every table and keep that list in step with the migrations. The tables are looked up in the catalog at every reset, so a table or a schema a migration adds is reset without a change to tomato.yml.

A schema named in schemas has to exist when the first scenario starts, which it does when the app creates it in its migrations at startup; otherwise the reset fails and says which schema is missing, rather than leaving its tables to leak state between scenarios. A pattern such as plugin_* may match no schema at all, since the schemas it stands for come and go: one a scenario creates is reset from the next scenario on. Quote a pattern in YAML.

Migration history tables are never truncated when the tables are found from a schema, in whichever schema they are, so migration tools still see their migrations as applied:

Tool	Tables
golang-migrate, Rails, sqlx	`schema_migrations`
goose	`goose_db_version`
Flyway	`flyway_schema_history`
Liquibase	`databasechangelog`, `databasechangeloglock`

Reference data that migrations insert (countries, roles, permissions) is truncated like any other table. Add those tables to exclude so scenarios don't have to re-seed them.

A table listed in tables is always truncated, even when exclude or the list above names it. So when tables is set without schemas, exactly those tables are truncated, and both exclude and the migration-table list are ignored.

All the tables are truncated in one TRUNCATE ... RESTART IDENTITY CASCADE statement, so a generated id starts again from 1 in every scenario.

The container field automatically provides the connection details - tomato resolves the container's host and port at runtime.

ScyllaDB / Cassandra¶

resources:
  app-db:
    type: scylladb          # or: cassandra
    container: scylla
    options:
      keyspace: app         # created if missing, then used for unqualified tables
      schema:
        - ./fixtures/schema.cql
      exclude:
        - countries         # reference data kept across scenarios

Every table in the keyspace is truncated before each scenario, except those in exclude. See ScyllaDB Configuration for all options.

Redis¶

resources:
  app-cache:
    type: redis
    container: redis
    options:
      db: 0
      password: ""
      reset_strategy: flush  # flush or pattern
      reset_pattern: "*"     # For pattern strategy

Kafka¶

resources:
  app-events:
    type: kafka
    container: kafka
    options:
      topics:
        - events
        - notifications
      partitions: 1
      replication_factor: 1
      reset_strategy: delete_recreate
      # Optional: Avro via Confluent Schema Registry
      schema_registry:
        container: schema-registry   # or url: http://localhost:8081

Field	Default	Description
`container`	—	The container tomato starts and connects to
`brokers`	—	Broker addresses, instead of `container`. A broker tomato does not start needs `allow_destructive_reset: true`, since its topics are recreated before every scenario

See Kafka: Avro and Schema Registry.

WebSocket¶

resources:
  app-ws-client:
    type: websocket
    url: ws://localhost:8080/ws
    # Or use container
    container: app
    options:
      port: "8080"
      path: /ws
      protocols:
        - graphql-ws
      headers:
        Authorization: Bearer token

gRPC¶

resources:
  app-grpc-client:
    type: grpc
    address: "localhost:9090"   # or: container: myservice
    options:
      timeout: 30s

The server must register the gRPC reflection service. See gRPC Configuration.

gRPC server¶

resources:
  users-stub:
    type: grpc-server
    options:
      port: 9995                              # fixed, so app.env can name it
      descriptor_set: proto/users.binpb       # buf build -o proto/users.binpb

app:
  env:
    USERS_ADDR: "{{.users-stub.address}}"     # localhost:9995

Stubs the gRPC services the app calls. See gRPC Server.

S3¶

resources:
  uploads-storage:
    type: s3
    container: minio
    options:
      buckets:
        - uploads
      reset_strategy: purge   # purge or delete

See S3 Configuration for MinIO and LocalStack setup.

Shell¶

resources:
  shell:
    type: shell
    options:
      timeout: 30s
      workdir: ./scripts
      env:
        PATH: /usr/local/bin

Hooks¶

Execute actions at different lifecycle points.

hooks:
  before_all:
    - sql_file: ./fixtures/schema.sql
      resource: app-db
    - shell: ./scripts/setup.sh

  after_all:
    - shell: ./scripts/cleanup.sh

  before_scenario:
    - sql: "DELETE FROM events"
      resource: app-db

  after_scenario:
    - exec: redis-cli FLUSHDB
      container: redis

Hook types¶

Type	Description
`sql`	Execute SQL query
`sql_file`	Execute SQL file
`shell`	Run shell command
`exec`	Run command in container

Features¶

Configure feature file discovery.

features:
  paths:
    - ./features
    - ./integration/features
  tags: "@smoke and not @slow"

Tag expressions¶

Expression	Description
`@smoke`	Scenarios tagged with @smoke
`@smoke and @api`	Both tags
`@smoke or @api`	Either tag
`not @slow`	Exclude slow tests
`@smoke and not @wip`	Smoke tests, excluding WIP
`(@smoke or @api) and not @slow`	Parentheses group terms

not binds tighter than and, which binds tighter than or. godog's legacy syntax (@smoke && ~@slow, , for or) is still accepted.

Environment variables¶

Use environment variables anywhere in the config:

containers:
  postgres:
    image: postgres:${POSTGRES_VERSION}
    env:
      POSTGRES_PASSWORD: ${DB_PASSWORD}

Only $VAR and ${VAR} are expanded. Shell default syntax such as ${VAR:-15} is not supported: the whole VAR:-15 is read as the variable name, so the value expands to an empty string. Set the variable, or keep the default in tomato.yml.