Testing what your service does when the network fails

resources:
  orders-db:
    type: postgres
    container: postgres
    database: orders
    options: { user: orders, password: orders }

  postgres-network:               # the app's link to Postgres
    type: network
    container: postgres
  redis-network:                  # and to Redis
    type: network
    container: redis

  ⇄ postgres-network  app → 127.0.0.1:61442 → postgres:5432  (seed 98866)
  ⇄ redis-network     app → 127.0.0.1:61443 → redis:6379     (seed 253200)

const db = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
  connectionTimeoutMillis: 1000,      // a hung database costs a second, not forever
  query_timeout: 1000,
});

const cache = createClient({
  url: process.env.REDIS_URL,
  disableOfflineQueue: true,          // fail now instead of queueing while Redis is down
  socket: { connectTimeout: 500, reconnectStrategy: () => 200 },
});

// A query that fails on the connection is tried again; a timeout is not.
async function query(sql, args) {
  for (let attempt = 1; ; attempt++) {
    try {
      return await db.query(sql, args);
    } catch (e) {
      const transient = ["ECONNREFUSED", "ECONNRESET", "57P01"].includes(e.code)
        || /Connection terminated/.test(e.message);
      if (!transient || attempt === 5) throw e;
      await sleep(250 * attempt);
    }
  }
}

// The cache is an optimisation, not a dependency: 200 ms, then Postgres.
async function product(id) {
  try {
    const hit = await within(200, cache.get(`product:${id}`));
    if (hit) return JSON.parse(hit);
  } catch {}
  return (await query("SELECT id, name, price FROM products WHERE id = $1", [id])).rows[0];
}

// The order id makes a retry safe.
await query("INSERT INTO orders (id, product, qty) VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING", …);

Feature: Placing orders when the database misbehaves

  Scenario: A database failover doesn't lose the order
    Given "postgres-network" resets connections for "2s"
    When "orders-client" sends "POST" to "/orders" with json:
      """
      {"id": "ord-1", "product": "tomato-seeds", "qty": 2}
      """
    Then "orders-client" response status is "201"
    And "orders-db" table "orders" has "1" rows
    And "postgres-network" has seen "4" connections

  Scenario: A hung database gets a quick answer, not a hang
    Given "postgres-network" stops responding
    When "orders-client" sends "POST" to "/orders" with json:
      """
      {"id": "ord-1", "product": "tomato-seeds", "qty": 2}
      """
    Then "orders-client" response status is "503"
    And "orders-client" response time is less than "2s"

Feature: Reading products when the cache or the database is slow

  Background:
    Given "orders-db" table "products" has values:
      | id           | name         | price |
      | tomato-seeds | Tomato seeds | 299   |

  Scenario: A cached product doesn't wait for a slow database
    Given "orders-cache" key "product:tomato-seeds" is:
      """
      {"id": "tomato-seeds", "name": "Tomato seeds", "price": 299}
      """
    And "postgres-network" adds latency "2s"
    When "orders-client" sends "GET" to "/products/tomato-seeds"
    Then "orders-client" response status is "200"
    And "orders-client" response time is less than "300ms"

  Scenario: Products are still served while the cache is down
    Given "redis-network" refuses connections
    When "orders-client" sends "GET" to "/products/tomato-seeds"
    Then "orders-client" response status is "200"
    And "orders-client" response json contains:
      """
      {"name": "Tomato seeds"}
      """

  Scenario: A hung cache costs 200ms, not the request
    Given "redis-network" stops responding
    When "orders-client" sends "GET" to "/products/tomato-seeds"
    Then "orders-client" response status is "200"
    And "orders-client" response time is less than "1s"

$ tomato run
6 scenarios (6 passed)
32 steps (32 passed)
11.2s

table orders has 1 rows, expected 0
while the network was faulty:
  postgres-network stops responding (step 1)

  Scenario: An order retried after a timeout is stored once
    Given "postgres-network" stops responding
    And "orders-client" sends "POST" to "/orders" with json:
      """
      {"id": "ord-1", "product": "tomato-seeds", "qty": 2}
      """
    And "orders-client" response status is "503"
    When "postgres-network" is healthy
    And "orders-client" sends "POST" to "/orders" with json:
      """
      {"id": "ord-1", "product": "tomato-seeds", "qty": 2}
      """
    Then "orders-client" response status is "201"
    And "orders-db" table "orders" has "1" rows

expected status 201, got 503
Body: {"error":"try again shortly"}

expected status 201, got 503
Body: {"error":"try again shortly"}
while the network was faulty:
  postgres-network resets connections (step 1, 2s left)
  postgres-network has seen 1 connection this scenario, 0 failed