PIR-0918: quorum held

Cluster corvid (config store) · Incident 2026-08-11 04:02–04:29 Impact none · Author platform on-call · Status review complete, one action item outstanding

What happened

A cooling failure in row 4 tripped the thermal cut-out on two of corvid's three members, corvid-a and corvid-b, within eleven seconds of each other. Both nodes powered off hard at 04:02:17 and 04:02:28.

A three-member cluster loses quorum when it loses two members. corvid did not lose quorum. Reads and writes continued through the event with no elevated latency, the config store served 2.1 M requests during the window, and the first anybody knew of the outage was the facilities alarm.

The cluster as drawn in the review

              row 4                             row 9
      ┌──────────────────┐             ┌──────────────────┐
      │  corvid-a        │             │  corvid-c        │
      │  192.0.2.71      │             │  192.0.2.73      │
      │  U14, rack 4-2   │             │                  │
      │  ── OFF 04:02:17 │             │  ── term 1       │
      └────────┬─────────┘             └────────┬─────────┘
               │                                │
      ┌────────┴─────────┐                      │
      │  corvid-b        │                      │
      │  192.0.2.72      │                      │
      │  U15, rack 4-2   │                      │
      │  ── OFF 04:02:28 │                      │
      └────────┬─────────┘                      │
               │                                │
               └────────────────┬───────────────┘
                                │
                        ┌───────┴────────┐
                        │  quorum 2 of 3 │
                        │  HELD 04:02:28 │
                        └────────────────┘

The box for corvid-c is drawn without a rack position because the reviewer could not find one. It is not in the asset register. Facilities walked row 9 during the review with the address in hand and found the U-positions either side of where it should be occupied by a patch panel and a blank.

Findings

F1. corvid-c has been a voting member since the cluster was built on 2021-04-06. The bootstrap log names three members. The procurement record names two servers.

F2. corvid-c has never been the leader. It has voted in all 1,904 elections and has never stood.

F3. Its term counter is 1. Every other member's term counter tracks the cluster's, which is currently 1,904. A member at term 1 should be rejected as stale by the other members. It is not, and the code path that would reject it is covered by a test that passes.

F4. Round-trip time from corvid-a to corvid-c, sampled every 15 seconds since 2021, has a standard deviation of 0.00 ms. It is 0.31 ms and it has been 0.31 ms 10,614,880 times.

F5. During the window in which corvid-a and corvid-b were powered off, the cluster committed 41 entries. The commit index moved from 8,441,902 to 8,441,943. All 41 entries replicated cleanly to corvid-a and corvid-b when they came back at 04:29.

Action items

A2 is outstanding. The register requires a serial number.

Related: stories/eleven-year-check, stories/the-fourth-nameserver.