INC-33871: single log line from kestrel-04

Severity 4 (informational) · Opened 2026-02-14 07:40 · Closed — Component Ledger (log aggregation) · Reporter on-call, ingest

Summary

At 03:17:09 on 2026-02-14, Ledger ingested one line attributed to kestrel-04.fen.internal. That host was decommissioned on 2019-11-02. Its drives went through the shredder on 2019-11-08 and the certificate for the shipper was revoked the same afternoon.

The line was accepted because it was correctly signed.

The line

Feb 14 03:17:09 kestrel-04 kernel: Command line: BOOT_IMAGE=/vmlinuz-6.1.0-18-amd64 \
  root=UUID=8c9a1f42-6b70-4a11-9d3e-c7f0a6b21d55 ro quiet net.ifnames=0

Ledger's own envelope around it:

ingest_ts   2026-02-14T03:17:09.441Z
source_ip   192.0.2.104
cert_sn     4A:19:C0:8E:2B:77
cert_cn     kestrel-04.fen.internal
cert_state  REVOKED 2019-11-08T15:22:41Z
verify      SIGNATURE OK
seq         1

Findings

1. The signature verifies. The private key was on the shredded drives. It was not escrowed; the escrow policy came in the following year. Ledger checks revocation and warns, but does not drop, which is why the line landed instead of being refused — that behaviour is being changed under WLN-2251 and is not the interesting part of this ticket.

2. 192.0.2.104 is not assigned. It was kestrel-04's address. The subnet is still routed; the address has been in the free pool since 2019 and has never been leased. The top-of-rack switch has no MAC in its table for it, and did not at 03:17 — we have the five-minute poll from 03:15 and from 03:20.

3. seq 1. The shipper numbers lines from one at each boot. This was the first line of a boot.

4. The kernel version in the boot line is 6.1.0-18-amd64. That kernel was released in 2023.

What we did

Held the line, pulled the ingest window either side of it (nothing), searched Ledger for any other line from that certificate since 2019 (none), searched for the UUID in the boot line across every filesystem in the estate (no match), and asked facilities to check the rack position. The rack position has held a blank panel since 2019 and holds a blank panel now.

Open

The ticket is open because I do not know what closing it would mean. If I close it as not reproducible, that is a claim about the future.

Ingest has been left as it is, deliberately. If there is a second line I would rather have it than not.

Related: stories/session-0, stories/eleven-year-check. On why the provenance fields on a record are the first thing to read and the last thing to trust, see machinery/provenance.