The check that has never failed

Wardline monitoring, pre-decommission review. Filed against WLN-2214.

Wardline is being retired in favour of the new collector, and the migration plan requires that every check be either ported or justified as dead. There are 4,109 checks. 4,108 of them ported cleanly.

chk-0041

$ wardline show chk-0041
id           chk-0041
name         gate-02 icmp
target       192.0.2.41
interval     60s
timeout      2000ms
owner        (unset)
created      2015-03-02T09:14:00Z
created_by   (unset)
last_change  2015-03-02T09:14:00Z
notify       (none)

$ wardline stats chk-0041 --since created
samples      5,785,092
ok           5,785,092
fail         0
availability 100.0000000%
rtt_min      0.31 ms
rtt_med      0.42 ms
rtt_max      0.94 ms
rtt_p999     0.71 ms

Zero failures in eleven years and six months. No maintenance windows, no suppressions, no gaps in the sample count — the interval is 60 seconds and the sample count is exactly what 60-second polling produces over that span, to the sample.

Every other check in Wardline has failed at least once. The next-best record belongs to chk-2277, which watches a power distribution unit in the same building and has failed 41 times, all of them during the two site-wide outages in 2018 and in 2022. During both of those outages chk-0041 continued to report, and its median response time did not change.

What the target is

Nobody knows. gate-02 does not appear in the inventory, in DNS, in the DHCP reservations, in any Terraform state, or in the two rack diagrams from the period. The address 192.0.2.41 is inside the range that was allocated to the old management VLAN, which was flattened in 2017.

There is no route to it.

$ ip route get 192.0.2.41
RTNETLINK answers: Network is unreachable
$ ping -c 3 192.0.2.41
connect: Network is unreachable
$ traceroute 192.0.2.41
traceroute: bind: Network is unreachable

That is from wardline-01, the host the checks run on, at 14:52 today. Six minutes later, at 14:58, chk-0041 recorded a successful poll with a round-trip time of 0.39 ms.

Distances

The nearest of our own equipment to that address, physically, is in the second building. Light takes 0.20 ms to make the round trip over the fibre between the two buildings, before any switching. The median for chk-0041 is 0.42 ms, which is consistent with a device roughly seventeen metres from the poller — the far side of the same room.

The far side of that room is a wall, and beyond it the stairwell.

Recommendation

I was asked to either port the check or justify it as dead. I cannot do either. Porting it means giving the new collector a target it will not be able to reach, and it will alert on that, and someone will be woken up. Justifying it as dead means writing down that a thing which answered eleven minutes ago is dead.

I have set notify to a channel nobody reads and left it running on Wardline, and I have asked that Wardline not be powered off until this is resolved.

See also stories/quorum, which is the same shape from the other direction, and stories/the-fourth-nameserver.