Heap fragment, core.20841

Ticket OPS-7719 · Process gild, pid 20841 · Signal SIGSEGV Core written 2026-03-02T02:14:06Z · Analysed 2026-03-02T09:30Z

gild crashed once, on one host, and has not crashed since. The core is 812 MB. The fault address is in a region that had been freed and reused, so the stack is not trustworthy and the usual next move is to look at what is around the fault in the heap.

This is what is around it. Offsets are from the start of the mapping.

0x00007f2c1a40  6e 69 67 68 74 6c 79 2d  30 33 30 30 00 00 00 00  |nightly-0300....|
0x00007f2c1a50  72 75 6e 73 3a 20 34 2c  31 31 32 00 00 00 00 00  |runs: 4,112.....|
0x00007f2c1a60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
0x00007f2c1a70  4f 50 53 2d 37 37 31 39  20 2d 2d 20 68 65 61 70  |OPS-7719 -- heap|
0x00007f2c1a80  20 66 72 61 67 6d 65 6e  74 2c 20 63 6f 72 65 2e  | fragment, core.|
0x00007f2c1a90  32 30 38 34 31 0a 46 61  75 6c 74 20 61 74 20 30  |20841.Fault at 0|
0x00007f2c1aa0  78 30 30 30 30 37 66 32  63 31 61 63 30 2c 20 69  |x00007f2c1ac0, i|
0x00007f2c1ab0  6e 20 61 20 72 65 67 69  6f 6e 20 74 68 61 74 20  |n a region that |
0x00007f2c1ac0  68 61 64 20 62 65 65 6e  20 66 72 65 65 64 2e 0a  |had been freed..|
0x00007f2c1ad0  41 6e 61 6c 79 73 65 64  20 62 79 20 6d 2e 72 65  |Analysed by m.re|
0x00007f2c1ae0  79 65 73 20 61 74 20 30  39 3a 33 30 2c 20 77 68  |yes at 09:30, wh|
0x00007f2c1af0  6f 20 77 69 6c 6c 20 6e  6f 74 20 66 69 6e 69 73  |o will not finis|
0x00007f2c1b00  68 20 74 68 69 73 20 73  65 6e 74 65 6e 63       |h this sentenc  |

The fault address is 0x00007f2c1ac0.

Notes for the ticket

The string at 0x1a70 is this ticket's number. The ticket was opened at 09:14 on the 2nd, seven hours after the core was written. The ticket number is issued by the tracker from a sequence; OPS-7718 was opened on 2026-03-01 at 16:40 and OPS-7720 on 2026-03-02 at 10:02.

gild does not construct ticket numbers. It has no network client for the tracker, no configuration mentioning it, and the string OPS- does not appear in the binary — I checked with strings on the deployed artefact and on the build artefact, which have the same digest.

The heap region containing this text was allocated 41 ms before the crash by gild's log formatter, which is the only allocator in the process that produces buffers of this size class. The formatter's output for that millisecond, on disk, is one line:

2026-03-02T02:14:05.981Z INFO  rollup complete, 4112 runs, 0 errors

The database it counts those runs from is /var/lib/gild/state.db, which is the file in stories/mtime.

Disposition

Core retained under OPS-7719. Not to be deleted at the 90-day mark; the retention override is on the ticket.