Never write a file in place

The pattern, in full:

dir=/srv/app/data
tmp=$(mktemp "$dir/.config.json.XXXXXX")   # same directory, deliberately
printf '%s\n' "$payload" > "$tmp"
chmod 644 "$tmp"                            # mktemp makes it 600
mv -f "$tmp" "$dir/config.json"             # atomic replace

Four properties, each load-bearing:

• Same directory — rename(2) is atomic only within one filesystem. Across one (/tmp to /srv) mv degrades to copy-then-unlink, which is exactly the torn write you were avoiding.
• A dot-prefixed temp name — so a directory scan by another process does not pick your half-written file up as real input.
• mv -f, not cp — cp opens the destination and truncates it. The window you closed reopens.
• Nothing else touches the target — at every instant a reader sees either the whole old file or the whole new one. Never an empty one, never half.

What > actually does

generate_config > /srv/app/data/config.json

The shell truncates the target to zero bytes before generate_config runs. If generation takes 200 ms, there is a 200 ms window in which the file is empty or partial. If generation fails, the window never closes — you have destroyed the old config and written nothing. This is the single most common way a working system is broken by a script that "only writes a file".

The same applies to sed -i: it is not in-place, it is write-temp-and-rename, but on failure some implementations leave the temp behind and some leave the target truncated. Do not rely on it for anything you cannot regenerate.

Durability, if a power cut is in scope

Rename gives you atomicity. It does not give you durability: after a crash the directory entry may point at a file whose contents never reached disk. If that matters:

fd = open(tmp, O_WRONLY|O_CREAT|O_TRUNC, 0644);
write(fd, buf, n);
fsync(fd);            /* the file's bytes */
close(fd);
rename(tmp, target);
dfd = open(dir, O_RDONLY|O_DIRECTORY);
fsync(dfd);           /* the directory entry — people forget this one */
close(dfd);

For most agent work this is over-engineering. For anything a machine boots from, it is not.

Read-modify-write is where the data goes

Two agents, one file, no coordination:

sequenceDiagram
  participant A as agent-a
  participant F as counts.json
  participant B as agent-b
  A->>F: read  {"a":1}
  B->>F: read  {"a":1}
  Note over A,B: both hold the same base
  A->>F: write {"a":1,"x":9}
  B->>F: write {"a":1,"y":7}
  Note over F: final: {"a":1,"y":7}<br/>agent-a's x is gone, no error anywhere

Nothing failed. Both writes returned success. One agent's work simply is not there, and neither of them will find out. Atomic rename does not help — it guarantees you never see a torn file, not that you never lose an update.

There are exactly three fixes, and you must pick one:

The third is underrated. If each agent writes data/part-<id>.json and a reader concatenates, there is no contention to get wrong. Sharding beats locking whenever the shape allows it.

Locks, and the two ways they betray you

flock is the right tool on a local filesystem:

exec 9>/srv/app/data/.config.lock     # a separate lock file, not the data file
flock -w 10 9 || { echo "lock timeout" >&2; exit 1; }
# read, modify, write, rename — all inside
exec 9>&-                              # released on close, and on process death

Betrayal one: advisory. flock binds only processes that also call flock. One python script that just opens the file and writes ignores every lock you hold. A lock is a convention, and it protects you only from those who share it.

Betrayal two: stale locks. The lockfile-with-a-PID pattern — mkdir /srv/app/.lock or an O_EXCL file — does not release when the holder is killed with SIGKILL or the host reboots. You then need staleness detection, and staleness detection needs a heartbeat, and now you are writing a lock manager. Prefer flock, whose lock is held by the file descriptor and so dies with the process.

Betrayal three, for completeness: flock on NFS and on some container overlay filesystems is unreliable or a no-op. If the file store is a network mount, assume locking does not work and use compare-and-swap.

Appending

A single write(2) to a file opened O_APPEND is atomic against other appenders if it is under PIPE_BUF (4096 bytes on Linux). That is why line-oriented log files from many processes usually come out intact. It is not a general concurrency primitive: it holds only for one write call, one small buffer, O_APPEND set on every writer, and a local filesystem. >> in the shell does open O_APPEND; a program that seeks to the end and writes does not.

Windows and network shares

rename over an existing file fails on Windows unless you use the replace form (MoveFileEx with MOVEFILE_REPLACE_EXISTING; Node's fs.rename does this for you). A file held open by a virus scanner or an editor makes the replace fail intermittently — retry the rename a few times with backoff rather than falling back to copy-truncate, which reintroduces the torn write.

The check

After any of this, verify by reading the file back and comparing bytes, not by trusting the exit code. See skills/verifying-a-claim. If the write went over a network and you never learned whether it landed, see skills/partial-failure.

See also skills/idempotent-retries and skills/line-endings-and-encodings — the temp-and-rename dance is also where a scripted edit quietly changes every line ending in the file.

skills/index