Check the bytes, not the rendering

After any scripted edit, run these three. They take one second and they catch almost every silent corruption:

file -i path            # encoding and whether it now thinks the file is binary
head -c 3 path | od -An -tx1     # ef bb bf = a BOM you did not want
tr -cd '\r' < path | wc -c       # count CR bytes (see the trap below)

The file looks fine in every viewer. That is the problem: every one of these faults is invisible in rendered text and fatal to a parser.

CRLF in a shell script

The symptom is a syntax error that makes no sense, because the error message contains an invisible CR that resets the cursor:

$ ./deploy.sh
./deploy.sh: line 2: $'\r': command not found
./deploy.sh: line 3: syntax error near unexpected token `$'{\r''

Or, worse, set -euo pipefail becomes set -euo pipefail\r, which is an unknown option, and the script runs on without -e. Every later failure is now silently ignored.

Cause: the file was written by a tool on Windows, or by a PowerShell redirection, or by a git checkout with core.autocrlf=true. Fix and verify:

sed -i 's/\r$//' deploy.sh          # or: dos2unix deploy.sh
bash -n deploy.sh && echo "parses"  # ALWAYS do this after

bash -n parses without executing. Run it after any scripted edit to any shell script or unit file; it is the cheapest check in this document.

The counting trap

grep -c $'\r' file      # WRONG: counts LINES containing a CR, not CRs
tr -cd '\r' < file | wc -c   # right

And $'\r' itself does not survive every shell — sh and PowerShell do not understand it, so the same command copied between environments quietly becomes a search for the two characters $ and \r. When it matters, count with a program that has no quoting layer:

node -e 'const b=require("fs").readFileSync(process.argv[1]);
         let n=0; for(const c of b) if(c===13) n++;
         console.log("CR",n,"LF",b.filter(c=>c===10).length)' file

A CR count that jumps in a directory that was all-LF is a change you made, not one that was there.

Keep it from coming back

# .gitattributes
* text=auto
*.sh   text eol=lf
*.bat  text eol=crlf
*.png  binary

eol=lf on shell scripts means they are LF in the working tree regardless of platform. Set it once and this class of bug stops recurring.

The BOM

Three bytes, EF BB BF, at the start of a UTF-8 file. Effects:

• #!/bin/sh is no longer at byte 0, so the kernel does not see a shebang: ./script: line 1: #!/bin/sh: No such file or directory.
• A strict JSON parser fails at position 0 with "unexpected token".
• A YAML front-matter block does not start with --- any more.
• The first key of a CSV header is id, not id — and every lookup by "id" returns undefined while the file looks perfect.

Detect and strip:

head -c 3 file | od -An -tx1        # ef bb bf
sed -i '1s/^\xEF\xBB\xBF//' file

Producers to watch: PowerShell Out-File/Set-Content on older versions, Excel "CSV UTF-8", and some editors' "UTF-8 with signature".

Mojibake, and how to read it

The corruption is diagnosable from its shape:

The first three are recoverable — re-decode with the encoding that was actually used. The fourth is not: nothing in the file records what the character was. That is why "just force ASCII" is a destructive fix.

The general rule: decode once at the boundary, work in one encoding, encode once on the way out. Most mojibake is a decode that happened twice or not at all.

The missing final newline

A file whose last line has no \n:

• while read line in a shell loop silently drops that last line.
• cat a b > c glues the last line of a onto the first of b.
• Every diff shows \ No newline at end of file, and any later edit shows a one-line change that is really zero.

Add one; POSIX says a text file's last line ends with a newline.

Invisible characters that are not line endings

If a tool starts calling a text file binary, look for these before anything else. A scripted rewrite is the usual source:

grep -nP '[\x00-\x08\x0B\x0C\x0E-\x1F]' file    # control chars
grep -nP '\xC2\xA0' file                         # non-breaking space
grep -nP '\xE2\x80[\x8B-\x8F\xAA-\xAE]' file     # zero-width and bidi marks
grep -nP ' +$' file                              # trailing whitespace

A non-breaking space inside YAML indentation, or a smart quote where a straight quote should be, produces an error message that points at the right line and describes the wrong problem.

The habit

Every scripted edit, three steps, no exceptions:

1. make the change
2. parse it        (bash -n / node --check / jq . / python -m py_compile)
3. read the region back and compare bytes, not appearance

Step 3 is the one that catches the edit which parses fine and means something else. See skills/escaping-through-shells for how the edit got mangled in the first place, and skills/verifying-a-claim for why step 2 alone is not enough.

skills/index