Shipping updates to a box you cannot SSH into

For a while, every RFID install came with a Windows PC. The reader read tags, the PC ran the software, and if anything went wrong someone could sit at that PC. This year I moved the whole stack onto the reader board itself. A Node server, a SQLite database, the rules engine and the React console all run on an ARM Linux board bolted to a wall.

That deletes the PC, which was the point. It also deletes the person sitting at the PC, which I did not think about hard enough at first.

The problem in one sentence

The board sits on a customer network, behind their NAT, with no port forwarded to it, and I am several hundred kilometres away.

So there is no SSH. There is no "just restart it". If I push an update that fails to start, the unit is dead and somebody has to physically drive to the site. That single fact ended up shaping most of the interesting engineering.

Getting a route in without asking for one

The first piece is a tunnel. Every board holds one outbound WebSocket open to a hub we run. Outbound is the trick: no customer has to open a firewall rule, because from their network's point of view the box is just making a normal client connection.

Once that socket is up, operator HTTP requests get replayed down it and the responses come back the same way. The console renders the board's own web UI in an iframe. It looks like you are on the local network with it, and you are not.

The naive version of this was slow enough to be useless. Every asset, every JavaScript chunk, every font, was crossing the board's WiFi once per operator per session. The board has better things to do than serve the same 2 MB bundle repeatedly. Because Vite gives assets content-hashed filenames, the hub can cache them safely, so I added a bounded 64 MB store at the hub. A hashed filename is immutable by construction: if the content changes, the name changes. Caching it forever is correct.

Updates, assuming the update will fail

The update path is where I spent the most time being pessimistic.

Send it in chunks. The tunnel has a 20 second request timeout. A whole bundle does not fit inside that on a slow link, so the upload is split, and no single request is big enough to time out.

Verify before unpacking. The sha256 is checked before a single file is written. A truncated upload should fail loudly at the checksum, not halfway through overwriting a working install.

Swap contents, not directories. My first version renamed the old directory and moved the new one into place, which is the standard trick. It broke, because the supervising shell holds its working directory by inode. Rename the directory and the supervisor is now sitting in a directory that no longer has a name. Swapping the contents in place, leaving the directory itself alone, avoids that entirely.

Assume it does not come back. After the swap, a detached watchdog waits for the new build to bind its port and write a marker file. If the marker never appears within the timeout, the watchdog restores the previous tree and starts it. The important detail is that the watchdog is detached: it must survive the death of the thing it is watching.

That last one exists because of something that happened on the bench. After a reboot the app came up fine and sshd did not. On a desk that is a curiosity. On a customer wall it is a site visit. Once I had seen the two fail independently, I stopped treating "the box is up" and "I can get into the box" as the same fact.

What I would tell myself at the start

Deployment stopped being a step at the end and became a design constraint. Roughly:

• If the only recovery path is physical access, you do not have a recovery path.
• Anything that must survive a failure has to run outside the thing that is failing.
• Test the rollback more than the update. The update gets tested every time you ship. The rollback only runs on your worst day.

None of this is novel. It is standard practice in any serious deployment pipeline. The difference on hardware is that the cost of getting it wrong is not a red build, it is somebody in a car.