There is exactly one development board on my bench. There are rather more things that want to touch it: automated builds, several coding agents running on different machines, and me, standing in front of it with a probe in my hand.

Two of those touching it at the same time is not a race you lose gracefully. This post is the lock that stops that – why it is built the way it is, and the four things that testing it under real contention proved were broken.

1. Why this needs a lock at all

The board is an EK-RA8D2, a Renesas evaluation kit: the microcontroller comes already mounted with its power, connectors and debug hardware wired up, so you can start writing firmware instead of designing a circuit board.

Programming it means writing into its MRAM – magnetoresistive RAM, the chip’s non-volatile program storage. Non-volatile means it survives power loss, the way flash memory in a USB stick does. When people say “flashing the board”, that is the memory they are writing.

Now picture two machines doing that at once. You do not get one program or the other. You get a corrupted mixture that is neither, and a board that may not boot far enough to tell you so.

The console makes it worse in a subtler way. The board reports what it is doing over a UART – a plain one-way stream of text, the same idea as a printer cable, and the thing automated tests read to decide whether a test passed. If two tests are watching that stream at once, their output interleaves. A test scraping for the word PASS can read somebody else’s PASS and report success for a build that actually failed. That is the dangerous category: not a crash, but a result that looks real and is not.

And the whole assembly is fed by a smart plug so a machine can power-cycle it remotely. A power cut in the middle of someone else’s measurement destroys the measurement without destroying the appearance of one.

The bench is also indivisible, which turns out to matter for the design. The debug probe on this board – a SEGGER J-Link, the device that halts the processor and writes its memory – is built onto the board rather than being a separate dongle, and the same USB connection carries the serial console. So you cannot hold “the console” while somebody else holds “the debugger”, because they are one wire. You cannot hold “the board” while somebody else cuts the power. There is one lock, and it covers the board, the ESP32-C6 radio module wired to its expansion header, the probe, the USB ports, and the plug.

2. The lock is a flock, which is the whole design

flock is short for file lock. It is a Linux facility, available both to C programs as a system call and from the shell as a command.

The idea is simple: you open a file, then ask the kernel for exclusive use of it. One process holds it; everybody else waits or is turned away.

flock /tmp/some.lock -c 'echo I have it; sleep 10'

Run that in two terminals and the second one waits. Nothing is ever written into the file – it is a place to meet and take turns, not storage.

What makes it the right tool here is when the lock ends. It is attached to the open file, not to the program. When the kernel hands you something you have opened it gives you a small number to refer to it by – a file descriptor – and the lock lives on that. The moment that descriptor goes away, the kernel releases the lock.

Which means: the program exits normally, released. It is killed outright, released. The machine loses power, released, because nothing survives a reboot. The dying program does not cooperate, does not run cleanup code, does not need to know it is dying. Somebody else pulled its plug and the lock was already gone before anyone noticed.

So the design is one sentence – the lock is a flock on the bench host and a holder is a live process – and the useful consequence is that a stale lock is not a state that can exist. Not “we handle stale locks well.” The situation is structurally impossible.

That matters more than it sounds. A stale lock is the failure that teaches people to work around a lock, and a lock people work around is worse than no lock at all, because it also supplies false confidence.

3. What I would have built instead, and why each is worse

The obvious first attempt is a lock file: create a file when you take the bench, delete it when you are done. It works until something crashes between those two steps, and then the file sits there forever and the bench is locked by a program that no longer exists.

The usual patch is to write your process ID into that file, so a later arrival can check whether the holder is still alive. But process IDs get recycled – after enough churn, the number that used to be a dead flashing script belongs to somebody’s text editor, and now your check says the bench is busy forever.

So people reach for a lease with an expiry, a TTL (“time to live”). Now you must choose the timeout, and both directions are bad. Too short and you seize the bench from a live actor halfway through writing memory. Too long and one crash blocks the bench for the whole duration.

A heartbeat – the holder periodically saying “still here” – just moves the same guess somewhere else. Now you decide how many missed pings mean death.

Every one of those is an attempt to infer whether the holder is alive. The kernel already knows, exactly, with no policy and no guessing. Rather than build a liveness detector, ask the thing that has the answer.

flowchart TD A["actor runs a bench command"] --> B["guard: may I have the bench?"] B --> C{"is the lock free?"} C -->|"yes"| D["hold taken, written to the journal"] C -->|"no"| E["print who holds it, and why"] D --> F["wait for a previous actor's tools to exit"] F --> G["the real work runs"] G --> H["kill anything of ours still running"] H --> I["file closes, kernel drops the lock"] E --> J["wait up to the budget, then refuse"]

4. Three answers, because two is not enough

Every bench command comes back with one of three results: it is yours, somebody else has it, or I could not tell.

That third one is the interesting one, and it is easy to leave out. If the tool cannot reach the bench host, or cannot read the record of who holds what, it does not get to shrug and continue. It refuses.

Written down that seems obvious. It is also exactly the thing that becomes “ignore the error and carry on” when somebody is in a hurry. And the failure it prevents is not a tool printing an error message – it is a tool that cannot see the bench, assumes therefore that nobody is on it, and programs the board underneath somebody who is.

I don’t know is a real answer. Folding it into either yes or no is how you end up confidently wrong.

5. Killing an ssh client releases the lock

This is the claim everything rests on, so it gets a test rather than an assertion.

Picture an agent on a laptop running a bench command over ssh – Secure Shell, which runs a command on another machine across the network. The lock lives on the bench host, not the laptop. If the laptop’s ssh program dies – terminal closed, network dropped, process killed – the far side has to notice, or the lock outlives the actor that owns it.

It works because of a small piece of plumbing. The remote program sits waiting on its standard input: the default channel a program reads from, which in this case is the ssh connection itself. Kill the client and the connection closes. A read on a closed connection returns end-of-file – the signal that nothing more is coming. The remote program exits, its file descriptor closes, and the kernel drops the lock. No cleanup handlers. No cooperation from the side that died.

Here it is measured. kill -9 sends SIGKILL, the one signal a program cannot catch, block or ignore – the kernel simply stops it, with no chance to tidy up. I sent that to the local client and left the remote program alone:

BEFORE:  remote holder alive; it owns the lock file; lock = HELD
ACTION:  kill -9 the LOCAL ssh client -- remote program untouched, no handlers
AFTER:   t+0.29s: FREE, holder process gone
         our own pipe to it was STILL OPEN, so the release cannot have come from us

That last line is the control, and it is the reason the measurement means anything. If our side had closed something, the release would prove nothing about the mechanism – we would just be watching ourselves let go.

It ships as a permanent self-test, plus a second case that kills a guarded script and requires the lock to drop within two seconds with no cleanup handler having run. Without that second condition, the file-closing mechanism could be quietly dead while a cleanup handler covered for it, and the test would still pass – which would be a test that proves the wrong thing.

6. A person is not a script

An automated agent that stops responding should lose the bench. A person who wanders off mid-debug to get coffee should not.

So the two are held differently. A person’s hold is a shell – you get a command prompt, and you keep the bench for as long as that prompt lives. Nothing takes it from you automatically, deliberately. An agent’s hold wraps one command and ends when that command ends.

Taking the bench from whoever currently has it has to be possible, or a forgotten shell blocks the bench indefinitely. But it is a separate command, it is recorded, and the confirmation phrase is long and awkward on purpose. A preemption you can type by accident is one that will eventually happen by accident.

Relatedly, the “give it back” command refuses to give back a hold it does not own. Releasing somebody else’s lock is not releasing, it is taking, and it goes through the command that says so out loud.

One bug from this area is worth telling. The preempt command originally hardcoded its actor type as human. So an agent typing one word would have seized the bench as a person – the one kind of holder nothing may take from without acknowledgement. It was found by testing what happens when the tool should say no, rather than testing that it says yes. That is a habit worth having.

7. What testing it under real contention found

Everything above I could have written in advance. The next four are why you test the thing instead of reasoning about it. All four were live, on a lock I believed was finished.

The harness drives four genuinely independent machines at the real bench, each running a real flash through the real guard, started together within 47 milliseconds.

7.1 There is no hardware interlock

The negative control – running the machines with the lock deliberately bypassed, to confirm the test could detect a collision at all – was supposed to be a formality.

All four got onto the debug probe simultaneously, for about fourteen seconds, with four programs reading one console.

More pointedly: two debug sessions on this probe both connect successfully and both report success. Neither notices the other. Nothing in the hardware or the driver stands in the way.

So the software lock is not a convenience on top of some deeper protection. It is the only thing between two actors and a corrupted board.

7.2 The guard had no fence

One entry point stopped its work when the holder died. The shared guard – the one all twenty-six bench scripts actually go through – did not.

Measured: a programming session ran seventy-five seconds past its own release, writing memory, while three other machines took the lock in turn and wrote underneath it.

The lock was working perfectly the whole time. It was handing the bench over on schedule, to the right actor, in the right order – while the previous occupant’s tooling was still running. Correct bookkeeping, corrupted board.

The guard now kills anything of its own still alive before the hold ends. That takes 0.07 seconds.

7.3 “You may have it” is not “they have stopped”

Related but distinct. The lock answers may I have the bench. It does not answer has the previous occupant actually finished. A handover could complete while the outgoing actor’s programming tool was still winding down.

So the bench host now waits for any leftover tools to exit before handing over – recorded, time-bounded, and refusing rather than proceeding if they will not go. The fence in 7.2 deals with a holder that is dead; this deals with one that is merely slow.

7.4 The wait knob was read but never set

The refusal message told you to wait for the bench. The length of that wait came from a setting that nothing, anywhere, ever assigned – so it was always zero, and every command failed instantly the moment the bench was busy.

The tool said “wait for it” and then refused to.

I like this one because it is not a subtle concurrency bug. It is a knob that was never connected, sitting in plain sight, doing nothing. It survived because nobody had run two things at once for long enough to be annoyed by it.

8. Proving exclusion without taking anyone’s word for it

Claiming mutual exclusion is easy. Proving it from the actors’ own logs is circular – an actor that silently skipped its work looks exactly like one that waited politely.

So who-did-what is worked out twice, by unrelated means, and the two are required to agree. Once from the programming tool’s own record of which memory addresses it wrote. Once from the kernel’s view of which network peer owns the process doing the writing. Neither derives from the lock’s own bookkeeping, which is the point: the lock does not get to be its own witness.

They agreed. Gaps of 2.4, 1.8 and 3.6 seconds between sessions, zero overlap, zero console collisions, and every programming session sitting inside the hold attributed to the same actor. Nineteen of nineteen claims held.

Fairness came out as clean rotation across twelve grants – each waiter served in turn, longest wait thirty seconds – and it held even when an unrelated fifth actor joined halfway through.

9. What is not proven

A writeup that lists only what worked is marketing.

The power-cut case is untested. A machine that vanishes without its network connection closing sends no shutdown packet, so nothing tells the far end it has gone. Only the ssh server eventually notices, through keepalive probes – periodic “are you still there” messages, set here to every 15 seconds with four allowed to go missing. I only ever staged kill -9, which closes the connection cleanly and politely. A real power cut is a different, slower path.

Scale is untested too: four machines, occasionally five, against a bench that nominally serves around twenty actors.

Fairness is an observation, not a promise. flock guarantees nothing about ordering; the clean rotation is a property of this kernel, not a contract. It matters little here, because an unfair queue would show up as a refusal when the wait budget runs out, never as a silent hang.

And interleaved programming was never deliberately staged. The negative control is read-only by design. The memory writes that landed under a dead holder’s live session in 7.2 were observed in passing during the pre-fix run, not produced by an experiment.

There is also one deliberate conservatism: a program started by a guarded script inherits the open lock file, and so keeps the hold alive. That is the safe direction to be wrong in, it is bounded by the wait budget, and it is written down where somebody will find it.

10. What generalises

Three things I would take to any shared-hardware setup.

Let the kernel decide who is alive. Every timeout and heartbeat scheme is an attempt to re-derive something the operating system already knows exactly.

Make “I don’t know” a distinct answer, and refuse rather than assume. Most of the harm in this class of system comes from a tool that cannot see the truth and picks the convenient answer.

Test what happens when the tool should say no. Every one of the four defects above was invisible to a test that takes the lock, does some work, and gives it back. They appeared only under real contention, from real machines, with the lock deliberately bypassed as a control. The lock appeared to work perfectly for the entire time it was allowing a dead holder to write memory underneath three other actors.