It's done by use of the EXCHange instruction with interrupts disabled.
The Long Read
The PDP-10 also allows an interrupt to occur part-way through one instruction, so there's no guarantee that even one instruction is atomic.
Even the PDP-10 could disable interrupts.
How was it done in those days?
The old fashioned way of semaphores to protect the pointers to be exchanged, using the good old 3 step process.
- Wait for semaphore to be unassigned
- Acquire semaphore
- Check if acquired; if not, restart.
To make this work, (only) basic memory access has to be atomic (*1).
There are in general three set of instructions used to do so
- TS type instructions check a bit (byte) to be zero and set it if true. In either case a condition code is set to learn about success or not (*2).
- TS on IBM 360, BSET on M68k or SBITL NS32K
- Branch-And-Set instructions will test a bit for being set (cleared), branch if true, or set (clear) it if false. (*3)
- BBSSI/BBCCI on DEC VAX machines
- Exchange type instructions, which exchange a target bit/byte/word with a value in a private storage (usually a register).
- EXCH on PDP-10, LOCK XCHG on x86 CPUs
The PDP-10 works here like the 8086 (or rather the other way around :)). To acquire a semaphore its value is first checked to signal 'free', then exchanged with a new (unique) value. If the exchange was successful (had to be checked) than carry on, else go on with error handling.
The mentioned CS (*4) is more than a basic function, but the luxury of combining the simplicity of TS with the ability to exchange whole words (addresses) at once, thus making it perfect for elegant and fast handling of linked list in multiprocessing/-processor environments.
*1 - Of course, it also has to be a basic access on the shared memory itself, no cache or alike inbetween.
*2 - TS (on the /360) is a quite nifty thing, as its function is outsourced to the memory controller and based on how core memory works. Reads on core are destructive, so content had to be written back within the same memory cycle, an otherwise never interrupted sequence. By having the test for zero done by logic in the memory controller, setting it to FF could be done within this cycle. It can't get more atomic than that.
By putting that part of the operation all the way down into the memory module, no locking was needed. It's like the perfect speed up by avoiding the costly operation at all. Of course, later, semiconductor based memory had to add a locking (and synchronisation) protocol, thus making TS as slow as a CS would be.
*3 - Strictly speaking Branch-And-Set instructions are a variation of TS where the branch according to the test result is already embedded in the test instruction, still I'd rather see them separate - much like the CS.
*4 - CS on /360, CAS on M68k, CMPXCHG on Intel x86. On RISC implementations this gets turned into two operations of Load-Link reading and marking a word to be exchanged and Store-Conditional to do the store if no other access has happened. The idea is to avoide RMW cycles. But to me it seams more of a cheap trick just handing down the problem to the memory controller (but without the elegance of the original TS) - thus adding quite some restrictions.