But the question is, how did DOS know what portions of the UMA where available?
DOS itself doesn't touch neither High-Mem (HMA) nor UMB without being ordered to do so. And even then it needs a driver capturing and providing these memory areas. Usually this is
If so, what if wanted to write a TSR that would monitor BIOS interrupts even before DOS is loaded?
Well, it can't be a standard TSR, as there is no DOS to load it. But yeah, as soon as there is memory above 640KiB it can be used to store a program surviving a restart and monitor activity before (and during) DOS load.
Would the OS overwrite my code or do I have any way to "protect" my TSR memory from being claimed later?
No, unless of course, you have it use
HIMEM.SYS, which in turn would not hesitate, but just grab whatever is available.
The first task is to find some memory. HMA might seem like a good idea, but as said before, then
DOS=HIGH can't be used at all. Better use a CGA as display adaptor, while installing some spare MDA (or vice versa), as they use different address areas in Upper Memory. OR even better, if at hand, use some (PC) memory card that allows the addition of RAM anywhere in the address space.
Now boot into DOS and start your 'special' TSR, which moves itself up into one of these additional RAM areas. The real work will be to find a combination of trigger that allows your code to keep itself in the loop during all stages of reset and boot.
If you found a way, it's all set to boot again and spy on DOS :)