But the question is, how did DOS know what portions of the UMA where available?
DOS itself doesn't touch neither High-Mem nor UMB without being ordered to do so. And even then it needs a driver capturing and providing these memory areas. Usually this is
If so, what if wanted to write a TSR that would monitor BIOS interrupts even before DOS is loaded?
Well, it can't be a standard TSR, as there is no DOS to load it. But yeah, as soon as there is memory above 640KiB it can be used to store a programm surviving a restart and monitor acivity before (and during) DOS load.
Would the OS overwrite my code or do I have any way to "protect" my TSR memory from being claimed later?
No, unless ofc, you have it use
HIMEM.SYS, which in turn would not hesitate, but just grab whatever is available.
The first task is to find some memory. HMA might seam like a good idea, but as said before, then
DOS=HIGH can't be used at all. Better use a CGA as display adaptor, while installing some spare MDA (or vice versa), as the use different address areas in Upper Memory. OR even better, if at hand, use some (PC) memory card that allows the addition on RAM anywhere in the address space.
Now boot into DOS and start you 'special' TSR, which moves itself up into one of these additional RAM areas. The real work will be to find a combination of trigger that allow your code to keep itself in the loop during all stages of reset and boot.
If you found a way, it's all set to boot again and spy on DOS :)