The IVT is at 0x0000-0x03ff while the BDA is at 0x0400-0x04ff but boot sectors are loaded at 0x7c00. What was at 0x0500-0x7bff that caused this convention? I'm also curious why some MBRs relocate themselves to 0x0600 but maybe that is a separate question.
As the least amount of memory needed to boot from floppy was 32 kilobytes, the first sector of the floppy, the FAT volume boot record, is loaded to just before the 32 kilobyte end of memory. However, it is loaded to 1 kilobyte before the end of memory, but the sector is only 512 bytes, so it could fit two sectors, or the remaining 512 bytes can be used for other purposes. It is possible that it could allow for support of 1 kilobyte sectors, as the IBM 5150 floppy format code suggests that it is possible to format floppies with 1 kilobyte sector size.
The area of 0x0500-0x7bff contains just memory. However, the 0x0500 area is not entirely free and safe to use, as at least there is one byte that BIOS uses as print screen flag - which is enough of a reason to avoid the area.
That is the reason why for example DOS hard drive master boot record relocates itself to 0x0600 area, and then load the next stage again to 0x7C00. It could relocate itself anywhere between 0x0501 and 0x7BFF, but it chooses to use the 0x7C00 downwards as stack area. Do note that when IBM PC was invented, there were no hard drives so the concept of MBR or relocating it did not exist, only the concept of floppy boot sector (volume boot sector) existed.
DOS itself, or rather IO.SYS, is loaded to memory starting from address 0x0070 by the FAT volume boot record. The VBR also uses area down from 0x7C00 as stack.
What was at 0x0500-0x7bff that caused this convention?
Nothing. There is no hard (-ware related) reason. And while loading an OS at top or bottom of memory is more of a philosophical question, it's one the OS to be loaded needs to decide for itself - nothing the BIOS should imply.
So lets better look at what are the requirements for a boot location adding the least restrictions
- The memory should be available in all memory configurations
So the lowest possible address, in case of the PC right after the BPB would might seem like a good idea, but in addition
- It should be an absolute location known to the block loaded to enable most compact code
Using 500h, like you assumed would be the first fitting both, except, Having a fixed boot load address right after BIOS data would prevent any dynamic allocation for that area - like lists of drives or alike, or future extensions. So, not really a great idea (*2). In addition,
- A boot sector usually contains code not needed later on
So it's a good idea to locate the boot sector at a fixed address far enough up to let it decide how to continue and where to put further stages during boot.
- Top of memory for the boot sector would be great, but
- A machine with variable memory setup does not have a hard top of memory
- Any address use should be as high up as possible, to give more room to act and
- Work with as many configurations as possible
- The original PC had memory sizes of 16, 32, 48 and 64 KiB
While disk loaded applications may be useful even with a system as low as 16 KiB (*3), it's not very likely. 32 KiB seem more like a reasonable lowest assumption for loading some kind of OS (*4) plus any kind of application.
- Loading it all the way to the top in minimum RAM configuration (32 KiB) would not leave any room for data OS might want to put there during boot (like stack space) without destroying the boot loader block. So leaving a bit room does come handy
Putting all together, loading the boot block at top of minimum memory configuration minus 1 KiB satisfy all of the above plus leaving room for prepared data structures for the loaded OS, and/or a buffer for another disk block.
I'm also curious why some MBRs relocate themselves to 0x0600 but maybe that is a separate question.
It may come rather obvious if you continue with above mindset of being as dynamic as possible.
*1 - I'll spare discussing advantages for either memory scheme here.
*2 - Systems working that way, like the Apple II, load right after a hardware defined address (screen memory) which will never move.
*3 - A 16 KiB application could well use ROM-BIOS for custom disk-I/O, but in general it was only meant as a cassette based (remember that the PC had a cassette interface) BASIC machine, put directly against the most basic Apple II+ model.
*4 - It's important to keep in mind that BIOS was written before and independent of DOS