CP/M, the operating system for Intel 8080 and Z80 (and others as well, but later)
CP/M is an 8080 operating system. Like DOS is an 8086 system. That other CPUs may as well execute it due to close compatibility with the 8080 is not CP/M's fault.
used to call 0005h for syscalls. [...]
It's the lowest memory address that can be assumed to be usable without giving up much functionality of the target system (or any at all).
CP/M, unlike developer-specific OSes, where machine and OS were developed in step, is an add-on product meant to be applicable to as many different designs as possible. As such it's important to be as unintrusive and adaptable as possible.
Address 5 is a great choice:
- Existing hardware interrupts can (continue to) use any RST
- Software can use all RST for whatever function it needs
- Only the last 3 bytes of RST 0 are reserved
- CP/M uses (has to use) RST 0 anyway
The last point is quite important, as RST 0 also dubs as warm start entry point for CP/M due being the RESET entry point (*1), so CP/M is already in control of these 8 bytes. It only needs the first 3 to jump to its warm boot code, leaving 5 bytes for other use ... like using 3 for a generic call interface without spending a rare RST resource (*1).
I'm proposing that CP/M could have used rst 8 or something,
There are only 8 RST locations available. Each the OS takes is one hardware and applications can't use.
This would have meant a saving of 2 bytes and six tacts per syscall, just by using the mechanism that the CPUs provide as standard.
"Six tacts per call" is not really a considerable benefit with syscalls usually taking at least several hundred to many thousands.
And it fits in with the scheme of using rst 0 for a warm reboot,
Using RST 0 for warm boot is a way of reusing RESET (*2).
rst 56 for the debugging.
I guess you mean RST 7? (*3) That's already a good example how a user program is benefiting from CP/M keeping its hands off RST instructions as much as possible.
Bottom line: Using address 5 comes with the least cost in terms of predefined system resources CP/M requires from a system to be ported to.
*1 - In fact, the other two bytes are used in a similar way, address 3 holds the I/O byte and 4 the disk byte. Cramping all RAM needs into the first 256 bytes, when a quarter (64 Bytes) is already reserved for the RST table and a whopping half (128 Bytes) for the file buffer, is a tough job.
*2 - It's all about how the system is built. RESET does start, like RST 0, execution at address 0. On a cold start an 8080 system needs to have some ROM there to start up. Since CP/M (usually) wants RAM at that location, that ROM will be mapped out at some point.
Thus CP/M can now (re)use that entry point as warm restart, so if any application executes an RST 0, it ends up in CP/M's warm start routine, while a follow-up RESET (usually) would start again from ROM code.
Even better, if the hardware somehow distinguishes between power-up reset and later reset, it may execute from address 0 in RAM instead, making RESET go direct to CP/M warm start.
What a nice way to kill (at least) two birds with one stone.
*3 - As 8080 software using Intel notation may be more appropriate - not to mention that it avoids any confusion about which addresses are legal or not.