Maybe a silly question but has there ever been a mainframe OS running on a Z80 system,
I'd say rather not - at least not as anything past some for fun/for learning projects.
of course with sufficient memory and memory management unit?
Of course - after all, a Z80 without extended address space (and thus some form of management) wouldn't work past emulating some very basic /360 models.
I'm thinking of a ported MVS running, with TSO, the classic OS/370 editor, a COBOL and/or FORTRAN compiler.
Erm ... the whole point of /370 and its software is that there is no porting needed. It's a 100% compatible environment. Everything model/hardware specific is hidden in some deep down abstraction layer within the OS.
So the only way to go is having the Z80 emulating the /370.
Or am I the only one having such weird ideas?
No, not at all. It's a very obvious idea. I'm as guilty as you (*1). The real issue here is speed. The Z80 is simply way too slow for any useful application - especially with that software burden.
One needs a lot of instructions to emulate. For /370 to Z80 I would assume a ratio of 1:100 (*2) for most of the emulation. With an average of 8 cycles per Z80 instruction (*3) this means the average /370 instruction will take about 800 Z80 cycles. At a time where this may have been interesting, say before 1980(*4), the Z80 was specced as Z80A at 4 MHz. So one could expect an upper boundary around 5,000 instructions per second. In fact it might be even less considering the address translation.
That is 1/3rd of the lowest full /360, the Model 30(*5) and about 1/5th to 1/10th of a /370-115(*6), the lowest end /370. Neither is capable of even loading MVS. To set this into context, by 1980 IBM's offerings were between 1.2 MIPS for a 3031 and 5 MIPS for a 3033u. So that Z80-based beast would have been 250 and 1000 times slower.
Even worse, all of that is without I/O - the part the /370 architecture was really good at - and micros never.
But hey, it might have been for a single user. Sure, but that doesn't change that any of the mentioned software packages has a code path of several 10k instructions per function, making anything EXTREMELY slow, up to simply not usable.
The 1982 Z80H wouldn't change that picture much. At that time way faster 16- and 32-bit capable CPUs would have been used, not a Z80. Even today's up-to-50-MHz Z80 compatible (EZ80, etc.) would still be way below acceptable in 1980, even more so when they finally were available.
Bottom Line:
Z80 were throughout all of their existence far too slow to be used as base for any useful /370 system. Memory, by contrast, would not have been an issue.
Maybe as a reference, a 2000s' Pentium III at 500 MHz running Hercules cranks out about 1.5-1.8 /370 MIPS. That's 20+ years of development later, clocked at >100 times faster than the Z80, having native 32-bit support and built-in hardware memory management, still barely hitting the low end 3031 CPU of 1977 Mainframe CPU (again not considering any I/O load).(*7)
*1 - I did so in 1980 for the 6502-based Apple 2. It was really more a fun issue. Other than intended, I didn't use it for any development.
*2 - For the non-FP instruction set and at the very minimum one can assume 10 instructions for each of
- Scheduler (using a table)
- Calculating an address (1-2 per instruction)
- each byte handled by that instruction (average 12 for character)
So with 1.5 Addresses and 8 bytes (average between word and character) that's about ~100 instructions. This is on purpose estimated toward the lower end.
*3 - A value gathered a very long time ago using an LA while running Multiplan.
*4 - In 1980, or later, much faster CPUs would have been chosen for such endeavour - after all, we are talking about customers that paid more for a terminal than a whole micro setup would cost.
*5 - The Model 30 peaked at 35 kOPS for basic 32-bit add. It also had a memory of 8..64 KiB of RAM.
*6 - Offered with 64 or 96 KiB RAM, so quite within reach of a 1976 Z80 system.
*7 - To finish the picture, although modern (non mainframe) CPUs have caught up quite a bit in the following 20 years, they still have not dethroned the dedicated mainframe CPU. But they have found a place in low end machines emulating /370.