What was the most popular operating system on the PDP-11 computers?
And is it known what percentage of PDP-11 computers used this operating system?
I'm only guessing, but I would guess in the late '70s and early '80s either
RSTS/E for traditional timesharing use
RSX-11M for real-time application use; maybe this transitioned to RSX-11M-Plus as larger memories became common
I have no data for this, just an impression of what was going on in DEC while I was there. I'd be interested in numbers if anyone can come up with them. The answer should in principle be knowable for DEC systems, since software came with a price tag.
In the USSR there were produced a lot of clones of PDP-11, some of them more exact and some of them much less.
As should be expected, the most of them ran variants of RT-11 operating system. Since those computers were widely used in schools, RT-11 was perhaps the most popular operating system for PDP-11 clones.
But if we consider home computers, the situation was different.
The most of operating systems were created for the computers of the BK series. I would claim it is the home computer with the greatest number of various operating systems available for.
Initially, BK-0010 was released without any disk drive support with only external storage being a tape recorder. But very soon many amateurs and cooperative firms found ways to attach disk drives to these computers. Below, is a table of some operating systems for BK series computers. The list is not exhaustive. I for instance, saw a ROM-cartridge-embeeded RT-11 variant for BK-0010.
As you can see from the table, the most of operating systems used the MicroDOS file system, which was named after an early OS of the same name.
BK had very limited amount of RAM. On BK-0010 the disk drive controller expansion cards usually added 4 to 16, or, very rarely, 32 (with paging) KB of memory, so the operating system had to fit this area, otherwise the usual programs for BK would not even work.
Thus, the operating systems had to choose, whether to prioritize command-line interface, which usually would leave the user without any file manager but allow batch files, or integrate the file manager into the kernel, providing virtually no command line interface.
The more advanced operating systems, such as ANDOS and MKDOS choose the modular approach, with resident kernel that would provide basic file operations (and in later ANDOS versions, a basic file chooser dialog that could be invoked from other apps), and an optional file manager, sometimes even with option to chose one.
Since the tape file format of BK allowed for 16-character long file names, and many programs and games would have the names of overlay files hardcoded, and the files often had names ending with their number like "01", "02", etc, the RT-11 6.3 file name format was not very suitable out of compatibility reasons.
Thus, more widespread became the MicroDOS file system with its 14-character file length and FAT12 with 8.3 format and complicated algorithms used by operating systems, like ANDOS to convert the file names, keeping their last characters that often contained file number.
So, a file named "НЕПОБЕДИМЫЙ" would become "НЕПОБЕДЙ" on the disk drive and when a program would open a file "НЕПОБЕДИМЫЙ", its name would be automatically shortened.
That said, I think, the most popular OS (at least, according to this recent lecture by the creator of the OS himself, Alexey Nadezhin) on home computer BKs was ANDOS. He also claims he had sold about 5000 copies officially but the majority, in excess of 100000 copies were pirated.
Andos 3.30 with Disk Master file manager:
Andos 2.23 with BK SHELL file manager:
The later versions of ANDOS could support up to 63 drives: hard drive partitions, floppies, tape and a RAM drive (on BK-0011(M)).
The file manager Disk Master included viewer, editor, calculator, music player, disk formatting tool, tree view, customizable user menus, backup and restore and other functions, even a screensaver.
It should be noted that all this was possible in just 16 KB (the shell occupied even less) due to the efficiency of the PDP-11 architecture.
The second most popular operating system on BK series was operating system MK-DOS (although according to Nadezhin, the creator of MK-DOS Mikhail Korolev thinks his system was more popular). It was the main competitor to ANDOS. It allowed to work with volumes up to 32 MB (unlike ANDOS which only allowed floppy-sized volumes) with hard drives up to 2 GB. Like ANDOS, it supported tape access and a RAM drive on BK-0011(M). MK-DOS supported true subdirectories, unlike ANDOS that only supported root directory on a drive but could visibly make illusion of different sub-folders. The distribution included Forth programming language.
Other notable operating systems included
AO-DOS with DOS-SHELL file manager
CSI-DOS with X-SHELL file manager
You can find more info about operating systems for BK series on this page in Russian Wikipedia.