Given the memory referencing limits of MS-DOS for compatibility purposes, why didn't games produced for the 286+ processors eschew MS-DOS and talk directly with the BIOS or IO hardware, instead of all the optimisations necessary to maximise the conventional memory available and provision of extended memory drivers, etc.?
You need to run the game somehow.
There were some early booter games, but as soon as you had a hard drive, users typically want to store programs, including games, on the hard drive, as files.
When your game is on a hard drive with DOS, the DOS must be started and running in order to run the game executable program file.
Which meant that your game needs to start running in an environment where DOS, any XMS and EMS drivers are already loaded, or other peripheral drivers like mouse driver or keyboard driver for using a specific keyboard layout for your language, in order to load the game and let it use the possibly hardware-specific memory drivers and hardware-specific mouse driver.
However, the video card and sound card were generally used directly by a 286 DOS game, as they rarely had any drivers that you needed to load under DOS (exceptions exist of course, such as 386-era sound cards like PNP Sound Blasters to enable usage of card, and for Gravis UltraSound to emulate a Sound Blaster or Roland MT-32 for games that don't support GUS directly).
And that is exactly what happened with 386+ CPUs. You had a game executable, which was really a DOS extender that set up a 32-bit protected environment for a payload which was the 32-bit game binary image.
We need to look into this in a bit more detail. There were indeed a limited amount of "direct boot" games that completely by-passed DOS and implemented their own low-level hardware support. (As you have limited your question to 286+ systems, these games, however, don't really fit into that scheme. They were mostly obsolete when 286 systems were common).
By-passing DOS, at least for storage, proved to be hampering support for specific hard disks, storing game state anywhere else but on floppy disks and really didn't save that much memory (DOS used to be, at least in the beginning, indeed, a very slim operating system). I would assume the reason for those self-booters was actually more copy-protection than anything else. Note most of these games date from the early 1980s, that is, when you basically could assume you didn't have to support more than a CGA card, maybe the joystick interface, the keyboard and the PC speaker - all hardware that had somewhat "standard" interfaces and hardware locations in the PC architecture, so it was relatively safe to directly access the bare metal. So, these games were indeed, for the most part, not only circumventing DOS (apart from storage), but also the PC BIOS.
Over the whole DOS era, there were a lot of (I would even claim, nearly all) games that by-passed DOS and went to BIOS (or, rather, directly to the hardware) for video (I would guess only very simple games like text adventures were actually using DOS screen I/O functions). But most of these didn't even use the BIOS as well but instead went directly to the hardware - the functions that the video BIOS offers to a game developer, are indeed, apart from detection and mode setup functions, really limited.
In the early times, the choice of possibly supported hardware was often limited to CGA video - That was easy to implement and didn't exactly require rocket science to produce something faster and more game-oriented than what the PC BIOS offered. This changed a bit with upcoming EGA, VGA and more obscure (Tandy, Amstrad PC, Plantronics,...) video standards that caused game developers to produce specific versions that might or might not have been delivered as specific packages or various versions in one package. During this time, you really had to read the fine print on any game you bought.
Most games took the easy way and used DOS to store game state or generally, DOS functions for access to storage media. This ensured you could save your games to specific media that required loadable DOS drivers.
For mouse support, most games used the "standard" driver that came with your system (I would consider these "loadable BIOS extensions", as there really wasn't ever anything DOS-specific in those drivers). Sound was a bit of a problem until AdLib/Sound Blaster were basically the card to have. These cards were typically accessed on bare-metal level by the games, by-passing both DOS and BIOS (neither had any support for sound anyways).
Towards the end of the DOS era, when games outgrew the memory supported by DOS*, they tended to come with DOS extenders like DOS/4G and DJGPP (and many more) that created a 32-bit environment on top of DOS, replacing most of the DOS functions with own code. Even if these still used DOS and BIOS for some functions - mainly for bootstrapping, they had to in fact by-pass a lot of it once they had entered protected mode, because neither DOS nor BIOS have support for protected mode applications.
*One of the milestone games that made everyone struggle with their DOS setup for conventional memory (and EMS/XMS as well) was the Wing Commander series. This really forced everyone to install memory managers for expanded and extended memory, optimize their driver load order and UMB/HMA usage to free up conventional memory, sometimes even causing people to implement specific "Wing Commander boot configurations", in short, was a bit of a nightmare to get running, clearly showed the "plain" DOS architecture was at its end to support large games and it was about time to - if you wanted to continue to make DOS games - jump on the DOS extender wagon (The games, however, were still extremely popular, even if they were a nuisance to get running).
You mean beside the fact that this would make it impossible, or at least quite complex, to install Games on hard disk - which was standard in AT times and later. Likewise for returning to DOS after the game.
Also, it's always important to keep in mind that DOS is in no way like OS that we know today. From today's PoV DOS isn't much more than a glorified boot loader that stays resident. GRUB got more complex features than DOS. Didn't provide much services beside abstract disk access (hiding hardware and file system). The same goes for drivers like mouse or EMS. All extreme bare bone and not much to save there - if that service is needed.
And that's what having different configurations were about slimming down all the overhead one loved to have for regular work (from extended keyboard handler and printer buffers all the way to sidekick and all the other TSR). One had a one for regular work and one optimized for gaming, selected during boot.
Given the memory referencing limits of MS-DOS for compatibility purposes, why didn't games produced for the 286+ processors eschew MS-DOS and talk directly with the BIOS or IO hardware,
Keep in mind that DOS was a rather slim. DOS 2.0 needs only about 24 KiB and even 3.3 is still less than 48 KiB. Eliminating DOS would not save much as the games would need to somehow provide the same services.
instead of all the optimisations necessary to maximise the conventional memory available and provision of extended memory drivers, etc.?
With games made for 286 and later and using_ 286+ features, those optimizations weren't really a thing, as those games used the memory above 1 MiB. All that CONFIG.SYS wrangling was only to help real mode games, those not using an extender, to work properly.
In fact, all those squeezings were way more common when it was about real mode applications than games. Get a few more pages into your word processor without swapping or lines in a spreadsheet.
Also, why should any game developer spend money to recreate disk services if they are already there - in a compatible way, supporting whatever configuration the user had?
Short answer: Because no one wanted to implement an OS in a game...
Long answer: Most games that required filesystem access used DOS (or later, on 386+ a DOS extender). Since version 2.0, DOS provided good UNIX-like handle-based file system support, and BIOS INT 13h services abstracted disk I/O, so the applications wouldn't need to support different disk controllers and disk types (MFM, IDE, SCSI from tens of vendors, floppy disks, network drives, and so on). While it is possible to implement your own disk I/O driver and a filesystem, it wasn't very practical, as it would mean writing drivers for multiple disk controllers and potentially leaving some controllers unsupported.
BTW, most games didn't use MS-DOS or BIOS when it comes to graphics, sound, and user input devices (perhaps except of mice). The API provided by MS-DOS didn't include any graphics support, and BIOS graphics support was slow (it might have been used to setup a graphics mode, but not to draw the picture). Sound support in BIOS and DOS wasn't a thing either.
Being now an old dinosaurus allows me to remember those days. Not only games were using BIOS, but a lot of common programs also did.
There were 3 main reasons for that:
- there was few MS/DOS level abstractions for sound for example besides sending a beep
- handling graphics at MS/DOS level was far from being as efficient as handling them directly as hardware level, and computers were not as performant as current ones are... This was specifically required for games to provide an acceptable user experience
- MS/DOS enforced no protection at all, so user level programs had direct hardware access
For example, I can remember writing a software getting values from meteorological stations through half duplex modems. Has MS/DOS had no provision for modem handling, my code directly dealt with the 8251 addresses...
On the other hand, I seldom used anything other than DOS for disk access... except for a tiny program able to rewrite the boot sector of floppy disks because of a virus that used to install itself there.
Hewing strictly to the question, the bottom line is that the BIOS was no more "extended-memory safe" than was DOS.
As others have said, much- arguably most- software on IBM PCs or strict compatibles went directly to the hardware for much functionality. The BIOS was used relatively little, except for- for example- putting a video card into a specific mode.