Back then, the operating system was much less powerful than you are used to today.
MS-DOS was a disk operating system. It primarily took care of the disk (floppy disks, mostly), including reading, writing and executing files, and handling directory structures. Sure, it had a few other functions which would wrap around lower level features of the system, but not too many. It did provide the command line interface and a lot of individual small programs - e.g. to copy files and such - but those aspects are irrelevant for the question at hand as a game would use none of those.
The other important system component back then was the BIOS (basic input output system) which an application could use for, well, text input and output - if the application had no particular performance requirements and preferred an easier API instead of low level access to the devices. BIOS text processing incurred a very much noticeable performance overhead, and generally limited what you can do a lot.
Especially iIn the later days of the 286, 386, 486 and with the advent of more memory, when paging or memory virtualization became more frequently necessary, there were added subcomponents for memory handling, which could be started or configured separately (see Upper Memory Area or DOS Memory Management for nice overviews and links to more information).
Aside from these system components, many programs, and certainly most if not all graphically intensive games, talked to the hardware directly. They had direct control over all aspects of the screen (including intricate timings in the phases where the electron beam moved back between lines, or from the bottom to top border), and the keyboard. Sometimes even direct or low-level access to the floppy drive (bypassing DOS), for copyright protection purposes. There was no preemptive multitasking like in every general purpose OS today - there simply was no "power" in the system which could influence the running program that much.
Speaking of which, there was always one running program. It would even be far-fetched to call it "process" in our modern meaning; there were none of the features we associate with processes today (priority, time scheduling, ownership of particular resources, separate user/kernel-level permissions etc.), although DOS did keep track of some of the resources and could reserve them for programs loaded at the same time...
... to be used, amongst others, for a type of application called TSR ("terminate and stay resident programs"), which meant that you could start small utilities which would stick around, and would usually intercept keyboard interrupts - so they could do some action on the press of a button, even if another program was currently active. This worked, mostly, but usually not very usefully while in a game which did fancy stuff with the screen. There was no way for a TSR to sanely take control, modify the screen, and restore everything to the previous setting. Of course one could try, but it was always a hack, nothing at all like today. There were no system-level video drivers which could do the state recovery, and so on.
If a program exited on ESC, that was purely a convention. Alt-F4 or other standard key combinations did not exist. Figuring out which keys a game used at all was always part of the fun, especially if you had a decentralized backup copy of a game without the manual...
EXE
, but aCOM
. Is this somehow related to its inability to exit?" No, it isn't. Besides you should really look at the file magic to determine whether it is an MZ executable or a flat .COM-style executable. That's what DOS does. It allows either filename extension for either type of executable. Only the presence of either an "MZ" or "ZM" signature (both work) indicates a non-flat executable.