I'm going to start by saying this was a long time ago, so my memory may fail me. And I did not use RM/COBOL, but I programmed for nearly 20 years starting in the mid 80's, migrating COBOL code from Data General ICOBOL to Unix systems which used Austec ACE Cobol (no longer in existence as far as I know) and then later we switched to one of the first versions of Acucobol, now owned by Micro Focus. Again on Unix, but then later we ported to DOS and Windows, which was a piece of cake because Acucobol did all the work with their runtime.
(We did later put a GUI on the software, using Acucobol, and that was a lot of work.)
Anyway Austec COBOL and Acucobol, like RM/Cobol, used ISAM files. Each Cobol file had a data file and an index file. No database server. These really weren't "databases", they were individual plain flat files with binary data structures inside them, and any relationship between them was in code and logic only.
As an aside, the data file, at least early on, simply contained fixed length Cobol records with a delete flag at the beginning of each record. You could read the data file sequentially if you had the layout (the "FD") of the Cobol record. Using the "FD", you could figure out the offset of each record, and where each field was. In the early days, I often had to write code to read the data file (with Cobol code) and write it back out to a new indexed file if the index file became corrupted or someone deleted a massive number of records from the data file, to recover the space. It was just a loop to READ the data file, MOVE the 01 level of the FD to the new indexed file FD and write the record.
In Windows, when the files were stored on a network server, you were either using a variant of Microsoft LAN Manager or Novell Netware.
MS LAN Manager implemented the SMB (Server Message Block) protocol. This was basically the equivalent of a "database server" on the system serving up the files. It maintained the locks.
This was all abstracted through system calls - the runtime made system calls to open a file, read a file, write a file, lock part of a file - and the OS determined whether or not it could do that on the "C" drive locally for instance, or if it had to send over some commands to a network server. That's the SMB protocol. It was invisible, for the most part.
That's why you can map a F: drive to a network server and still use DIR or COPY from a DOS prompt the same as a C: drive.
You can read about the SMB Protocol (which later evolved into the CIFS protocol) here:
https://learn.microsoft.com/en-us/windows/win32/fileio/microsoft-smb-protocol-and-cifs-protocol-overview
You can also learn some interesting things by looking at Samba, which is an open source SMB Server for Unix systems. This was originally written by reverse engineering the SMB protocol, which wasn't documented publicly, using a packet sniffer. The docs provide some insights, as might the source code.
https://www.samba.org/samba/docs/old/Samba3-HOWTO/locking.html
