Belated ascendancy of dynamic linkers

Question

Old computer systems were supplied—by our present notion—with very little memory, thus conservation of both RAM and storage room has been tremendously important during those years of austerity. Meanwhile a multitask operating system ran numerous processes which often kept the same static data in memory (but separately). Until c. 1992 putting libraries into every executable file was a common practice and a simple Hello-Worldish binary produced on early Unix System V releases may easily take 64 KiB (compared to about 12 KiB for modern OSes); it slowed computing and wasted the room.

In mid-1980s memory mapping was already a known concept, multitasking was understood as the future (even for personal computers), and developers of the day couldn’t be oblivious to a great economy that placing one copy of a common library to RAM (instead of many separate copies) promised. In fact, Microsoft jumped on shared libraries from the beginning of Windows. What hampered introduction of dynamic linkers elsewhere?

Answer 1

I have to disagree, to some extent, with the framing of your question. While it is correct that limited RAM in early micros made it a valuable resource to conserve, it is not the case that shared code libraries weren't used to accomplish this. Shared code libraries were very prevalent in the early micros, and were generally embedded as ROM firmware, as a direct solution to the problem of RAM being limited.

If you think of the typical home micro with BASIC in ROM with this framing, it is clear that BASIC is just a shared run-time library. What it does is make possible many sophisticated (if somewhat slow!) applications with very modest RAM requirements. This is true whether the run-time is located in RAM or ROM, but is much more effective if you can ROM it and leave all the system RAM for the app.

Apart from BASIC, many systems included a full set of reusable application libraries in ROM. The original Mac, Atari ST, and Amiga all did this. On the Amiga, the code in the ROM was literally comprised of shared libraries in the modern sense of that term.

If you were to accept that shared code libraries have been a consistent feature from the earliest RAM-constrained micros, then your question sort of morphs into why did it seem to come later to the systems we use today, like Windows and Unix derivatives. That is a totally separate question, and I think it relates more to the unintended side-effects of this approach - colloquially referred to as DLL Hell. That is a problem best solved in the OS, but routinely solved with add-ons that succeed to varying degrees.

In defense of the early implementations that exposed this problem with dependency issues, perhaps, the problem size of doing dependency management well was often underestimated. Also, OS developers likely couldn't have predicted the proliferation of 3rd party shared libraries that became de-facto additions to the OS, but without the corresponding and disciplined revision/testing/release process.

Answer 2

On the BESM-6 (a 1960s Soviet mainframe) the most widespread programming environment was dynamically linking by default. Directly loaded application executables were not typically pre-built; the "executing loader" would link the program in memory and jump to it. Overlays were dynamic as well.

For example (the compiler output is edited for brevity; note the loader printout after *EXECUTE)

                                                       27.03.21 M1
 Ф O P T P A H
   /16.07.73/
                PROGRAM MAIN
                PRINT 1
             1  FORMAT(’ MAIN’)
       2        CALL LOADGO(’FOO’)
       3        CALL LOADGO(’BAR’)
                END

                SUBROUTINE FOO
                PRINT 1
             1  FORMAT(’ FOO’)
       2        RETURN
                END

                SUBROUTINE BAR
                PRINT 1
             1  FORMAT(’ BAR’)
       2        RETURN
                END

           *EXECUTE
        MAIN       01000        SWRITT     03150        AHID/*   E 04372
        PROGRAM  E 01000        LUNMUN   C 03153        IOSKIP*  C 04377
        BCDWRIT*   01025        LUN*MON    03154        IOXGIVEM   04400
        FT*621   E 01025        OCTTDEC    03162        GIVEMASK E 04401
        BCDENC*  E 01031        TABWT/*    03214        IOXXRPCK   04430
        FT*611   E 01031        RWTB/*     03221        IOXXUPCK E 04431
        FT*571   E 01033        PRINT8     03304        MON*ITOR   04502
        FT*561   E 01036        PRINT80  E 03304        PLBEG    E 04511
        NEXTLET* E 01043        IOXXTTWT   03342        PLCLO    E 04517
        RK*      E 01060        TTPRINTD E 03346        ASAVE*   E 04525
        WSY*     E 01330        TTPRINT  E 03346        SAVE*HID E 04532
        IOCONT*    01513        TTPRIKS  E 03360        FOR*ALL  E 04547
        IOAC*    E 01545        /IP*     E 03426        OH*      E 04551
        FT*002   E 01560        IOXXFMR*   03442        OH*1     E 04556
        FD*642   E 01561        IOXXFMW* E 03443        ISO/GOST   04565
        FD*722   E 01561        IOXXER/M   04040        TSTATE*  C 04634
        FC*722   E 01561        IOXXER/O E 04041        *ICHECK* C 04635
        FC*642   E 01561        IOXXPKWT   04062        GIVELEXX   04643
        FC*002   E 01561        BCDPUN   E 04063        SAVELEXX E 04745
        IOEND*   E 01566        COLPUNC* E 04064        RD/BT      04765
        FT*003   E 01566        COLPUNB* E 04121        WR/BT    E 04766
        IOSVFR   E 02075        COLPUNE* E 04121        SDEC*      05005
        SUBPERR* E 02107        IBCDCTR* E 04126        *IOXLSW* C 05045
        STOP*      02511        IOXXLPWT   04226        CLEARLEX   05046
        BCDBEG*    02532        ERRIOM     04231          CBOБOД   05060
        FT2*     E 03041        ERRLUN   E 04311
        KONV1*   E 03054        HID/*      04367

 MAIN

        FOO        05070
          CBOБOД   05106

 FOO

        BAR        05070
          CBOБOД   05106

 BAR

What happens immediately after *EXECUTE is akin to what ld.so does after an executable starts. The LOADGO procedure dynamically creates an overlay and jumps to it (a la dlopen+dlsym+subroutine call) with caching, i. e. two consecutive LOADGO calls with the same name will just do the jump, otherwise an analog of dlclose will be done first.

Speaking of memory-mapped shared libraries: address space used to be scarce enough to discourage wastage due to page alignment of libraries, and not all ISAs allowed for efficient position-independent code.

Answer 3

The question seems to cover multiple areas at once, including

• Dynamic linking at load time by the OS
• Runtime linking controlled by the application
• Shared libraries, in form of system-wide (or per-user) libraries

and

• Shared code, either as preloaded by OS or
• Shared code loaded by application when needed, but only loaded once

In any case, all of that was already in use, since the 1970s (*1), in mainframe software. Applications were not only developed in modules and linked to a static binary, but also only partially linked, deferring the final linking to load time. Some OSes did offer link loading as default program start. Here, an executable was not linked at all, but started from a library and linked while loading.

Since the linker was part of the OS runtime, any linking could be done later during program runtime. This included, of course, unloading of modules as well. Underlying this was always some library management system, usually also part of the OS.

In the early days of real memory operating systems there was no difference between shared and not shared, as sharing was just exchanging a memory pointer. With virtual memory this was supplied by the OS. Again, during load and later.

Last but not least were functions where the OS could be made to pre-load certain modules (including whole applications) at startup or by operator command. Quite handy in a multi-user environment with many users using the same applications, like editors, languages or other tools. The main program would be loaded only once at startup. Any user starting 'his' editor, would not load the whole application but only a stub opening local data storage (where the files go) but otherwise run the pre-loaded, shared code.

All of this was developed essentially with the beginning of multi-tasking and multi-user support, as RAM was always too small (*2), putting massive emphasis on space saving.

But the same is true for mainstream micro computers. Sure, OSes started out less sophisticated than mainframes 20 years before, but they soon caught up. This not only includes the mentioned Windows, but also Amiga OS, which was (like Windows) essentially a collection of libraries (*3). Well, and of course various unixoide OSes. Even before that, there was OS/9, offering quite sophisticated ways to share code at runtime. And that's on an 8-bit CPU.

Conclusion: I have a hard time to see the 'not use' of dynamic linking and shared code as implied in the question. The abilities were present all during the 1980s and easy to use. So the assumption about not being offered is wrong.

Looking closely, it seems as if the underlying question is rather why were single-user single-application OSes - like MS-DOS - the main environment used during that time. Right?

It's the classic question why Romans didn't have gas stations: A solution without a need.

Usage in a single-user single-program environment was about one application at a time (*4), which in turn was able to use the whole memory for whatever needed. More often memory was already too small for all functions of that single application, so overlay swapping within itself was needed, not sharing between parallel applications.

Bottom line: Never forget when looking back, that any solution, no matter how helpful it is nowadays, also needs a problem to be solved - often the answer is simply: the problem didn't exist back then.

---

*1 - Technically even before that, but, like with many other developments, it wasn't until the 1970s that an overall structure was established for generic use.

*2 - In 1970, a machine with 128 KiB was considered large. The largest (civilian) installation in Europe in 1972 was the information system for the 1972 Olympics in Munich, with an unbelievable 2 MiB of online RAM.

*3 - Brian H. may go into more detail.

*4 - Maybe enhanced with a few background helpers.

Answer 4

Without virtual memory, you basically must have all library code loaded into physical memory at once to use dynamic linking. Systems using processors that supported virtual memory were uncommon before about the mid 1980s. But even if you had virtual memory, it was still problematic. Imagine working on MIT's Multics in 1972. You need to call the sine function. It's only a couple of dozen words, but you have to drag in a four kiloword page to get it. The whole machine only had a couple of hundred pages, shared among dozens of users. All this dynamic stuff was slow. There were big performance advantages to creating a self-contained statically-linked binary that only needed a few pages loaded rather than calling dynamically-linked functions spread over many pages.

You might think of virtual memory and dynamic linking as conserving memory, but they only work well when you already have plenty of memory.

Answer 5

With single-user (non-multi-tasking systems) systems, an application could use any memory that hadn't been allocated before it launched, and any such memory that an application didn't use would generally sit idle. While it was possible for applications to open a DOS shell, which could use any memory the applications didn't, it wouldn't be terribly common to have two or more programs in memory that shared a substantial amount of code.

There were a few situations where it was helpful to have some code such as communications "drivers" shared among applications, but under MS-DOS that was accomplished by loading a FOSSIL driver and then loading the applications that would communicate with it. In such scenarios, however, an application that wanted to communicate wouldn't dynamically link to the driver. Instead, such a driver would often set up an interrupt handler, allowing an application to set up some registers and trigger the appropriate interrupt.

Personally, I'm not a big fan of dynamic linking. If a calls to an application's printf are dynamically linked to an OS function by that name, that will make the application sensitive to how the OS function opts to process various constructs. If one had some code that expected the %p format specifier to output a pointer value as 16 hex digits preceded by 0x, and some other code that expected it to output 16 hex characters without a prefix, an implementation which chains to the OS function will only be able to support whichever program is expecting whatever behavior the OS implements. By contrast, if printf were statically linked, one could build each application with an implementations whose library printf function handled the %p function in the required fashion, and run both programs on the same OS.