24

The 8086 is a 16 bit processor with a 20 bit segmented address space, famous for being used in the IBM PC. It is also infamous for being tricky to program for.

I am currently writing what is eventually going to be an operating system written in C for the 80286. For this project, I would like to know...

  • what conventions did programmers employ to deal with segmentation?
  • what extensions exist for common programming languages to accommodate the segmented address space?
  • what techniques exist to write relocatable code for the 8086 and 80286?
  • are there any free/open source compilers that generate code that uses more than one data segment I can use for studying?
  • 1
    I'm not familiar with it, but it seems to me a lot of implementations defined a non-standard "far" keyword that you could apply to pointers to make them composed of both the segment and the address rather than just the address. – Muzer Feb 27 '17 at 17:18
  • 2
    near pointers are relative to the DS segment, whatever value is has at the time. – peter ferrie Feb 27 '17 at 18:01
  • 8
    I'm only 35 but somehow this question makes me feel old... – Jason C Feb 28 '17 at 0:25
  • 5
    By the way check out en.wikipedia.org/wiki/Intel_Memory_Model. For search keywords, any searches for "intel 16 bit memory model" or "near far pointers" will generally yield you a wealth of info on the topic, in addition to the answers below. – Jason C Feb 28 '17 at 0:27
  • 1
    @FUZxxl What I'm trying to say is, here's some extra helpful google keywords too if you find them of any use, and maybe you won't, and also ha ha I feel old. The reason we are not on the same page is you felt the need to defend your other knowledge for some reason. It was just... some info that I thought might be helpful. You may wish to reflect in the shower why you jumped to mild defensiveness when none was needed. It's a little strange! If you didn't already know about memory models or whatever, that'd be OK too... what you know/don't know is not interesting to me at all. – Jason C Feb 28 '17 at 1:40
36

What it boils down to is that, unlike in a flat addressing model, there are multiple types of pointers:

  • near - 16-bit (16-bit offset only.)
  • far - 32-bit (16-bit offset + 16-bit segment)
  • huge - The same asfar, but the compiler generates code to automatically manage the segment when doing pointer arithmetic. (This is slow, but can convenient, because it means the programmer can ignore segments when dealing with structures >64K).

Memory models represented different default types of pointers for both code and data. IIRC:

  • tiny - everything is near, and code and data are both in the same segment. DOS .COM files used this by default:
  • small - everything is near, with one segment for code and another for data.
  • compact - One code segment, but multiple data segments.
  • medium - One data segment, but multiple code segments. Different .c files emitted as different code segments. For function pointers to be general, they had to be far. (This model was commonly used.)
  • large - Multiple data segments, multiple code segments.

The medium model wound up being one of the more commonly used models, because it presented a nice set of trade-offs. It allowed for more than 64K of code, but kept the quick access to a single data segment. Some systems provided the ability to malloc and free against a separate heap outside the default segment, which gave selective access to >64K data as well.

Practically speaking, what this meant is this:

  • Compilers/linkers had flags to set the memory model in use. (Along with separate libraries, etc. for each... people can and did save disk space by not installing memory models they weren't using.)
  • Pointers were annotated with vendor-specific anotations for near, far, and huge. (And sizeof for pointers would vary between the different types.)
  • There were macros that allowed segment/offset pointers to be assembled from component parts and broken out into separate segments and offsets.
  • There were 4096 different valid far pointers for each location in the address space. (So pointer equality could be strange.)

One specific bit of trickery around this was the old MakeProcInstance function in Win16. It generated code at runtime that would bind an instance of a function to a particular data segment.

https://blogs.msdn.microsoft.com/oldnewthing/20080207-00/?p=23533

It's also worth mentioning that the 286 was rather different in protected mode. Rather than the segment being directly shifted and added to the offset to form the final pointer, the segment number was an offset into a table that contained information (including the segment offset) for each segment managed by the OS. The indirection made it possible to relocate and protect segments.

To make it possible to easily traverse blocks of memory >64K, Windows would allocate blocks of segments where the increment between one segment and the next was well known:

http://www.midnightbeach.com/jon/pubs/huge-model.htm

Note that as tricky as all this sounds, IMHO, it's not nearly as complex as some of the bank switching schemes that were used. The Apple ][ was bad about that, but in DOS, EMS 3.2 was no picnic either. EMS 3.2 took a 64K block of memory in the space between 640-1024K and divided it into 4 16K chunks. The chunks then served as windows into a larger address space (megabytes) managed through the EMS memory mapping. An EMS memory reference thus was a far reference to a 16K window mapped to some portion of another address space...

  • 1
    Great answer! Especially the last link is just what I was looking for. – fuz Feb 27 '17 at 22:01
  • Despite their quirks, these machines were a lot of fun. (In an alternative history spirit, I remember wondering at one point what would've happened if the 80286 had introduced a variant of real mode that shifted the segment by 8 bits rather than by 4. This would've given a way to get to the full 24-bit address space from something that looked an awful lot like legacy 8086 real mode. It's not like 286 boxes didn't sell, but it would've made it easier to use their capabilities from DOS.... ) – mschaef Feb 27 '17 at 22:18
  • 1
    I'm actually trying to write a protected mode operating system for the 80286, which is why I'm asking about all this stuff. No existing toolchain satisfies my needs, so I'm currently writing my own. – fuz Feb 27 '17 at 22:50
  • 1
    @mschaef You've mixed up compact and tiny models. Tiny is CS=DS(=SS), and what .com files use; compact is one CS and multiple DS, most common for .exe files. – peter ferrie Feb 28 '17 at 6:26
  • Peter, FUZxxl, thanks for the correction/edit. – mschaef Feb 28 '17 at 14:33
12

That's a lot of parts for a single question.

•what conventions did programmers employ to deal with segmentation?

•what extensions exist for common programming languages to accommodate the segmented address space?

These seem to be related.

Compilers use memory models that make it transparent. The models have names like "tiny", "compact", "large", and "huge" (and there are even more than that). They generate memory accesses using pointers that are "near" or "far". For "tiny" models, code, data, and stack all must fit in the same segment; "compact" has one 64kb block for code, and multiple segments for data, and the compiler dynamically assigns the proper value for DS; "large" has multiple 64kb blocks for code and data, and the compiler dynamically assigns the proper values for both CS and DS; "huge" is like "large" but allows data structures to exceed 64kb is length, and the compiler uses pointer trickery to deal with it.

Linkers grouped all of the global variables into the data segment(s), and depending on the memory model. To access the variables, the compiler would generate either a direct move from the current DS, or an LES/BX-style access and then an ES-overridden memory access.

Calling into a far routine required a "far" (generally 0x9A segment:offset) call, and the CPU takes care of saving the calling segment on the stack. The callee performs a "far" return and the CPU pops the segment register from the stack to reverse the operation.

•what techniques exist to write relocatable code for the 8086 and 80286?

The "MZ" (.exe) file format carries a relocation table that is generated by the compiler, and allows the file to be relocated by the operating system on load. There is no need to perform explicit position-independent coding if you are using a high-level language.

•are there any free/open source compilers that generate code that uses more than one data segment I can use for studying?

Watcom C++ (http://openwatcom.org/) is free and open-source.

Borland (now Embarcadero) Turbo C (http://edn.embarcadero.com/) has a free option after registration on their site.

  • How did linkers distribute global variables over segments? Was this done manually or is there some sort automatismus? How do calling conventions deal with segment registers? Are they caller or callee saved? – fuz Feb 27 '17 at 19:15
  • updated my answer to include these – peter ferrie Feb 27 '17 at 19:35
  • 2
    I don't know of books, but the user manual for Borland C, at least, probably covers all of it. At that time, everything was described in the manuals because nothing was online. – peter ferrie Feb 27 '17 at 19:53
  • 1
    If you register on the Embarcadero site, you will get access to the software and its manuals. They're probably in PDF these days. – peter ferrie Feb 27 '17 at 19:59
  • 1
    I am trying to write an assembler, linker, and C compiler for the 80286 and am unsure how to handle segmentation. Allowing for more than 64k of code and data is a definitive goal. – fuz Feb 27 '17 at 20:13
4

In terms of operating system design concepts like memory models are a bit of a red herring. Operating systems like MS-DOS and 16-bit Windows didn't care about these things. They provided a mechanism to load executables and provided an API that those executables could use to invoke system services. How programs dealt with segmentation was left to the programmers and their development tools.

This means the requirements on your 16-bit OS are fairly small. Your OS needs to support loading executables with multiple segments. It would need to allocate space and a selector for each segment. Your API would be need to use far pointers, with a 16-bit selector and 16-bit offset, wherever it passes or returns an address. With that you've pretty much all you need to handle however programs want to manage segmentation.

Relocations are maybe the only tricky part, but they're potentially easy to handle. Segmentation actually makes this easier, since in protected mode you can relocate entire segments without changing either the selector used to access them or the offsets of anything in the segment. The only relocation type you need to support is one that sets any segment references in the code and data in the executable to the selector you've allocated for them when you load the executable. (This is similar to how MS-DOS MZ executables work, with only segment relocations. Windows 16-bit NE executables support more relocations types, but this appears to be in order to support DLLs.)

In fact you can get away with not having any relocations at all by assigning each program its own LDT. This means a program can assign its own fixed LDT entries for segments and use the corresponding fixed selector value in the executable. The major drawback with this is that no OS I know of worked that way, so you won't find any development tools that will create executables like this. This also doesn't support shared libraries (eg. DLLs) as they can't depend selectors not being allocated by some other shared library. (Though you could do something like 32-bit Windows where executables don't normally have relocations, but DLLs do.)

I'd also add that memory models didn't quite work the way some of the other answers suggest. Their primarily purpose was to affect the default size of pointers, whether object pointers and function pointers were by default near or far. They had less of an effect on the layout of segments than you might think, and different compiler implementations could behave differently. For example when using a far code model (large or medium) compilers normally put each translation unit in its own code segment, however this was often undone by linkers that would try to pack these code segments together. In the far data models (large, compact) compilers normally put all statically allocated data in the same default data segment, the one which DS was assumed to always point to. You would have to use the far keyword explicitly in a declaration to have it exist in a different segment (or alternatively enable some compiler option that would do this automatically).

  • 1
    Thanks for posting this detailed answer. I hope you continue to share your knowledge. Perhaps you would be interested in related questions (visible down the right-hand side of the page). – wizzwizz4 Mar 3 '17 at 7:36
3

One thing that hasn't been addressed yet: relocation.

It's actually easier when you're using a segment:offset architecture. The segment registers take care of most of the work for you. A dos .COM file didn't have any relocation capability--it's simply a memory image, nothing more. It would be loaded wherever the operating system felt like loading it, the segment registers initialized to the segment it was loaded to and that's that.

Even when you were using a larger memory architecture most references would be near and not need relocation. When relocation is needed the program is compiled for a fixed address and a table written into the file telling the OS what needs patching (add the load address to the value found.) I never wrote anything big enough in assembler to dig into how it actually worked, though.

  • In a program with multiple code segments, I haven't quite figured out how to get shared libraries working without having to fix up relocations. – fuz Mar 3 '17 at 14:10
  • @FUZxxl - the standard approach was to fix up relocations. DOS .exe files had a list of locations where segment numbers were directly referenced, and when the file was loaded DOS added the segment address it was loaded at to whatever was in those locations. Windows real mode added a bit of complexity by supporting direct linking between files, but again this was performed by using fixups (lazy ones: the compiler would output code that invoked a relocation service and was followed by an id for the symbol that was required; on first execution this would be overwritten with a direct reference) – Jules Aug 8 '18 at 23:23
  • I wonder to what extent having the loader handle relocations was better than simply reading a blob of code into consecutive memory, puts the address of the end of the blob into a register pair, jumps to code at the start of the blob? The code at the start of the blob could then build the executable in memory as needed. This approach would work especially well with one-shot "source to executable" compilers, since they wouldn't need to back-patch anything on disk. If an address isn't available when a piece of code is compiled, the compiler could simply... – supercat Mar 27 at 15:50
  • ...keep track of where the fixup was needed and then output the fixup information at the end of the file after the address was available. This would take up a little extra space on disk, but any RAM holding the fixup information could be recycled by the application once the fixups were applied. – supercat Mar 27 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.