3

For a game reversing project, I am trying to undo linking with the C library performed by the MSC 5.1 C compiler+linker.

To that end, I have created a simple executable that does nothing except reference the same libc routines as my game, hoping to be able to tell what data is added by libc and where:

/* uninitialized (BSS) data */
int x;
int y;

int main() {
    fclose();
    fopen();
    // [...]

    x = 1;
    y = 2;
    return 0;
}

It was relatively easy to find the initialized data added by libc using some sentinel values. However, the unitialized area (BSS) is a different story. When linking with just the startup code (crt0), the layout of the BSS is the following when viewed in IDA:

dseg:01F2                 db 0FFh
dseg:01F3                 db 0FFh
dseg:01F4                 db 0FFh            ; 3x 0FFh, libc initialized data end marker
dseg:01F5                 db 0Bh dup (?)     ; 11 bytes of uninitialized data, presumably from libc startup code
dseg:0200 word_10720      dw ?               ; x
dseg:0202 word_10722      dw ?               ; y
dseg:0204                 align 10h
dseg:0204 dseg            ends

However, once I link in any of the file I/O-related functions (fopen/fread/fflush), the layout changes drastically:

dseg:0346                 db 0FFh
dseg:0347                 db 0FFh
dseg:0348                 db 0FFh
dseg:0349                 db 407h dup(   ?)       ; 😭😭😭
dseg:0750 word_11CD0      dw ?                    ; x
dseg:0752 word_11CD2      dw ?                    ; y
dseg:0754                 db 20Ch dup(   ?)
dseg:0754 dseg            ends

My sentinel data gets pushed to a seemingly random offset, and the BSS grows by 0x3fc (0x407-0xb) before the data, and 0x20c afterwards.

These two areas don't seem to be referenced anywhere (especially not within any libc function) in the executable, I can't see them when dumping the object files of the associated functions extracted from libc, and I can't tell where they came from. The only place where there is a reference to BSS, is in the startup code which zeroes it out before executing main().

Without knowing the origin of these blocks, I'm unable to locate them in the actual game executable, and consequently to undo the linking by leaving out the uninitialized data introduced by libc functions.

5
  • 1
    I assume you're aware that streaming I/O needs some management like lists of FILE structures, buffers, etc. ? While most goes into dynamic allocated memory, global information/root pointers need to go into static BSS memory. They will (usually) not be touched by your code but the linked library, possibly by using pointers. You might want to dissect the lib to see more.
    – Raffzahn
    Jan 7 at 15:09
  • On a more general note, are you sure that this is linked with the default C-Lib? Would be unusual for a game project that large not to build their own, optimized libraries which may organize functions and code chunks in more appropriate manner. ame goes for the assumed 'overloading' of functions - they rarely are to provide non existent functions, especially not as basic ones as itoa, but rather offering optimized ones. Usually leaving off stuff the application doesn't need to save space or gain speed. Non-Fat -Lib were an art form back then :))
    – Raffzahn
    Jan 7 at 15:22
  • 2
    Back when I cared about such things (but Windows, not DOS), the C RTL sources were on the MSDN discs. I imagine it's possible to locate them even now.
    – dave
    Jan 7 at 16:43
  • 1
    @Raffzahn I was fully expecting libc to use these areas, the problem is that looking over the code of all libc functions that made it into this executable, I can't see any usage of it. I checked all immediate offset values, and the only place referencing the BSS is the code of start from crt0.asm. Edited to clarify. Jan 7 at 16:54
  • 3
    @Raffzahn I am sure the game is using the version of libc shipping with MSC 5.1 based on the Microsoft copyright watermark left in the executable, I also located code of libc routines from that library in the exe using IDA signatures, and the initialized data of the game also contains error strings and other data that match with what was linked in from libc into this example. Jan 7 at 16:57

2 Answers 2

4

Turns out the MSC linker can put BSS data in an executable that is not referenced anywhere.

I inspected the map file generated by the linker and found some public names (__bufin/__bufout/__buferr) around my sentinel data:

0158:0350       __bufout
0158:0550       __bufin
0158:0750       _x
0158:0752       _y
0158:0754       __buferr
0158:0960       _end

Using dmpobj from OpenWatcom on all object files extracted from the MSC libc, found the COMDEF record defining these symbols in _file.obj, the sizes add up to 1536 bytes:

ninja@dell:slibce$ dmpobj _file.obj
[...]
COMDEF(b0) recnum:13, offset:000000b0h, len:002ah, chksum:f7h(f7)
    2 - '__bufin' Type 0, NEAR, Size:00000200h
    3 - '__bufout' Type 0, NEAR, Size:00000200h
    4 - '__buferr' Type 0, NEAR, Size:00000200h

This object file (containing only data) must have been pulled in by the linker to satisfy some dependency for fopen and friends. Anyway, by inspecting other object files referencing these symbols, I was able to locate code (__flsbbuf) and data spelling out the exact locations of these buffers. With this information, the layout of the BSS is as follows:

dseg:0346                 db 0FFh
dseg:0347                 db 0FFh
dseg:0348                 db 0FFh
dseg:0349                 db    ? ; probably alignment
dseg:034A bss_start       db    ? ; crt0 data     ; DATA XREF: start+66
dseg:034B                 db    ? ;
dseg:034C                 db    ? ;
dseg:034D                 db    ? ;
dseg:034E                 db    ? ;
dseg:034F                 db    ? ;
dseg:0350 bufout          db 200h dup(   ?)       ; DATA XREF: __flsbuf+9A
dseg:0550 bufin           db 200h dup(   ?)       ; DATA XREF: dseg:012E
dseg:0750 word_11CD0      dw ?                    ; x
dseg:0752 word_11CD2      dw ?                    ; y
dseg:0754 buferr          db 200h dup(   ?)       ; DATA XREF: __flsbuf:loc_10BBA
dseg:0954                 db    ? ; probably more alignment to end on paragraph boundary
dseg:0955                 db    ?
dseg:0956                 db    ?
dseg:0957                 db    ?
dseg:0958                 db    ?
dseg:0959                 db    ?
dseg:095A                 db    ?
dseg:095B                 db    ?
dseg:095C                 db    ?
dseg:095D                 db    ?
dseg:095E                 db    ?
dseg:095F                 db    ?
dseg:095F dseg            ends

The crux of the issue was that I was investigating a minimal example which had the data but not the code using it, which I didn't think was possible. After adding in the missing functions I was able to find the link between them and the BSS data, and eventually carry over the results to my game reconstruction.

9
  • 1
    Detailed writeup of the investigation: neuviemeporte.github.io/f15-se2/2024/01/07/unlink2.html Jan 7 at 23:52
  • 4
    Not that surprising. I think early linkers tended to pull dependencies object-file by object-file, not symbol-by-symbol. Jan 8 at 9:11
  • 1
    With names like "bufin", "bufout", "buferr", I'm pretty much 100% certain they are the buffers associated with the predefined stdin, stdout and stderr instances of FILE.
    – JeremyP
    Jan 8 at 15:18
  • 2
    @Raffzahn: technically yes, but what the question was about was where the buffers are coming from, or barring that - how to determine their origin. That they serve a purpose, and that they must be used by libc was not in dispute. But I could not find a routine using them which didn't make sense and was the reason for the question. Jan 8 at 15:48
  • 1
    @neuviemeporte Pulling dependencies in linkers usually works at best at section granularity (or object file granularity), not symbol granularity; COMDEF notwithstanding, linkers usually do not even have enough information to extract individual symbols (especially their sizes) from object code, and there may be internal references within the object file that were already resolved at assembly time, which the linker cannot see; so it has to pull a whole section or file. Jan 9 at 14:32
2

TL;DR: Linker Links Without Judgeing

A linker doesn't know about programs and functions and even less about usage. A linker combines objects, including all segments and resolving open relation. If a data area is defined in an object it will be included in the resulting object. The linker has no way to determinate if it's used or not - nor is it his job to do so.


The Basics

The issue seems to be originated in an in complete picture about what a compiler creates, a library stores and how a linker handles it. So lets take a detour to look about the different things we like to subsume under compiling and linking(*1):

The Compiler

To start with, a compiler does not create programs or functions, but object modules. Program and function are source level categorises programmers of a certain language will use. And while the compiler will understand their meaning within its language and handle it accordingly during source handling, his output will be nothing but an object modules or objects for short. Usually one per compile run. Objects are language independent items. An object does not hold 'functions' and 'data tags' as distinct elements, but a collection of segment, each a list of size and content as well a list of names associated with offsets within those segments and fixup information containing how to modify content when reordering those elements into a new object.

All this is stored as a sequence of records - a structure that dates back to the earliest days of computing. In case of Microsoft tools it's based on the Intel object format (*2) with some MS-specific modifications.

C-compiler are build with the assumption of creating a standard set of segment names (BSS,TEXT,DATA,STACK) according to the way C structures programs, but the object format couldn't care less. Here a segment name is without any meaning, beside being a unique identifier. What it does and how it's handles is defined by attribute records.

Important point to remember: C specific ideas and concepts stop here. Every handling once an object is created is independent of language and compiler.

The Library

A library is just a container of objects - much like a ZIP archive.

I most systems a compiler outputs the object created as standard file (*3) - the one C likes to call a.out (*4). That output may, depending on OS, be directly run. But more often than not it's just a part of a larger project or some generic utility, like C functions to handle streams. Back in ye olde days creating a file was a complex issue and having dozens if not hundreds of files almost impossible. To handle that libraries were invented, collection of various items that could have been files on their own. One special case were object libraries, especially suited to store and quickly retrieve object modules.

Using libraries simplified handling as well as software delivery. Including run time support like the C-Lib provides. What was true in the 1960s is still true today. Also, while file clutter when installing isn't uncommon on today's small systems, the use of a libraries is still of help to reduce such.

Microsoft libraries are rather simple structures consisting of a library header followed straight appended object modules.

The Linker

While it's possible to write a single object for a single program, we all know and appreciate the ability to work on independent modules and clear interfaces - in our own projects as well as for using components others provide. Someone has to assemble them: The Linker.

Unlike often assumed a linkers primary job is ot to create a loadable program, but to join multiple objects into a new combined object, which then can be put to further use - including being executed.

To do so the linker works solemnly with the information given in the objects involved. As described they do not contain any information about the source level structure (functions etc.). Only description of segments, their attributes and noteworthy offsets within to be modified. The objects to be linked are given by command interface. In addition libraries can be given, allowing the linker to search those for objects that fit - simplifying the process of delivering a group of modules quite a lot :))

It will try to resolve whatever can be resolved (combined). The result will always be another - hopefully one with less unresolved references :)

It's important to keep in mind that the linker has no idea what a TEXT, DATA or BSS segment is. They will be processed like any other random segment name. Only their attributes do count.

The Other Kind of Linking

An object module may already be an executable program. That's the case if the OS-loader simply used the same object format as compiler/linker produce (*5). With others may need a converter. MS-DOS is of the later kind. While it's EXE ('MZ') format is not a memory image, but an object format that still needs adjustment before execution, only it's format differs from the format MS defined for objects its compilers create (*6)

In fact, even this part does not really know about C style segment names. It will blindly output all groups as attributed and update the header accordingly.

Those Basics Applied

Now let's look at the issue using above view of object modules and how they may create the observed result (*7):

So imagine the developers writing the C-Lib for their C-compiler structuring the various function for streaming I/O (stdio) into multiple objects but using common header files defining global data structures used by some of them.

Let's say they create two objects 'FOPEN' and 'FREAD' each just consisting of the C-Lib function they are named after (fopen(); fread()). Both are compiled separate, but both include a common global data definition (say via a .H file). One coudl think of a single read/write buffer plus maybe some global flag(s) (*8).

Now lets assume of them only READ will also use the buffer, FOPEN not. Still, if either object gets linked into a new object, the buffer and any other global data defined by that common include will be part of that new object. The linker doesn't care if the data item is used or not. It's defined as part of both and will thus be included.

All as expected.

Why?

Of course one can now ask why they didn't split the global definitions into multiple files and include only those really needed for each object - like defining he buffer only in FREAD, not in FOPEN. Sure, would work, but what's the use case? Open without any Read (or Write) in a program is at best a strange fringe case (*9).

On the other hand, we all know, spreading information over evermore include files is a first class source for problems. So, it's rather understandable of MS using a single data definition for all related functions of a package - even if compiled separate.


Bottom line:

*A linker does not judge usefulness of the items within an object to be linked (10). It simply processes it as defined.


*1 - I will try to stay with only a rough description necessary to see what happened above, not documenting every detail.

*2 - The well known Intel-Hex format is a subset of the full object format well known for handling ROM data.

*3 - Technically a Microsoft object file can contain more than one object (something Assembler programmers enjoyed to use). Objects are structured as records, ending with a MODEND record, any following records will be interpreted as a (hopefully) new object module.

4 - Other, like all TSOS offspring provide a special unnamed container ('') to free scripts from reserving and assigning storage space - something that can be quite cumbersome in vastly different environments - by using virtual memory to hold compiler output.

*5 - Some OS even offer Link-Loaders which are not only able to load a single object module into memory and execute it, but also perform linking with (standard) libraries at load time. Quite handy to reduce code size on disk as well as making sure OS interface libraries are always up to date. This is related to dynamic linking, but different from dynamic loading at runtime under program control.

*6 - The object format is a carry over from earlier usage. CP/M as well as stand alone products, while the EXE format is a new creation for MS-DOS to support the increased capabilities of the 8086 while not going all the way of directly handling objects. After all, it was to be integrated into the DOS kernel, so compactness was a plus :)

*7 - Note this is just an example how it most likely worked. I have not dissected the library used (not linked anyway), so the exact structure may diverge. More so, there are other configuration that may lead to the same result, due the same mechanics, but using different compartization.

For example, only FOPEN declaring those sections, even though not using, After all, if one tries to split the streaming functions into smaller chunks so only needed ones are included, FOPEN would be always needed ... well or not in case of standard handles. Or in other words, there are countless ways to organize that. For a definite answer one would need to dissect the library used.

*8 - Think error code or active or whatsoever.

*9 - Yes, i know, one could build a 'size' utility that only does fopen(); fseek(,,SEEK_END);fclose() to get a file size without using any other stream functions, but lets stay serious...

*10 -Having said that, there are optimizing linkers that try to be smarter than the programmer they serve. They only work sufficient reliable within very strict regulated environments. Though, problems they create are, in my experience very hard to detect and correct. There is a reason reliable linkers are very conservative creations, working almost unchanged since the1960s.

2
  • if FOPEN and FREAD are compiled into separate objects and "both include a common global data definition", then they can't be linked together because that means multiple definitions for the same symbol. Did you mean declaration? Jan 10 at 11:40
  • @neuviemeporte o. You're still thinking on language level. The linker works on object level - where names are considered the same. It might be helpful if you insert a detour and learn about the Intel/Microsoft object format and how the linker works first. Use an object lister on the modules involved (start by looking up what's inside the C-Lib you use) to see what records there are. A first reading might be (as so often) the fine MS-DOS Encyclopedia, especially article 19&20.
    – Raffzahn
    Jan 10 at 13:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .