I'm trying to allocate a 64KB buffer in Watcom C 16-bit DOS. I'm using the "compact" memory model as it states [1] that the code segment is limited to 64KB through a near pointer (just offset) but that the data segment will be allocated through a far pointer (segment:offset).

From the documentation I also get the impression that using malloc in the compact memory model is simply an alias for _fmalloc.

However, when I try to allocate 64KB, or as close as I can get (1024 * 64 - 1, the maximum representable with a 16-bit size_t) the malloc fails. But when I allocate 64KB minus 32 bytes (i.e. 1024 * 64 - 32), the malloc succeeds.

How can I allocate 64KB (1024 * 64 - 1, i.e. 65535/FFFFh bytes)? And why does my malloc succeed when I subtract 32 bytes?

[1] https://users.pja.edu.pl/~jms/qnx/help/watcom/compiler16/wmodels.html

  • good point, so malloc(0x10000) == malloc(0x0) on this system. – PaulHK Nov 5 at 9:28
  • Are you sure it's 32 and not 26? I'm looking at the OpenWatcom code, getting 26 by arithmetic, and failing to find my error. – Tommy Nov 6 at 22:42
  • @Tommy Yes, definitely 32 bytes. Are you in "compact" memory model? I'm not on OpenWatcom, but the original Watcom v11.0 (I will try to upgrade to 11.0c just in case). – Luke Nov 7 at 8:02
  • @Tommy Just tested with Watcom 11.0c, definitely have to subtract 32 bytes. – Luke Nov 7 at 8:09
  • Well I got my answer working forwards from github.com/open-watcom/open-watcom-v2/blob/… but obviously miscalculated sizeof(heapblk). If I can find my error I'll turn this into an answer. – Tommy Nov 7 at 14:29

Typically memory allocators need some extra data in addition to the requested allocation size. This is for maintaining a list of allocated blocks on the heap, it's usually a link list node along with a couple of other bits of info like if the block is currently allocated or free. This tiny structure usually goes behind the pointer returned to you by malloc and is typically rounded up to some power-of-2 size to ensure pointers returned by malloc are aligned.

Because of this, you cannot allocate exactly the size of your heap because we need additional space for that block's link structure.

For example, 8 x malloc(8192) calls would also fail on the 8th call because each allocation would occupy, in your case, 8kb+32bytes.

  • 5
    I'm not entirely convinced that eight 8K allocations would fail. There's no real reason why these would have to come from a single 64K segment. The arena data could be used not just to link together blocks in this segment but also link together whole other segments (32 bytes is pretty chunky for a block detail section when all you really need is a size, forward and back pointers and possibly a sentinel). – paxdiablo Nov 5 at 10:17
  • It depends on the implementation, I was deducing it was a 32 byte overhead from the question above, which seems high to me, but it could be the OP didn't exhaustively test every size until it starts to fail. I suppose you could test it by measuring the address gap for 2 consecutive malloc calls on a pristine heap, or better yet dump the memory around your allocate to detect the presence of a embedded allocation link structure. I have no information on the actual malloc() strategy used here other than what the original question put forward. – PaulHK Nov 5 at 10:20
  • 1
    Thanks Paul, for your answer. I'm sure you are correct but I'm looking for actual documentation from/by Watcom that would confirm this. – Luke Nov 6 at 8:01
  • You could inspect the memory around the pointer that malloc returns: e.g. void *p1 = malloc(256); memset(p1, 0xcc, 256); void *p2 = malloc(256); memset(p2, 0xdd, 256); --> Stop your debugger and dump memory around 'p2' -- You should be able to see if there is a gap between the 0xcc's and 0xdd's that the malloc subroutine inserted for housekeeping data. – PaulHK Nov 6 at 8:33

How can I allocate 64KB (1024 * 64 - 1, i.e. 65535/FFFFh bytes)?

I have not tried it but the following trick might work:

You can use halloc(0x1000, 0x10) to allocate a "huge" memory block of 0x1000*0x10 bytes (= 64 KiB) size. According to the manual this block will be 0x10-byte aligned if the second argument is 0x10.

halloc() can allocate more than 64K but returns a huge memory pointer, not a far memory pointer (see below).

In "Real Mode" ("normal" MS-DOS) the linear address is calculated as 0x10*segment+offset so if the offset is a multiple of 0x10 (if the address is aligned to 0x10 bytes) you can calculate a far pointer that points to the same memory using the following method:

You take the offset, divide it by 0x10 and add the result to the segment. Then you set the segment to 0.

Example:

union {
    struct {
        unsigned short off;
        unsigned short seg;
    } s;
    void far *ptr;
} u;

u.ptr = (void far *)huge_pointer; // Example: 0x1234:FED0
u.s.seg += u.s.off >> 4;
u.s.off = 0;
far_ptr = u.ptr; // should be 0x2221:0000

Both 0x1234:FED0 and 0x2221:0000 point to the same memory.

However there are two differences between the pointers:

  • hfree() and similar memory management functions must be called with the "original" pointer returned by halloc(), not with the "calculated" one

  • The difference between huge and far memory pointers is that far pointers do not allow crossing segment boundaries: A far pointer to X:Y can only access (0x10000-Y) bytes of memory. So a far pointer to 0x1234:FED0 can only access the first 304 bytes of the memory block (0x10000-0xFED0 = 304) while a far pointer to 0x2221:0000 can access the first 64 KiB (note that halloc() can allocate more than 64K).

  • This looks like a good solution to me. The code for halloc is here and it looks as though it always allocates aligned to a paragraph boundary, whatever the size is, anyway. This is definitely a better solution than the one I'd have suggested (making sure not all DOS memory is allocated to the heap -- I'm not familiar with Watcom's tools, but there should be an option for how much heap to allocate somewhere -- and then using INT 21h/AH=48h to perform the allocation). – Jules Nov 6 at 21:13
  • Another thing to note from the halloc code is that the returned pointer always has its offset set to zero, which should simplify the translation to a far pointer somewhat... – Jules Nov 6 at 21:15

A community wiki answer because it includes a lot of useful evidence, but I seem to flub the numbers somewhere:

Per OpenWatcom, the definition of _fmalloc is this. Notably:

if( amt == 0 || amt > - ( sizeof( heapblk ) + TAG_SIZE * 2 ) ) {
        return( NULL );
}

Allowing for integer rounding and given that size_t is 16-bit, that asserts that if the block you asked for is larger than 65536 minus sizeof( heapblk ) + TAG_SIZE * 2 then the allocation will fail.

heapblk is a struct:

struct heapblk {
    tag                 len;                /* size of heap (0 = 64K) */
    heapptr             prev;               /* segment selector/offset for previous heap */
    heapptr             next;               /* segment selector/offset for next heap */
    freeptr             rover;              /* roving pointer into free list */
    unsigned int        b4rover;            /* largest block before rover */
    unsigned int        largest_blk;        /* largest block in the heap  */
    unsigned int        numalloc;           /* number of allocated blocks in heap */
    unsigned int        numfree;            /* number of free blocks in the heap */
    freelist            freehead;           /* listhead of free blocks in heap */
#if defined( __WARP__ )
    unsigned int        used_obj_any :1;    /* allocated with OBJ_ANY - block may be in high memory */
#endif
};

And from line 50 of the same file, TAG_SIZE is just sizeof(tag).

On line 106 tag turns out to be a typedef for unsigned int so it's also 16 bit.

heapptr is a union defined on line 115, of a __segment and a heapblk_nptr (a.k.a. a _WCNEAR). So it's two bytes as nptr means near pointer, and a __segment means 16 bit.

freeptr is a union defined on line 120, of a freelist_nptr and an unsigned int. So also two bytes.

freelist is a struct defined on line 125 consisting of a tag, and two freeptrs. So six bytes.

So a heapblk is 22 bytes of meaningful storage:

  • 8 bytes for the four unsigned ints;
  • 2 bytes for the tag (because it's another unsigned int);
  • 6 more for the freelist;
  • 2 for the freeptr; and
  • 4 in total for the heapptrs.

That plus another 4 for the 2*TAG_SIZE is 26.

Therefore as posited elsewhere, you cannot allocate a full 64kb because some storage is reserved for heap management; however I seem to total that to 26 bytes, not 32.

  • I can’t see how in this case, but alignment issues sometimes explain surprising struct sizes... – Stephen Kitt Nov 7 at 18:30
  • Yeah, I kind of assumed that since everything is two bytes in size and it's a 16-bit machine, everything was naturally aligned. But maybe I'm wrong that everything is two bytes in size, or there's an additional alignment factor that I'm unaware of? – Tommy Nov 7 at 18:42
  • I can’t think of anything. But wouldn’t sizeof( heapblk ) refer to the heapblk typedef rather than struct heapblk? (Yeah that doesn’t help solve the mystery at all...) – Stephen Kitt Nov 7 at 18:50

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.