8086 stack segment and avoiding overflow in interrupts

Question

This is a followup to Could the Intel 8086 CPU have many segments in memory of the same type?

In the event of a hardware interrupt or user interrupt through the INT instruction, was there a risk that the user program had left the stack pointer close to wrapping round, so that the interrupt itself (which pushes three words onto the stack) or the code running in the interrupt would overwrite part of the stack segment?

Was there a standard value of stack that had to be reserved in order to handle interrupts safely?

Was there provision made in MS-DOS to switch stacks in order to run more complex code than the reserved space allowed?

Other systems had multiple stacks: Motorola 6809 two stacks design, the ARM FIQ system. Protected mode and multiple-ring systems also generally use separate stacks making this a non-issue.

(It's been a very long time since I've written real mode 8086 code, and I can't remember having to worry about this)

Answer 1

was there a risk that the user program had left the stack pointer close to wrapping round, so that the interrupt itself (which pushes three words onto the stack) or the code running in the interrupt would overwrite part of the stack segment?

Yep. The 8086 required space to put 6 bytes on the stack:

• The processor status
• The current instruction pointer
• The current code segment

If the stack had less than 6 bytes of space left, an interrupt would corrupt it (or something else if the program hadn't allocated 64k for the stack).

I would expect an operating system that needed to use the stack during the interrupt routine to change the stack segment to some other area to protect the user stack from overflow. This is fairly normal on operating systems for computers where the stack can be moved (i.e. not the 6502).

Having said the above, if the user stack was within six bytes of overflowing, there was almost certainly going to be a stack overflow anyway even without the interrupt.

Answer 2

We need to distinguish here between the "stack," which is a data structure in the imagination of the programmer, and what the CPU does with stack registers and the memory pointed to by them, which is a mechanical process. When the CPU pushes a value on the stack it simply changes the values in the stack pointer register and certain memory locations; whether this action changes what the programmer decided was the "stack" or some other area of memory depends on the program.

Later processors such as the 80826 offered hardware memory management that allowed the programmer to tell the CPU what the top and bottom addresses of his stack are and the hardware would generate an exception if an instruction using the stack pointer tried to write to memory outside this range. The progammer thus reserved memory for the stack by use of these settings.

But earlier processors offered no such thing; a programmer "reserves" an area for the stack only in his mind, and then attempts to make sure that his code and any other code called during the course of running his code (such as interrupt routines) won't write outside that area.

The stack pointer wrapping around may or may not be an issue depending on how the programmer set up the stack (though it usually would be). If the first free location on the stack is 0000 and you push two bytes, these bytes will be stored in location 0000 and FFFF and the next free stack location will be FFFE. Whether this is a problem depends on if FFFF was being used for something else. E.g.:

• If the programmer had started the stack at FFFF, the write toFFFF would be overwriting information earlier in the stack and you will have a problem if this is used at some point later on.
• If the programmer had started the stack at 7FFF and the memory above that was being used for other purposes, whatever was using that memory location pointed to by SS:FFFF (even though they may not have been using the Stack Segment register to access that location) will have its data changed, again likely causing problems.
• If the programmer had started the stack at 7FFF, but nothing was using the location FFFF 32K above that until after that value was popped off the stack, the programmer's expectations might be violated but no harm would come from this since it was "free" memory anyway.
• If the programmer had started the stack at 7FFF, but considered the entire 64K stack segment to be "stack" anyway, things would be just fine. The "top half" of the stack would be 0000-7FFF and the "bottom half" would be 8000-FFFF.

Programmers always had to have some awareness of what interrupt code was running in the system and what its stack usage requirements were likely to be. For example, normally the 8254 system timer calls INT 08h on a regular basis, and the default handler in the BIOS will need a little bit of stack space. A handler similar to this one from the Bochs emulator would use four bytes of stack when it pushes AX and DS, and an additional six bytes would have been used by CPU automatically pushing the flags, instruction pointer and code segement register. That handler also invokes INT 1Ch, which by default points to just an IRET instruction and so that would use an additional six bytes of stack, for a total of 16.

But this INT 1Ch handler could be, and often was, replaced by something else that wanted to be called on a regular basis, such as with a TSR (terminate and stay resident) utility. Multiple TSRs might stack on each other, each one replacing the value for INT 1Ch with its own value and then doing both its own thing and invoking the previous handler when called.

There were also typically several other interrupts regularly happening in the system (e.g., for disk drives, serial ports, and the like), some of which might even be called when the system was already handling an interrupt.

The combination of non-determinancy in the "standard" set of hardware interrupts along with some random and unknown collection of TSRs and similar software being loaded by the user before running a program meant that it's nearly impossible to know how much spare stack space you really needed. And if you overflowed your stack space, the program or user might not even know about it for some time, depending on how the stack was set up and what else was using areas of memory that might be referenced by the stack pointer.

So the basic approach you'd take is:

1. Leave as much extra space on the stack, beyond your program's requirements, as reasonably possible. If you get close to using 64K in your own stack, consider using multiple stack segments to keep a reasonable amount of free space in every one.
2. Try to ensure that any interrupt handler routines you write use the minimum possible amount of the interrupted program's stack. Consider allocating your own stack area and switch the stack to that shortly after entering the handler if you use more than a few dozen bytes of stack space.
3. Hope for the best.

Early MS-DOS didn't offer any special facilities for switching to a different stack, though of course there's nothing stopping anybody writing an interrupt handler from just saving the old SS:SP pair and loading new values pointing to an area of memory they've allocated. But starting in 3.2 MS-DOS could be configured to allocate additional stack areas and switch to them after receiving a hardware interrupt but before calling its handler. See Eric Towers' answer for details.

Multiple Stack Registers and ARM FIQ

This isn't really related to the core of your question, but since you mention these, I'll briefly elucidate on them.

As you mention, MC6809 had two "stack" registers, the system stack pointer S and the user stack pointer U. Only the system stack pointer was used by the CPU interrupt system. Whether U was even a separate stack depended on how the program decided to use it; it could also be used as another pointer into the main stack, as described in this answer to the question you linked, or it could even be used as just another index pointer, without using its push and pull operations at all.

The ARM implmentation of faster interrupts had a separate set of hardware registers that replaced r8 through r14 during an FIQ, meaning that you don't need to push them on the stack at all, saving the time it takes to do that. (It's actually a little more complex than this, due to user vs. system mode and other things, but none of that is really relevant to this question.)

Answer 3

Responding only to

Was there provision made in MSDOS to switch stacks in order to run more complex code than the reserved space allowed?

Yes, sort of. MS-DOS (3.2 and later) permitted the CONFIG.SYS command stacks=n,s, where n and s were small integers. n specified the number of stacks and s specified their size in bytes. On the IBM PC, PC/XT, and PC-Portable, these integers defaulted to 0 and 0. On anything else, n was 9 and s was 128, specifying nine stacks, each of 128 bytes.

The MS Windows 3.1 and 3.11 installers added "stacks=9,128" to CONFIG.SYS. (This was in response to potentially excessive stack usage by a combination of EMM386.EXE, SMARTDRV.EXE, the mouse driver, and TSRs hooking the keyboard interrupt.)

Documentation of this stack switching exists, describing that the stack switching was specifically to address the problem of interrupts occurring on nearly full stacks.

Beginning in MS-DOS version 3.2, MS-DOS added code to swap the stack whenever an asynchronous hardware interrupt occurs, before passing control to the original interrupt handler. (Interrupt vectors 02H, 08-0EH, 70H, and 72-77H, corresponding to IRQs 0-15.) Before MS-DOS 3.2, interrupt handlers received control with the stack set to that of whatever program or device driver was executing when the interrupt occurred. Thus, if a program stack is nearly full and a hardware interrupt occurs, random and unpredictable behavior will result (typically a system hang).

...

If stack switching is enabled, MS-DOS ensures nested interrupts do not get the same stack. Upon completion of the interrupt processing, the stack is released so that MS-DOS may use it for a future interrupt.

Having too low values could be diagnosed by

If the STACKS values are not equal to 0,0 and you see a "Stack Overflow" or "Exception error 12" message, increase the number or size of the stacks.

or

If stack switching is enabled and MS-DOS runs out of free stacks, the following message will be displayed:

FATAL : Internal Stack Failure, System Halted.

If the above message is displayed, try a higher number for n in STACKS=n,m. If you experience random hangs or other strange behavior, try increasing the m value in STACKS=n,m.

One last thing on hardware interrupt handlers. There is conflicting documentation on how to use stacks in the stacks pool. Some say you cannot:

If you are hooking interrupts with the MS-DOS get/set interrupt vector services, stack switching will not be done before your interrupt handler gets control. Any stack switching necessary must be done by your interrupt handler.

I am not aware of a method to request a stack from the stacks pool from MS-DOS, so you will have to create your own stack in the usual way. (malloc() a chunk of memory and use it as a stack. Examples.)

Others say it happened automatically on using DOS function 25h.

Function 25h Set Interrupt Vector

...

note ... 4) When you use function 25 to set an interrupt vector, DOS 3.2 doesn't point the actual interrupt vector to what you requested. Instead, it sets the interrupt vector to point to a routine inside DOS, which does this:

1. Save old stack pointer

2. Switch to new stack pointer allocated from DOS's stack pool

3. Call your routine

4. Restore old stack pointer

The purpose for this was to avoid possible stack overflows when there are a large number of active interrupts. IBM was concerned (this was an IBM change, not Microsoft) that on a Token Ring network there would be a lot of interrupts going on, and applications that hadn't allocated very much stack space would get clobbered.

So whether use of the hardware interrupt stack pool happens automatically depends on which reference you believe. Some experimentation may be required.

However, the above (solely about hardware interrupt handling) only addresses half of your query. You also asked about DOS function calls. For these, MS-DOS (3.1 and later) maintained three stacks and their use was complicated. Simplifying, entering a DOS function, one is on stack 3, then for certain functions, further execution switches to stack 1 or stack 2, depending on the function. I am not aware of any mechanism (other than the methods described above to switch stacks for hardware interrupt handlers and the use of three internal stacks to spread around the stack overhead of interrupts) used by MS-DOS to prevent/detect overflows of these stacks.

Answer 4

Unless your IRQ routine was large you'd expect enough space to be left on the stack. You could swap to your own dedicated stack before carrying on. Most of the time you would usually just be copying data over from the interrupting device anyway (or setting the next DMA buffer).

Terminate and stay resident code (e.g. SideKick) that hooked keyboard vectors would normally swap to it's own stack once it had decided to popup and take over the system.

For example the source code to print.com https://github.com/microsoft/MS-DOS/blob/80ab2fddfdf30f09f0a0a637654cbb3cd5c7baa6/v2.0/source/PRINT_v211.ASM has this code to swap the stack once it's decided to do some work.

GOAHEAD:
        PUSH    AX              ;Need a working register
        MOV     [SSsave],SS
        MOV     [SPsave],SP
        MOV     AX,CS
        CLI
;Go to internal stack to prevent INT 24 overflowing system stack
        MOV     SS,AX
        MOV     SP,OFFSET DG:ISTACK
        STI
        PUSH    ES

Answer 5

1) indeed there is a risk if user program leaves very little stack space. Hardware interrupt handlers, DOS and BIOS calls and TSRs have to assume there is "enough" stack space so they can work.

2) it would be difficult to say what is a standard value that must be available all times, as there are a lot of manufacturers for device drivers, video BIOSes, SCSI BIOSes, network packet driver TSRs etc.

3) MSDOS hooks some interrupts to run via a trampoline to change stack so that user given stack is not used during the interrupt.

Answer 6

Was there a standard value of stack that had to be reserved in order to handle interrupts safely?

No, setting a standard is next to useless (beyond wanting to have at least room for on entry, i.e. 3 words for an 8086)

Was there provision made in MSDOS to switch stacks in order to run more complex code than the reserved space allowed?

No, that's something that has always been up to the task in question (user program, interrupt routine, etc.)

So far this has been as well answered by several other good writeups. But it sounds as if there's an implied assumption that there can be an absolute protection against stack overrun, which I want to address:

Other systems had multiple stacks: Motorola 6809 two stacks design , the ARM FIQ system. Protected mode and multiple-ring systems also generally use separate stacks making this a non-issue.

Separating doesn't solve the problem of a stack overflow. It is only meant to simplify OS programming, as it removes the ned to juggle with user stacks during mode switch and/or switching stacks between user tasks.

It's been a very long time since I've written real mode 8086 code,

A stack overrun can happen in any configuration and setup - unless the absolute nesting level of all concurrent users/uses of a stack is known. Something even hard to calculate for small and well defined embedded systems - real world is is always up to screw ambitions, isn't it:))

Last but not least, even with a protected mode system able to detect stack overrun (like by bound values or segment sizes) and allocate more stack on the fly can not prevent all possible situations, as their workings rely on the ability to interrupt a such instructions, but there may be interrupt situations with higher priority than stack monitoring.

So full protection is only possible on systems without a hardware stack.

Answer 7

On some systems, context is stored during interrupts using mechanisms that are entirely separate from the subroutine call stack. An application on such a system may safely move the stack pointer to arbitrary locations without disabling interrupts, without risk of any storage near the stack pointer being unexpectedly disturbed (obviously if an application is going to make any subroutine calls or push anything on the stack, the stack pointer would have to be at a suitable location for such purposes).

On many microprocessors and microcontrollers that, like the 8088/8086, are not designed to support multi-user operating systems, however, interrupts will cause information to be stored in addresses that are computed relative to whatever the stack pointer happens to be when they are triggered. This effectively means that except when interrupts are disabled, the contents of storage in the vicinity of the stack pointer (typically at addresses below, but not including, the stack pointer) must be presumed capable of changing in arbitrary fashion at arbitrary times. Typically the amount of information stored by the processor itself for a given interrupt will be small, and the number of interrupts that can nest will likewise be small. Thus, if if no interrupt-handling code stores anything beyond that, the amount of storage one must leave available at all times would be fairly modest.

In many cases, however, interrupt-handling code will need to store the states of more registers than are used by the underlying application, and may be written in a language like C or Pascal which expects to store information on the stack. There are two ways this can be accommodated: (1) have the interrupt handler try to keep its stack usage reasonably small (under 256 bytes or so, and preferably under 100), and hope that the underlying application has enough stack space available, or (2) have the interrupt handler reserve its own storage for keeping such information, and have it start with something like:

savedSP: .ds 1
savedSS: .ds 1
mySP: .ds 1
mySS: .ds 1
IrqEntryPoint:
    mov [cs:savedSP],sp
    mov [cs:savedSS],ss
    lss sp,[cs:mySP]
    jsr far myInterruptHandler
    lss sp,[cs:savedSP]
    reti

The two main problems with this approach are: (1) it will fail very badly if the interrupt ends up being invoked recursively; (2) if other interrupts that don't use their own stacks might be triggered during this one, it will need to leave enough stack to accommodate those. In general, having all interrupts use the same stack as the application will be most time- and memory-efficient if applications leave enough space for the purpose, a situation which could have been improved if there were a standard means by which drivers could let the system know how much stack space they require, along with a means by which applications could ask the OS how much stack space must be left for all registered drivers, combined. Having all interrupts use their own stacks would be the second-best approach--not as efficient, but still robust. Using a hybrid, which is what actually happened, is the worst approach, but things somehow tended to work pretty well anyway.