The "large" memory model of x86 allowed a program to have multiple code segments and multiple data segment, using far calls to jump across code segments, and far pointers to access data from several data segments. But how about the stack segment?
Stack is a segment like any other. Thus changing SS allows to have multiple stack segments.
How did 16-bit MS-DOS programs use a large (> 64KB) stack?
They did not, as there can be none. Segments are always restricted to a maximum size of 64 KiB. That's true for all segments. The fact that its possible to manage larger structures than 64 KiB by manipulating segment registers as if they are address pointers is not in any way supported by the CPU, but handled under program control.
For the stack as a CPU controlled data structure (PUSH/POP/CALL/...) this means that no single stack can exceed 64 KiB. But like with data segments, a program can have as many as it wants (*1) and reload the segment register as they wish.
Of course, a procedure doing so should restore the original value of all registers before returning. Also, if stack is used to return data, teh procedure needs to do so on the original (callers) stack, not the new one. All of that is not supported by any language runtime I remember and would require some assembly.
Was there an additional calling convention where SS was adjusted at the beginning and the end of a procedure?
No. If needed for some reason, it had to be done 'manually'.
Or maybe was it the case that every program did avoid using more than 64 KB of stack?
It's hard to imagine a use case for more than 64 KiB of stack - or even 64 KiB at all - with the PC as intended target. For a PC that's a good tenth of the whole memory. An application needing that much stack space might be one more tailored for a mainframe or sole larger mini computer.
The only real use case for a program/procedure is when it can't 'trust' the caller to provide the stack space it needs to run. In that case the procedure would save the callers SS/SP and set them to it's own stack space.
Situations like that may happen with libraries, background (interrupt) functions and of course the OS itself. And in fact does MS-DOS for all of its more complex functions switch to its own stack, and back when returning.
Changing stack does add several pitfalls, especially in small systems like DOS.
- Changing stack adds an overhead
- Copying from and to the caller stack needs far addressing and careful handling
- Private stack can usually not be allocated dynamically
- Private must be static allocated, increasing memory requirement
- The pointer to a private stack must be kept in a unique location addressable after entry, thus
- either an absolut address (not exactly what user programs get),
- or kept (as daa) in the code segment, aka modifying it.
- Locking is required for certain operations - especially switching
- Additional care needs to be taken to allow reentrance
All of this makes stack switching restricted to very special cases. DOS itself being the only common case, but here as well quite restricted. That's why DOS is for many functions non-reentrant, and as well the reason why parameter passing is done in registers - that way all pitfalls about stack handling can be avoided.
*1 - Well, they still should fit into memory and be not more than 65536 thereof ;)