Is there some kind of upscaling happening for the lower res program,
No.
or each one retains it own res?
Yes.
I cannot understand how would be possible to send such a mix of resolutions to a monitor.
It works, as for the monitor both are the same resolution frame structure (resolution): 256 lines within a 50 (Europe) or 60 (US/Japan) Hz Frame.
TL;DR: For the Monitor 320x256x32 is the very same as 640x256x4
How about a 640x512 interlaced screen and a 320x256 [non interlaced] one?
(from comment)
To the screen that's still the very same resolution. Interlaced resolution is generated by sending out two pictures in succession, one with all even lines, one with all odd lines. The sluggishness of human perception composes both into one.
Analogue video signal does not know such thing as 'pixel'. It's structures as frames containing a (fixed) number of lines, all structured by timing.
Within a line the signal is not quantified into discrete objects (pixel) with discrete values (intensity, colour) but analog. That is a continuous signal of intensity and colour which may change at any rate (or not at all). The fact that a logic generating this signal is using distinct values (pixel) with certain limitations ('width' and 'number of possible colours') is neither transmitted nor of interest to the Monitor.
As long as the total number of lines within a frame is the same, the monitor doesn't care if the horizontal resolution is four pixel per line or 4k pixel. Ok, given, with 4k per line the picture may get blurry (*1) as the electronics may not follow fast enough, but that's a different story.
On the generator side it's similar.
(Caveat, description extreme simplified)
The Amiga's video circuit does, while generating a video line, continuously read video memory. It reads 40 bytes (=320 bit) per line for a two colour (1 bitplane) 320 pixel wide picture. For more colours more bytes (80 for 4, 120 for 8, 160 for 16 or 200 for 32) need to be read per line. In theory this is unlimited.
In reality it's limited by memory speed. This means there is an upper limit of bytes that can be read during the time available for a line. A resource shared between pixel and colour per pixel. And this is where a lower resolution comes in handy. For 160 pixel per line it needs only to read half the number of bytes (20) per line and bitplane, freeing up time to read more bitplanes, thus producing more colours.
When it comes to sending out that 'pixel' to the analogue monitor, the difference between 320 and 160 is simply that for 160 the same value is kept unchanged twice as long.
The Amiga's video generation can switch between different horizontal resolutions (and colour depths) within the duration of one line, enabling to stack areas with different resolutions on a single screen (*2). In Amiga Lingo these are called ViewPort.
Combining a 'non-interlaced' ViewPort and an interlaced one simply results in the non-interlaced lines being send out to the screen with each frame, while the interlaced one will have its even and odd lines send to alternating frames.
Now, if you are really interested on the Amiga side of all of this, the Amiga OS Documentation Wiki offers a quite good writeup of all details.
*1 - It simply smoothes out all transitions - one may think of it as analogue anti-aliasing :))
*2- Always with a blank line in-between as the first picture nicely shows, where this blanked line goes right thru the mouse pointer.