According to the very same link you provided:
To perform this feat, Oktalyzer loaded eight channels in memory, mixed
them in real time down to four channels, and sent the result to the
Amiga sound chip. This was a processor-intensive task which degraded
sound quality, but was more than made up for with doubled channels.
Oktalyzer could also be run in 4-channel mode to suit more
which makes more sense to me that splitting each channel into two 4-bit channels.
So the process is quite straightforward. During the time between two VBlank interrupts (these occurs normally each 20 ms), the CPU has to take two sample buffers, mix them and write the result to a channel buffer that will be used to feed Paula.
This is quite processor intensive, as for each sample in a 20 ms buffer, the CPU has to calculate what sample to fetch (depending upon the playing frequency), check if it reached the end of sample so it will start at repetition begin. With both samples gathered, each one has to be scaled according to its volume setting, then the CPU has to add them and possibly, apply some compression to avoid clipping while keeping the original dynamic range (adding both sample and then dividing into 2 would result in too quiet sound)
I've just digged into the source code of an early release of Oktalyzer and found this macro: (comments of my own)
move.b (a0,d0.w),d4 ;get sample from buffer addressed by a0
add.b (a1,d1.w),d4 ;add sample from buffer addressed by a1
eor.b d5,d4 ;d5=-128. Fast way to add -128 to result
move.b d4,(a5)+ ;store result into output buffer
add.l d2,d0 ;calculate offset for next sample, buffer a0
add.l d3,d1 ;ditto for buffer a1
Which pretty does much of what I've explained (minus the volume scaling operation).
From this source code, I guess that initial buffers pointed by
a1 already contain scaled samples, as there is no code in the macro to check for clipping. I also assume that these original samples are 7 bits unsigned samples. Adding them won't produce clipping, but we will end up with a 8-bit unsiged sample, which is not suitable for Paula, which expects 8-bit signed samples. To get that, we have to substract 128 from the sample value, that's to say, add -128. Adding -128 to a 8-bit number is just toggling its MSb, so d5 is preloaded with -128 and then it is XORed with the calculated sample.