As you can see from the schematic, the Armstrad uses a 6845 compatible display controller together with a custom-made Gate Array or ASIC for video output. The Gate Array also controls the screen modes using two bits of a register.
We'll never know exactly what happens until someone reverse engineers the Gate Array, and while it has been decapped and photographed in April 2016, so far I don't know of any available schematics.
But to illustrate the principle, we can design our own gate array for Modes 0 to 2, and then see what happens if we switch to Mode 3.
To design a gate array, we need to write down boolean equations for the part we are interested in and convert them to disjunctive or conjunctive normal form while at the same time simplifying them, ignoring inputs we don't care about. The simplified equations then form the basis for programming the gate array.
The 6845 produces the RAM address of each data byte d we want to display in turn, and it's our task to convert this data into a pen-table index p, depending on the mode m. With max. 8 pixels per byte, let's use an internal 3-bit counter b for the bit position in the byte. We'll just look at the first nibble, the second nibble works in the same way. Here are tables for our 3 modes:
Counter Mode 0 Mode 1 Mode 2
b2 b1 b0 p3 p2 p1 p0 p3 p2 p1 p0 p3 p2 p1 p0
-------- ----------- ----------- -----------
0 0 0 d3 d2 d1 d0 0 0 d1 d0 0 0 0 d0
0 0 1 d3 d2 d1 d0 0 0 d1 d0 0 0 0 d1
0 1 0 d3 d2 d1 d0 0 0 d3 d2 0 0 0 d2
0 1 1 d3 d2 d1 d0 0 0 d3 d2 0 0 0 d3
So in Mode 0 we use always the same 4 bits for the pen index for a single wide pixel, while in Mode 2 we use each bit of the nibble in turn for four narrow pixels.
Writing / for "not", * for "and" and + for "or", we can read off the simplified equations in DNF:
p3 = d3 * /m0 * /m1 # d3 in mode 0
p2 = d2 * /m0 * /m1 # d2 in mode 0
p1 = d1 * /b1 * /m1 # d1 for b=0,1 in mode 0 & 1
+ d1 * b1 * /m0 * /m1 # d1 for b=2,3 in mode 0
+ d3 * b1 * m0 # d3 for b=2,3 in mode 1
p0 = d0 * /b1 * /b0 # d0 for b=0
+ d0 * * /m0 * /m1 # d0 for mode 0
+ d1 * /b1 * b0 * /m0 * m1 # d1 for b=1 in mode 2
+ d2 * b1 * m0 * /m1 # d2 for b=2,3 in mode 1
+ d2 * b1 * /b0 * /m0 * m1 # d2 for b=2 in mode 2
+ d3 * b1 * b0 * /m0 * m1 # d3 for b=3 in mode 3
Now let's set m0=1 and m1=1, and see what happens in Mode 3:
Counter Mode 3
b2 b1 b0 p3 p2 p1 p0
-------- -----------
0 0 0 0 0 0 d0
0 0 1 0 0 0 d0
0 1 0 0 0 d3 0
0 1 1 0 0 d3 0
So for our implementation, Mode 3 is 320x200 with 2 colors, using bits d0, d3, d4 and d7 to select the pen for the color. (I've probably made a mistake somewhere, but it still illustrates the principle).
Now the real Gate Array uses a different design: Maybe a shift register instead of a counter, or CNF instead of DNF, or internally negated values, or ... But the effect is the same: Simplifying the boolean equations leads to an unintended mode that "sort of, but not really" does something useful.
(BTW, a similar thing happens for the "undocumented" opcodes of the 6502).
