The Wiki Entry says that the N64 calculated with 24-bit color, but output 21-bit.
Right,that's the way it is.
The 15 or 24 Bit colour is what the Reality Display Processor can produce (see Video Interface on p.46). Internally it works with a 32 Bit RGBA based pipeline. Results are stored as 15 or 24 Bit values before being send to the DAC. This is where the 21 bit come into play, as the interface between the RDP and the DAC offers only 7 bit per colour. Here is a nice schematic showing how the data dan be decoded with standard IC can be used to decode the data stream (Thanks to traal for finding a more information)
However, the programming manual contradicts this; on page 210, it says that the console indeed calculated with 24-bit color, but output 15-bit.
The Framebuffer mentioned on page 210 isn't memory image of a whole picture. The closest equivalence of what is shown would be a sprite buffer. Although these sprites also have coverage and Z buffer information, so the RDP can decide if a pixel mentioned there is to be taken into account. And if, how the Colour Composer and Blender stages have to handle it.
All colour computation is done with 32 Bit RGBA values.
The resulting image is of course again stored in memory as a frame buffer, so the DMA can feed the DAC to generate a picture. It doesn't matter if the picture was stored in 15 or 24 bit format, it always get send as 21 bit RGB to the DAC. Since the N64s memory is organized as two 2 Mi by 9 RAMs, 15 bit format fits nicely into an 18 bit word, while 24 bit does need two words.
Switching to 15 bit format is effectively halving the memory need for a picture without losing much detail. With 'only' 4 MiB (9-bit-bytes) total RAM, the need for 'only' ~600 KiB instead of up to 1.2 MiB is a quite substantial saving. Even more helpful in full PAL with 900KiB/1.8MiB storage needed for a picture.
And no, I have no idea why they didn't go with an 18 bit format which would have offered the same space savings. My only guess is that it nicely goes with the (also space saving) 16 bit RGBA data format for textures.