I wonder why the Commodore 64 memory map was created as it was.
As so often it helps to look at a machine's predecessor; in the case of the C64 that's not the VIC 20, but the Max Machine, a very minimalist game system/computer. After all, the C64 is a quick hack, filling that design with 64 KiB of RAM, a second CIA for user port/serial (*1,2) and on-board ROM with BASIC. Thus the C64 memory map is based on the Max Machine's:
$0000-07FF 2 KiB RAM
$0800-7FFF free for expansion (*3)
$8000-9FFF 8 KiB for ROM module
$A000-BFFF 8 KiB for ROM module
$C000-DFFF 8 KiB for I/O
$E000-FFFF 8 KiB for ROM module
The upper 32 KiB address space of the Max Machine was basically divided into four 8 KiB Blocks with 3 of them for ROM, except there was no ROM on board. ROM was provided on cartridges, including any character ROM (*4).
The C64 was designed to keep maximum compatibility to the Max Machine (*5). That's also why there is the
/GAME signal on pin 8 of the cartridge port. Grounded on the Max cartridges, it was pulled high on the C64 to serve as input to the PLA to put the machine into 'Ultimax' mode, making it fully compatible. That way the C64 was able to run all those many Max cartridges that were expect to come (*6).
/GAME pulled high, all internal ROM (and RAM above $1000) was disabled and to be replaced by cartridge. To still allow the addition of C64 cartridges that can take advantage of all RAM and ROM another signal
/EXTROM was added, which when pulled would default to an an 8 KiB area at $8000 for external ROM but everything else as with the default C64. With both pulled, a second 8 KiB at $A000 would be provided instead of BASIC.
Bottom line: The seemingly strange C64 banking is all about compatibility with the Max.
Is it known why the real arrangement was chosen instead of the one I described?
In hindsight much could have been improved with the C64, not just the memory map. Back then it was all about getting it out as fast as possible.
Was there a conscious decision to keep 4K of non-banked RAM outside of the reach of BASIC to be able to store machine-code subroutines there?
As mentioned, compatibility. Easy to see when noting that in Ultimax Mode those 4 KiB are not mapped.
*1 - A second I/O chip was needed to offer the serial IEC as well an user port.
*2 - The expedited development is also the reason why there are two CIAs with two clocks instead of a CIA and a VIA - it was less risky to use the same chip twice.
*3 - For example, used to add 2 KiB of RAM with the 'full' BASIC cartridge.
*4 - This is also the reason why the C64 has the added complication of the character ROM in parallel to I/O: They had to find a place that could not have been used with other cartridges. A place 'outside' the map, but still visible to the VIC. The 4 KiB I/O space is the place the VIC will least likely access for character definition data in RAM or ROM, making that the perfect place to put it.
*5 - After all, the Max was expected to be sold for a longer time, while the C64 was only intended as a short time stop-gap measure until the TEDs were done.
*6 - This turned out to be a gift to piracy. Ultimax mode meant that a cartridge can take over all vectors, which freezers made good use of :))