[Part of what is described here can be found on Herb Johnson's great site about CP/M history, the other is experience of 30+ years in mainframe procedure]
CP/M was, Dave explains, heavily influenced by DEC's TOPS-10 operating system, one Kildall was quite familiar with. Its 6.3 structure nicely fits the 36-bit nature of the PDP-10 (as well as other DEC machines).
But to get to CP/M, the story took several intervening turns.
Kildall wrote 8008 emulators and a PL/M compiler for Intel (later used in the upcoming ISIS operating system). The compiler was developed on a PDP-10 in Fortran under TOPS-10 - Intel used PDP-10 before the MDS-800 (the first 8080-based development system, running ISIS) was ready.
At that time, Kildall had already started a project to make PL/M a self-hosted system on an 8008 and later 8080. He tried to sell this to Intel, but Intel refused, as they believed then that all development would stay on minicomputer systems (notably PDP-10). Systems like the Intelec-8 were only intended for debugging and as a flexible environment for prototypes.
This changed soon and Intel opted to develop the Multibus and the MCS-800 as a stand-alone 8080-based development system - with ISIS as its designated operating system. Except Intel opted to develop in-house, not using Kildall's offer.
Being, for the most part, a down-port of the tools used on the PDP-10, it came naturally that ISIS used many of the conventions of TOPS-10, including a file system with a 6.3 name convention.
Kildall in turn acquired one of the first MDS-800 systems (running ISIS) to finish his CP/M project for generic 8080 machines. The very first known version of CP/M has a BIOS fitting the MDS-800.
While proven to work fine in many DEC installations, the 6.3 scheme had a huge shortcoming when transferring/accessing IBM files, as IBM used a basic 8-character file name for tape files and in turn also for (floppy) disk (*1). Transferring an IBM file to TOPS-10 more often than not ended up crippling the name - especially if the extension had to be spared for TOPS-10 usage. It's easy to imagine how this resulted in endless problems when exchanging more than one file at a time.
Increasing the file name size to 8.3 added legroom to reduce such issues.
Equally importantly, the CP/M catalogue entry simply had room to do so. One catalogue entry is 32 bytes, of these the first 16 are used to store the metadata, while the second store up to 16 block numbers the file uses.
3 bytes were needed for organisation, which would leave 13 for a file name. Instead of using all, two were kept as reserved (*2), resulting in the 8.3 structure we know (*3).
So in the end it's simply what was possible without spending everything.
*1 - Well, no, it's not 8, but 17 in a volume label, or 44 in a catalog entry, which are again made up of one or multiple times 1..8 characters joined by a '.' giving something we would today call sub-directories - except they weren't really ones. Confusing? Well, not really, but an explanation would need quite some room to describe each environment. In the end and for all practical means, exchange files were usually kept to 8 character names.
*2 - Quite a good idea as one of these was later (CP/M 2.x) used to allow files larger than 512 KiB.
*3 - It's maybe important to keep in mind that these were NOT 8-bit characters, but 7-bit. This fact was used in CP/M 2.x to store various flags in the high bits of the file type for flags (Read Only and Hidden) and in CP/M 3.x to add a backup bit, while the second 'reserved' byte now added an 'incomplete record' counter, which finally allowed that an exact file length could be managed - before that, files were always multiples of 128 and CTRL-Z used as EOF marker ... if possible.