However, who came up with the idea and the design? Was it Sinclair themselves, or did they steal/license the idea from someone else?
Endless loop audio cassettes, or cartridges as they were called, were already an old and proven technology, way before Sinclair did the Spectrum. Evidently the first audio storage device to use this principle was the Tefifon of 1936. Here, an endless plastic ribbon carried grooves much like a record (vinyl) with a single groove going around the ribbon several times.
The first commercial successful magnetic tape version was the Fidelipac of 1954, playing mono, which evolved into the Muntz Stereo-Pak and finally into the well-known (in the US) 8-Track of 1964. There were many similar developments, like the Playtape of 1966. Even much later, when the Compact Cassette had taken over, Fisher-Price tried to use the concept with their Pocket Rockers, despite already offering CC players for children (*1).
The ZX Microdrive was just another application of the principle of an endless tape wound around a single spool. There is no principal difference to using a compact cassette as many home computers did. Recording can be done as sound, or digital. Additionally, rewinding is not needed. It's a lot like a floppy disk with a large single track.
Sinclair wasn't the first to apply it to computers. Already at the beginning of the micro computer 'age' in the mid '70s, 'Stringy Floppies' were seen as a cheap alternative to the expensive floppy drive.
Exatron presented an ESF drive for S100 and TRS-80 computers at the second West Coast Computer Fair in 1978. Compared to floppies that could easy surpass US$1000 including a controller, the US$250 Stringy Floppy was extremely appealing. Everything that was possible with a floppy could be done as well. Transfer speed was quite comparable, except that seek operations did take additional time.
Sinclair's Story starts with a suggestion by Andrew Grillet, who proposed to use a solution similar to the 8-track as storage for 'a' computer. This was already a long time before the ZX-80 and it's not really known if and how it stayed alive until 1982 when David Southward, managing the ZX Spectrum peripheral development, assigned Ben Cheese to work out the electronics and John Williams to do the mechanics for what later became the Microdrive.
The Microdrive was announced in 1982 with the Spectrum. It was meant to cost only £50 - a Commodore 1540 disk drive was at that time still over £350. So like the Spectrum itself, it was incredibly cheap. It took until summer of 1983 until the drive really materialised with its buyers.
I remember the wafers being rather unreliable, and flimsy and prone to having tape spill out of them.
While they did need some TLC, there weren't as bad as it was said. I did work quite some time with Microdrives on a QL and was quite happy. From a user view they where quite comparable to floppies on home computers and even PC.
Why were they chosen, and/or designed in the first place?
Sinclair's lever was the price. He sold computers at rates that others couldn't even print the manual.
Just the idea of them sounds terrible.
The idea is great and worked reliably for decades in music as well as for computers. If at all, a point can be made that the designers wanted to make the wafers as small as possible, which was maybe a bit too small for the parts used.
If they dreamed up the idea internally and designed them themselves, then it is a bit more understandable, and maybe they did not turn out as well as had been expected.
As with their computers, they took an existing idea, one that others already had working (see Exatron) and then stripped it to the minimum to enable the cheapest possible solution to offer a certain performance.
One could have a Spectrum with a Microdrive for less than a VIC20 with a cassette drive. If that wasn't appealing, I don't know what would have been.
*1 - Another of the quite common attempts to create a locked-in product.
