While Stephen Kitt's answer already hits the core, I believe it needs a bit more history, as direct access magnetic storage did start quite a while before the IBM 350.
Drums and Disks
First there were drums. Drums were huge cylinders with a magnetic surface, and a separate head for each track - something that would have been rather impossible with a disk. Drums are somewhat located at the the borderline between main memory and external storage. Except that the idea of external storage wasn't even formulated back then the way we see it today (*1). The drum was the main memory of early computers and the CPU was built around its timing. There was no memory to hold even a single track in direct access (*2).
The IBM 650 Magnetic Drum Data-Processing Machine of 1953 is a good example. It was built around a drum with up to 80 tracks holding 50 words of 10 decimal digits each, so lets just make that 5 Bytes, giving ~20 KiB in today's units.
Drums were in use way until the 1970s and gained sized up to 200 and more heads. while also enhancing data density per track, resulting in drives with several megabytes. On a drum, heads were neither flying nor movable. Each track had it's own head. Thus there was no seek time, no head settling, at maximum waiting for one turn to pass was needed. Which at 10,000 RPM (and more) wasn't much.
It wasn't fun to adjust heads on a drum. A little too low and the drum was dead (head crash); a little too high, and there was no usable signal. This is something that carried over to disk (see later).
While Univac added bigger (and additional) drums in 1956, IBM chose the lower-cost approach of a disk. Here only a small number of heads were needed and less electronics to handle a way bigger amount of data at lower cost. On the negative side, disks were incredibly slow compared to drums.
When IBM then added the 355 disk unit (a variant of the mentioned 350) to the 650, a controller was also needed (653) which housed the first core memory to store (part of) a track, so access within a single track could be sped up.
Unlike often assumed, the IBM 305 RAMAC wasn't so much a disk based system. The CPU was still based around a drum. The 350 disk just added a layer of storage; faster than tape, slower than drum. Due to the (almost) standard disk, the 305 could operate with a smaller and cheaper drum, while handling data sizes similar or even bigger than the 650, albeit slower.
From here on IBM walked the disk 'track'. While drums as main memory were replaced by core, drums as secondary storage vanished rather quickly, but not everywhere and not completely. In fact, DEC offered their faster PDPs (like the 11/45) with a drum as fast paging memory (*3) in addition to disks. To some degree drum plus disk was what hybrid disks (HD + Flash Cache) are today.
Early Disks used basically the same technology as drums with heads at fixed height. Part of the inherent adjustment problem was that they couldn't really go close to the surface and thus density was quite limited which again resulted in even lower speed. In comparison, the 650's drum did rotate at 12,000 rpm with about 2000 bits in one track (50 words á 10 decimal digits), that multiplies to roughly 400kbit/s transfer rate. The 355 disk rotated at 1,200 rpm (less than a 10th of the speed); data transfer rate was 8,800 characters per second - with 6 bits/character that's about 52 kbit/s, roughly an 8th of what the drum did. In addition there were only 3 head mounts (350 had two) with two heads each, all covering the same 100 surfaces. Track to track time (one track) was almost 100ms, surface to surface could reach 2s and above. So yes, faster than handpicking a punch card, but in no way even near how a drum performed.
But boy were they cheap.
In the end a way more capable technology was replaced by a cheaper, lower performing technology for fast storage. What followed was much like later small hard disks: a race for lower cost small drives, but also for more storage at the same cost. Just 5 years later the 353 reached a capacity of 16 MiB and a transfer rate of 8 Mbit/s. Much of this was possible due now self flying heads allowing a more than 10-fold storage density. Even access time was way down as every disk side got a dedicated head.
Next step were Storage Module Device disks - a series of disks modeled after the IBM 1316 disk pack, a 6 deep stack of (now) 14 inch disks. Exchangeable in less than two minutes and whopping 2 MiB per stack. By 1964 the capacity had grown to 7 MiB and many companies provided compatible devices and for at least until 1980 they became standard - sizes increased up and beyond 300 MiB per stack.
Exchangeable disk stacks were standard. Hard disks fell completely out of use (while drums were still made and used). It wasn't until 1973 that IBM changed the design with what was called the Winchester drive. And no, it wasn't meant as a fixed built-in in hard disk like we use today. It was still an exchangeable media, except that now the head mount (and heads) were part of the disk pack. The idea was to keep the disks away from dust by being completely encapsulated. Electronics, drive motor and head motor were still part of the drive, not of the disk pack. The main goal was again a higher density, not so much for faster transfer (but they didn't mind) but to make it cheaper. Capacity of these 'Data Packs' was 30 to 70 MiB.
It took a few more years until the improved drives were made into a non removable setup with a tenfold increase in storage (>300 MiB). While SMD drives did compete size-wise this first generation of 'new' hard disks were faster - but users were not really convinced. With 'traditional' drives they could store a multitude of data on a few drives by exchanging packs, while not a drive had a fixed size. Large, but fixed, and no way for easy backup and restore by just putting in a disk pack (*4).
IBM developed (*5) the floppy as a way to replace punch cards at data entry stations. Here cost was the main concern, as punch cards were cheap, proven and equipment was existing. Outside this process, there was no real need for a diskette like media. For program and data exchange tapes were perfect; they were fast and held large amounts of data with small reels being handy to transport. A floppy wasn't smaller, but stored way less data and required new drives.
It was the Minicomputer of the mid 1970s that made the floppy a basic media. While up to 1 MiB storage was negligible for mainframes, it was huge for minis. And while it was incredibly slow in mainframe terms (card readers were faster), it was an incredible gain for minis. And way cheaper compared to disks.
For the upcoming micros during the second half of the 1970s, the floppy was heaven sent. And that's why they started out with floppies - and mainframes didn't.
And here history loops: With minis and micros growning in demand, it suddenly became feasible to develop small (in size), small (in cost), small (in capacity) hard drives again. 1979 the first 8" HD with 'just' 10..60 MiB and in 1980 Seagate with the 5 MiB 5,25" ST506.
*1 - The terminology classic IBM systems use do reflect this until today, as disk is still seen not as a separate entity, but another level of system memory. If at all, tape does qualify as such.
*2 - The needed part of a track was sent directly into the corresponding registers as required by the instruction processed.
*3 - Guess why BSD's swap is traditionally on /dev/drum :))
*4 - A common backup scheme was to copy the whole data disks onto a new disk pack by reorganizeing the database, then store the old and continue work with the (now defragmented) new ones. In case of a deadly disk crash, the system could start from the old disk within minutes.
*5 - Developed is the key word here. While the original design was for a cheap way to load microcode for the Merlin, it was data entry that transformed it into a general purpose storage method.