What prevented optical drives from being used as the dominant secondary storage like the magnetic disk drives, in PCs? Was it entirely technical limitation or other issues like late development and marketing? By optical drive I do not mean CD or DVD which has to follow the standards, but some kind of enclosed "hard-disc" with optical read and write technology and re-writable disc platters. I assume these also would have been cheaper to produce and less susceptible to damage from dust and vibrations because of the absence of microscopically closely placed read-write heads.
For the simple reason that until relatively recently, it was very difficult to make a rewritable optical medium, but it was easy to make a rewritable magnetic medium. Magnetic tape as a recording medium has been a practical technology since the late 1930s, predating the digital computer. For comparison, CD-RW was introduced only in 1997, more than a decade after the pressed (non-writable) CD format.
Write speed and endurance.
Optical drive technology has been much slower to write to than magnetic Hard Disk Drives (HDDs).
The erasable optical technologies that made it to mass market were much slower in write time than HDDs and in erase time. Their life before failure was in the 100's of erases.
Optical drive use declined with the advent of USB flash sticks and internet connectivity, which made transporting files between machine smaller and far more convenient. HDD costs dropped to the point where schemes like RAID became a practical proposition. HDDs, singly or in RAID, offer relatively reliable backup and fast write and retrieval access. Optical drives never experienced such progress in their evolution.
What prevented optical drives from being used as the dominant secondary storage like the magnetic disk drives in PCs?
- Technological advantage of Disks
- Did I mention speed and density?
Even a Blue Ray does only pack 25 GiB of data on a (5.25) side. Actual HD do up to 700 GiB per (3.5) side. At times various ways of optical storage or optical guided storage, from MOD to Floptical have been tried. They could deliver remarkable results - and in theory great ones, but they never could catch up with HD development.
Was it entirely technical limitation or other issues like late development and marketing?
All of them. They had to catch up and market had to accept - not at least by price. The LS120 drives (in the US mostly known as SuperDisk) for example did make a good entry and provided reliable storage in reasonable size, still, it was outrun by CD as CD could distribute at extreme low price, becoming a must, while CD-R(W) was available as upgrade for those wit writing need. Not to mention the system battle with ZIP drives ... one both lost in the end.
Optical recording technologies have (do) come up every now and then, offering promising results, but there is never really a niche allowing to catch up with magnetic recording. Likewise holographic storage.
I assume these also would have been cheaper to produce and less susceptible to damage from dust and vibrations because of the absence of read-write heads.
Even an optical drive has a read-write head, how else should they read and write? To get acceptable density (and speed) they will run into exactly the same issues as HD do.
The only technology that has emerged are SSDs - simply because their speed advantage was so overwhelming that even the small size of early SSD (a few dozend MiB) wasn't a hurdle, giving them time to evolve. It still took them about 20 years (starting ~1990) to become an acceptable device for mainstream PC usage.
On a side note, there are applications of optics in magnetic recording - lie using a laser to heat up a disk surface to make it writable. A quite clever combination. So optical means are not gnored but integrated - just not simply as a separate class.
and less susceptible to damage from dust and vibrations because of the absence of microscopically closely placed read-write heads
That, exactly that (in reverse) made magnetic storage to integrate the media with the r/w heads and later their controller, enclose them in a dust-proof box and call the whole thing "hard disk". Optical media, being read and written from a distance was OK rotating up to "52x" speed in whatever device you insert it. When it got scratched, it wasn't in the RW device and wasn't that bad in the first place (the scratches are way out of focus).
Now, about the "secondary storage" (in your meaning).
In the begining, there was a music CD. All the data written in a single spiral-shaped track (just like the vinyl disc it was meant and later succeeded to replace). It was created at once from start to finish and then photocopied (almost like the vinyl).
Then, they glued a filesystem over the track. It was pretty much adapted to the single-create-and-then-photocopy process. Read-only at the design level. Seeking was rather slow, 1/2 second seek was normal.
Then, the CD-R appeared. It had to be compatible with the CD. Writting a CD-R is a royal hassle - prepare a single file with the very read-only filesystem inside, then write it at once to the CD-R (and welcome to the buffer underrun hell). The device needed few tens of seconds just to start writting and the main reasons were the spiral and the unknown optical media response that was calibrated (using part of the media itself) at each write. Later developments improved the experience, but not the method.
CD-RW didn't change anything except the possibility to erase the media (at once!).
DVD-R, RW, RAM, whatever, blue discs added data density and speed, but the spiral structure and the unknown media properties were still the main features. Seeking is still slow, writing is still a project and not a task. Way better than a tape, but still bad as a random-access media.
And in the meantime, HDD technology outpaced the optical media in regard to storage space, reliability and price. (The networks also improved and it became easier to send a dvd worth of data instead of writing a dvd and sending it to someone.)
Magneto-optical things never gained traction. They were more "magneto" than "optical" in regard to the read/write process in close proximity and contact between the media and RW head. The few competing "standards" and rather low reliability didn't helped either.
Light isn't small enough. Let's compare a couple of numbers for the present day:
The magnetic surface of each platter is divided into small sub-micrometer-sized magnetic regions, each of which is used to represent a single binary unit of information. A typical magnetic region on a hard-disk platter (as of 2006) is about 200–250 nanometers wide (in the radial direction of the platter) and extends about 25–30 nanometers in the down-track direction (the circumferential direction on the platter)
For a Blu-ray Disc, the spot size is 580 nm. ... The lasers are GaN (gallium nitride) laser diodes that produce 405 nm light directly. CDs use 780 nm near-infrared lasers.
The size of a "bit" on a hard disk is much smaller than the size of a bit on a blu-ray disc.
It is extremely difficult to make an illuminated spot that is smaller than the wavelength of the light used to illuminate it, or to focus on it when you have illuminated it. This is the "diffraction limit". Additionally, as your spots get smaller and move faster, in order to see them properly you need to provide more illumination.
This is why optical drives haven't kept up with the capacities of magnetic drives. Blu-Ray in particular required the development of cheap blue laser diodes, just as blue LEDs were invented a long time after red LEDs.
They were, at one time. Admittedly, optical tape drives rather than disc, which had several advantages : removable media for one.
Here's one : part of Colossus but I have used them in the 1970s and early 1980s where they were sometimes used to interchange data between computers and EPROM programmers.
I haven't had a HD head crash since 1995.
In 1998, a CD-ROM drive was $55. Two years prior it was still over $100, only coming down when manufacturers were "forced to compete on price instead of performance" where "the advances have been virtually undetectable to consumers". – Cnet: CD-ROM prices plummeting
In 1997 the "Price per GB of Hard Drive Storage:" was $100. – notebooks.com, Hard Drive Prices Over Time: Price per GB from 1981 to 2010
Optical media has been obsolete since its inception, having no use beyond throwaway pieces of plastic on which you may purchase The White Album again, to be inserted into devices which made easy the digital rights management of the content at a regional level.
Besides all the other reasons optical media sucks, no one who grew up with a computer in the 80s and thrived on bootleg VHS tapes wants that.
The most important reason would be that optical drives, or the whole optical data storage in general, are rubbish. They're admittedly slightly less rubbish nowadays, but back then... bleh. I am not even sure why it was such rubbish, it needed not be.
Writing to tape or harddisk is a no-brainer. You plug it in (or have it installed internally), if it is a removable medium (floppy, ZIP, hd, whatever) you may have to insert a medium, too. But either way, all in all it's "write and forget".
Tape does come with the annoyance of being somewhat, err, linear in terms of access, so usability isn't paramount, but on the other hand side it's cheap as dirt. On the other hand, anything that spins, somehow, is mighty fine because it is random-access, reasonably fast, relatively trouble-free, and still affordable.
Enter optical. You need a high-spec computer, and preparing the disk is a holy ritual for which you need special software, and which is best not interrupted lest the spiritual energy be lost. The computer cannot possibly do anything else while writing, either, you had better not even press a key or touch the mouse. Because any ever-so-slight disturbance will cause the magic laser beam to have a hiccup and the medium will be corrupted beyond repair. No, I'm not joking. It was more than a decade before drives with "buffer underrun protection" were available. Really, now? Why do I even have to care about any such thing as buffering when all I want is to write data to storage? It should be trivial enough? Well no, it's high science, and involves a lot of meta-magic. And no, of course drives that depend on such things cannot have a buffer built-in. It's why you need a high-spec computer, and it's why you are not to touch it.
Then there's those special filesystems and their extensions, and neither one is really compatible with each other or compatible with the "normal" filesystem that you're used to, and neither one can map files and filenames the same as you're used to it. Well sometimes it can, only just... not always. On a PC that wasn't an issue because English is the only language in the world anyway, but on Mac it sure was an issue.
Ah right, and of course, you cannot boot from optical like you can from magnetic store, either. Except when doing yet another most obscure magic dance that emulates a floppy, and whatnot. Sure enough, loading a floppy emulation image (i.e. many sectors) is supported, but loading a single boot sector is not. That would be just too easy.
So it says 4x or 8x speed on your drive, which sounds awesome (must be pretty fast!?), but of course it is not. Plus, it's ever so painful to figure out how long it actually takes to burn a disk. And of course, what's written on the medium must match what's written on the drive, although your local expert tells you that it works either way. And sure enough, you're trying, and it certainly works either way. Until it doesn't.
Seeing how the process of burning is not a 10-second thing, you do want to know how long it takes because you'll walk away and come back later. So you did your math (or are lucky enough to have the program show you an estimate) and it was, let's say 10 minutes. You come back 11 minutes later, only to find out that the drive took 3 minutes to warm up and another minute doing nothing, and now the process is 31% done, and at the end it will take another 3 minutes to "finalize" the disk (totalling around 25 mins). Oh, and then there's a good chance it will tell you "finalizing failed" the last second...
Wait, what, finalize? What's that, why do I need this? Nobody can tell for sure. A non-finalized disk works just fine. In the same drive, or in a similar drive from the same manufacturer. Or in some others. But it won't work in all. Why? Nobody knows.
Finalizing means you can no longer append data to the disk. Well, that sucks, but at least you know that the disk will work with all drives.
Surprise. There happens to be this one drive that you have which will still not read the disk. Oh and of course, the CD player in your car's radio won't play some industrially-manufactured CDs that you legitimately bought (for very substantial money at that time) because Sony shipped deliberately broken CDs in the 90s (you dirty pirate, eh?) and Pioneer players wouldn't play those. Fun fact, you actually had to pirate-copy the disk that you legitimately bought, in order to listen to the music...
Oh, and of course there's "plus" and "minus" disks, and half a dozen of other subtypes, which all look more or less the same and have more or less the same names. Some are more expensive than others, without being obvious why. Being an expert power-user, you never buy the wrong type. Except rarely. Every now and then, you find out that your particular drive doesn't work with one type. Sometimes it's a case of "Duh, read the label, it says so on the box!", and sometimes it just doesn't work anyway. Correct medium, for all you can tell, does not work.
So you managed to burn a disk, and successfully. Let's use it! Put in your optical medium, and hold your breath. Here come the 30 seconds during which the drive makes some funny noises and does something (what exactly, actually?) and then you either get to read the contents, or you're not getting to read it. Really, why does sequentially reading in the first dozen kilobytes that hold the filesystem structure take so long and cause so many seeks? Oh right, because that's not what is being done! Sure enough, with a read-only medium that has abysmal seek times, buffering the directory structure is a no-go, as well as reading it in sequentially.
CDs have a durability of around 20 years, probably much longer. And they have Reed Solomon error correction, so your data is really, really safe. Even if you drill a hole through the disk, the error correction can restore the data!
Alright. Save your data on CD, keep it in the closet, and put the CD into the drive 6 months later, only to find out that it's unreadable. Awesome. Well, luckily you burned 5 of them, just in case...
Now, meanwhile, optical drives have improved a lot. But they still suck, in direct comparison. BluRay, even double-layer, simply has too little storage to be useful for backup, and it still takes ages to write one disk. Plus, the mediums that actually have a fair chance of lasting a decade (manufacturer claims 1000 years, this remains to be proven) are so expensive that you can almost pay someone in a low-cost country to chisel the data into stone.
In comparison, store data on NAS with redundancy, and pull your backups onto a harddisk which you pull out at the end and put into a foam cover.
More comfort, more speed, less cost, no trouble. And both backup and restore work in 1/10th of the time.