Let’s start by addressing a few misconceptions.
I have a new external Western Digital USB HDD here and I saw in Wireshark that it uses SCSI.
That’s because storage over USB uses SCSI-style protocols. This doesn’t imply anything about the drive’s native protocol; you computer is communicating with a storage controller over USB, not with the drive itself.
Also the recent Linux kernel translates all ATA commands to SCSI (that is since /dev/hda has been renamed to /dev/sda).
It’s the other way round: it translates SCSI commands to ATA.
Can someone explain how that came and why HDDs and SSDs never switched to SCSI?
Hard drives and SSDs did use SCSI (and SAS, SCSI’s serial descendant).
To address your main question, it’s all about history.
The first PC hard drives used a bespoke interface, which ended up named after one of them: ST-506. SCSI was developed simultaneously, and was made public after the ST-506 became available, so it’s understandable that Seagate didn’t use SCSI.
When IDE was developed, software compatibility with the existing ST-506 interface was a more important consideration than compatibility with the emerging SCSI standard. SCSI was also more complex and expensive to implement, which left room for IDE. Thus we ended up with IDE hard drives (in most PCs) and SCSI hard drives (in Macs, and PCs where higher performance or support for larger numbers of drives was needed, and workstations etc.).
PATA was good enough for most uses, so it survived in the mainstream, and SCSI survived in the enterprise; similar considerations apply to SATA and SAS later on.
CD-ROM drives were introduced in a messier fashion, with multiple competing proprietary interfaces; initially, Philips / LMSI and Hitachi interfaces, both in 1985. SCSI CD-ROMs came later, in 1987, and relied at first on cards providing translation from SCSI to the drives’ native protocols. In the mid-nineties, when the ATAPI Council started working on allowing CD-ROM drives (and tape drives etc.) to be attached using a PATA-style interface (and the same physical connector), they presumably thought that the SCSI command set made sense and that it would be easier to re-use that than come up with yet another new standard.