It seems the question is mixing up physical disk access (CHS) with a logical access scheme used at a higher level. LBA is and always has been an issue at the OS level. The fact that some disk controllers also present their storage in an abstract fashion does not change how a disk works, it only relieves the OS (or better, its low-level drivers) from handling that abstraction.
When was the CHS (cylinder - head - sector) system invented
With the very first disk and before that on drum drives (*1,2).
and what was before it?
All disk type data storage (*3) needs to position to a cylinder and select a head to read a certain sector when it passes by. There is no other way.
As long as CHS was not compatible with Zone Bit Recording,
Err, why? CHS and ZBR go quite well together. There is no 'collision' of any kind. With or without ZBR, data on a disk is accessed by positioning to a cylinder, selecting a head and reading a certain sector when passing by.
it was finally replaced with LBA.
LBA is, on a disk still based on CHS. After all, each and every disk still has to position to a cylinder, select a head and read a certain sector passing by this head.
(From a comment)
As far as I can see, CHS and ZBR may be compatible only if the disk controller intentionally provides fake information about disk geometry to the operating system.
Not sure what fake geometry that should be. It's no dark magic for an OS to consider that tracks 1..39 have a different number of sectors than tracks 40..79, or is it?
It's not very convenient and I'm not ever sure if it was possible before the controllers moved from the computer's backplane into the disk itself
First, the controller would have to move into the computer, wouldn't it? For classic mainframes, the controller was never part of the computer but an external storage device attached via an I/O channel and addressed as controller/drive/CHS.
(it looks like it happened in 1980s after ST-406).
Not really. For one, since it's an abstraction layer it's about the OS/driver or OS/controller interface, so different driver architectures as well as different controller interfaces provided LBA addressing at different times. For example, SCSI (SASI) was LBA based from the very beginning. Much like some mainframe systems.
But the PC...
Despite asking in a generic way and for early technology, the question does seem to be more or less inspired by the IBM-PC. But that machine is an odd branch in disk development. Or more exactly, how compatibility was handled by add-on developers squeezing out more of a given interface.
The original ROM-BIOS API offers disk access via addressing of 10-8-6 bits for C/H/S. It does not feature any restriction of having the same number of sectors over all cylinders (or head for that matter), thus ZBR is quite possible with that arrangement.
Unrelated to that, disks grew bigger, eventually reaching more than 1024 cylinders (or 64 sectors) in the late 1980s. Thus a way was needed to integrate them. For DOS this was essentially a non-issue, as DOS used already its version of LBA when requesting a device driver action. It would have been up to the disk (or better controller) manufacturer to offer disk drivers that handle those disks - much like SCSI and others did.
Except disk controller manufacturers tried to outsmart DOS by lying at the BIOS API level, presenting the disk in different configurations by flunking bits around. A disk with 1200 cylinders and 4 heads was presented at the API level as one having only 600 cylinders but 8 heads instead. Doing so allowed to slip such drives below DOS without providing drivers. Quite a bad solution from an engineering point of view, but providing a plug&play feature seriously supporting sales.
Even worse, when IDE came in the mid-1980s - usually quoted as integrating the controller onto the disk - this was not just made on a BIOS API level, but on hardware, register interface, cementing it even further.
At some point, a bit of bit magic wasn't enough anymore and controller manufacturers switched to complete fantasy CHS numbers and called them LBA. The ATA standard (ca. 1994) formalized this. All still using the original 22 bits. When that no longer was sufficient (disks growing past 2 GiB), interfaces were tweaked again, but this time in conjunction with changed BIOSes and changed disk drivers.
CHS/LBA on the PC is kludgery very specific to the PC. It bears its very own history and is not related to any other system or historical development.
*1 - Except that drums are only a single cylinder, thus no cylinder selection is needed.
*2 - This includes of course drums used as main memory.
*3 - Let's ignore the special case of CDs and floppies like the Sony/Canon 2" VideoFloppy with their spiral tracks.