Update: Well, with the question being edited to ask for kilobytes per second,
So my question is: What was the highest capacity format used on an audio cassette, in terms of kilobytes per second?
the whole answer becomes rather simple (*1):
Divide the baud rate by the average number of bits a format used per data byte to get an average KiB/s value.
Thus, it again comes down to the Baud number (see all the way down).
Cassettes used with computers tended to have a far smaller capacity, typically 15 minutes or less. But they were still labelled according to their length, not their data capacity. One reason for this is that different computers used different formats, despite an attempt in the 1970s to create a universal standard.
No, it wasn't. These cassettes featured nothing computer specific. They are simply music cassettes (*2), thus made to run in the same devices (cassette recorders) at the same speed, 4.7625 cm/s (or 1.875 inch/s), thus ~2.85 meter equals one minute, or 42.75m for a C15.
That standard, the Kansas City standard (and the related Computer Users' Tape Standard)
CUTS is Kansas City (see this Question/Answer) - the naming just depends on what association was intended (or avoided). Kansas City if association with Byte magazine was OK, CUTS if not - or in the case of Processor Technology, because they defined it under that name. For more information see this Question/Answer.
used a 300 baud data rate. After wrapping every 8 bits with a start bit and two stop bits, each byte takes 36.7ms of tape, making for a raw capacity of ~27.2 bytes per second, or ~1.6 kilobytes per minute.
Part of the failure of the Kansas City
You wanted to state that CUTS wasn't a failure, as it had been adopted by many manufacturers and sold with millions of computers, right?
So my question is: How much data could these other formats fit on a 15-minute cassette?
Since there in an almost infinite number of formats and use cases, it would be way too broad to answer this. But there's a simple way to get close for machines you're interested in:
Step 1: Multiply the baud rate (as its bit/second) of the format in question by 900 (eliminating time and reducing it to the medium in question) to get a gross capacity in bits.
Step 2: Divide that by the number of bits this format records per byte (like 10 for many) and you'll get the gross capacity in bytes.
Step 3: Subtract the overhead and you'll get the net storage capacity in bytes.
Step 3 may be the most difficult, as block structure and length may depend a lot on what is stored. One large file or many smaller, each with their own headers. Similarly, how much space a user will leave between files to find them later on..
Writing one large block with next to no header will result in almost the gross capacity, while a recording like those used by Commodores, with small blocks, long headers and double recording (for program storage), will yield way less than 50% - with more than one program per tape easily as low as 20%. Since a tape can also store data as well as programs, and data may even be formatted in its own way, it's a pure guessing game.
All of this makes it almost impossible to give even a close number without exact specification of the existing/intended usage.
Following on from that, Which cassette format could store the most data on a C15 cassette?
Look for the one with the least bits per byte and the least overhead. In any case it'll be less than the gross capacity calculated in step 2. In some cases maybe 10% less, in others more than 50%. Your guess is as good as mine.
Bottom line, for a rough comparison, the baud rate will do the job - everything else is application specific.
With that in mind a baud rate table like this can be made:
It might be noteworthy, that depending on recorder and tape material the theoretical maximum when using real cassette material and existing heads is somewhere between 20,000 and 35,000 Bd. But that would need encoding techniques way past 1980s micros.
(Microcomputer) tapes aren't a blocked random access media like disks (or mainframe tapes) - they are in themselves random :))
*1 - Which in turn makes it somewhat ridiculous.
*2 - Except eventually being sold at a higher price per minute :))