It is well-known that binary floating point is unsuitable for financial applications, because 0.1 is not exactly representable in binary.
It's less the binary part, than floating point in general. Financial application want fixed precision. And they want to have exceptions handled according to established rules - rules not set by scientists, but accountants, and the tax office.
For this reason, early computers often had hardware support for decimal arithmetic. This was usually in the form of BCD, 1 decimal digit per 4 bits, e.g. the 6502 and Z80, tightly constrained by transistor count as they were, provided BCD support as a matter of course.
Historical it is called a packed format when two decimal digits are stored within a byte, as BCD refers to all representations of decimal numbers in a binary format. Of course, packing was made possible only with bytes made up from 8 bits, a combination made popular with the IBM /360 (*1).
Before that bytes where, if present at all, made up from various number of bits, in many cases 6 (*2). For example the IBM 7090 stored six 6-bit characters in one 36 byte word. There was no difference between a BCD number and a character. Similar Univacs 1100 series - here even offering the handling of six 6 bit characters/BCD (called sixth word) or 9 bit ones. The later sound like a candidate for the question, except operations where limited.
It seems to me that you could pack 3 decimal digits into each 10 bits, thereby getting e.g. 9 rather than 8 digits in a 32-bit word, with a couple of bits spare. I'm not sure an 8-bit CPU could implement fast arithmetic for this format, but I don't see any reason a 32-bit CPU shouldn't be able to.
Except, why should they do so? Unlike BCD, where handling is inherent decimal and packed BCD where storage density is increased by a simple shift operation, 9 bit 3 digit bytes do require decimal to binats conversion plus an adder structure made to roll over at 1000. 60 bit machiens like the CDC ones would have been great candidates, but they again did rather support 10 6 bit BCD per word than the 6 10 bit your format would have required.
Did any historical CPUs provide hardware support for this format?
Not that I can remember any. There where several 9 bit per byte machines, including mini computers, but none with an explicit 10 bit 3 decimal digit format.
If not, why not?
As usual, that's not a serious question to be asked.
If at all, it must be seen in context with the development before the 360, BCD operations where done on 6 bit units. This means 54 of 64 combinations are unused (*3). With the introduction of an 8 bit byte and packed BCD, this was reduced by 1/3rd while keeping the simplicity of BCD.
Simplicity here means as well that transfer from decimal (ASCII, EBCDIC or punch card) and back was done in a straight forward fashion, using shift (multiplex) operations, not calculation of any kind.
The same 8 bit byte could now as well hold a 7 bit character and doing away with clumsy 6 bit encoding without wasting much code space. So 8 bit per byte is a great compromize both ways (not to mention the usage of another power of two, which sounds total obvious to us nowadays).
*1 - The /360s packed format, BTW, included a sign, encoded in the least significant nibble. So 4 bytes did not hold 8 digits, but 7 plus sign. All operations on packed numbers did operate signed.
*2 - And that can get prety weired: CDC used for their 6600 series 60 bit words, divided into 12 bit bytes, each able to hold two 6 bit chars :))
*3 - Not entirely correct, as some where used for sign and other special encodings.
