Really early computers like the Mark I and ENIAC didn't have enough memory to attempt to handle text; also the use-case was mostly calculations.
A number of decimal IBM computers used characters (with 5 or 6 usable bits) as the basic unit, and decimal digits were just a special usage of those characters:
The IBM 1401 computer, and its compatible successors such as the 1410 and 7010, used seven bits in memory to represent a 6-bit character plus a one-bit word mark. Decimal arithmetic was based on numbers build from those characters.
The IBM 1620 computer organized memory into two-digit cells, with four bits for the digit, and one flag bit. In alphanumeric mode, those 5 bits were used for characters. Wikipedia has a table of the encoding.
The IBM 700/7000 series used a memory consisting of six-bit characters. The zone bits of the characters were used to indicate the sign of a number, and the last digit of a number contained its sign, also indicating the end of the number.
This also explains the funny way EBCDIC encodes letters, because it was influenced by those encodings.