Was `wchar_t` ever widely adopted by the Unix culture in actual practice?

Question

My very rough understanding of character encoding history as it relates to the Unix family of platforms/languages is that:

1. They started using single-byte (7/8/9-ish bit) character sets like ASCII/EBCDIC/КОИ-8/etc./etc.
2. Then early (16-bit) versions of Unicode came along and a big push was made in the Unix world to add wchar_t support to All The Things™
3. But Unicode relatively quickly retconned its early work as merely the "Basic Multilingual Plane" and added 16 more planes, becoming ± a 21-bit character set at its core

Is it historically correct to say that Unix programmers, liking to pretend that character strings and byte arrays were the same thing anyway, never really adopted wchar_t and in practice wide use was never made of wchar_t support? Did most developers end up skipping straight from using their local ASCII equivalent to using UTF-8, never really bothering with the …wc API variants?

I do note several significant programming languages whose string support got awkwardly saddled by the 16-bit Unicode era (including Java, JavaScript, and even Objective-C's Foundation framework comes to mind) — but would consider those as platforms of their own and not part of "Unix culture".

---

Update: to be a bit more specific, I'm wondering if wchar.h additions like fgetws and fwprintf ever really got significant adoption in the Unix world? What percentage of "built in" userspace utilities, commonly distributed executables, popular off-the-shelf software, etc. migrated to the wchar_t APIs instead of (or before) migrating to UTF-8?

Or am I completely off base here, and UTF-8 didn't so much replace the wc stuff but rather became the de-facto standard for mbtowc/wcrtomb conversions (e.g. with any serious text processing idiomatically being done internally against on arrays of wchar_t values)?

Answer 1

Is it historically correct to say that Unix programmers, liking to pretend that character strings and byte arrays were the same thing anyway, never really adopted wchar_t and in practice wide use was never made of wchar_t support?

Yes and no. For one, Unicode did make quite an inroad for Unix in the 90s - and more often than not using a 16 bit type. But of more importance before Unicode, and that seems to be missing in your list, were Multi Byte Character Sets (MBCS). The most popular were the ISO-2022 series.

ISO-2022 (and its predecessors) have been used since the mid 1970s to enable Asian character sets in 7-bit environments. And it was quite well used with Unix systems in Asia. It was the de facto standard for that region into the 1990s and still today many mail clients use it as default.

---

MBCS and Unix

MBCS have been around since the early 1970s and for sure been used with many Unix applications.

MBCS are a generic term for character sets that (may) use multiple bytes to encode a single character. They are, like all character sets standardized as mean of interchange between applications. Internal representation is, as always, the application's choice (*1).

MBCS usually work by switching between various 7 bit planes (Basic ASCII being one of them) by means of escape codes (*2). Either with a rather specific handling, like IR-167 or more generically as defined by ISO-2022. Each plane offers 94 code points - all printable chars between x'21' and x'7E'.

Using only these 94 characters plus ESCape allowed

• clean use of 7 bit transmission lines
• 8 bit storage

And quite importantly here:

• Any file system that allowed the use of ESCape (x'1B') within file names automatically supported localized file names without further means.

Also, to some extend,

• Any tool that handles strings simply as chars will be (to some degree) usable with MBCS encoded data.

Unix was always able to include 'special' chars in file names, including ESCape. With terminals supporting ISO-2022 (directly or via translation in shells / IO drivers) handling was as seamless as for 7-bit characters - unless broken by some programmers caring too much for cleaning strings of "weird characters", that is.

MBCS and wchar

Essentially all ISO-2022 enabled applications I have ever seen used internally a 16-bit (wchar-like) type to handle characters. After all, switching all character handling from 8 to 16 bit is a rather easy and low effort (*2) process.

While the basic standard is, like UTF-8, a stateful encoding, there was a variant encoding using mixed 8/16 bit character representation. All basic 7 bit ASCII characters were encoded as is with high bit off, all extended characters had their high bit set in both bytes. Scanning and separation could be done quickly using only byte operations. Internal storage depends, much like with Unicode, on application specific needs, but 16 bit (wchar) was quite common.

All functions that most programmers now associate with UTF-8 (mblen, mbtowc, wctomb, etc.) have been defined before to be used with generic MBCS.

---

Why this usage may have been less visible to anyone not involved

Of course, doing MBCS applications is a bit past entry-level programming, so it's not the stuff seen in examples everywhere, but usually hidden in projects.

In the Windows world it's simply more visible as MS included native support for 16-bit character sets - originally meant to simplify MBCS handling, but soon mostly identified with Unicode.

Then again, I think I remember even DEC supporting MBCS handling as 16 bit in Ultrix - which IIRC is a BSD variant, so BSD may have similar support.

---

*1 - Just think how the Apple II and PET used different character encoding for video or keyboard.

*2 - No surprise here for some that remember how DEC implemented multi lingual character sets in some terminals.

*3 - Keep in mind, this is about applications where adding a few dozen lines of code to replace C library functions is peanuts.

Answer 2

I think your question must be put a bit differently for a first step towards an answer, like "was it common to use localized Unix applications from 70ies to nineties?" - In my opinion, the answer to that is already "no" (although improving towards the end of the period). The simple reason for that that most applications were specialized, low-volume software that came in English and only in English. If you use the "C" locale only, you can very well live with 7-bit ASCII. (I remember being really impressed when I received my first Sparc Classic in 1992 that was, in my memory, the first Sun machine that - out-of-the-box - didn't freak out when you entered accented characters into its vi or shell. Thus, was "8-bit clean")

While English can get away with 7-bit-ASCII, most European languages can represent most of the needed characters in 8 bits, and (predominantly) ISO encoding. They really didn't absolutely require a wide character type. It is, however, different with Asian and Arabic languages - I do remember we wrote a news distribution application for Unix for a Chinese press agency early-1990. This one didn't use a single char type, but was plastered with wchar_t occurencies. And we had to contract (rather, "lend" from the customer) a native-speaking Chinese Unix programmer, because no-one in the team had the slightest idea how to handle Mandarin.