11

My very rough understanding of character encoding history as it relates to the Unix family of platforms/languages is that:

  1. They started using single-byte (7/8/9-ish bit) character sets like ASCII/EBCDIC/КОИ-8/etc./etc.
  2. Then early (16-bit) versions of Unicode came along and a big push was made in the Unix world to add wchar_t support to All The Things™
  3. But Unicode relatively quickly retconned its early work as merely the "Basic Multilingual Plane" and added 16 more planes, becoming ± a 21-bit character set at its core

Is it historically correct to say that Unix programmers, liking to pretend that character strings and byte arrays were the same thing anyway, never really adopted wchar_t and in practice wide use was never made of wchar_t support? Did most developers end up skipping straight from using their local ASCII equivalent to using UTF-8, never really bothering with the …wc API variants?

I do note several significant programming languages whose string support got awkwardly saddled by the 16-bit Unicode era (including Java, JavaScript, and even Objective-C's Foundation framework comes to mind) — but would consider those as platforms of their own and not part of "Unix culture".


Update: to be a bit more specific, I'm wondering if wchar.h additions like fgetws and fwprintf ever really got significant adoption in the Unix world? What percentage of "built in" userspace utilities, commonly distributed executables, popular off-the-shelf software, etc. migrated to the wchar_t APIs instead of (or before) migrating to UTF-8?

Or am I completely off base here, and UTF-8 didn't so much replace the wc stuff but rather became the de-facto standard for mbtowc/wcrtomb conversions (e.g. with any serious text processing idiomatically being done internally against on arrays of wchar_t values)?

11
  • 7
    I'm not an expert with widespread knowledge of Unix software repositories - but I've never seen wchar_t in the wild. I've never worked for a company where it was part of the coding standards. (For Windows software, yes, it is ubiquitous for use with the Unicode (UTF-16) enabled APIs.) (Not that the C library works properly with UTF-16 even if you are using wchar_t ...)
    – davidbak
    Oct 20 at 1:29
  • 3
    Ditto what @davidbak said. Can't give an official answer, but I've been working with Unix systems since the late 1980s, and I've never seen any Unix software that supported 16-bit characters except as an export/import format. UTF-8 was invented pretty early in the history of Unicode. It existed well before Unicode was widely accepted, and the purpose of it was to be compatible with all of the multitude of Unix-like, encoding agnostic, 8-bit text handling programs and library routines. Oct 20 at 2:05
  • 2
    There is actually wchar_t type in standard includes of linux or *BSD OSes. In linux, it has sizeof(wchar_t) equal to 4 and there are few functions like wcscpy() or wcscat().
    – lvd
    Oct 20 at 5:05
  • 4
    Your history misses out 1A: stateful 8-bit encodings (euc-JP, ISO-2022, etc). Oct 20 at 7:35
  • 4
    The UTF-8 origin story. Oct 20 at 12:23
2

I think your question must be put a bit differently for a first step towards an answer, like "was it common to use localized Unix applications from 70ies to nineties?" - In my opinion, the answer to that is already "no" (although improving towards the end of the period). The simple reason for that that most applications were specialized, low-volume software that came in English and only in English. If you use the "C" locale only, you can very well live with 7-bit ASCII. (I remember being really impressed when I received my first Sparc Classic in 1992 that was, in my memory, the first Sun machine that - out-of-the-box - didn't freak out when you entered accented characters into its vi or shell. Thus, was "8-bit clean")

While English can get away with 7-bit-ASCII, most European languages can represent most of the needed characters in 8 bits, and (predominantly) ISO encoding. They really didn't absolutely require a wide character type. It is, however, different with Asian and Arabic languages - I do remember we wrote a news distribution application for Unix for a Chinese press agency early-1990. This one didn't use a single char type, but was plastered with wchar_t occurencies. And we had to contract (rather, "lend" from the customer) a native-speaking Chinese Unix programmer, because no-one in the team had the slightest idea how to handle Mandarin.

7
  • "most applications were specialized, low-volume software that came in English and only in English." I'm not really sure where you got this impression. True, 'low volume' is a relative term, but I wouldn't call Unix neither low volume, nor English only application. During a good part of the 1980s Siemens was Europe's #1 seller with their Unix systems. Selling 6 digit figures of units (computers, workstations, terminals) per year. They had a good hold with high volume governmental customers around the world (including the UK). Not localizing application in that area is not possible.
    – Raffzahn
    Oct 21 at 8:25
  • 1
    P.S.: while Arabic can of course easy be handled (and still is) with 8 bit (8859-6), it was also done as 7 Bit MBCS, like IR-169, combining Arabic, French and German. Also, Japan had quite a lot of Unix usage already early on, with Fujitsu maybe being the main proponent (and still is with their own SPARC CPUs as well). Using 2022 was mandatory to handle their characters.
    – Raffzahn
    Oct 21 at 8:46
  • Certainly, as a non-Unix progammer in the 1980s, writing code to handle a variety of one-byte character sets (ASCII, DEC 169, the various 8859-x sets) and a variety of multibyte character sets (generally, CJK codings) was a pain in the rear. From that point of view, the combination of Unicode and wchar_t (meaning UCS-2 at the time) in the early 1990s was a vast improvement. The joy didn't last too long, though, once Unicode and ISO 10646 got chummy. Oct 21 at 11:47
  • @Raffzahn It might have been different in the non-technical application space, and, admittendly, governmental use. My experience is limited to Unix workstation in technical applications (HP, Sun, SGI,...). Localized applications were very, very rare there.
    – tofro
    Oct 21 at 15:51
  • @tofro Given, we're way more frugal in engineering - but like with Mainframes before and PCs now, the majority or usage falls mundane clerical jobs. No postal service anywhere in the world would buy a system no capable to handle local names and addresses - nor would any other commercial user. We may be the ones that create, program and control technology, but they use it (and pay us) :)
    – Raffzahn
    Oct 21 at 17:15
1

Is it historically correct to say that Unix programmers, liking to pretend that character strings and byte arrays were the same thing anyway, never really adopted wchar_t and in practice wide use was never made of wchar_t support?

Yes and no. For one, Unicode did make quite an inroad for Unix in the 90s - and more often than not using a 16 bit type. But of more importance before Unicode, and that seems to be missing in your list, were Multi Byte Character Sets (MBCS). The most popular were the ISO-2022 series.

ISO-2022 (and its predecessors) have been used since the mid 1970s to enable Asian character sets in 7-bit environments. And it was quite well used with Unix systems in Asia. It was the de facto standard for that region into the 1990s and still today many mail clients use it as default.


MBCS and Unix

MBCS have been around since the early 1970s and for sure been used with many Unix applications.

MBCS are a generic term for character sets that (may) use multiple bytes to encode a single character. They are, like all character sets standardized as mean of interchange between applications. Internal representation is, as always, the application's choice (*1).

MBCS usually work by switching between various 7 bit planes (Basic ASCII being one of them) by means of escape codes (*2). Either with a rather specific handling, like IR-167 or more generically as defined by ISO-2022. Each plane offers 94 code points - all printable chars between x'21' and x'7E'.

Using only these 94 characters plus ESCape allowed

  • clean use of 7 bit transmission lines
  • 8 bit storage

And quite importantly here:

  • Any file system that allowed the use of ESCape (x'1B') within file names automatically supported localized file names without further means.

Also, to some extend,

  • Any tool that handles strings simply as chars will be (to some degree) usable with MBCS encoded data.

Unix was always able to include 'special' chars in file names, including ESCape. With terminals supporting ISO-2022 (directly or via translation in shells / IO drivers) handling was as seamless as for 7-bit characters - unless broken by some programmers caring too much for cleaning strings of "weird characters", that is.

MBCS and wchar

Essentially all ISO-2022 enabled applications I have ever seen used internally a 16-bit (wchar-like) type to handle characters. After all, switching all character handling from 8 to 16 bit is a rather easy and low effort (*2) process.

While the basic standard is, like UTF-8, a stateful encoding, there was a variant encoding using mixed 8/16 bit character representation. All basic 7 bit ASCII characters were encoded as is with high bit off, all extended characters had their high bit set in both bytes. Scanning and separation could be done quickly using only byte operations. Internal storage depends, much like with Unicode, on application specific needs, but 16 bit (wchar) was quite common.

All functions that most programmers now associate with UTF-8 (mblen, mbtowc, wctomb, etc.) have been defined before to be used with generic MBCS.


Why this usage may have been less visible to anyone not involved

Of course, doing MBCS applications is a bit past entry-level programming, so it's not the stuff seen in examples everywhere, but usually hidden in projects.

In the Windows world it's simply more visible as MS included native support for 16-bit character sets - originally meant to simplify MBCS handling, but soon mostly identified with Unicode.

Then again, I think I remember even DEC supporting MBCS handling as 16 bit in Ultrix - which IIRC is a BSD variant, so BSD may have similar support.


*1 - Just think how the Apple II and PET used different character encoding for video or keyboard.

*2 - No surprise here for some that remember how DEC implemented multi lingual character sets in some terminals.

*3 - Keep in mind, this is about applications where adding a few dozen lines of code to replace C library functions is peanuts.

3
  • Some interesting context but I'm not sure if even "used internally a 16 bit type to handle characters" really answers my question. I'll add this to my OP too, but I guess I'm wondering more if specifically functions like fputwc and fgetwc or their wrappers ever saw significant use? (And by MBCS, I'm assuming you mean things like UTF-8 itself and/or more stateful character encodings? I.e. not necessarily treated at the I/O level as wchar_t even if internally that may have been a convenient storage type?)
    – natevw
    Oct 20 at 21:42
  • 1
    @natevw Err, what else is using a wchar type then handling internally as 16 bit? Next, no application past text book stage (I know) would scatter any fput around source files. serialisation function are usually well wrapped in centralized functions that handle any conversion from internal representation (wchar) to whatever the storage representation is - essentially the definition of fputwc, isn't it? Last: UTF-8 is a MBCS, but the issue is way older than Unicode. MBCS were developed in the 70s to tackle exactly this issue. All professional Unix use in Japan was based on MBCS support.
    – Raffzahn
    Oct 21 at 8:36
  • In all the 32-bit and 64-bit Linux/Unixes I've used, wchar is 32 bits.
    – cup
    Oct 21 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.