Could we have avoided the whole UTF-16 fiasco? [closed]

Question

Anyone who has studied Unicode and is honest will admit that UTF-16 was kind of a mistake. It was born from the early assumption that 16 bits would be enough for all of Unicode. Then a hack was invented to extend it to 21 bits using "surrogate pairs". This led to lots of software not working correctly for these higher valued characters, which continues to this day, especially for display and editing-oriented applications (the dreaded "double backspace bug").

There is also an issue of "big-endian" vs "little-endian" requiring a "byte order mark".

Much later, UTF-8 was invented and it has now become a consensus that UTF-8 is the best solution for pretty much all applications, since there is no special treatment of higher-valued characters and no byte-order-mark needed. As a bonus, it's backwards-compatible with 7-bit ASCII.

All of the problems with UTF-16 could have been avoided by being more forward-thinking at the beginning (I know hindsight is 20/20). But would it have been conceivable for the inventors of Unicode to foresee that 16 bits would not be sufficient, to think of UTF-8 at the beginning, and only specify UTF-8 and UTF-32 as valid encodings for Unicode? Is there any history to suggest that perhaps some people were more forward-thinking but their views were disregarded?

Answer 1

The designers of Unicode did consider whether 16 bits would be sufficient, and decided that they would be, based on their assumptions:

Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 2¹⁴ = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally useful Unicodes.

The 16-bit choice was based in part on the idea that Unicode would only encode modern characters. UCS-2 (the original encoding, initially called “Unicode”) was designed with this in mind.

Unicode was however extended to include “legacy” characters a few years after its introduction, which broke the initial assumption and required more code points than could be represented with 16 bits. If the Unicode designers had wanted to include legacy characters, they would presumably have used a different encoding. I’m not aware of any particular difficulty which would have made UTF-8 harder to design in 1988 than in 1992.

UTF-16 was designed as an extension to UCS-2 to address the latter’s limitations, and was published quite late in the game (1996, with Unicode 2.0). UTF-8 was designed in 1992, which answers your question, but raises the question of why UTF-16 was used; the answer to that is that it was deemed an acceptable compromise to fix problems with UCS-2, and that it was important to continue with something similar to UCS-2, given that a fair amount of software already used it (including Windows NT).

Answer 2

Cultural factors played a role. Western creators of Unicode expected that unifying East Asian ideographs would be similar to unifying Western fonts (roman, italics...) and that they get to decide where the line between modern and unusual characters lies. This proved to be too optimistic. For the unification, the devil was in the details, but also in the need of processing of multi-lingual documents and databases: variation selectors were already breaking the 16 bits per character principle, like UTF-16 later did.

In 2000, PRC published their GB 18030 standard as mandatory for all software applications marketed for China. At this point, at the latest, it became obvious that Unicode Consortium wasn't going to have a free hand in inclusion of characters if they want their standard to remain relevant world wide.

All this is a part of string representation evolution. Starting with external and internal encoding to be a simple character array, through a phase where fixed character width was preferred for internal encoding and variable character width for storage and transmission, to a final phase where the complexity of the problem requires variable character width everywhere. We are now somewhere between the middle phase and the last one. Was such evolution foreseeable? Maybe. But the arrays of characters have done a lot of useful work in their own time due to their simplicity of processing.

Answer 3

Unicode represented a compromise among a number of competing objectives. Probably its biggest failing is its presumption that a single storage format could be suitable for string storage, string concatenation, text rendering, and text editing. It sacrificed considerable storage density so that text editing could be performed without having to be aware of all the details of the character set, but then evolved in such a way as to negate that advantage. If Unicode had specified a representation where all composite glyphs wre marked with start and end markers, then code which needs to format a piece of text could recognize composite glyphs and avoid splitting them without needing to have any idea of what they are. As it is, however, Unicode has gone so far afield that accurately determining whether a line may be split between two code points may require scanning an arbitrary quantity of text before it and having an up-to-date list of all supported country flags.

Answer 4

But would it have been conceivable for the inventors of Unicode to forsee that 16 bits would not be sufficient, to think of UTF-8 at the beginning, and only specify UTF-8 and UTF-32 as valid encodings for Unicode? Is there any history to suggest that perhaps some people were more forward thinking but their views where disregarded?

The problem was indeed foreseen, and the views of those who wanted to start with a much larger number of code points were indeed put forward and disregarded.

The 1990 draft of ISO 10646 offered a 2,147,483,648 code point space¹ and the following encodings:

• UCS-4, encoding all characters in single 32-bit values.
• UCS-2, encoding only the Basic Multilingual Plane.
• UTF-1, a variable-length encoding.

Note that while they did offer a 16-bit encoding UCS-2, this is not the same as UTF-16 because it's not variable length (it's simply impossible to encode all the code points in it). Arguably, even though UCS-2 likely would have been extensively used, this would still have been simpler than the situation in which we find ourselves today with UTF-16.

However, according to Wikipedia:

In 1990, therefore, two initiatives for a universal character set existed: Unicode, with 16 bits for every character (65,536 possible characters), and ISO 10646. The software companies refused to accept the complexity and size requirement of the ISO standard and were able to convince a number of ISO National Bodies to vote against it. The ISO standardizers realized they could not continue to support the standard in its current state and negotiated the unification of their standard with Unicode....

Meanwhile, in the passage of time, the situation changed in the Unicode standard itself: 65,536 characters came to appear insufficient, and the standard from version 2.0 and onwards supports encoding of 1,112,064 code points from 17 planes by means of the UTF-16 surrogate mechanism. For that reason, ISO 10646 was limited to contain as many characters as could be encoded by UTF-16 and no more, that is, a little over a million characters instead of over 679 million. The UCS-4 encoding of ISO 10646 was incorporated into the Unicode standard with the limitation to the UTF-16 range and under the name UTF-32, although it has almost no use outside programs' internal data.

---

¹_{Certain code points that involved numbers in "control character" ranges were not allowed to be used, so the total number of code points in this space that were available to be encoded was 679,477,248.}

Answer 5

It would've been an incredibly hard sell. What a lot of people don't realize, even back in the day, was how fundamental 16-bit values were to the original Unicode standards. Until version 3.0 was published in September 1999, Unicode required that all processing be done using 16-bit values. This meant that neither UTF-8 nor UTF-32 could actually be called Unicode, and any program that worked on these encodings as-is without converting them a 16-bit representation for processing weren't Unicode conformant.

Even the distinction between UCS-2 and UTF-16 was considered unnecessary. There was quite a bit of doublespeak in the standard in order to maintain the fiction that the original goal of having one character per 16-bit value hadn't been broken by the addition of surrogate pairs.

Given this insistence on 16-bit values by the Unicode Consortium for almost entire decade, despite the much more practical and honest approach of the ISO 10646 committee, despite how UTF-8 was proving itself to be what could actually replace ASCII, it's hard to see how anyone could've convinced them to take a different direction. I'm sure there's still people involved with the Unicode Consortium that still think 16-bit values are all you really need.

Answer 6

Yes, absolutely, if not for some big players in the Unicode consortium heavily pushing an agenda for a 16-bit fixed-size character set early-on. On the ISO 10646 side, it was always assumed that a 16-bit character set would be insufficient. They had other things wrong that kept them from gaining traction though, and in 1990 a unification with Unicode was negotiated. Since the unification, Unicode has always been larger than 16-bit.

Your comment that

Much later, UTF-8 was invented...

is ahistorical. UTF-8 was invented soon after, in 1992, before Unicode was in widespread use. But at the time Microsoft was still pressing ahead with their UCS-2 agenda, treating the non-BMP characters as unimportant, because their most important goal was to displace the standard text-based protocols and file formats of the emerging internet with their own creations they could control. Obsoleting 8-bit code-unit text was a big part of that.

Then of course Sun and Java came along and carried UCS-2/UTF-16 forward.

If either of these things hadn't happened, it's very plausible that we could have gone straight to UTF-8 without all the UTF-16 headaches.

Answer 7

TL;DR: UTF-8 and any other variable-length encodings are good "external", "exchange" format, but bad "internal processing" format.

You start from a very wrong premise that "UTF-16 was kind of a mistake" and "UTF-8 is the best solution for pretty much all applications". It isn't.

UTF-8 is very convenient in that you don't need to think about new code points—it can and will encode whatever new appears in Unicode standard and that its compatibility with lower part of ASCII often simplifies things even more. Thus UTF-8 is very good for data storage and transfer: you implement/use relevant lib in your program or write "UTF-8 encoded string" in your protocol/API spec once and it just keeps working.

However when you need to actually process text and do it fast, you will get hit by the greatest drawback of UTF-8: it is a variable length encoding. With any fixed length encoding you can trivially jump to an arbitrary character or n characters back/forward, and find string length from buffer length. More complex operations like slicing strings or fast substring search algorithms directly depend on this. In UTF-8, it is way more complex: you need to scan buffer for character positions all the time or index them soo many high-performance tricks just simply don't work. BTW, endianness is not a problem in internal representation, since library/VM simply "knows" its internals and doesn't need to "figure out" anything about it.

So there you are: a range of applications where UTF-8 is far from best and fixed length encodings shine. So, how do VMs and string/regexp libraries—two of the most common types of programs that implement some kind of internal string representation—get around this problem? Most often they either completely limit itself to some kind of fixed width or implement several different string representations with specific width and try to fit any incoming string in whatever smaller form it fits best and then introduces separate internal implementations for all necessary functions for each width where authors can use all the usual 8-bit era tricks trivially adjusted, but constant number of bits. Sometimes UTF-8 is also added to the list just to be sure in hope that most strings will fit into internal fixed width representations and whatever won't and will need to fall back to slower algorithms will be minority. Since most UTF-16 text doesn't use anything out of 65536 range, you only need to scan it once—which is very fast and can be greatly optimized thanks to its fixed width nature—to make sure it fits, mark type in internal representation and then go ahead and use fast functions. Sometimes implementations go even further and simply treat all the UTF dangling bits—like surrogates—as entirely separate character. Why? The entire JavaScript string spec is written that way!

Some examples:

• Perl VM tries to use 8-bit and falls back to convoluted mess of locale functions and UTF-8 strings (no UTF-16 here, but it is still same scheme of trying to fit a string into fixed width representation and only falling back when necessary).
• Mozilla internals use 8-bit and 16-bit strings.
• Python 3 has 8, 16, and 32-bit fixed width and UTF-8 logic.
• ICU library uses 16-bit (it allows surrogates, but they're treated as separate characters instead of forming single code point).
• As already said, ECMAScript/JavaScript describes string behavior pretty much the same as ICU above.
• PCRE compiles separate functions for 8, 16 and 32-bit.

Answer 8

It could have been avoided if Unicode had taken a more reasonable path, but being Design by Committee, that was practically guaranteed not to happen.

If desired, the BMP (= 16 bits of Unicode) plus some metadata, would have been way enough for everybody, forever.

Now, I said "reasonable path" above, but what is reasonable? That's of course debatable, and if you ask 5 people, you will get 6 opinions!

The Unicode guys deemed the goal "one encoding for anything and everything a human has ever written" reasonable, and desirable, and achievable. Which is pretty much the opposite of what ASA/ANSI had in mind a few decades earlier, which was more or less "English is the only language in the world".

I am personally baffled at how humans seem to live and think in extremes, are always over-reacting to extremes, and they never learn, or behave reasonably and with moderation. You see this prominently not only in computing, but also in many other fields (financials, politics, healthcare). I guess that's just how humans are.
_{Note the parallel with IPv4 and IPv6 where you have "4 billion is way more than we will ever need" versus "100,000 IP addresses per square meter (i.e. 64 bit) is not enough, we definitively need 128 bits". Both are ridiculous extremes.
Also note the parallel with present pandemia handling in EU (including Germany, where I live). Letting three months pass without any precautions, border controls, or preparations. Stock up masks or disinfectant? Build hospitals or extend ICUs, order a few extra respirators? Nah, what for, problem is in China. Let's instead have Carneval! And then, one day you wake up, completely surprised that people are dying and there's neither enough masks nor disinfectant, nor intensive care capacity. Now it's total panic, unreasonable measures totally over board, invading and limiting the freedom of millions in a way that hasn't been seen since the Third Reich, and driving entire industries and even countries to the brink of ruin. Sure enough, there was no way of doing it any better.}

Now, human non-reason aside, the facts are that a mere 8 bits worth of characters are annoying for everybody except English speakers who do not need to do anything but "normal no-special text" whereas 16 bits are way enough for everybody except someone wishing to process ancient Chinese texts which contain symbols that most Chinese do not even recognize.

Heck, 16 bits are sufficient to accomodate languages that are spoken by a few hundred people worldwide, or have not been spoken by anyone for a thousand years at all. You can use Cherokee or Okham just fine with "only 16 bit" Unicode. You can write in Bamum, if you even know what that is (I had to look it up). Can you read Vedic Sanscrit? Well congrats if you can, and guess what, despite being dead for ~3000 years, it fits just fine into the BMP.
And, to top it off, you can encode the exact same glyph in several redundant ways, some of which do not really make much sense as separate characters (such as number 2 with or without a circle, or a filled circle around it, or as superscript and subscript). Much of what's in Unicode could as well, and would likely be much better encoded, by metadata/markup.

The assumption is that it's a good thing you can just type some mathematical symbols and Arab characters or Sumerian in English text, and it "just works" and it's all the same, you do not even need to waste a thought. The assumption is that you actually need this every day, and that not having to think about it is a good feature.
The assumption is that you want to embed symbols of a bronze-age artefact in your Whatsapp messages.

The assumption is that people like Donald Knuth and Leslie Lamport are idiots because they require you to add dollar signs when embedding a mathematical formula. You can just insert the symbols right away, can't you!

While there is some truth in the idea that it's generally cool to just type away and not care, reality has it that most of the time you do not really need this feature, and in fact it would be better if you were switching the equivalent of "context" or "code page" with some kind of control sequence. Think TEX dollar signs, think escape codes, think xml tags, whichever you prefer, all the same.

Did you ever wonder why in the beginnings of the web, the makers of HTML would create a feature like <p lang="fr"> ... </p>? Why would one want to do this? Who cares? Well, search engines do care, and accessibility services (e.g. text to speech) do. It is much less of a challenge for an accessibility service that intends to read for a blind person to correctly pronounce words if it is told what language a word or paragraph is written in, rather than having to guess from the character codes.

It wouldn't even be necessary (or desirable) to go all the way down to a single target language on the character level. French people write Italian or German or English too, at times. Koreans write Japanese at times. They both use a few hanja at times. So, while desirable as metadata in markup, it is probably not desirable to have the actual character encoding too tensely bound to one particular language.

However, if nothing else, one could for example have distinguished very roughly between these cases:

• normal text for everybody in "living" languages LTR
• normal text for everybody in "living" languages RTL
• math, music, technical and electric symbols etc.
• other dingbats, smileys, whatever
• ancient texts, non-chinese
• ancient texts, chinese
• klingon, sindarin, morgul and what you like

The impact in terms of having to include escape characters (implicitly done by the word processor, or explicitly by writing markup, whatever) would be non-existent for 95% of all people, and very acceptable for the 5% writing scientific papers. On the contrary, explicitly pointing out that a formula is indeed a formula or that a foreign word is indeed just that is very helpful for processing (laying out, indexing, searching, translating, TTS) the document. Being able to tell immediately that a ³ is indeed a 3, only just in superscript is much preferrable to hoping that a program that processes some funny weird characters will figure it out.

It would have taken a mere dozen or so (or two dozen, depending on how elaborate one wants it) escape codes to do such a thing.

Unicode (necessarily, being forced) partially did go that route with e.g. the LRM/RLM markers. How could they otherwise have embedded the information that Arab text needs to be written the "wrong way around"?

But they did, in my opinion, not do it properly. Turning on RTL without context is a mis-feature if you ask me. Writing e.g. German right-to-left is just meaningless. On the other hand side, writing Arab LTR is equally meaningless, so why is this possible? Also, the ability of setting up homograph spoofing websites that read exactly the same as the original is not truly a feature, either. Nor is the ability to write the exact same thing with either combining or non-combining characters not a feature, but a nuisance.

Unicode also partially went the markup route with e.g. s̶t̶r̶i̶k̶e̶t̶h̶r̶o̶u̶g̶h̶, only just not properly. There is no way of knowing that s-t-r-i-k-e-t-h-r-o-u-g-h is actually strikethrough, and it's both a text processing nightmare, and a user annoyance (try and copy/paste that!).

 
. . L̶͔͍̪͕̯̼̪͙͒͋̽͂̆̅̚͠͝ì̡͕̖̳̞͊̉̈̀͊͘k̢̪̯̹̟̓͊̽͂͆͐̐́͢ȩ̶̳̙͔̭̙̪͗͒̔̄͟͡͡ẃ̸̳͇̬̟̯̾̽̈́͒̈́̋̔į̛̤͇̞̘͚̹̰̝̾̄̐̀͜ş̴̯͖̺̹̖͔̯͈̾̆͛̂̑̎́̅̂e̸̢̨̙͎̣̻͋̾́̔̾͜,̵̭̞̯̥͕̗̪͔̼͇̓̇́͛̽̽̓̉̚ Z̛͔͖͎̠͆̅̓̀͆̂̀͘͢͝ḁ̵̥̻̣̦̘̣̻̠͆̊̀̽̍̄́̓̃͋ͅḻ̶̡̣̜̬̗͙͔̫͊̌͆̐̔̚̚͜͡ĝ̘̜͚̟̟͍̬̜͉̈͛̌̿͡͠ö͕̳͉͈̝͉͕̟́̉̇̑̔͋̆̾̓͜͞ ṱ̠͓͍̜͈̾͋͘̕͡e̸̹̞͍͓̥̮͔͔͆̓̊͐͘͡x̶̰̱̖͎̰͕̤̺̙̞̑̂̄͑͡͝͡ț̶̗͕͍̗̖̎̔̑̌̄̓͡ i̡̺̠͕̟͚̝̲͂̆͐͋́̀͠͝s̸̛̛͕̻͖̺͕̘͔̱͛̈̓̒͑̚͢͢͡͡ ņ̖̞̔̌̑͊̌͒̓͢͜͝͠͡ò̶̗͙͖̫̟͚̞̱͌̄͛̓̇͝t̘͖̤̉͆̂̂̉͘͢͟͢͠ v͕̮̣̱̥̗̱͐̽̌̓̀̃̕̚͟͝è̷̡̢̦̞̜̺̿͂͛͛͂́͘r̸̨͓̫̳̆͆͆̀͗̈́͆ͅẏ̹̠̠̪̯̳̗̯̀͒͊̂͋̂̈̉̍͜ ų̡̺̖̥̲̓͗͛̑͟͡͠͝s̢̡̫̜̙̞͓̄̈́̾̌̀̅͂̏͟͟͞͞ͅe̶̯̫̹̠͔͗̍͌̄̓͊̐͐͋̚͢f̸̳̫̝̭̀̄̾͐̔̓͒͘͘͜u̻̤̠͙͕̯̲̟͍̐̄̔̿̈́̇̈́l̶̦͕̮̝̬͂̏͋̈̾̀͘ͅ.̷̢̛̙̻̙̜͋̒̃̇̆͜ͅ . . ⟵ Likewise, Zalgo text is not very useful

 
So, long story short, yes, it could have been avoided, but alas, that didn't happen. More than just 65535 code points are needed in one consistent representation, so you either need 4-byte characters, or UTF-8 (which despite its disadvantages works reasonably well for most things) or something like UTF-16 (which has all the disadvantages of UTF-8, plus some more, but no advantages).