54

Anyone who has studied Unicode and is honest will admit that UTF-16 was kind of a mistake. It was born from the early assumption that 16 bits would be enough for all of Unicode. Then a hack was invented to extend it to 21 bits using "surrogate pairs". This led to lots of software not working correctly for these higher valued characters, which continues to this day, especially for display and editing-oriented applications (the dreaded "double backspace bug").

There is also an issue of "big-endian" vs "little-endian" requiring a "byte order mark".

Much later, UTF-8 was invented and it has now become a consensus that UTF-8 is the best solution for pretty much all applications, since there is no special treatment of higher-valued characters and no byte-order-mark needed. As a bonus, it's backwards-compatible with 7-bit ASCII.

All of the problems with UTF-16 could have been avoided by being more forward-thinking at the beginning (I know hindsight is 20/20). But would it have been conceivable for the inventors of Unicode to foresee that 16 bits would not be sufficient, to think of UTF-8 at the beginning, and only specify UTF-8 and UTF-32 as valid encodings for Unicode? Is there any history to suggest that perhaps some people were more forward-thinking but their views were disregarded?

EDIT: I think the most concise criticism of UTF-16 is that it is a much more complex design than UTF-8, making it much more likely to be poorly understood by developers and thus poorly implemented and tested. It's not so much that the design itself is "broken".

  • Comments are not for extended discussion; this conversation has been moved to chat. – Chenmunka Mar 20 at 14:06
  • 1
    What's the "double backspace bug"? – wjandrea Mar 20 at 21:08
  • 2
    @wjandrea: In some text editors (in particular, older versions of Windows Notepad) for UTF-16 platforms, deleting a non-BMP character requires pressing Backspace twice. – dan04 Mar 20 at 22:25
45

The designers of Unicode did consider whether 16 bits would be sufficient, and decided that they would be, based on their assumptions:

Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 214 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally useful Unicodes.

The 16-bit choice was based in part on the idea that Unicode would only encode modern characters. UCS-2 (the original encoding, initially called “Unicode”) was designed with this in mind.

Unicode was however extended to include “legacy” characters a few years after its introduction, which broke the initial assumption and required more code points than could be represented with 16 bits. If the Unicode designers had wanted to include legacy characters, they would presumably have used a different encoding. I’m not aware of any particular difficulty which would have made UTF-8 harder to design in 1988 than in 1992.

UTF-16 was designed as an extension to UCS-2 to address the latter’s limitations, and was published quite late in the game (1996, with Unicode 2.0). UTF-8 was designed in 1992, which answers your question, but raises the question of why UTF-16 was used; the answer to that is that it was deemed an acceptable compromise to fix problems with UCS-2, and that it was important to continue with something similar to UCS-2, given that a fair amount of software already used it (including Windows NT).

| improve this answer | |
  • Comments are not for extended discussion; this conversation has been moved to chat. – Chenmunka Mar 19 at 18:48
  • 8
    The thing is that it was the wrong decision already in 1988. The Japanese had been supporting larger character sets via JIS, Shift-JIS, etc for quite some time. I worked in 1983 with an Australian company who used these encodings for in-memory strings - a thing of horror forever! But it worked with all Eastern char-set standards, unlike Unicode. Supposed to "cover all the scripts of the world" it was known not to do that even from the outset. – cliffordheath Mar 20 at 5:56
  • 7
    Considering that there are more than 50,000 Chinese origin characters alone this seems like an incredible short sighted decision. Then again they completely screwed up CJK so I'm not surprised. – user Mar 20 at 11:27
  • 3
    @user, many Chinese characters are not in everyday use. If you're taking your sample from magazines, newspapers, and books, you're going to completely miss the huge set of characters that are used mainly for personal names. – Mark Mar 20 at 20:51
  • @Mark So as long as the universal character encoding covers the basics it doesn't need to worry about some obscure Chinese ones? – user Mar 22 at 15:33
22

Cultural factors played a role. Western creators of Unicode expected that unifying East Asian ideographs would be similar to unifying Western fonts (roman, italics...) and that they get to decide where the line between modern and unusual characters lies. This proved to be too optimistic. For the unification, the devil was in the details, but also in the need of processing of multi-lingual documents and databases: variation selectors were already breaking the 16 bits per character principle, like UTF-16 later did.

In 2000, PRC published their GB 18030 standard as mandatory for all software applications marketed for China. At this point, at the latest, it became obvious that Unicode Consortium wasn't going to have a free hand in inclusion of characters if they want their standard to remain relevant world wide.

All this is a part of string representation evolution. Starting with external and internal encoding to be a simple character array, through a phase where fixed character width was preferred for internal encoding and variable character width for storage and transmission, to a final phase where the complexity of the problem requires variable character width everywhere. We are now somewhere between the middle phase and the last one. Was such evolution foreseeable? Maybe. But the arrays of characters have done a lot of useful work in their own time due to their simplicity of processing.

| improve this answer | |
  • Comments are not for extended discussion; this conversation has been moved to chat. – Chenmunka Mar 20 at 14:06
  • the complexity of the problem requires variable character width everywhere - I don't think that's true; UTF-8 external and fixed-width UCS-4 internal (32-bit char codepoints) would still be a possible choice, assuming that 32 bits is enough for everyone. Merely sub-optimal for a lot of cases where you write the string out again without doing much indexing / looping over chars, so the cost of UTF-8 decode / encode would be for no benefit. Also, UTF-8 internal saves memory space / mem bandwidth / cache footprint at the expense of (or in parallel) CPU cycles, which is usually a good tradeoff. – Peter Cordes Mar 20 at 21:16
  • (I'm assuming that we're processing text with many single-byte characters, so UTF-8 is much denser than UCS-4). If you were still talking about languages like Java that unfortunately hitched their wagons to UCS-2 and then got stuck with UTF-16, yeah that sucks for them; worst of both worlds when processing mostly English. Or were you talking about composite glyphs, where a sequence of 2 code-points means something special and needs to be kept together in that order, (like supercat's answer mentions)? – Peter Cordes Mar 20 at 21:18
  • 4
    @PeterCordes Except that 32 bits isn't enough for everyone. You'll probably want to read Manish Goregaokar’s Let’s Stop Ascribing Meaning to Code Points but the gist of it is that, in some languages, code points only become meaningful characters when combined into grapheme clusters. Eevee's Dark corners of Unicode is one I'd also recommend. – ssokolow Mar 20 at 22:28
  • 1
    My main point was that it wasn't complexity of the problem, it was implementation efficiency tradeoffs that lead to those choices. But yes, since multiple codepoints can be one character (grapheme), then there's no standard encoding with fixed-width characters. (I edited that in to my last comment after scrolling down more; I hadn't realized / had forgotten it was a thing.) – Peter Cordes Mar 20 at 22:36
14

Unicode represented a compromise among a number of competing objectives. Probably its biggest failing is its presumption that a single storage format could be suitable for string storage, string concatenation, text rendering, and text editing. It sacrificed considerable storage density so that text editing could be performed without having to be aware of all the details of the character set, but then evolved in such a way as to negate that advantage. If Unicode had specified a representation where all composite glyphs wre marked with start and end markers, then code which needs to format a piece of text could recognize composite glyphs and avoid splitting them without needing to have any idea of what they are. As it is, however, Unicode has gone so far afield that accurately determining whether a line may be split between two code points may require scanning an arbitrary quantity of text before it and having an up-to-date list of all supported country flags.

| improve this answer | |
12

But would it have been conceivable for the inventors of Unicode to forsee that 16 bits would not be sufficient, to think of UTF-8 at the beginning, and only specify UTF-8 and UTF-32 as valid encodings for Unicode? Is there any history to suggest that perhaps some people were more forward thinking but their views where disregarded?

The problem was indeed foreseen, and the views of those who wanted to start with a much larger number of code points were indeed put forward and disregarded.

The 1990 draft of ISO 10646 offered a 2,147,483,648 code point space¹ and the following encodings:

  • UCS-4, encoding all characters in single 32-bit values.
  • UCS-2, encoding only the Basic Multilingual Plane.
  • UTF-1, a variable-length encoding.

Note that while they did offer a 16-bit encoding UCS-2, this is not the same as UTF-16 because it's not variable length (it's simply impossible to encode all the code points in it). Arguably, even though UCS-2 likely would have been extensively used, this would still have been simpler than the situation in which we find ourselves today with UTF-16.

However, according to Wikipedia:

In 1990, therefore, two initiatives for a universal character set existed: Unicode, with 16 bits for every character (65,536 possible characters), and ISO 10646. The software companies refused to accept the complexity and size requirement of the ISO standard and were able to convince a number of ISO National Bodies to vote against it. The ISO standardizers realized they could not continue to support the standard in its current state and negotiated the unification of their standard with Unicode....

Meanwhile, in the passage of time, the situation changed in the Unicode standard itself: 65,536 characters came to appear insufficient, and the standard from version 2.0 and onwards supports encoding of 1,112,064 code points from 17 planes by means of the UTF-16 surrogate mechanism. For that reason, ISO 10646 was limited to contain as many characters as could be encoded by UTF-16 and no more, that is, a little over a million characters instead of over 679 million. The UCS-4 encoding of ISO 10646 was incorporated into the Unicode standard with the limitation to the UTF-16 range and under the name UTF-32, although it has almost no use outside programs' internal data.


¹Certain code points that involved numbers in "control character" ranges were not allowed to be used, so the total number of code points in this space that were available to be encoded was 679,477,248.

| improve this answer | |
  • What problems would there have been with saying that anything beyond roughly 50,000 characters would be encoded using "entity start", "entity middle", and "entity end" codes, and then representing each composite glyph as a sequence of 6-bit values identifying the components thereof, eliminating the need for any other composite-glyph handling? – supercat Mar 19 at 16:33
  • 2
    @supercat That's probably worth its own question, and certainly quite different from this topic. However, off-hand it seems to me to increase the number of illegal codepoint sequences, making string processing more difficult, as well as making strings longer. – cjs Mar 20 at 1:04
  • There would be an unbounded number of byte sequences that could, but don't represent glyphs. Given that the extremely vast majority of computer-processed characters are within the ASCII range (even if the text within an HTML document is in Japanese, all of the characters in all of the tags are going to be ASCII) using 16 bits for each ASCII character will render any additional size increase from the described approach relatively moot. I'm not sure why you think string processing would be more difficult, since implementations wouldn't need to know or care about what characters... – supercat Mar 21 at 16:16
  • ...represent valid glyphs. – supercat Mar 21 at 16:17
  • @supercat Now that I re-read your questions, it sounds like you may be talking about encoding (as opposed to code points), which is a separate thing from composite characters and their glyphs. Regardless, SE questions and especially comments are not a good place for open-ended questions; they're better suited for chat or another forum such as retrocomputingforum.com. (Feel free to replace your comments with a link should you decide to post elsewhere.) – cjs Mar 22 at 0:31
7

It would've been an incredibly hard sell. What a lot of people don't realize, even back in the day, was how fundamental 16-bit values were to the original Unicode standards. Until version 3.0 was published in September 1999, Unicode required that all processing be done using 16-bit values. This meant that neither UTF-8 nor UTF-32 could actually be called Unicode, and any program that worked on these encodings as-is without converting them a 16-bit representation for processing weren't Unicode conformant.

Even the distinction between UCS-2 and UTF-16 was considered unnecessary. There was quite a bit of doublespeak in the standard in order to maintain the fiction that the original goal of having one character per 16-bit value hadn't been broken by the addition of surrogate pairs.

Given this insistence on 16-bit values by the Unicode Consortium for almost entire decade, despite the much more practical and honest approach of the ISO 10646 committee, despite how UTF-8 was proving itself to be what could actually replace ASCII, it's hard to see how anyone could've convinced them to take a different direction. I'm sure there's still people involved with the Unicode Consortium that still think 16-bit values are all you really need.

| improve this answer | |
  • 2
    Well, UTF-8 is certainly NOT Unicode even today. It is one of possible Unicode representations in bytes. – Oleg V. Volkov Mar 19 at 15:54
  • 1
    @OlegV.Volkov It used to be that program that took UTF-8 as input, processed it as a sequence of 8-bit values, and then output UTF-8 couldn't be described, according to the standard, as using Unicode at all. – Ross Ridge Mar 19 at 16:05
  • @RossRidge: Of course, one of the advantages of UTF-8 is that if one has multiple UTF-8 strings whose code points behave in context-independent fashion, an application can perform many kinds of basic string manipulation without having to know or care about character encodings or Unicode. Unfortunately, properly handling even something like the string concatenation necessary to say The answers are "Woozle" and "Wozzle", in cases where code points don't behave in context-independent fashion, would require more code than many computers used to have RAM in total. – supercat Mar 19 at 20:19
7

Yes, absolutely, if not for some big players in the Unicode consortium heavily pushing an agenda for a 16-bit fixed-size character set early-on. On the ISO 10646 side, it was always assumed that a 16-bit character set would be insufficient. They had other things wrong that kept them from gaining traction though, and in 1990 a unification with Unicode was negotiated. Since the unification, Unicode has always been larger than 16-bit.

Your comment that

Much later, UTF-8 was invented...

is ahistorical. UTF-8 was invented soon after, in 1992, before Unicode was in widespread use. But at the time Microsoft was still pressing ahead with their UCS-2 agenda, treating the non-BMP characters as unimportant, because their most important goal was to displace the standard text-based protocols and file formats of the emerging internet with their own creations they could control. Obsoleting 8-bit code-unit text was a big part of that.

Then of course Sun and Java came along and carried UCS-2/UTF-16 forward.

If either of these things hadn't happened, it's very plausible that we could have gone straight to UTF-8 without all the UTF-16 headaches.

| improve this answer | |
  • 1
    If it had been true that 16 bits were enough, then getting rid of variable-length 8-bit encodings (the multibyte disasters of this world) for a fixed-length 16-bit encoding would have been a substantial improvement. I believe the 16-bit dam burst with Unicode 2.0, in 1996. Until then, UTF-8 as a solution was needed only for legacy byte-oriented systems. Now, I agree, UTF-8 is the best available solution, because UTF-16 combines the disadvantages of both systems. – another-dave Mar 20 at 22:07
  • @another-dave: "If it had been true" is really pointless to speculate about, since it wasn't, and would have become even more untrue over time. Moreover the benefits of fixed-length encoding are pretty much nonexistent. This has been discussed to death elsewhere. – R.. GitHub STOP HELPING ICE Mar 20 at 22:13
  • Certainly Unicode treated 16 bits as enough, and therefore it was reasonable for NT to adopt UCS-2 in line with the published standard, And, as a practicing programmer, I certainly see benefits in fixed-length encoding. – another-dave Mar 20 at 23:04
  • 1
    @another-dave: Anything you write with fixed-length encoding in mind is almost certainly broken with zero-width/combining-mark characters which are essential to many languages. The unit of "character" just isn't really as useful as it seems. What you want is almost always "string". – R.. GitHub STOP HELPING ICE Mar 20 at 23:29
6

TL;DR: UTF-8 and any other variable-length encodings are good "external", "exchange" format, but bad "internal processing" format.

You start from a very wrong premise that "UTF-16 was kind of a mistake" and "UTF-8 is the best solution for pretty much all applications". It isn't.

UTF-8 is very convenient in that you don't need to think about new code points—it can and will encode whatever new appears in Unicode standard and that its compatibility with lower part of ASCII often simplifies things even more. Thus UTF-8 is very good for data storage and transfer: you implement/use relevant lib in your program or write "UTF-8 encoded string" in your protocol/API spec once and it just keeps working.

However when you need to actually process text and do it fast, you will get hit by the greatest drawback of UTF-8: it is a variable length encoding. With any fixed length encoding you can trivially jump to an arbitrary character or n characters back/forward, and find string length from buffer length. More complex operations like slicing strings or fast substring search algorithms directly depend on this. In UTF-8, it is way more complex: you need to scan buffer for character positions all the time or index them soo many high-performance tricks just simply don't work. BTW, endianness is not a problem in internal representation, since library/VM simply "knows" its internals and doesn't need to "figure out" anything about it.

So there you are: a range of applications where UTF-8 is far from best and fixed length encodings shine. So, how do VMs and string/regexp libraries—two of the most common types of programs that implement some kind of internal string representation—get around this problem? Most often they either completely limit itself to some kind of fixed width or implement several different string representations with specific width and try to fit any incoming string in whatever smaller form it fits best and then introduces separate internal implementations for all necessary functions for each width where authors can use all the usual 8-bit era tricks trivially adjusted, but constant number of bits. Sometimes UTF-8 is also added to the list just to be sure in hope that most strings will fit into internal fixed width representations and whatever won't and will need to fall back to slower algorithms will be minority. Since most UTF-16 text doesn't use anything out of 65536 range, you only need to scan it once—which is very fast and can be greatly optimized thanks to its fixed width nature—to make sure it fits, mark type in internal representation and then go ahead and use fast functions. Sometimes implementations go even further and simply treat all the UTF dangling bits—like surrogates—as entirely separate character. Why? The entire JavaScript string spec is written that way!

Some examples:

| improve this answer | |
  • 5
    UTF-16 isn't fixed width. The use of UTF-16 instead of UCS-2 implies the support of surrogate pairs to define a single character. – Ross Ridge Mar 19 at 16:13
  • 1
    For what purpose are code-point boundaries relevant? Glyph boundaries would be relevant for many purposes, but the rules for those are subject to constant change no matter what encoding one uses. – supercat Mar 19 at 16:15
  • 2
    But both UTF-8 and UTF-16 are variable length! So what advantage does UTF-16 bring to the table beside confusion and fake news that it is fixed length? – JoelFan Mar 19 at 16:38
  • 4
    @StephenKitt, easier to support incompletely, which is the root of the problem. Easier to make it seem like it works when it's really broken. – JoelFan Mar 19 at 17:30
  • 3
    The drawbacks of UTF-8 aren't as bad as @Oleg makes out. Unlike JIS/Shift-JIS, you can jump anywhere into a string and find a character boundary forward or backward. Although you need to count to index characters, by remembering the total count, and remembering the offset to the last-used position, you can optimally get to the required character 99% of the time (moving forward or backward from start, end or remembered point), because string algorithms are usually sequential. That's how we did it, and studies showed it was very effective. – cliffordheath Mar 20 at 6:03
1

It could have been avoided if Unicode had taken a more reasonable path, but being Design by Committee, that was practically guaranteed not to happen.

If desired, the BMP (= 16 bits of Unicode) plus some metadata, would have been way enough for everybody, forever.

Now, I said "reasonable path" above, but what is reasonable? That's of course debatable, and if you ask 5 people, you will get 6 opinions!

The Unicode guys deemed the goal "one encoding for anything and everything a human has ever written" reasonable, and desirable, and achievable. Which is pretty much the opposite of what ASA/ANSI had in mind a few decades earlier, which was more or less "English is the only language in the world".

I am personally baffled at how humans seem to live and think in extremes, are always over-reacting to extremes, and they never learn, or behave reasonably and with moderation. You see this prominently not only in computing, but also in many other fields (financials, politics, healthcare). I guess that's just how humans are.
Note the parallel with IPv4 and IPv6 where you have "4 billion is way more than we will ever need" versus "100,000 IP addresses per square meter (i.e. 64 bit) is not enough, we definitively need 128 bits". Both are ridiculous extremes.
Also note the parallel with present pandemia handling in EU (including Germany, where I live). Letting three months pass without any precautions, border controls, or preparations. Stock up masks or disinfectant? Build hospitals or extend ICUs, order a few extra respirators? Nah, what for, problem is in China. Let's instead have Carneval! And then, one day you wake up, completely surprised that people are dying and there's neither enough masks nor disinfectant, nor intensive care capacity. Now it's total panic, unreasonable measures totally over board, invading and limiting the freedom of millions in a way that hasn't been seen since the Third Reich, and driving entire industries and even countries to the brink of ruin. Sure enough, there was no way of doing it any better.

Now, human non-reason aside, the facts are that a mere 8 bits worth of characters are annoying for everybody except English speakers who do not need to do anything but "normal no-special text" whereas 16 bits are way enough for everybody except someone wishing to process ancient Chinese texts which contain symbols that most Chinese do not even recognize.

Heck, 16 bits are sufficient to accomodate languages that are spoken by a few hundred people worldwide, or have not been spoken by anyone for a thousand years at all. You can use Cherokee or Okham just fine with "only 16 bit" Unicode. You can write in Bamum, if you even know what that is (I had to look it up). Can you read Vedic Sanscrit? Well congrats if you can, and guess what, despite being dead for ~3000 years, it fits just fine into the BMP.
And, to top it off, you can encode the exact same glyph in several redundant ways, some of which do not really make much sense as separate characters (such as number 2 with or without a circle, or a filled circle around it, or as superscript and subscript). Much of what's in Unicode could as well, and would likely be much better encoded, by metadata/markup.

The assumption is that it's a good thing you can just type some mathematical symbols and Arab characters or Sumerian in English text, and it "just works" and it's all the same, you do not even need to waste a thought. The assumption is that you actually need this every day, and that not having to think about it is a good feature.
The assumption is that you want to embed symbols of a bronze-age artefact in your Whatsapp messages.

The assumption is that people like Donald Knuth and Leslie Lamport are idiots because they require you to add dollar signs when embedding a mathematical formula. You can just insert the symbols right away, can't you!

While there is some truth in the idea that it's generally cool to just type away and not care, reality has it that most of the time you do not really need this feature, and in fact it would be better if you were switching the equivalent of "context" or "code page" with some kind of control sequence. Think TEX dollar signs, think escape codes, think xml tags, whichever you prefer, all the same.

Did you ever wonder why in the beginnings of the web, the makers of HTML would create a feature like <p lang="fr"> ... </p>? Why would one want to do this? Who cares? Well, search engines do care, and accessibility services (e.g. text to speech) do. It is much less of a challenge for an accessibility service that intends to read for a blind person to correctly pronounce words if it is told what language a word or paragraph is written in, rather than having to guess from the character codes.

It wouldn't even be necessary (or desirable) to go all the way down to a single target language on the character level. French people write Italian or German or English too, at times. Koreans write Japanese at times. They both use a few hanja at times. So, while desirable as metadata in markup, it is probably not desirable to have the actual character encoding too tensely bound to one particular language.

However, if nothing else, one could for example have distinguished very roughly between these cases:

  • normal text for everybody in "living" languages LTR
  • normal text for everybody in "living" languages RTL
  • math, music, technical and electric symbols etc.
  • other dingbats, smileys, whatever
  • ancient texts, non-chinese
  • ancient texts, chinese
  • klingon, sindarin, morgul and what you like

The impact in terms of having to include escape characters (implicitly done by the word processor, or explicitly by writing markup, whatever) would be non-existent for 95% of all people, and very acceptable for the 5% writing scientific papers. On the contrary, explicitly pointing out that a formula is indeed a formula or that a foreign word is indeed just that is very helpful for processing (laying out, indexing, searching, translating, TTS) the document. Being able to tell immediately that a 3 is indeed a 3, only just in superscript is much preferrable to hoping that a program that processes some funny weird characters will figure it out.

It would have taken a mere dozen or so (or two dozen, depending on how elaborate one wants it) escape codes to do such a thing.

Unicode (necessarily, being forced) partially did go that route with e.g. the LRM/RLM markers. How could they otherwise have embedded the information that Arab text needs to be written the "wrong way around"?

But they did, in my opinion, not do it properly. Turning on RTL without context is a mis-feature if you ask me. Writing e.g. German right-to-left is just meaningless. On the other hand side, writing Arab LTR is equally meaningless, so why is this possible? Also, the ability of setting up homograph spoofing websites that read exactly the same as the original is not truly a feature, either. Nor is the ability to write the exact same thing with either combining or non-combining characters not a feature, but a nuisance.

Unicode also partially went the markup route with e.g. s̶t̶r̶i̶k̶e̶t̶h̶r̶o̶u̶g̶h̶, only just not properly. There is no way of knowing that s-t-r-i-k-e-t-h-r-o-u-g-h is actually strikethrough, and it's both a text processing nightmare, and a user annoyance (try and copy/paste that!).

 
. . L̶͔͍̪͕̯̼̪͙͒͋̽͂̆̅̚͠͝ì̡͕̖̳̞͊̉̈̀͊͘k̢̪̯̹̟̓͊̽͂͆͐̐́͢ȩ̶̳̙͔̭̙̪͗͒̔̄͟͡͡ẃ̸̳͇̬̟̯̾̽̈́͒̈́̋̔į̛̤͇̞̘͚̹̰̝̾̄̐̀͜ş̴̯͖̺̹̖͔̯͈̾̆͛̂̑̎́̅̂e̸̢̨̙͎̣̻͋̾́̔̾͜,̵̭̞̯̥͕̗̪͔̼͇̓̇́͛̽̽̓̉̚ Z̛͔͖͎̠͆̅̓̀͆̂̀͘͢͝ḁ̵̥̻̣̦̘̣̻̠͆̊̀̽̍̄́̓̃͋ͅḻ̶̡̣̜̬̗͙͔̫͊̌͆̐̔̚̚͜͡ĝ̘̜͚̟̟͍̬̜͉̈͛̌̿͡͠ö͕̳͉͈̝͉͕̟́̉̇̑̔͋̆̾̓͜͞ ṱ̠͓͍̜͈̾͋͘̕͡e̸̹̞͍͓̥̮͔͔͆̓̊͐͘͡x̶̰̱̖͎̰͕̤̺̙̞̑̂̄͑͡͝͡ț̶̗͕͍̗̖̎̔̑̌̄̓͡ i̡̺̠͕̟͚̝̲͂̆͐͋́̀͠͝s̸̛̛͕̻͖̺͕̘͔̱͛̈̓̒͑̚͢͢͡͡ ņ̖̞̔̌̑͊̌͒̓͢͜͝͠͡ò̶̗͙͖̫̟͚̞̱͌̄͛̓̇͝t̘͖̤̉͆̂̂̉͘͢͟͢͠ v͕̮̣̱̥̗̱͐̽̌̓̀̃̕̚͟͝è̷̡̢̦̞̜̺̿͂͛͛͂́͘r̸̨͓̫̳̆͆͆̀͗̈́͆ͅẏ̹̠̠̪̯̳̗̯̀͒͊̂͋̂̈̉̍͜ ų̡̺̖̥̲̓͗͛̑͟͡͠͝s̢̡̫̜̙̞͓̄̈́̾̌̀̅͂̏͟͟͞͞ͅe̶̯̫̹̠͔͗̍͌̄̓͊̐͐͋̚͢f̸̳̫̝̭̀̄̾͐̔̓͒͘͘͜u̻̤̠͙͕̯̲̟͍̐̄̔̿̈́̇̈́l̶̦͕̮̝̬͂̏͋̈̾̀͘ͅ.̷̢̛̙̻̙̜͋̒̃̇̆͜ͅ . . ⟵ Likewise, Zalgo text is not very useful

 
So, long story short, yes, it could have been avoided, but alas, that didn't happen. More than just 65535 code points are needed in one consistent representation, so you either need 4-byte characters, or UTF-8 (which despite its disadvantages works reasonably well for most things) or something like UTF-16 (which has all the disadvantages of UTF-8, plus some more, but no advantages).

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.