It could have been avoided if Unicode had taken a more reasonable path, but being Design by Committee, that was practically guaranteed not to happen.
If desired, the BMP (= 16 bits of Unicode) plus some metadata, would have been way enough for everybody, forever.
Now, I said "reasonable path" above, but what is reasonable? That's of course debatable, and if you ask 5 people, you will get 6 opinions!
The Unicode guys deemed the goal "one encoding for anything and everything a human has ever written" reasonable, and desirable, and achievable. Which is pretty much the opposite of what ASA/ANSI had in mind a few decades earlier, which was more or less "English is the only language in the world".
I am personally baffled at how humans seem to live and think in extremes, are always over-reacting to extremes, and they never learn, or behave reasonably and with moderation. You see this prominently not only in computing, but also in many other fields (financials, politics, healthcare). I guess that's just how humans are.
Note the parallel with IPv4 and IPv6 where you have "4 billion is way more than we will ever need" versus "100,000 IP addresses per square meter (i.e. 64 bit) is not enough, we definitively need 128 bits". Both are ridiculous extremes.
Also note the parallel with present pandemia handling in EU (including Germany, where I live). Letting three months pass without any precautions, border controls, or preparations. Stock up masks or disinfectant? Build hospitals or extend ICUs, order a few extra respirators? Nah, what for, problem is in China. Let's instead have Carneval! And then, one day you wake up, completely surprised that people are dying and there's neither enough masks nor disinfectant, nor intensive care capacity. Now it's total panic, unreasonable measures totally over board, invading and limiting the freedom of millions in a way that hasn't been seen since the Third Reich, and driving entire industries and even countries to the brink of ruin. Sure enough, there was no way of doing it any better.
Now, human non-reason aside, the facts are that a mere 8 bits worth of characters are annoying for everybody except English speakers who do not need to do anything but "normal no-special text" whereas 16 bits are way enough for everybody except someone wishing to process ancient Chinese texts which contain symbols that most Chinese do not even recognize.
Heck, 16 bits are sufficient to accomodate languages that are spoken by a few hundred people worldwide, or have not been spoken by anyone for a thousand years at all. You can use Cherokee or Okham just fine with "only 16 bit" Unicode. You can write in Bamum, if you even know what that is (I had to look it up). Can you read Vedic Sanscrit? Well congrats if you can, and guess what, despite being dead for ~3000 years, it fits just fine into the BMP.
And, to top it off, you can encode the exact same glyph in several redundant ways, some of which do not really make much sense as separate characters (such as number 2 with or without a circle, or a filled circle around it, or as superscript and subscript). Much of what's in Unicode could as well, and would likely be much better encoded, by metadata/markup.
The assumption is that it's a good thing you can just type some mathematical symbols and Arab characters or Sumerian in English text, and it "just works" and it's all the same, you do not even need to waste a thought. The assumption is that you actually need this every day, and that not having to think about it is a good feature.
The assumption is that you want to embed symbols of a bronze-age artefact in your Whatsapp messages.
The assumption is that people like Donald Knuth and Leslie Lamport are idiots because they require you to add dollar signs when embedding a mathematical formula. You can just insert the symbols right away, can't you!
While there is some truth in the idea that it's generally cool to just type away and not care, reality has it that most of the time you do not really need this feature, and in fact it would be better if you were switching the equivalent of "context" or "code page" with some kind of control sequence. Think TEX dollar signs, think escape codes, think xml tags, whichever you prefer, all the same.
Did you ever wonder why in the beginnings of the web, the makers of HTML
would create a feature like <p lang="fr"> ... </p>
? Why would one want to do this? Who cares? Well, search engines do care, and accessibility services (e.g. text to speech) do. It is much less of a challenge for an accessibility service that intends to read for a blind person to correctly pronounce words if it is told what language a word or paragraph is written in, rather than having to guess from the character codes.
It wouldn't even be necessary (or desirable) to go all the way down to a single target language on the character level. French people write Italian or German or English too, at times. Koreans write Japanese at times. They both use a few hanja at times. So, while desirable as metadata in markup, it is probably not desirable to have the actual character encoding too tensely bound to one particular language.
However, if nothing else, one could for example have distinguished very roughly between these cases:
- normal text for everybody in "living" languages LTR
- normal text for everybody in "living" languages RTL
- math, music, technical and electric symbols etc.
- other dingbats, smileys, whatever
- ancient texts, non-chinese
- ancient texts, chinese
- klingon, sindarin, morgul and what you like
The impact in terms of having to include escape characters (implicitly done by the word processor, or explicitly by writing markup, whatever) would be non-existent for 95% of all people, and very acceptable for the 5% writing scientific papers. On the contrary, explicitly pointing out that a formula is indeed a formula or that a foreign word is indeed just that is very helpful for processing (laying out, indexing, searching, translating, TTS) the document. Being able to tell immediately that a 3 is indeed a 3
, only just in superscript is much preferrable to hoping that a program that processes some funny weird characters will figure it out.
It would have taken a mere dozen or so (or two dozen, depending on how elaborate one wants it) escape codes to do such a thing.
Unicode (necessarily, being forced) partially did go that route with e.g. the LRM
/RLM
markers. How could they otherwise have embedded the information that Arab text needs to be written the "wrong way around"?
But they did, in my opinion, not do it properly. Turning on RTL
without context is a mis-feature if you ask me. Writing e.g. German right-to-left is just meaningless. On the other hand side, writing Arab LTR is equally meaningless, so why is this possible? Also, the ability of setting up homograph spoofing websites that read exactly the same as the original is not truly a feature, either. Nor is the ability to write the exact same thing with either combining or non-combining characters not a feature, but a nuisance.
Unicode also partially went the markup route with e.g. s̶t̶r̶i̶k̶e̶t̶h̶r̶o̶u̶g̶h̶, only just not properly. There is no way of knowing that s-t-r-i-k-e-t-h-r-o-u-g-h is actually strikethrough, and it's both a text processing nightmare, and a user annoyance (try and copy/paste that!).
.
.
L̶͔͍̪͕̯̼̪͙͒͋̽͂̆̅̚͠͝ì̡͕̖̳̞͊̉̈̀͊͘k̢̪̯̹̟̓͊̽͂͆͐̐́͢ȩ̶̳̙͔̭̙̪͗͒̔̄͟͡͡ẃ̸̳͇̬̟̯̾̽̈́͒̈́̋̔į̛̤͇̞̘͚̹̰̝̾̄̐̀͜ş̴̯͖̺̹̖͔̯͈̾̆͛̂̑̎́̅̂e̸̢̨̙͎̣̻͋̾́̔̾͜,̵̭̞̯̥͕̗̪͔̼͇̓̇́͛̽̽̓̉̚ Z̛͔͖͎̠͆̅̓̀͆̂̀͘͢͝ḁ̵̥̻̣̦̘̣̻̠͆̊̀̽̍̄́̓̃͋ͅḻ̶̡̣̜̬̗͙͔̫͊̌͆̐̔̚̚͜͡ĝ̘̜͚̟̟͍̬̜͉̈͛̌̿͡͠ö͕̳͉͈̝͉͕̟́̉̇̑̔͋̆̾̓͜͞ ṱ̠͓͍̜͈̾͋͘̕͡e̸̹̞͍͓̥̮͔͔͆̓̊͐͘͡x̶̰̱̖͎̰͕̤̺̙̞̑̂̄͑͡͝͡ț̶̗͕͍̗̖̎̔̑̌̄̓͡ i̡̺̠͕̟͚̝̲͂̆͐͋́̀͠͝s̸̛̛͕̻͖̺͕̘͔̱͛̈̓̒͑̚͢͢͡͡ ņ̖̞̔̌̑͊̌͒̓͢͜͝͠͡ò̶̗͙͖̫̟͚̞̱͌̄͛̓̇͝t̘͖̤̉͆̂̂̉͘͢͟͢͠ v͕̮̣̱̥̗̱͐̽̌̓̀̃̕̚͟͝è̷̡̢̦̞̜̺̿͂͛͛͂́͘r̸̨͓̫̳̆͆͆̀͗̈́͆ͅẏ̹̠̠̪̯̳̗̯̀͒͊̂͋̂̈̉̍͜ ų̡̺̖̥̲̓͗͛̑͟͡͠͝s̢̡̫̜̙̞͓̄̈́̾̌̀̅͂̏͟͟͞͞ͅe̶̯̫̹̠͔͗̍͌̄̓͊̐͐͋̚͢f̸̳̫̝̭̀̄̾͐̔̓͒͘͘͜u̻̤̠͙͕̯̲̟͍̐̄̔̿̈́̇̈́l̶̦͕̮̝̬͂̏͋̈̾̀͘ͅ.̷̢̛̙̻̙̜͋̒̃̇̆͜ͅ
.
. ⟵ Likewise, Zalgo text is not very useful
So, long story short, yes, it could have been avoided, but alas, that didn't happen. More than just 65535 code points are needed in one consistent representation, so you either need 4-byte characters, or UTF-8 (which despite its disadvantages works reasonably well for most things) or something like UTF-16 (which has all the disadvantages of UTF-8, plus some more, but no advantages).