-6

Some say UTF-8 was the best solution.

The price you pay is that it basically makes all parsing optimizations that rely on a fixed relationship of byte offset to character position unusable.

Compilers, Interpreters, XML/HTML parsers and similar parsing-heavy applications already existed in the Pentium Pro era - and behaved much less sluggish than a modern equivalent would on a machine with even 5 times the processing power of a machine from that era. These usually used ISO-8859-xx fixed character size encodings and similar. Windows used fixed 16bit (wide char) formats internally in many places from early on. UTF-8 was considered as newfangled and impractical as IPv6 still is back then.

How much is this innovation related to an explosion in needed processing power for such tasks?

Clarification: The main question is whether going just all 32-bit, memory usage or not, would have in the end been the better way.

12
  • 1
    (Haven't voted for closure by now - not at least because I do not really know what reason to pick, as it is a) not asking for any old development (ok, given, Unicode is last century :)), but about modern results, b) without any proof for the claim made and c) rather opinion based, thus asking for opinions. So, what to do?
    – Raffzahn
    Jul 4 at 22:02
  • 1
    1) It is asking for new stuff, not old, 2) It does not support any proof for the claim made (like numbers that the very same algo runs slower when made for UTF-8) 3) asking for any proof for/against your own hypothesises asking for opinions. So pick any of the three - or all - and 'work around (though, I'd suggest rather basic rethinking than just patching for compliance)
    – Raffzahn
    Jul 4 at 22:11
  • 3
    It's amusing that a question mentioning XML thinks that a character encoding is the reason for bloat :-) Jul 4 at 22:13
  • 1
    The core of my question is trying to understand bloat - but asking generic question tends to get vague answers, that's why I considered ballooning hypotheses a better way ;) Jul 4 at 22:14
  • 1
    Regarding the clarification: see What topics can I ask about here?, in particular “Almost all "What If?" or alternate history questions, as they are fictitious in nature and rarely bring genuine insight.” which is in the “What should I avoid asking?” section. Jul 5 at 6:53
9

It's not.

The price you pay is that it basically makes all parsing optimizations that rely on a fixed relationship of byte offset to character position unusable.

Okay, but what parsing optimisations do? “Character” is a meaningless concept for almost every syntax you might want to parse. If you use bytes, so long as your special characters (e.g. ={}()[];) are in ASCII, you can treat as a three-byte variable name, and you can use the same algorithm.

That is the big selling point of UTF-8: unless you're messing with the high bit, almost every ASCII-aware algorithm will keep working perfectly. If you were to run a Pentium Pro-era compiler or interpreter, you'd find it runs as quickly as you'd expect it to on modern hardware (even if you feed it UTF-8 input).

UTF-8 was considered as newfangled and impractical as IPv6 still is back then.

I'm not so sure. UTF-8 is entirely backwards-compatible with ASCII. If anything, UCS-2 is the equivalent of IPv6: a backwards-incompatible protocol adopted by the new-fangled NT operating system, where all characters take up more space but, in return, many more characters can be used.

4
  • 1
    The high-bit convention in UTF-8 is at odds with 8-bit codes like the various 8859-x, in use in European countries. Backwards compatibility with ASCII is not a hallelujah to them. But I agree with your observations on scanning for separator characters, which more often than not come from the common ASCII subset. Jul 4 at 21:08
  • 1
    @another-dave But it's unlikely that the ensuing mojibake will have a significant effect on the compiler / interpreter / parser, since most punctuation characters these things use are in ASCII.
    – wizzwizz4
    Jul 4 at 22:19
  • Oh, I agree with that part- just let's not oversell the backwards-compatibility. Something expecting UTF-8 gets nutty when fed 8859-1 (usually by clueless programmers). That's all I meant. Jul 4 at 22:24
  • @another-dave Yup: but the compatibility is quite peachy going the other way. (Most of the time.)
    – wizzwizz4
    Jul 4 at 23:50
7

No.

As an American, which I'm guessing you are, you probably operated in an ASCII bubble. As a Western European, which you might be, you at least had some not-quite-ASCII variant of a 7-bit character code, or maybe some 8-bit ISO code, that preserved the character/byte equivalence at the cost of having to associate an encoding with the text and/or some incompatibilities with code written by ASCII chauvinists.

The rest of the world was already dealing with the non-equivalence of bytes and characters, and careful programmers even in the US needed to take that into consideration.

So: this was a pre-existing condition. To determine character counts required parsing a string. The C RTL has long since had routines to do that -- showing it has long since been necessary. UTF-8 improves the situation by having one common bytes-to-characters mapping and by its self-synchronizing structure; it is not the cause of the problem.

At one time I would have sung the praises of UCS-2, believing that 65536 characters ought to be enough for anyone, but since that was long ago shown to be untrue, UTF-8 is the best we have got. We might long for some past time when life was simpler (and indeed this forum is the place where we do a lot of that longing), but UTF-8 did not initiate the Fall.

5

I'll express a somewhat dissenting opinion to the rest of the provided until now answers: yes, UTF-8 (or, rather, more generally Unicode at large) is partially and indirectly responsible for the bloat in the required CPU resources. But its contribution is just a drop in the ocean, small enough for most people to say "no, it isn't". Rather, it's the mechanism by which it adds overhead is what's responsible for a large (but not dominant) part of that extra processing cost.

If you want to pin specifically basic processing of the multi-byte sequences in Unicode as the culprit for increased processing resources usage, you'll have to look at 20-25 years ago instead of merely 10-15. 10-15 years ago the various Unicode encodings were already common place - mind you, that was Windows XP era, which is Unicode-native. And as others have already pointed out, UTF-8 (or any other Unicode encoding) adds little to no processing overhead, especially compared to the existing encoding schemes for CJK languages.

What wasn't quite as common ten to fifteen years ago is the use of encryption everywhere, video, animation, UI effects, and a much stronger focus on security than ever before.

  • Until Windows Vista desktop GUI visual effects were mostly limited to things like showing the whole window content while it's being moved/resized, shadow under mouse pointer, etc. Since Windows Vista - shadows under Windows, smooth scaling animations, and other effects became much more common, and they all need CPU and GPU processing time.
  • If you don't have sufficiently new CPU with AES-NI or similar hardware implementation of encryption, modern-day trends for encrypting everything will impact your CPU usage as well. E.g., the difference between system with and without full disk encryption on something like Core 2 Duo or a first-generation Core i7 is noticeable even without any benchmarks, but do the same on second-generation Core i7 with AES-NI and performance difference is nigh-imperceptible. Same applies to various TLS-encrypted connections (e.g., HTTPS), where you typically have AES-encrypted session, except for both end points to agree on symmetric key they must first use un-accelerated asymmetric encryption for initial key exchange. Combine that with the fact that modern websites usually pull resources from multiple hosts and thus need multiple HTTPS connections...
  • Nowadays it's very common to use video where in the past an animated GIF would be used. Many modern websites show videos in a loop at the top of their main pages, and sometimes even multiple videos spread throughout the page. More animations, more visual effects - it all adds up very fast.

Finally, to your point about more sluggish behavior of modern XML parsers - XSLT is Turing-complete; C++ templates are Turing-complete; PostScript is Turing-complete; C++ exception stack unwinding is Turing-complete; ELF metadata is Turing-complete; MMU (memory management unit) page faults are Turing-complete; return-into-libc is Turing-complete enough to use it as attack vector; CSS is (almost) Turing-complete; SVG (which is XML-based) is Turing-complete; MS PowerPoint animations are Turing-complete; Speculative execution in modern CPUs is Turing complete (Ross Mcilroy, Jaroslav Sevcik, Tobias Tebbi et al, 2019-02-14), providing essentially a "shadow computer" of sorts for executing side-channel attacks. TrueType fonts are Turing-complete. And last but not the least, Unicode itself is Turing-complete as well, with its bidirectional algorithms being complex enough to support tag system and implement logical gates.

In general, the more "universal" some system is intended to be, the more likely it is to end up being Turing-complete. If we still used language-specific character encodings we wouldn't need bidirectional algorithms and thus there would be fewer opportunities for Turing-completeness. The problem with all of those unexpected and surprising examples of Turing completeness is that they all have security implications - once an attacker gains sufficient control over data being fed into those Turing-complete systems they can compromise your system and do anything they want with it. So mitigations pile up, checks for input sanitation, barriers that make optimizations like speculative execution less effective, etc. They all have performance impact, slowing our systems down. Specifically Unicode, and especially UTF-8 cause only a small fraction of all the security vulnerabilities out there that need mitigation, but they certainly do their part.

TL;DR: a system doesn't always have to be Turing complete to be a source of security issues. And accordingly, not everything that is a source of security issues is Turing-complete. But everything that's Turing complete can be a source of security issues. And everything that has security issues causes the necessity for those security issues to be mitigated. And almost every mitigation incurs a performance cost. UTF-8 is a particular encoding for Unicode. UTF-8 itself has security issues despite not being Turing complete. Unicode is Turing complete... Hilarity ensues.

2
  • UTF-8 is not Turing-complete, so I wonder which vulnerabilities you are thinking about.
    – user253751
    Jul 5 at 10:44
  • @user253751 UTF-8 by itself is not Turing complete indeed, but it's not without it's own security issues. unicode.org/reports/tr36/#UTF-8_Exploit provides some examples. P.S.: added a summary of sorts, hopefully that clarifies some things.
    – moonwalker
    Jul 5 at 23:43
4

Some say UTF-8 was the best solution.

No. Not really (*1). Of course it may have influence on some low level operations, but they are marginally. In addition, they are the very same as with any other multi byte character encoding. Not to mention that there is an essential classic version of this: Escape Codes :))

The price you pay is that it basically makes all parsing optimizations that rely on a fixed relationship of byte offset to character position unusable.

Which isn't so much at all. It's more of a psychological inconvenience. A string is still a string. When parsed into a token it will still result in a variable length structure, managed and handled as one. The only difference will be that string length is now in bytes, while number of characters may differ (*2). any further handling is as a token and as such independent of encoding.

Compilers, Interpreters, XML/HTML parsers and similar parsing-heavy applications already existed in the Pentium Pro era - and behaved much less sluggish than a modern equivalent would on a machine with even 5 times the processing power of a machine from that era.

Mind to provide some data supporting that it's due UTF-8, not simply other bloat creep?

UTF-8 was considered as newfangled and impractical as IPv6 still is back then.

This seems deeply opinionated. For one, variable length character encoding is something that is already around since the 1970s (*3), and second, success of IPv6 is not related nor is it a failure (*4).

How much is this innovation related to an explosion in needed processing power for such tasks?

Not at all. I would wonder if a difference could be detected at all.


*1 - Well, I do not like it, but that isn't in any way related to UTF-8 but a general dislike of all variable length encoded charsets. I try to stay with UCS-2 or -4 whenever possible, but that's just me being an old fart.

*2 - But can easy be calculates (almost for free) when its structure is filled.

*3 - ECMA-35, which eventually ended up as ISO 2022 was first codified in 1971. BTW with heavy involvement of DEC.

*4 - In essentially all environments I touch (except retro that is), IPv6 is a done deal since severalyears.

8
  • You may be surprised to know that UCS-2 and UCS-4 are variable-length encodings. How many bytes do you suppose are occupied by the "s̷͇̫̳̓̃͂̈̃̽͛̀̈" character in "t̷̡͚̘͕̘̰̲͖̯̲̼̹͎͋̎̇͂̽e̴̬̥̥̟͕͕̩͚͓͚̪̙̺̪͂̀s̷͇̫̳̓̃͂̈̃̽͛̀̈ț̵̡̨̆̑́̇̋̈͆͛̐́̾̇̚͝"? It's not 2, and it's not 4! Although, Unicode tries to pretend not to have this problem, by redefining the word "character" so that "s̷͇̫̳̓̃͂̈̃̽͛̀̈" officially counts as several characters. But we all know it's one character.
    – user253751
    Jul 5 at 10:45
  • @user253751 It would have been a good idea to simply describe what the issue, as your example(s) do not show is a recognisable way: i.stack.imgur.com/Gk124.png :) So I can not see the single character you mention and only guess your point is not about surrogate characters, but combined. Right? Well, that isn't a charset issue, but a use case by use case one. In one use an acute redefines a character in another it only helps reading. Unicode is not to decide when is a mark part of a character, forming a new one and when is it (just) a markup (like underline or italic)? More important...
    – Raffzahn
    Jul 5 at 12:14
  • Then just take s̷̓̃ - Unicode says it is 4 characters, but it is clearly 1. That character being "latin small letter s with a bunch of accents"
    – user253751
    Jul 5 at 12:16
  • @user253751 Unicode is by definition a bit schizophrenic, as it's basic was made to have a 1:1 representation of as many characters of existing charsets as possible (unlike SMS7 which goes strict or the glyph). This ofc includes charsets that worked with modifications. So, long story short: No, combined characters are not a single character, but a character with markup added to describe a modified glyph. (Also, it helps to wait for a follow up, when one is already marked to come)
    – Raffzahn
    Jul 5 at 12:16
  • By that reasoning a is not a character either, but no less than 15 (alternatively written as <b><i>a</i></b>)`?
    – user253751
    Jul 5 at 12:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.