21

It seems like using either UTF-16 or UTF-8 would have been a lot more straightforward than what Java decided on for storing strings in classfiles and JNI payloads. It is a monstrosity that combines aspects of UTF-16 and UTF-8 formats. Is there any rational explanation for choosing this option?

See: https://docs.oracle.com/en/java/javase/13/docs/specs/jni/types.html

It seems to me that this unique format is what you would get if you started with UTF-16, then passed that through a "UCS-2 to UTF-8" converter... in other words, something you'd only produce by mistake... a variation of https://en.wikipedia.org/wiki/Mojibake (albeit one that is lossless).

I have since discovered that there is a name for this format: https://en.wikipedia.org/wiki/CESU-8. (However the Java string format is technically a small extension of CESU-8 to add "overlong null bytes" and this extended version of CESU-8 has its own [vague] name: ‘Modified UTF-8’).

9
  • 8
    Would the fact that Java was originally designed to work with UCS-2 be a clue as to why things might be stored as they are?
    – supercat
    Commented Feb 27, 2023 at 20:31
  • 4
    I believe Java (first released in 1995) predates UTF-16 (standardized in 1996), but I'm not sure if it also predates UTF-8 (I see one RFC from 1996, but I don't know if it existed before). I do believe that UCS-2 was also the internal encoding for the Windows OS to properly support Asian languages, but I don't know how prevalent UCS-2 was elsewhere, and I also don't know what the specific reason was why Java chose it - but my educated guess (hence not an answer): In 1995, UCS-2 was the best thing available to support international languages without resorting to codepages. Commented Feb 27, 2023 at 21:14
  • 3
    this isn't retro is it? why not ask this over at SO itself?
    – davidbak
    Commented Feb 27, 2023 at 21:15
  • 4
    The nonstandardness is in the handling of UTF-16 surrogate pairs. "Proper" UTF-8 would require you to combine the surrogates to get one codepoint, then UTF-8-encode that. The Java approach independently encodes each surrogate (and thus ends up with two 3-byte units rather than one 4-byte unit). When the standard programmer-visible datatype 'appears to be' UTF-16, this is probably a win. Having written my share of encoding-converters, it seems to me to be a reasonable approach.
    – dave
    Commented Feb 27, 2023 at 22:02
  • 7
    @davidbak, SO is notoriously hostile to "why" questions, especially involving "history"
    – JoelFan
    Commented Feb 27, 2023 at 23:16

6 Answers 6

33

It seems to me that this unique format is what you would get if you started with UTF-16, then passed that through a "UCS-2 to UTF-8" converter.

As I understand it:

From a modern viewpoint UTF-16 and UCS-2 are different encodings, but historically it makes more sense to think of them as different versions of the same encoding. What is now called "UCS-2" was historically just called "Unicode".

Back in 1989 the ISO had proposed a Universal character set as a draft of ISO 10646, but the major software vendors did not like it, seeing it as over-complicated. They devised their own system called Unicode, a fixed-width 16-bit encoding. The software companies convinced a sufficient number of national standards bodies to vote down the draft of ISO 10646 and ISO was pushed into unification with Unicode.

This original 16-bit Unicode was adopted as the native internal format by a number of major software products. Two of the most notable were Java (released in 1996) and Windows NT (released in 1993). A string in Java or NT is, at its most fundamental, a sequence of 16-bit values.

UTF-8 was devised in 1992, as a format for encoding Unicode/UCS values as a self-synchronising sequence of bytes in a form suitable for use in Unix filenames, on the web and in other ASCII-based environments. In most cases, it was also smaller than the 16-bit fixed-width representation.

It became clear that 16 bits was not enough. ISO wanted to expand the code space to 32-bit. Unicode did not like this idea because of memory usage concerns and because software was already using 16-bit. There was a compromise: Unicode 2.0 in 1996 expanded the code space to just over 20 bits, and introduced UTF-16

UTF-8 was able to encode all of the new characters directly, and this was the standard way to do things. However, it did create problems for software that used UCS-2 internally and UTF-8 as a storage format.

Adopting "standard UTF-8" for such software would cause compatibility issues. If a new version of the software wrote a supplementary character, the old version of the software would have no way to handle the 4-byte sequence. It would likely either give an error, or mangle the text.

Whereas by using the UTF-16 encoded in UTF-8 format, there would be no need to change the routines that saved and loaded strings at all. There would be no need to change formats at all. From the perspective of old software, the new characters would just be regarded as pairs of unknown characters.

Java's variant has one other change. It uses a 2-byte code for code point zero. My understanding is that this was done to allow the string to be processed safely with standard C string handling routines, even if the Java string contained null characters.

1
16

At the time Java was designed, it was not at all clear that UTF-8 would emerge as the most commonly used Unicode representation. It was generally assumed that "Unicode" meant "two byte characters", as implemented for example in the Win32 API of Windows NT (released 1993). This can also be seen in the Java 16-bit char type (released 1995). Java and the JVM were originally completely implemented using 16-bit characters. Java began life as the "Oak" programming language in 1991, but it's hard to say how much changed between then and 1995. The precursors to UTF-8 (FSS-UTF) were only invented around 1992-1993. UTF-8 itself was not finally adopted by the IETF until 1998.

As it turned out, 16-bit characters was not enough for what Unicode eventually became, which meant Java had to expand their representation while still being compatible with the older 16-bit character representation. I can't find specific references to when Unicode "surrogate pairs" were developed, but they were introduced to Windows in Windows 2000.

In short, Unicode character encodings were going through a number of different transformations in the 1990s. Java and Windows are two major pieces of software developed during that time, and the legacy of their various choices are still very visible today.

2
  • 3
    Looks like surrogate pairs were part of Unicode 2.0 (July 1996)
    – JoelFan
    Commented Feb 27, 2023 at 23:19
  • It looks like Java was started in 1991 and the UTF-8 encoding designed in late 1992. They probably didn't have time to reconsider this later, when Java was put in the Netscape browser. Commented Feb 28, 2023 at 19:22
5

The main thing is that a Java string with their modified UTF-8 encoding can have no embedded null (zero) bytes. This is for C compatibility (in the implementation, and in the JNI) java strings are C-strings with null termination). But the code point \000 is valid UTF-8. They just chose to use a non-standard encoding of the code point \000 and IMO UTF-8 should have done it that way anyway given the prevalence in the real world in production of C-style APIs.

The other difference - no 4byte encodings - is most likely because in those days there was nothing interesting in that part of the Unicode spec - beyond the BMP - and thus it was an implementation complexity of no value. In those days implementation complexity was a big deal because of where Java was originally intended to be used/deployed: toasters and other devices with dinky computing resources.

Internally they used fixed size 16 bit characters because the approach Sun adopted for Unix - fixed size 32 bit characters - was totally a non-starter for their use cases. And nobody uses variable length coding internally because multi-byte character sets are a correctness and performance nightmare.

12
  • 3
    Rust uses UTF-8 internally and utf8everywhere.org argues for everyone to do it because doing anything with codepoints rather than either bytes or extended grapheme clusters is wrong 99% of the time. (See also eev.ee/blog/2015/09/12/dark-corners-of-unicode and manishearth.github.io/blog/2017/01/15/…) Raku does some kind of thing with ropes and baked-in Unicode tables to allow O(1) indexing and iterating by extended grapheme clusters.)
    – ssokolow
    Commented Feb 27, 2023 at 21:57
  • 1
    @JoelFan - sometimes misguided people do things like encode multiple independent strings into one "string" thing, separate by nulls. Only works of course for strings with counts, not C-strings. Sometimes other misguided people think they can use a "string" thing to hold a binary blob. Only works of course for strings with counts, not C-strings. And then, of course, there are the hackers ... who take advantage of the fact that sometimes a string with count is passed to an API that takes a C-string, fun ensues.
    – davidbak
    Commented Feb 27, 2023 at 23:25
  • 1
    @davidbak: If a language fails to provide a binary blob types other than strings, programmers who need binary blobs shouldn't be blamed for using strings for that purpose.
    – supercat
    Commented Feb 27, 2023 at 23:47
  • 9
    UTF-8 should have done it that way anyway Why? The need for UTF-8 to embed the null character is exactly the same as the need for ASCII to embed the null character. Some system designers design systems that don't permit embedded nulls. Why should UTF-8 go out of its way so that a system designed to not permit embedded nulls can have embedded nulls?
    – dave
    Commented Feb 28, 2023 at 0:01
  • 2
    @davidbak I was replying to "And nobody uses variable length coding internally because multi-byte character sets are a correctness and performance nightmare." with counter-examples.
    – ssokolow
    Commented Feb 28, 2023 at 2:13
4

[Throughout the remainder of this post, I'm going to use "UCS-2", "UCS-4", "UTF-8" (etc.) as they're current interpreted, even though most of those terms didn't exist at the time.]

When Java was new, there were basically only two choices available. Unicode promised (but in the long term, didn't deliver) encoding everything in 16-bits per code point (UCS-2).

The alternative was ISO-10646, which didn't specify encoding, as such. But it used 32-bit code points, and lacking a specification of anything else, at the time that was generally interpreted as meaning UCS-4.

Each of those has problems, but the problems with ISO-10646 were serious in the short term, while the problems with Unicode were much longer term.

The big controversy with Unicode at the time was something called "Han unification". Chinese and Japanese alphabets contain quite a few similar characters. Han unification used a single code point for both the Chinese and Japanese versions of many of those. Quite a few people (especially Chinese and Japanese people) didn't like that much (like: hated it with the blinding fury of a supernova).

The big problem with ISO-10646 (as it was seen at the time) was that it needed 32 bits per character. Hard drive prices were falling to the point that using more space for text was seen as somewhat reasonable, but quadrupling the size of nearly all text files was generally seen as just a bit too much.

UTF-8 had been invented at the time, but it wasn't particularly well known. Worse, Shift-JIS (among others) had created a strong distaste for variable width encoding schemes among many programmers. While UTF-8 avoided many of the worst problems, the simple fact that it was a variable-width encoding was enough for many programmers to reject it out of hand.

So, at the time Sun was faced with two (or conceivably three) choices: ISO-10646/UCS-4, ISO-1646/UTF-8, or Unicode/UCS-2.

I'm going to go on record as saying that if they'd chosen either of the ISO-10646 options, Java would have been still-born--long since dead and forgotten except in a footnote in an appendix about "otherwise promising projects killed by a single mistake that was understandable but still utterly bone-headed given the constraints of the time".

So in the short term, UCS-2 was really their only choice. Not a great choice by any means, but at least one that allowed Java to remain viable long enough for its early choices to turn into problems people still cared about decades later. And yes, they were faced with a bit more than just UCS-2 vs. UTF-161. By the time they dealt with making that move, quite a bit of software had been written taking advantage of lots of quirks in both UCS-2 itself, and how Java happened to implement parts of it. They needed to account for both, which resulted in something that's fairly kludgy at best.


1. Or at least a fairly reasonable approximation of each.
4

TL;DR: conforming UTF-8, as presently defined, cannot encode some valid (to Java) Java Strings, on account of disallowing code sequences encoding surrogate code points.


In the beginning, Java chose 16-bit Unicode code points for representing characters. That was perhaps short-sighted, but 16 bits was sufficient for all of Unicode at the time. Correspondingly, Java's primitive char type is 16-bits wide, and, uniquely among Java's primitive types, it is unsigned.

Such a representation is a bit unwieldy for a C-based JVM implementation, however, because raw Java character data tends to contain lots of null bytes, making it unsuitable for use with C string functions. To much the same degree, the Java representation also tends to be larger, which is unfortunate for storage-constrained devices. Such devices were definitely among Java's initial targets.

The UTF-8 of the time addressed both of these issues, which made it a good choice for the native-code representation of Java string data. It's unclear why Sun chose a variant encoding of code point 0 (which still contains a null byte), but there are at least two plausible explanations:

  • it helps distinguish between embedded Java char's with value 0 and C string terminators; and
  • it makes certain kinds of potential JVM string-handling bugs easier to troubleshoot.

These are not mutually exclusive.

And since the Unicode code point space did not provide for code points wider than 16 bits, and no code points were reserved (e.g. for surrogates), that one variant encoding was the only difference between Java's character encoding and standard UTF-8. Moreover, it's a pretty benign difference, because a UTF-8 decoder has to go out of its way to reject the Java variant-zero. A naive UTF-8 decoder will just decode it to code point 0, which is exactly what it in fact represents.

So Java pretty much chose UTF-8, not some amalgamation. And then UTF-8 was yanked out from under it.

When Unicode widened the code point space and created surrogate pairs and UTF-16, it also specified that UTF-8 encodings of surrogate code points were invalid code sequences. That made Java's (and others') existing practice non-conforming. But why should Sun have cared? They had working software, and the character representation Java used internally was, after all, internal. They stuck with it.

And Sun didn't particularly pivot on Strings, either. To this day, they are sequences of 16-bit unsigned integer values corresponding to Unicode code points in the BMP (including surrogates). Java does produce and consume UTF-16 surrogate pairs inside strings, but it does not forbid arbitrary placement of surrogate code points within. In this sense, Strings' character-data nature is not quite complete. It was much later that Java introduced code-point-based interfaces to Strings, which must effectively perform UTF-16 decoding of the character data, and the traditional char based interfaces are still available and widely used.

So, Sun just continuing to do what they were already doing is what moved Java from slightly-quirky-if-that UTF-8 to the bona fide-variant UTF-8 now known as CESU-8. But they couldn't move to UTF-8 even if they wanted to do, because Java Strings' allowance for unpaired surrogates makes them impossible to encode in conforming UTF-8. I suppose it's a bonus that it's easier to transcode Java String data to CESU-8 and back, exactly because there is a one-to-one correspondence between CESU-8 code sequences and Java chars. No interpretaion of surrogate pairs is required.

4
  • Why couldn't they have used a version of UTF-8 that simply allowed unpaired surrogates, instead of creating the much-less-standard CESU-8?
    – JoelFan
    Commented Feb 28, 2023 at 23:25
  • Obviously, @JoelFan, they could have chosen any encoding they wanted. I can't tell you for sure why they made a decision different from what you suggest, but I can say that it's likely the same decision I would have made under the circumstances. Again, they started with pretty much standard UTF-8, and changed nothing from that point on. I don't see how changing to an encoding such as you describe would have served any useful purpose for them. And I guess it's a matter of opinion whether such an alternative is well characterized as more standard than CESU-8. Commented Feb 28, 2023 at 23:36
  • 1
    Because then the coding from UTF-16 into UTF-8 is context-sensitive: the translation of the current char depends on the following char. If a proper pair is present, translate as the codepoint represented by the pair. Otherwise (unpaired, or reverse-paired) translate independently.
    – dave
    Commented Feb 28, 2023 at 23:37
  • 1
    It's also worth emphasizing, @JoelFan, that Java does not use CESU-8 to encode Unicode code point sequences. That would indeed be weird. Java uses it to encode sequences of 16-bit code units, for which it seems a more natural fit than standard UTF-8. Commented Feb 28, 2023 at 23:58
2

Unicode for Java is (or at least was at the time) strictly 16 bit character - as Unicode was as well a 16 bit code space. Within this 16 bit code surrogate code points (pairs) opened a way to extend the 2^16 character points by another 2^20 ones.

CESU-8 in turn is an encoding that for one allows to use the same address space, but at the same time guarantees that any conversion to Unicode will result in one or two valid 16 bit characters. As a result any program using a 16 bit character type will be able to handle them.

It essentially transposes the same mechanic 16 bit Unicode uses to UTF-8.

4
  • While Java was originally strictly 16 bit character, it long ago gained 32 bit codepoint counterparts to char APIs, e.g. String.codepointAt() vs String.charAt()
    – ninjalj
    Commented Feb 28, 2023 at 12:30
  • 1
    @ninjalj So? JNI need to be compatible.
    – Raffzahn
    Commented Feb 28, 2023 at 12:42
  • Unicode for Java is (or at least was at the time) strictly 16 bit character My point was that the word is there should probably be was
    – ninjalj
    Commented Feb 28, 2023 at 12:49
  • @ninjalj Sorry, you lost me. It's there, you even cite it.
    – Raffzahn
    Commented Feb 28, 2023 at 12:55

Not the answer you're looking for? Browse other questions tagged .