9

The CP-1252 (sometimes called Windows-1252 or many more stupid names) encoding has five unused codepoints, 81h, 8Dh, 8Fh, 90h, 9Dh. The placement of these is not immediately obvious to me.

Are they left out for any particular reason? There's enough vacant codepoints to support North Sami, or Karelian, or Czech, or another language (just the first languages that came to mind that need five or less extras, I'm not making a statement about these being the most obvious choices). I would have thought whoever picked the graphemes would have given some thought to this, so I must be missing something.

8
  • CP-1252 did not happen "all at once", characters got added in several steps, presumably in answer to immediate needs. At some point past about 1990, there ceases to be much point in further additions, because Unicode.
    – dave
    Sep 14, 2022 at 12:06
  • 6
    Seeing that Czech needs (at a glance) č, ď, ě, ň, ř, ť, ů and their capital variants, no way it would fit. And both Sámi and Karelian are rather unimportant languages (in the eyes of most US and Europeans even at the very tail of possible languages to support), the answer is obvious. OTOH, adding č would be somewhat useful and adding ć and (lowercase only) đ would cover Serbocroatian (and Slovene). Though the latter addition would bite back the Unicode folk badly some years later... Sep 14, 2022 at 12:26
  • I recall older versions of CP125x (certainly the Windows 3.x ones) having no Euro sign. I suppose the unused code points were reserved for contingencies like this. Sep 14, 2022 at 12:38
  • 3
    @user3840170 The Euro sign was only designed in 1996, so even Windows 95 would have had no chance to include it. According to Wikipedia, CP 1252 dates to Windows 1.0, ten years before that.
    – IMSoP
    Sep 14, 2022 at 17:21
  • 1
    @OmarL the same argument can be made about Finnish and Estonian, Danish, Dutch and others. Spending them on any particular language will only have a minor over all effect - unless you're using exactly that language.
    – Raffzahn
    Sep 15, 2022 at 14:03

3 Answers 3

10

CP1252 grew out of a combination of ECMA-94 (which was derived from DEC's MCS) and Apple's Mac Roman (*1), these 5 positions are reserved for future expansion, a future that never happened. (*2)


It was common back then to have positions reserved for future extension (*3). Spare codepoints would be held for important additions otherwise not possible - like when assigning the Euro symbol in the late 1990s (*4). Filling them with random letters just to use them would be a short-sighted waste of this extremely limited resource.

A single code page is not really able to fit all Latin-based languages, not even the European ones. Adding 5 more characters would not be enough to make even a single additional language work. Even more so as that issue was already resolved by using language-specific code pages.

It might also be not very likely that they will ever be filled, as there are other ways in use for more than a decade. CP1252 is a relic for compatibility.


Some background/timeline to Paul Humphrey's mentioning of Office as base (and as usual way too long for a comment)

  • While MS Office wasn't a thing until August 1989, MS Word was.

  • Microsoft Word was published for DOS in 1983 using its own character set, based on DEC's MCS.

  • MCS is a precursor for the 1985 ECMA-94 character set, which in turn was adopted by ISO as 8859-1 in 1987, so long after the definition of CP-1252.

  • Word was ported early on to the Mac, published in spring of 1985.

  • Considering this it's natural that Windows 1.0, published in November 1985, uses a charset that supports their major publishing product Word.

  • Though, Word wasn't sold with Windows 1.0 as Word, but as Microsoft Write - this was most likely due a deal between Gates and Sculley (*5).

  • All symbols found in at rows 8x/9x are either to be found in the DOS-Word character set or the original (*1) Mac Roman character set.

  • 22 of the 27 characters in row 8x/9x of today's CP1252 are present in neither 8859-1 nor CP437 nor CP850, but are in Mac Roman.

  • Most likely, the remaining 5 are later additions - as Another-Dave mentions as well. The Euro symbol introduced in 1996 certainly is, as user3840170 notes.

Further history:

  • It wasn't until 1989 that an actual Word, based on a then up to date Mac Version, was released for Windows 2.11.(*6)

*1 - The original Mac Roman charset had only 237 characters and did grow from there on. Most additions happened until Mac OS 6, the Euro symbol for sure later.

*2 - So far :))

*3 - At least for somewhat forward looking designs.

*4 - In fact, the Euro symbol is a great argument for keeping some spare characters, as ISO's attempt to reform 8859-1 as ISO-IR-204 to add it simply didn't work out.

*5 - Sculley threatened Gates to stop delivery of Windows 1.0 due to being too similar to the Mac, which Gates countered by pulling WORD and EXCEL for the Mac, which were the leading products in their categories. The agreement led to Word being 'not published' for Windows.

*6 - Obviously felt Gates no longer bound by that agreement as Apple sued Microsoft anyway for Windows 2.0 in 1988 .

10
  • 2
    MS Write was not MS Word it was a much simple editor for RTF files. Word had its own format and language and much more formatting.
    – mmmmmm
    Sep 15, 2022 at 7:26
  • @mmmmmm True, it offered reduced functionality, compared with full Mac Word, but was nonetheless based on Word sources. Likewise was reading and writing of .DOC files enabled soon. In fact, not long after (1987?) MS even added Write for the Mac which was fully compatible with Word, but with limited functions, much like on Windows. It was intended as cheap package positioned below full Word.
    – Raffzahn
    Sep 15, 2022 at 14:18
  • I am not certain of all that - Write for Mac is according to wikipedia a cut down Word for Mac - but Write under Windows was released the same time as Mac Word but Windows Word took 3-4 years to come out - OK the first 2 might not count as Windows 1.0 would not have been capable enough and Write for Windows has always been part of Windows - I thought it was more an example of using different fonts in an app (ie we need some sources rather than guesses.) My knowledge is only as a user of Windows 1.0
    – mmmmmm
    Sep 15, 2022 at 14:29
  • @mmmmmm yes, full Word for Windows was only delivered short before Win 3.0 came - during the time the Apple vs. MS case were at its peak. Most likely part of their fight.
    – Raffzahn
    Sep 15, 2022 at 14:38
  • Adding the Euro sign was an especially big deal because (1) The 11 (now 20) countries that adopted it have a lot of combined economic clout, and (2) There was still a lot of non-Unicode-aware software back in 1999, so the € symbol had to be added to all the legacy 8-bit code pages for European languages. Since then, many other currency symbols (₭₮₯₰₱₲₳₴₵₶₷₸₹₺₻₼₽₾₿) have been added to Unicode without much fanfare.
    – dan04
    Sep 15, 2022 at 17:11
8

The Windows-1252 character set gradually evolved. Per Wikipedia:

  • The original version, distributed with Windows 1.0 (1985), was based on an early draft of ISO-8859-1 that did not include the × and ÷ symbols. (These code positions were originally planned for the French letter Œ/œ, but withdrawn by a French ECMA delegate who (incorrectly) insisted that this letter was a mere typographic ligature (like ff, fi, fl, ffi, ffl) instead of a linguistic one.) It also did not define any characters in the 0x80-0x9F range, leaving 34 unused codes.
  • Windows 2.0 (1987) added × and ÷ (compatible with the now-finalized ISO-8859-1), as well as the curly single quotes and (introducing the first incompatibility with ISO-85591-1). This left 30 unused codes.
  • Windows 3.1 (1992) added 22 characters ŒœŠšŸƒˆ˜–—‚“”„†‡•…‰‹›™, bringing the number of unused codes down to 8.
  • Windows 98 (1998) added the Euro sign and the Slavic letter Ž/ž, leaving the 5 unused codes we know today.

Are they left out for any particular reason?

Probably not. Microsoft just never had any immediate reason to assign characters to those byte values.

There's enough vacant codepoints to support North Sami, or Karelian, or Czech, or another language

Let's see exactly which other language could be supported.

Keep in mind that most Latin-script letters come in paired upper and lower case variants, so 5 code points is really only enough room for 2 extra letters. Unless one is a weird special case like German ß, which was a lowercase-only letter (never occurring at the start of a word) until Unicode introduced a capital in 2008.

If we take Unicode's exemplar characters as the official definition of “the alphabet” in each language, then candidate languages to add support for are:

  • Akan (ɔ and ɛ)
  • Asturian ( and )
  • Breton (ʼ)
  • Colognian (ė and ů)
  • Embu (ĩ and ũ)
  • Ganda (ŋ)
  • Hungarian (ő and ű)
  • Jola-Fonyi (ŋ)
  • Kamba (ĩ and ũ)
  • Kikuyu (ĩ and ũ)
  • Kurdish (ş)
  • Langi (ɨ and ʉ)
  • Luba-Katanga (ɔ and ɛ)
  • Meru (ĩ and ũ)
  • Nama (ǀ, ǁ, ǂ, and ǃ)
  • Nigerian ( and )
  • Quechua (ʼ)
  • Slovenian (č)
  • Turkmen (ň, ş)
  • Uzbek (ʻ and ʼ)
  • Walser (č and ũ)
  • Wolof (ŋ)
  • Yoruba (ɔ and ɛ)

And sure, Microsoft could decide, for example, to add the letters ŐőŰű to its code page mapping, so that you could encode Hungarian text in Windows-1252. But there's little point in doing so, because you can encode Hungarian text in Windows-1250. And more importantly, you can encode it in Unicode (either UTF-8 or UTF-16). Nobody spends the effort to update single-byte codepages anymore, because it's a “legacy” technology anyway.

1

CP-1252 is an extension of the ISO 8859-1 character set with extra typographic symbols, presumably for the benefit of things like Microsoft Office.

ISO 8859 is more or less the character set of DEC VT 220 dumb terminals IIRC. However I believe the codes 0x80-0x9f were used for extra terminal control codes instead. Much as 0x00-0x1f are for purposes like newlines or tabs.

Obviously, Windows runs on PCs not dumb terminals so the extra space was used for a few more characters instead. Perhaps the unused, reserved characters would eventually have been used for something had unicode not come about.

6
  • 1
    This all feels very speculative. The mention of Microsoft Office, for instance, doesn't match the (admittedly unreferenced) claim on Wikipedia that the code page dates back to Windows 1.0, several years before the "Office" brand was invented.
    – IMSoP
    Sep 14, 2022 at 17:28
  • 4
    0x80 to 9f (i.e., the "control chars with the high bit set) are the C1 controls standardized by ECMA et. al.
    – dave
    Sep 14, 2022 at 23:00
  • I did read somewhere once that the requirements of ms word were a factor, but can't recall now where I read it or if I'm recalling correctly. So yeah it's fair to call it speculative. Sep 15, 2022 at 20:21
  • 1
    @IMSoP: CP1252's inclusion of the typographic dashes (–—, versus ASCII - hyphen-minus), curly quotes (‘’‚“”„), angle quotes (‹›), ellipsis (), and bullet (, versus ASCII *) to me suggest an orientation towards WYSIWYG word-processing, even if not Word specifically.
    – dan04
    Sep 16, 2022 at 15:32
  • @dan04 "Word" is much older than "Office"; the connection is plausible, but the wording demonstrates a lack of detailed knowledge.
    – IMSoP
    Sep 16, 2022 at 20:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .