10

For the purpose of this question, a legacy textfile contains characters in the range 0x20 through 0x7e, with each line terminated by an OS-specific combination of 0x0d and/or 0x0a; it might be terminated by 0x1a. It does not contain markup or metadata enclosed by <>, a leading dot, or anything similar, and does not contain UTF-8 etc. encoded characters.

Is anybody aware of any historic attempt to add something to this that indicates what codepage is to be assumed if a byte in the range 0x7f through 0xff is encountered?

I'm obviously aware that email-style headers and things like JCL do it, but these are outside the file per se. What I'm interested in is any attempt to add something comparable with a Unicode BOM at the start of the file, or metadata appended after an explicit 0x1a EOF.


As further background: I've been researching the mess that surrounded the ALGOL-68 standardisation process, Wirth's withdrawal, and his design of Pascal. I believe that somebody on the committee acted in bad faith, that Wirth was in a position where he couldn't object openly, and that Pascal was his attempt to present the World with a working language before ALGOL got its act together: in practice ALGOL never really get its act together so his haste was unjustified.

As part of that, and as a longstanding Pascal user, I'm sketching out what I believe were Wirth's mistakes: many of which he later corrected (Modula-2, Oberon). However if I ever get to the position of working on a personal replacement, I'd need to sort out the character set (codepage) issue.

Wirth was on the cusp between 6-bit character sets, EBCDIC, and ASCII. Focusing on ASCII, there have been multiple codepages which contained the special ALGOL characters (assignment arrow etc.), which in many cases were also used by APL. Looking at the Free Pascal Compiler forum and mailing lists, codepages remain a major issue both from the POV of literals in the sourcecode and interpretation of textfiles as they are being read.

Taking that into account, if I ever get round to actually coding anything I'll need some way of handling both Unicode and arbitrary codepages, and I'm blowed if I'm going to make FPC's mistake and try to handle everything as UTF-8 internally.

I'm thinking along the lines of using 0x1a as a codepage marker, followed by CP and the codepage number or name, followed by another 0x1a. That would make sense in both ASCII and EBCDIC, and could easily be distinguished from data by looking at the overall length of the file.

35
  • 6
    Basically all schemes I am aware of either have this information as part of an already existing wrapper (which you rule out, "no metadata"), or outside of the file (e.g. as language settings on the system). And given the sheer amount of possible encodings (which goes beyond "codepages", e.g. think IBM EBCDIC), I'd be really surprised if any standards exist that doesn't require you to specify this as some kind of human-readable metadata.
    – dirkt
    Feb 2, 2023 at 10:44
  • 2
    That confirms my experience, but I'm interested if anybody has actually come across anything otherwise... including on e.g. an IBM DisplayWriter or a Redactron. I carefully avoided consideration of EBCDIC (or for that matter the multitudinous 6-bit codes) since (a) any metadata prefix/suffix would presumably have to itself adhere to some character encoding and (b) EOF markers etc. are likely to differ. Feb 2, 2023 at 11:14
  • 3
    So you assume ASCII based encodings? So PETSCII text file can be left out? All computer systems seemed to be a bit special and not even fully ASCII compatible. Files were generally system and/or OS and/or program specific.
    – Justme
    Feb 2, 2023 at 22:21
  • 1
    Regarding Wirth and Pascal: Wirth was probably well aware of the laguage/country specific versions of ascii. That was most probably the reason for allowing '(' and ')' instead of '{' and '}' and '(.' and '.)' instead of '[' and ']'.
    – UncleBod
    Feb 3, 2023 at 9:01
  • 1
    @UncleBod No, those were a restriction of the IBM 029 cardpunch. He started off on an IBM (704?) at Berkeley, moved on to a B5000 (six-bit character set) at Stanford followed by an IBM 360, and then various machines in Europe including CDC. However almost everybody in those days bought or rented (sometimes rebadging) the desk-sized IBM punches. More interesting is := as an alternative to a special assignment glyph, which goes back to Zuse and IAC. Feb 3, 2023 at 9:12

4 Answers 4

16

The closest thing to a universal convention (in the sense that it can be applied to any kind of text file, not necessarily universal adoption) that I know of is Emacs’ file-local variable declarations:

-*- coding: cp437 -*-

This was first released in GNU Emacs in Emacs 20.1, released in 1997. It may have existed earlier in MULE.

The vim editor has a similar feature, called modelines:

vim:set fileencoding=cp437

This was introduced in Vim 5.2, released in 1998.

However, I can see some materials on the Web claiming that the latter does not work particularly well for encodings. I don’t actually use either editor daily, so I cannot say anything about that.

Both served as inspiration for the Python encoding declaration standard, PEP 263, adopted in 2001, which I guess is just barely old enough to count as ‘retrocomputing’ here.

1
  • As an aside, if the user ends up doing this as a pet project, this convention is reasonably widely supported in editors (or at least, PEP263 is). The difficult part of languages is always documentation and tooling.
    – Dannie
    Feb 9, 2023 at 12:11
15

When 8-bit codepages were in use, files were rarely transferred from one computer to another that used a different character set. So all files on the filesystem were assumed to contain text in the system's codepage.

The MIME system used for email (https://en.wikipedia.org/wiki/MIME#History) was one of the first systems that had in band signalling of what charset was in use.

10
  • 1
    My experience selling and maintaining software from the late 70s onwards is that it /does/ matter, since apart from anything else when software is shipped it typically (unless written by linguistic chauvinists) includes tailored documentation etc. Feb 2, 2023 at 13:01
  • 4
    "Rarely?" A lot of files was transferred from one computer to another just in the Win9x era that I remember – floppy disks to carry documents between home computer and workplace, SMB shares at work. And even here, as late to the Internet as we were, web and email was definitely a thing in 1998. (Websites had content-type tags, but MS Word documents were already being sent around that didn't have anything of that sort.)
    – grawity
    Feb 2, 2023 at 21:05
  • 4
    MS-DOS systems here went through five different national "system codepages" for the same language (with TSRs to switch between 770, 771, 773 on the fly), then Windows added two more – older computers running Win3.11 had a switcher to select between two "system codepages" as 1257 was still new; newer ones running Win98 had a MS Office addon for converting .doc's between the two as well as from DOS codepages to Windows ones. (Not to mention documents that needed Russian codepages, or even half-773/half-866 in the same file...)
    – grawity
    Feb 2, 2023 at 21:09
  • 3
    @user1686 Plus there was also the widely-popular Kermit which was regularly used to move data around within an institution, and of course BBSes and Compuserve etc. And a particular issue- predating PCs- was the couple of characters which were redefined relatively late in ASCII: ^ and _ vs up- and left-arrow (used e.g. in Smalltalk) if my memory is correct. Feb 2, 2023 at 21:29
  • @grawity: That's a full 30 years after the late 60s ALGOL/PASCAL stuff being talked about here. PCs did not exist yet at that time.
    – Chris Dodd
    Dec 28, 2023 at 6:48
5

Browsers and other programs that attempted to automatically deduce the character encoding of text files would scan them for patterns, such as opening and closing quotation marks, or common words in the language. This sometimes went hilariously wrong, with the most famous example being the version of Windows that misread the line “Bush hid the facts” as UTF-16 and turned it into mojibake. (You had to be there.)

The closest thing to this in wide use is the HTML element

<meta http-equiv="Content-Type" content="text/html; charset=..." />

This originally told the browser to start over and, this time, parse the document as if the server had sent that MIME type in the HTTP header, and was therefore usually put as close to the top as it could be. It was later simplified to <meta charset=... /> in HTML 5. However, if you tell a browser that your document is in the Latin-1 encoding, it (as the standard mandates) will actually assume it is in Windows code page 1252.

I’m still in the habit of saving my C and C++ source files with a byte-order mark, since without one, Microsoft Visual Studio 2008 would interpret them as the currently-active code page, and that version provided no other way to override this. The current version of MSVC does attempt to auto-detect the encoding, but why take chances? GCC assumes UTF-8 but provides command-line options to override this, and Clang, last I checked, only reads UTF-8, so UTF-8 with a byte-order mark works on all modern compilers with no flags needed, and is the only encoding all of them understand.

1
1

The general way this has been done for pretty much forever, is to have a set of 'patterns' (perhaps rexexps, perhaps something else) that are run against the file and the first (or highest priority) one that matches sets the encoding to use for the file.

This may be a fixed set encoded directly in each application that wants to access files, or there might be a standard database on the system that all applications use -- it depends on the OS. There might be a magic or .mime.types file somewhere that has this database and may be extendible by users to deal with any weird file they might come across, or might be in some proprietary form that is not (easily) extended.

2
  • That's the general way for file formats (as in structure), but I don't think I've ever seen magic being used to distinguish text encodings in structureless text files. Do you know of a specific way to tag different encodings that's commonly used (which was OP's question) or are you saying it's possible to detect different encodings using just regex?
    – grawity
    Dec 29, 2023 at 5:35
  • @grawity OP here. I think the closest possible thing is the use of modelines (etc.) in the accepted answer, even though when I posed the question I was hoping for something (e.g.) embedded in the EOF marker. The problem with any form of magic is that the characters that distinguish various CPs are likely to be relatively rare, /except/ when distinguishing between e.g. Western and Middle/Far-Eastern character sets. The specific case I was originally wondering about was pre-Unicode representation of APL, for which at least two ASCII-based fonts exist plus at least one EBCDIC. Dec 29, 2023 at 11:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .