Is there a common convention to describe the encoding of a legacy text file?

Question

For the purpose of this question, a legacy textfile contains characters in the range 0x20 through 0x7e, with each line terminated by an OS-specific combination of 0x0d and/or 0x0a; it might be terminated by 0x1a. It does not contain markup or metadata enclosed by <>, a leading dot, or anything similar, and does not contain UTF-8 etc. encoded characters.

Is anybody aware of any historic attempt to add something to this that indicates what codepage is to be assumed if a byte in the range 0x7f through 0xff is encountered?

I'm obviously aware that email-style headers and things like JCL do it, but these are outside the file per se. What I'm interested in is any attempt to add something comparable with a Unicode BOM at the start of the file, or metadata appended after an explicit 0x1a EOF.

---

As further background: I've been researching the mess that surrounded the ALGOL-68 standardisation process, Wirth's withdrawal, and his design of Pascal. I believe that somebody on the committee acted in bad faith, that Wirth was in a position where he couldn't object openly, and that Pascal was his attempt to present the World with a working language before ALGOL got its act together: in practice ALGOL never really get its act together so his haste was unjustified.

As part of that, and as a longstanding Pascal user, I'm sketching out what I believe were Wirth's mistakes: many of which he later corrected (Modula-2, Oberon). However if I ever get to the position of working on a personal replacement, I'd need to sort out the character set (codepage) issue.

Wirth was on the cusp between 6-bit character sets, EBCDIC, and ASCII. Focusing on ASCII, there have been multiple codepages which contained the special ALGOL characters (assignment arrow etc.), which in many cases were also used by APL. Looking at the Free Pascal Compiler forum and mailing lists, codepages remain a major issue both from the POV of literals in the sourcecode and interpretation of textfiles as they are being read.

Taking that into account, if I ever get round to actually coding anything I'll need some way of handling both Unicode and arbitrary codepages, and I'm blowed if I'm going to make FPC's mistake and try to handle everything as UTF-8 internally.

I'm thinking along the lines of using 0x1a as a codepage marker, followed by CP and the codepage number or name, followed by another 0x1a. That would make sense in both ASCII and EBCDIC, and could easily be distinguished from data by looking at the overall length of the file.

Answer 1

The closest thing to a universal convention (in the sense that it can be applied to any kind of text file, not necessarily universal adoption) that I know of is Emacs’ file-local variable declarations:

-*- coding: cp437 -*-

This was first released in GNU Emacs in Emacs 20.1, released in 1997. It may have existed earlier in MULE.

The vim editor has a similar feature, called modelines:

vim:set fileencoding=cp437

This was introduced in Vim 5.2, released in 1998.

However, I can see some materials on the Web claiming that the latter does not work particularly well for encodings. I don’t actually use either editor daily, so I cannot say anything about that.

Both served as inspiration for the Python encoding declaration standard, PEP 263, adopted in 2001, which I guess is just barely old enough to count as ‘retrocomputing’ here.

Answer 2

When 8-bit codepages were in use, files were rarely transferred from one computer to another that used a different character set. So all files on the filesystem were assumed to contain text in the system's codepage.

The MIME system used for email (https://en.wikipedia.org/wiki/MIME#History) was one of the first systems that had in band signalling of what charset was in use.

Answer 3

Browsers and other programs that attempted to automatically deduce the character encoding of text files would scan them for patterns, such as opening and closing quotation marks, or common words in the language. This sometimes went hilariously wrong, with the most famous example being the version of Windows that misread the line “Bush hid the facts” as UTF-16 and turned it into mojibake. (You had to be there.)

The closest thing to this in wide use is the HTML element

<meta http-equiv="Content-Type" content="text/html; charset=..." />

This originally told the browser to start over and, this time, parse the document as if the server had sent that MIME type in the HTTP header, and was therefore usually put as close to the top as it could be. It was later simplified to <meta charset=... /> in HTML 5. However, if you tell a browser that your document is in the Latin-1 encoding, it (as the standard mandates) will actually assume it is in Windows code page 1252.

I’m still in the habit of saving my C and C++ source files with a byte-order mark, since without one, Microsoft Visual Studio 2008 would interpret them as the currently-active code page, and that version provided no other way to override this. The current version of MSVC does attempt to auto-detect the encoding, but why take chances? GCC assumes UTF-8 but provides command-line options to override this, and Clang, last I checked, only reads UTF-8, so UTF-8 with a byte-order mark works on all modern compilers with no flags needed, and is the only encoding all of them understand.

Answer 4

The general way this has been done for pretty much forever, is to have a set of 'patterns' (perhaps rexexps, perhaps something else) that are run against the file and the first (or highest priority) one that matches sets the encoding to use for the file.

This may be a fixed set encoded directly in each application that wants to access files, or there might be a standard database on the system that all applications use -- it depends on the OS. There might be a magic or .mime.types file somewhere that has this database and may be extendible by users to deal with any weird file they might come across, or might be in some proprietary form that is not (easily) extended.