For the purpose of this question, a legacy textfile contains characters in the range 0x20 through 0x7e, with each line terminated by an OS-specific combination of 0x0d and/or 0x0a; it might be terminated by 0x1a. It does not contain markup or metadata enclosed by
<>, a leading dot, or anything similar, and does not contain UTF-8 etc. encoded characters.
Is anybody aware of any historic attempt to add something to this that indicates what codepage is to be assumed if a byte in the range 0x7f through 0xff is encountered?
I'm obviously aware that email-style headers and things like JCL do it, but these are outside the file per se. What I'm interested in is any attempt to add something comparable with a Unicode BOM at the start of the file, or metadata appended after an explicit 0x1a EOF.
As further background: I've been researching the mess that surrounded the ALGOL-68 standardisation process, Wirth's withdrawal, and his design of Pascal. I believe that somebody on the committee acted in bad faith, that Wirth was in a position where he couldn't object openly, and that Pascal was his attempt to present the World with a working language before ALGOL got its act together: in practice ALGOL never really get its act together so his haste was unjustified.
As part of that, and as a longstanding Pascal user, I'm sketching out what I believe were Wirth's mistakes: many of which he later corrected (Modula-2, Oberon). However if I ever get to the position of working on a personal replacement, I'd need to sort out the character set (codepage) issue.
Wirth was on the cusp between 6-bit character sets, EBCDIC, and ASCII. Focusing on ASCII, there have been multiple codepages which contained the special ALGOL characters (assignment arrow etc.), which in many cases were also used by APL. Looking at the Free Pascal Compiler forum and mailing lists, codepages remain a major issue both from the POV of literals in the sourcecode and interpretation of textfiles as they are being read.
Taking that into account, if I ever get round to actually coding anything I'll need some way of handling both Unicode and arbitrary codepages, and I'm blowed if I'm going to make FPC's mistake and try to handle everything as UTF-8 internally.
I'm thinking along the lines of using 0x1a as a codepage marker, followed by CP and the codepage number or name, followed by another 0x1a. That would make sense in both ASCII and EBCDIC, and could easily be distinguished from data by looking at the overall length of the file.