How is PETSCII used by assembly-language programs?

Question

First, a bit of context. I wrote an interactive disassembler for 6502 code that, among other things, attempts to identify ASCII strings automatically. Appropriate source code for different assemblers is generated. This works well for ASCII and the "high ASCII" often found in Apple II code, but having never worked with Commodore machines I'm at a bit of loss when it comes to PETSCII.

What I know comes from wikipedia, which has a couple of graphics showing the "shifted" and "unshifted" character sets. These show the letters in the 0x41-5a range (upper-case only) or the 0x41-7a range (mixed case, with lower case coming first). The article also mentions "text mode", in which lowercase letters run from 0x41-5a, and uppercase letters run from 0xc1-da. This latter mode seems to be what the 64tass assembler is aware of, e.g. in this example from the manual:

 .enc "petscii"      ;define an ascii->petscii encoding
 [...]
 >1000 93 d4 45 58 54 20 49 4e    .text "{clr}Text in PETSCII\n"
 >1008 20 d0 c5 d4 d3 c3 c9 c9 0d

Confusing me somewhat is an example of using "screen encoding", in which lower-case letters are in the range 0x01-1a:

                                .enc  "screen" ;screen code mode
 >1000 13 03 12 05 05 0e 20 03  .text "screen codes"
 >1008 0f 04 05 13

This doesn't seem to match up with anything on the wikipedia page.

My goal is to auto-detect PETSCII strings, present them in the disassembled output in a useful way, and generate assembly source code that is readable and generates the correct output. This is clearly important for string constants, but also matters for immediate character operands like:

CMP #'m'

I think there needs to be a general "here there be PETSCII" flag, which would change the way the disassembler handles character data and be used to set command-line flags for the cross-assembler (e.g. ca65's -t flag). I'm concerned that it might be insufficient. If there are multiple character sets in use, and they change from one line to the next, then some support for character set mode tagging may be required.

The ca65 assembler has a PETSCII conversion table that seems to correspond to "text mode", and is enabled with a command-line switch. However, the assembler also provides a .charmap pseudo-op that lets you override the global setting.

The approach I'm leaning toward mirrors the way cross-assemblers like ca65 and 64tass work, with a global "treat characters as PETSCII" flag. The other character modes would either be handled as hex data or as an explicit special-case format. Before I settle on this I want to be sure my understanding of PETSCII usage is accurate.

Various questions arise:

1. How do assembly-language programs on Commodore systems commonly deal with text? Is "text mode" the dominant form?
2. Is it common for programs to use multiple encodings, e.g. "screen codes"? Which system does keyboard input use?
3. Does regular ASCII get used with any regularity? Is it reasonable to expect that all character data will be some flavor of PETSCII?

EDIT: Just to untangle myself a bit, the C64 has two character sets in ROM. The system starts up with a set that only has upper-case letters and symbols. You can toggle it to a different set where various symbols have been replaced with lower-case letters. I have seen this second mode referred to as "shifted mode" (because you switch to it by holding Shift and then tapping the Commodore key), "lower case mode", or "text mode" (because text looks nicer in mixed case).

The way the characters are stored in the text screen buffer is similar but everything is shuffled around a bit. All of the symbols are represented from 0x00-7f. If the high bit is set, the character is shown with its colors inverted. This is what's meant by "screen codes".

The charts in https://www.aivosto.com/articles/petscii.pdf do a good job of expressing all this.

Answer 1

I'd say you're a bit out of luck with your objective, unfortunately. I'll focus on the C64, which is the only Commodore machine I programmed regularly, but it's probably quite similar on other machines using PETSCII as well:

The PETSCII encoding is a variation of ASCII, this is used by the builtin operating system for text. But at least the C64 has fonts in a completely different encoding, called "screen codes". The OS routines for text output do the conversion.

So, you will find two things kind of widespread:

• programs storing all their texts in PETSCII because they use OS routines for input/output.
• programs storing all their texts in "screen code", because they don't bother with the OS and directly write to screen memory.

I already doubt you can come up with a decent heuristic to distinguish between them. But of course, it can get worse: The C64 (and other Commodore machines) allow you to use custom fonts. I used this possibility in a document viewer recently, with a font "re-pixelled" in 8x8 after the CGA font, so the encoding is CP-437. Of course, all the text is in CP-437 encoding as well. Although you probably won't find many programs using neither PETSCII nor "screen code", it is possible.

Is it common for programs to use multiple encodings, e.g. "screen codes"? Which system does keyboard input use?

As for input, the situation is somewhat similar. The OS routines will return characters in PETSCII encoding, so a program using them expects input in PETSCII. A program using the hardware directly (scanning the keyboard matrix) will get matrix positions (row/column) of the key pressed and translate that itself. It might translate to PETSCII, to "screen code", or to anything else.

Answer 2

1. Most text is shown in text mode, as opposed to being drawn in bitmap mode.

2. Yes, programs can very easily use more than one encoding. PETSCII is what the OS APIs all use, including the ones to print things to the screen, but when actually printing to the screen it is converted to screen codes. Some software might decide to forgo the OS APIs and print to the screen directly, in which case it makes sense to store strings as screen codes.

3. Regular ASCII might come up in GEOS, but in programs that run in the BASIC environment, you can probably expect everything to be PETSCII. As software starts banking out the ROMs and replacing more and more of the OS with its own routines, these assumptions become a bit less reliable, and you can't really make as many assumptions. Games commonly define their own character sets, for example.

Answer 3

How do assembly-language programs on Commodore systems commonly deal with text? Is "text mode" the dominant form?

Assuming that you mean screen encoding by "text mode": If the programmer wants to put the text on screen by copying the string from memory, screen codes is the prefered storage method, since no conversion is necessary during the copy routine.

If the color of the characters should be also set or if control codes (color set, cursor movement, carriage return, etc) one would store the text in Commodore's version of ASCII which is called PETSCII and output the text character by character via the $FFD2 routine (which actually points to $F1CA on a C64). This, however, is much slower

Which system does keyboard input use?

The get character function in the ROM ($FFE4) gives back the PETSCII code.

Please note also that PETSCII is often used to emphasize rather the specific character graphics of Commodore computers rather than indicating the possibility to map control codes.

Does regular ASCII get used with any regularity? Is it reasonable to expect that all character data will be some flavor of PETSCII?

I would say yes. Unless one writes a program for a Commodore computer that should read ASCII files I don't see many applications for using ASCII instead of PETSCII. ASCII is also missing control characters for setting text color, moving cursor (other than down via line feed), and clear screen.

Other than that there is quite some overlap between ASCII and PETSCII, as long as one switches uppercase and lowercase characters.

Answer 4

The Wikipedia page on PETSCII is unfortunately very wrong. I advise you to have a look at the "talk" page. For reference I'm copying my own contribution there with some elaborations:

Instead of "shifted" and "unshifted", the two text modes were commonly called "lowercase" (or sometimes "business") and "graphics" mode.

Since the shift key sets the high bits in the character, the way the wikipedia article mentions the repeating characters is the wrong way around. Normal characters are 0x20-0x5F (as in old-style ASCII with only uppercase letters) and with shift they range 0xA0-0xDF. It is the others who were repeats. The reason for the repeats is that the PETSCII characters all map to 128 different "screen codes" (in the video memory) and they don't map 1-to-1, but the conversion code didn't care about that. In screen codes, the high bit is used for reverse video.

On this image you can nicely see how each shifted character relates to its unshifted one.

In the keyboards after this, there were fewer keys, so the nice regular mapping of the shift key to the high bit was partially lost.

About screen code values: In BASIC programs, and for file I/O, PETSCII values are used which as mentioned above are derived from an older ASCII version that only contains upper case letters, and has an up-arrow instead of ^, and a left-arrow instead of the underscore. So printable characters start at value 32. But in the character cell screen memory, it would be wasteful to not use the first 32 values. So the first screen code, 0, is used for the @ character (so screen codes 0-31 correspond to PETSCII 64-91, then screen codes 32-63 correspond to PETSCII 32-63, then the following 64 screen codes are the shifted variants of the first 64, and finally everything is repeated in inverse video)

To make things more confusing, in the very first PET model the screen codes for upper and lower case letters were swapped (compared to later versions, including the Commodore 64). So if you switched to lower case mode, your upper case letters remained on screen as they were, and to get lower case letters you had to press shift. Presumably because that was inconvenient, the character generator rom was rearranged, and that meant that when you changed into lower case mode, your existing on-screen uppercase letters became lower case (and some graphics became upper case letters), as it is known also on the 64.

Going from PET to VIC-20 and 64, the backslash was changed into a pound symbol.

So to get to your questions:

1. If an assembler program contains something like CMP #'m', what most assemblers would do is simply lift the given character code from the source and use that value. Since generally the program is written in PETSCII this would result in a PETSCII encoding

2. Many programs would likely use screen codes, but it would depend on how they show text to the user. If it stores the bytes directly into screen memory, it would need to use screen codes. Machine language programs would often fall into this category. But if the author preferred to use the ROM file I/O functions, they would use PETSCII. It would be slower, but easier since you don't need to take cursor positions and movement into account. BASIC programs would do this a lot (but for instance games which don't want to bother with moving the cursor around to have random access to screen positions would often include some POKEing to screen memory as well).

3. Most programs would in my expectation not use "regular ASCII", unless they would try to be compatible with the outside world. Say a terminal program such as Kermit. But since PETSCII includes the "uppercase only" parts of ASCII, a limited compatibility exist anyway.