Why does the C64's LIST command choke on REM statements with a Shift+L?

Question

On the Commodore 64 and VIC-20, the LIST command will abort with a ?SYNTAX ERROR on any line that contains a REM statement with a Shift+L character. A minimal program triggering the error would look as follows:

10 rem L

Some programmers deliberately started or peppered their BASIC programs with such "killer REM" statements to hide the source code. (Of course, the protection this provided was only rudimentary, and could be easily bypassed.)

As far as I know, this quirk affects only the C64 and VIC-20; the C128, C16, Plus/4, and various PET models don't care about the argument of REM statements. What I don't understand is why this quirk exists and how it works. Is this a bug or a feature of CBM BASIC? If it's a feature, was it ever documented anywhere by Commodore, and why is it present only on the C64 and VIC-20? If it's a bug, what exactly causes the LIST command to choke on a shifted L in a REM statement?

Answer 1

What I don't understand is why this quirk exists and how it works.

As usual, lousy programming. It's a routine that exists in next to every Microsoft Basic, but often modified by the receiving company. It's used to list a line. On the 6502 version space was a premium, so they tried to cut down as much as possible in tests. And lets be serious, a 'real' terminal can not produce codes above 128 so there is no chance someone would enter a token code somewhere. And what computer engineer in his right mind would create a machine using other codes than plain ASCII?

Is this a bug or a feature of CBM BASIC?

It's a combination of several little bits of 'less than perfect' ending up creating a bug.

and why is it present only on the C64 and VIC-20?

Because here the token table is exact 127 bytes of token names plus one byte of zero.

If it's a bug, what exactly causes the LIST command to choke on a shifted L in a REM statement?

As described already, the list routine iterates over all program lines and in each it displays the line number and then parses all token/text and print it. Tokens are recognized by having the high bit set. Tokens on the C64 reach from $80 (END) to $CA (GO) plus $FF for PI. Whenever a token is detected, the token table is searched for (PI is handled separately) in the most primitive way: $7F is subtracted and then tokens are scanned and skipped with that value decremented until zero is reached. Then this word pointing at is printed. Not really fast, but who cares during listing.This all works fine and stable - even with unused tokens. Try enter a Shift + M, and you get a FOR ($81) listed.

Just $CB fails, as it points to the combination of a single zero and an 128 Byte overflow.

In conclusion it's a combination of a missing upper boundary check together with an addressing problem that only appears when using a character set not designed for and an input routine not catching it either.

The original routine can be found in all MS-Basic 1.0. For newer MS-Basics the token table is of a different length and the routine is modified - usually by using a 16 Bit Pointer instead of a fixed address plus offset - as their token tables are larger than 127 bytes.

Answer 2

Internally, a BASIC program isn't represented as the text you see when you list it, but as a tokenized data structure where each of the language keywords are represented in an optimized 1-character form. Basically, if the upper bit is set in a character byte (i.e. values >= 128/$80), it is processed as a token.

Note that this does not only apply to the shift-L case, fx entering "10 REM Shift + A" will list as "10 REM ATN". If you consult a list of the predefined BASIC tokens, you will find that 203/$CB is the last defined; this will match Shift + K. After this, you are presumably (I'm speculating, haven't actually pored over the ROM listing) referencing a non existing or otherwise erroneous token, and the interpreter bails.

That you can enter tokens directly would seem like a bug or at least design limitation that was found acceptable, but I've never seen any official stance on that. Earliest mention of this token quirk I could find was in a 1983 issue of Compute!.

Answer 3

The answer by @Raffzahn is close, but it has at least one important detail wrong: the total length of the token names has nothing to do with it.

The following facts all all relate to the issue.

• Commodore Basic (any Microsoft Basic really) stores programs in a tokenized form in memory. That means here that all Basic keywords are replaced by a single byte, and other text is stored as PETSCII (Commodore's variant of ASCII).
• The token values start at $80 (0x80).
• In particular on the first PETs, there is a very strong relationship between the PETSCII value of a key and its shifted variant. Pressing shift adds $80. On later models, keyboards have been rearranged, and this connection is much less clear (but the character encoding has not changed).
• Text that follows the REM statement is just copied into the program, without tokenizing, but also without checking if it is pure text.
• Normally, bytes of $80 and higher that are meant to be text are only allowed in string denotations. When LISTing, strings are treated separately from non-strings.
• The first version of Basic allowed spaces in keywords while tokenizing. So if you write GO TO 10, this gets tokenized the same way as GOTO 10.
• This led to problems for instance with the statement IF ST AND 64 THEN..., which gets tokenized as IF S TAN D 64 THEN... which is of course a syntax error.
• The second version did not allow the spaces any more. But apparently it was deemed important that people could still write GO TO 10, so an extra keyword GO was added. TO already existed: FOR I=1 TO 10:...
• Now it was discovered that if you used the new GO keyword on a computer with the newer Basic and transported it to the older version, you would get the ?SYNTAX ERROR when listing the program there.
• The token value of GO is 203 (75 + 128 or shift-K).
• The reason was that the GO token was (there) just 1 past the allowed tokens (for that Basic version).
• The same problem exists on the new version, just with a token value 1 higher: 204, aka 76 + 128, aka shift-L.
• The VIC-20 and the Commodore 64 both use this version, substantially unchanged.
• This shows that the total length of the token strings being some magic value is NOT the cause of the problem: in the two BASIC variants, this total differs.
• The problem also exists for an even later version of Basic (BASIC 4.0 with disk management keywords added). Here the problematic value is 219 (91 + 128, or shift-[). Unfortunately, on keyboards of emulators, or the ones supplied with the PET model 8032, this is difficult or impossible to type. But if you enter a program 10 REM X and then POKE 1031,219 (which overwrites the X, if you didn't add anything extra), you can see the effect.

So what does cause the problem?

• The full text of the tokens is stored sequentially, in numeric order. The last letter of each token has $80 added to it.
• The table is terminated by a 00 byte, so the tokenizer can recognize the end when tokenizing.

Here is the code from LIST in the Basic ROM which prints out a token (Commodore 64 version): (source: https://www.pagetable.com/c64disasm/#A724 comments from Lee Davison)

The idea here is to start with a pointer (the Y index register) to the beginning of the token table. Subtract $7F from the token, which leaves the lowest value now at 1 (not 0). Decrement the token number (X register). If now 0, Y points to the correct token now, and skip ahead to print it. Else, skip until past the next byte with the $80 bit set and go back.

.,A724 38       SEC            else set carry for subtract 
.,A725 E9 7F    SBC #$7F       reduce token range to 1 to whatever 
.,A727 AA       TAX            copy token # to X 
.,A728 84 49    STY $49        save index for line 
.,A72A A0 FF    LDY #$FF       start from -1, adjust for pre increment 
.,A72C CA       DEX            decrement token # 
.,A72D F0 08    BEQ $A737      if now found go do printing 
.,A72F C8       INY            else increment index 
.,A730 B9 9E A0 LDA $A09E,Y    get byte from keyword table 
.,A733 10 FA    BPL $A72F      loop until keyword end marker 
.,A735 30 F5    BMI $A72C      go test if this is required keyword, branch always found keyword, it's the next one

So what happens with our 1-too-high token?

Well, our token pointer Y will have skipped past the last valid token text (and Y points at the terminator 00 byte). When decrementing the token number in X we end up at 00, which must mean we're at the right token text, so we decide to print this text. See A737.

Note how the BNE $A737 is marked "branch always". However, with our $00 byte that we just printed, this doesn't actually happen. The "print character" routine at $AB47 helpfully restores the status flags for the just-printed character.

                               print keyword

.,A737 C8       INY            increment keyword table index 
.,A738 B9 9E A0 LDA $A09E,Y    get byte from table 
.,A73B 30 B2    BMI $A6EF      go restore index, mask byte and print if byte was end marker 
.,A73D 20 47 AB JSR $AB47      else go print the character
.,A740 D0 F5    BNE $A737      go get next character, branch always

So as a result, we fall into the interpreter routine for the FOR keyword. It will try to fetch bytes from the program and interpret them as following a FOR keyword. However, what it gets isn't valid (the relevant pointer is pointing just past the LIST command we just issued), and hence: "?SYNTAX ERROR".

                               perform FOR

.,A742 A9 80    LDA #$80       set FNX
.,A744 85 10    STA $10        set subscript/FNX flag
.,A746 20 A5 A9 JSR $A9A5      perform LET
.,A749 20 8A A3 JSR $A38A      search the stack for FOR or GOSUB activity
.,A74C D0 05    BNE $A753      branch if FOR, this variable, not found
                               FOR, this variable, was found so first we dump the old one
.,A74E 8A       TXA

Here is the print character routine, shown because it explicitly restores the Zero flag.

                               print character

.,AB47 20 0C E1 JSR $E10C      output character to channel with error check
.,AB4A 29 FF    AND #$FF       set the flags on A
.,AB4C 60       RTS