The answer by @Raffzahn is close, but it has at least one important detail wrong: the total length of the token names has nothing to do with it.
The following facts all all relate to the issue.
- Commodore Basic (any Microsoft Basic really) stores programs in a tokenized form in memory. That means here that all Basic keywords are replaced by a single byte, and other text is stored as
PETSCII (Commodore's variant of ASCII).
- The token values start at $80 (0x80).
- In particular on the first PETs, there is a very strong relationship between the PETSCII value of a key and its shifted variant. Pressing shift adds $80. On later models, keyboards have been rearranged, and this connection is much less clear (but the character encoding has not changed).
- Text that follows the REM statement is just copied into the program, without tokenizing, but also without checking if it is pure text.
- Normally, bytes of $80 and higher that are meant to be text are only allowed in string denotations. When
LISTing, strings are treated separately from non-strings.
- The first version of Basic allowed spaces in keywords while tokenizing. So if you write
GO TO 10, this gets tokenized the same way as
- This led to problems for instance with the statement
IF ST AND 64 THEN..., which gets tokenized as
IF S TAN D 64 THEN... which is of course a syntax error.
- The second version did not allow the spaces any more. But apparently it was deemed important that people could still write
GO TO 10, so an extra keyword
GO was added.
TO already existed:
FOR I=1 TO 10:...
- Now it was discovered that if you used the new
GO keyword on a computer with the newer Basic and transported it to the older version, you would get the
?SYNTAX ERROR when listing the program there.
- The token value of
GO is 203 (75 + 128 or shift-K).
- The reason was that the
GO token was (there) just 1 past the allowed tokens (for that Basic version).
- The same problem exists on the new version, just with a token value 1 higher: 204, aka 76 + 128, aka shift-L.
- The VIC-20 and the Commodore 64 both use this version, substantially unchanged.
- This shows that the total length of the token strings being some magic value is NOT the cause of the problem: in the two BASIC variants, this total differs.
- The problem also exists for an even later version of Basic (BASIC 4.0 with disk management keywords added). Here the problematic value is 219 (91 + 128, or shift-[). Unfortunately, on keyboards of emulators, or the ones supplied with the PET model 8032, this is difficult or impossible to type. But if you enter a program
10 REM X and then
POKE 1031,219 (which overwrites the
X, if you didn't add anything extra), you can see the effect.
So what does cause the problem?
- The full text of the tokens is stored sequentially, in numeric order. The last letter of each token has $80 added to it.
- The table is terminated by a 00 byte, so the tokenizer can recognize the end when tokenizing.
Here is the code from
LIST in the Basic ROM which prints out a token (Commodore 64 version): (source: https://www.pagetable.com/c64disasm/#A724 comments from Lee Davison)
The idea here is to start with a pointer (the Y index register) to the beginning of the token table.
Subtract $7F from the token, which leaves the lowest value now at 1 (not 0).
Decrement the token number (X register). If now 0, Y points to the correct token now, and skip ahead to print it.
Else, skip until past the next byte with the $80 bit set and go back.
.,A724 38 SEC else set carry for subtract
.,A725 E9 7F SBC #$7F reduce token range to 1 to whatever
.,A727 AA TAX copy token # to X
.,A728 84 49 STY $49 save index for line
.,A72A A0 FF LDY #$FF start from -1, adjust for pre increment
.,A72C CA DEX decrement token #
.,A72D F0 08 BEQ $A737 if now found go do printing
.,A72F C8 INY else increment index
.,A730 B9 9E A0 LDA $A09E,Y get byte from keyword table
.,A733 10 FA BPL $A72F loop until keyword end marker
.,A735 30 F5 BMI $A72C go test if this is required keyword, branch always found keyword, it's the next one
So what happens with our 1-too-high token?
Well, our token pointer Y will have skipped past the last valid token text (and Y points at the terminator 00 byte). When decrementing the token number in X we end up at 00, which must mean we're at the right token text, so we decide to print this text. See
Note how the
BNE $A737 is marked "branch always". However, with our $00 byte that we just printed, this doesn't actually happen. The "print character" routine at $AB47 helpfully restores the status flags for the just-printed character.
.,A737 C8 INY increment keyword table index
.,A738 B9 9E A0 LDA $A09E,Y get byte from table
.,A73B 30 B2 BMI $A6EF go restore index, mask byte and print if byte was end marker
.,A73D 20 47 AB JSR $AB47 else go print the character
.,A740 D0 F5 BNE $A737 go get next character, branch always
So as a result, we fall into the interpreter routine for the FOR keyword. It will try to fetch bytes from the program and interpret them as following a FOR keyword. However, what it gets isn't valid (the relevant pointer is pointing just past the LIST command we just issued), and hence: "?SYNTAX ERROR".
.,A742 A9 80 LDA #$80 set FNX
.,A744 85 10 STA $10 set subscript/FNX flag
.,A746 20 A5 A9 JSR $A9A5 perform LET
.,A749 20 8A A3 JSR $A38A search the stack for FOR or GOSUB activity
.,A74C D0 05 BNE $A753 branch if FOR, this variable, not found
FOR, this variable, was found so first we dump the old one
.,A74E 8A TXA
Here is the print character routine, shown because it explicitly restores the Zero flag.
.,AB47 20 0C E1 JSR $E10C output character to channel with error check
.,AB4A 29 FF AND #$FF set the flags on A
.,AB4C 60 RTS
lis replaced by
10 REM CONCAT