Retrocomputing Stack Exchange is a question and answer site for vintage-computer hobbyists interested in restoring, preserving, and using the classic computer and gaming systems of yesteryear. It only takes a minute to sign up.
Sign up to join this community
I'm looking over the table of tokens for Sinclair BASIC and I'm struck by two things. One is that it's not ASCII. Some parts are in the same order, but others are not, so it would seem conversion is non-trivial.
The real question though, is why does it have all the letters, digits and punctuation in the table twice? For instance, A is in 38 as well as 166. What is the purpose of this?
You're right; the ZX81 does not use ASCII. That's because it uses the CPU to generate the displayed picture, from a compressed format in memory. The way this works is an absolute freakshow*, but from what I understand it involves the CPU actually executing the on-screen text. The code-point is fetched from video memory as the CPU runs a program (the program counter points somewhere in the display file), and the codepoint is looked up in ROM, so that the bitmap is loaded to a shift register. But what the CPU actually sees is one NOP instruction for each character on that row. But when it reaches the end of the line, the CPU sees a HALT instruction. So the NEWLINE character by necessity has the same codepoint as the HALT instruction, which is why the ZX81 doesn't use ASCII.
Conversion to/from ASCII is finicky, but not hard to do: If you have the space, do a lookup table; otherwise do some range-checking and adding/subtracting and things. But why bother with that? Just use the native encodings.
But what you're seeing is not a duplicate token. It's just some more clever bit fiddling that means some characters have more than one codepoint, because the uppermost bit is ignored for some of them in some contexts. This is used for terminating constant strings in the ROM, and I believe also the filenames on tape. (It has other uses as well: the bit can also be used to show inverse text or graphics as other answers have already said)
Terminating strings in this way is more space efficient than using a sentinel value at the end of the string (C takes this approach), or storing the length of the string or other metadata along with its contents (like UCSD Pascal, or Python).
Take for an example the letter A. Its codepoint is 0x26. All printable single characters fit in seven bits, so to store a string like PRINT is five bytes for the string itself, since you can fit the string terminator in the unused bit of the last character, T, with the added benefit that the Z80 can very cheaply detect the end-of-string condition with its sign flag. (The PRINT token in the BASIC program is of course stored as a single byte, but to list the program, the string is copied from the ROM).
* I do in no way mean to detract from the brilliance of the design by calling it a freakshow
That encoding is not just the token table but as well the BASIC charset and the screen charset which offers each of the 64 printable characters as Black-on-White (00h..3Fh) or White on Black (80..BFh). Much like the Commodore PET or the Apple II does.
One is that it's not ASCII.
Who needs ASCII anyway?
Neither Commodore nor Apple did go all out ASCII with their encoding.
Some parts are in the same order,
Which are for things that commonly need to be ordered by increasing representation, like
Numbers and Punctuations are both lower in value than Letters, much like with ASCII. In fact, the ZX81 yields an advantage over ASCII by having all 4 types in distinct and closed groups, not mixing Punctuation at random places around Numbers and Letters
so it would seem conversion is non-trivial.
Since the machine doesn't operate in ASCII no need for conversion exists:
CHR$ 38 yields A andCODE "A" gives 38.The real question though, is why does it have all the letters, digits and punctuation in the table twice?
Its the same reason why the Apple II or the PET has them as well each twice (*1): To display each in two versions: Normal or Invers
For the ZX81 token list and character set and display character set is put into the same code space.
The character set consists of only 32 character existing in two versions:
This is way better to be seen when looking at an ordered image like this
(Image taken from Wikipedia)
Or more orderly shown in a nice table at the ZX81 Character Set Wiki page.
What is the purpose of this?
Simplifying the whole character and screen handling.
As already mentioned, Apple II and Commodore PET feature similar duplication within their screen character set. Those machines must parse every output according to a ASCII to Screen Code translation while observing modal settings. In case of the Apple two the screen character subset is selected according to a general setting made by the BASIC commands NORMAL, INVERS or BLINK, while on the PET the string to be printed has additionally to be parse for the control codes of RVS and RVS OFF, each changing the character translation for any subsequent character PRINTed.
The ZX81 does not have to flip encoding according to some mode, nor does it need to provide additional BASIC commands, like the Apple, or parse output, like the Commodore, to handle modal setting. A character within the Display File (*2) will simply be output the way it is (*3)
Looking at above Wiki table might leave one wondering what the codes in row 7x are, as the 'printout' image shows them as question marks.
They are, with partitial exception of New Line (76h), and regarding string output exactly what their top bits suggest: unused tokens. These codes are only produced and relevant due keyboard input. They control the editor but may as well be inputted (read) by a BASIC program using INKEY$.
Within the Display File New Line is used to mark the end of a line, filling the actual screen line after it with nothingness.
*1 - Well the Apple II even 4 times, including blinking and a redundant white on black copy.
*2 - Display File or D_FILE is what Sinclair called the structure holding the picture to be shown. In many way similar to a screen buffer, in others, as seen quite different.
*3 - And the (PRINT) routine inserting a character into the display file does only have to check against a a fixed bit mask to differentiate between characters and tokens. Anything with x0xx.xxxx ist a character to be copied, while all other are potential tokens and decoded into their representational strings (Like 0DEh into THEN) accordingly (*4), or if not found into a question mark.
*4 - Which in turn simplifies the LIST command a lot, as most tokens only need to be PRINTed without any further evaluation.
{|}~`.
What the linked table does not show is that the letters with the high bit set are displayed in white on black rather than black on white. The equivalent table at Wikipedia is clearer in this respect.
If a program contains a SAVE instruction with a literal string such as
SAVE "CHECKERS"
When the instruction is executed, the high bit is set on the last character of the filename and so it then appears in the listing in reverse video.
why does it have all the letters, digits and punctuation in the table twice? For instance, A is in 38 as well as 166. What is the purpose of this?
In ZX Spectrum 0x90-0xA4 characters are actually UDG (User-Defined Graphics). By default they have the same apperance as 0x20–0x7F, but they are different symbols. Programmers can define these symbols to use as custom graphic tokens or national symbols.
