14

I'm looking over the table of tokens for Sinclair BASIC and I'm struck by two things. One is that it's not ASCII. Some parts are in the same order, but others are not, so it would seem conversion is non-trivial.

The real question though, is why does it have all the letters, digits and punctuation in the table twice? For instance, A is in 38 as well as 166. What is the purpose of this?

6
  • 10
    Maybe the high bit was used in certain places as a flag? Same token, but flagged or not. (note: 166-38=128)
    – davidbak
    Sep 12, 2023 at 17:53
  • 5
    Ah yes, that proved to be useful. Googling high bit sinclair got me the answer: its used to flag the last character in a variable name if you use the long-name format. Wow, that seems like QUITE a waste of valuable ROM real estate! Sep 12, 2023 at 18:44
  • 4
    What do you mean by "quite a waste"? The tokens are not duplicated, the letters are not tokens, and the letter's code takes 7 bit, so the most significant bit is normally an indicator of the last letter in the string. They're not stored twice in the rom.
    – Vlad
    Sep 12, 2023 at 19:19
  • 6
    You might want to specify "ZX-81" BASIC. The ZX-Spectrum's BASIC is still "Sinclair BASIC" (and the QL's somewhat as well), but both use close-to ASCII character encoding.
    – tofro
    Sep 12, 2023 at 21:45
  • 2
    @MauryMarkowitz The high bit handling is only for text constants in ROM and BASIC marking multi letter variable names. In BASIC code and string data the second highest bit (2^6) selects between printable characters and tokens while the high bit (2^7) marks normal or invers for the 64 printable characters (00xxxxxx = Black on White, 10xxxxxx is White on Black, x1xxxxxx are tokens or unused). Codes in the 7x range are neither characters nor tokens but keyboard codes used for control function. They are treated as undefined tokens (x1xxxxxx) within strings or any output.
    – Raffzahn
    Sep 13, 2023 at 1:56

4 Answers 4

20

You're right; the ZX81 does not use ASCII. That's because it uses the CPU to generate the displayed picture, from a compressed format in memory. The way this works is an absolute freakshow*, but from what I understand it involves the CPU actually executing the on-screen text. The code-point is fetched from video memory as the CPU runs a program (the program counter points somewhere in the display file), and the codepoint is looked up in ROM, so that the bitmap is loaded to a shift register. But what the CPU actually sees is one NOP instruction for each character on that row. But when it reaches the end of the line, the CPU sees a HALT instruction. So the NEWLINE character by necessity has the same codepoint as the HALT instruction, which is why the ZX81 doesn't use ASCII.

Conversion to/from ASCII is finicky, but not hard to do: If you have the space, do a lookup table; otherwise do some range-checking and adding/subtracting and things. But why bother with that? Just use the native encodings.

But what you're seeing is not a duplicate token. It's just some more clever bit fiddling that means some characters have more than one codepoint, because the uppermost bit is ignored for some of them in some contexts. This is used for terminating constant strings in the ROM, and I believe also the filenames on tape. (It has other uses as well: the bit can also be used to show inverse text or graphics as other answers have already said)

Terminating strings in this way is more space efficient than using a sentinel value at the end of the string (C takes this approach), or storing the length of the string or other metadata along with its contents (like UCSD Pascal, or Python).

Take for an example the letter A. Its codepoint is 0x26. All printable single characters fit in seven bits, so to store a string like PRINT is five bytes for the string itself, since you can fit the string terminator in the unused bit of the last character, T, with the added benefit that the Z80 can very cheaply detect the end-of-string condition with its sign flag. (The PRINT token in the BASIC program is of course stored as a single byte, but to list the program, the string is copied from the ROM).


* I do in no way mean to detract from the brilliance of the design by calling it a freakshow

11
  • 6
    Just like WordStar. Sep 12, 2023 at 20:36
  • 1
    @manassehkatz-Moving2Codidact, I don't know what that means, but you could be right Sep 12, 2023 at 20:36
  • 9
    In the versions of WordStar prior to the 5.0 release, the 8th bit or "high" bit of ASCII characters, which are usually reserved to extend the character set, were instead used to store print and formatting information. loc.gov/preservation/digital/formats/fdd/… Sep 12, 2023 at 21:11
  • Nitpick that BASIC keywords (including LPRINT) are stored as single-byte tokens in ZX-80 BASIC. But the general explanation is still valid on the advantages of using the 7th-bit like that, especially on extremely limited memory environments (the ZX-81 shipped with 1024 bytes of RAM) Sep 12, 2023 at 23:51
  • 1
    @Rich Oh for sure, it's brilliant. Uses the program counter, and the Interrupt Vector Base register, of all things to generate the display addresses. And then encoding the NEWLINE as a HALT instruction. I do in no way mean to detract from the brilliance of the design by calling it a freakshow, certainly not. Sep 15, 2023 at 8:05
7

TL;DR:

That encoding is not just the token table but as well the BASIC charset and the screen charset which offers each of the 64 printable characters as Black-on-White (00h..3Fh) or White on Black (80..BFh). Much like the Commodore PET or the Apple II does.


Details

One is that it's not ASCII.

Who needs ASCII anyway?

Neither Commodore nor Apple did go all out ASCII with their encoding.

Some parts are in the same order,

Which are for things that commonly need to be ordered by increasing representation, like

  • Numbers (0..9),
  • Letters (A..Z) and
  • Graphic symbols (bit wise encoding to allow SET/RESET operations)

Numbers and Punctuations are both lower in value than Letters, much like with ASCII. In fact, the ZX81 yields an advantage over ASCII by having all 4 types in distinct and closed groups, not mixing Punctuation at random places around Numbers and Letters

so it would seem conversion is non-trivial.

Since the machine doesn't operate in ASCII no need for conversion exists:

  • CHR$ 38 yields A and
  • CODE "A" gives 38.

The real question though, is why does it have all the letters, digits and punctuation in the table twice?

Its the same reason why the Apple II or the PET has them as well each twice (*1): To display each in two versions: Normal or Invers

For the ZX81 token list and character set and display character set is put into the same code space.

The character set consists of only 32 character existing in two versions:

  • 000..063 Black on White
  • 128..191 White on Black

This is way better to be seen when looking at an ordered image like this

enter image description here

(Image taken from Wikipedia)

Or more orderly shown in a nice table at the ZX81 Character Set Wiki page.

What is the purpose of this?

Simplifying the whole character and screen handling.

As already mentioned, Apple II and Commodore PET feature similar duplication within their screen character set. Those machines must parse every output according to a ASCII to Screen Code translation while observing modal settings. In case of the Apple two the screen character subset is selected according to a general setting made by the BASIC commands NORMAL, INVERS or BLINK, while on the PET the string to be printed has additionally to be parse for the control codes of RVS and RVS OFF, each changing the character translation for any subsequent character PRINTed.

The ZX81 does not have to flip encoding according to some mode, nor does it need to provide additional BASIC commands, like the Apple, or parse output, like the Commodore, to handle modal setting. A character within the Display File (*2) will simply be output the way it is (*3)


One More Thing:

Looking at above Wiki table might leave one wondering what the codes in row 7x are, as the 'printout' image shows them as question marks.

They are, with partitial exception of New Line (76h), and regarding string output exactly what their top bits suggest: unused tokens. These codes are only produced and relevant due keyboard input. They control the editor but may as well be inputted (read) by a BASIC program using INKEY$.

Within the Display File New Line is used to mark the end of a line, filling the actual screen line after it with nothingness.


*1 - Well the Apple II even 4 times, including blinking and a redundant white on black copy.

*2 - Display File or D_FILE is what Sinclair called the structure holding the picture to be shown. In many way similar to a screen buffer, in others, as seen quite different.

*3 - And the (PRINT) routine inserting a character into the display file does only have to check against a a fixed bit mask to differentiate between characters and tokens. Anything with x0xx.xxxx ist a character to be copied, while all other are potential tokens and decoded into their representational strings (Like 0DEh into THEN) accordingly (*4), or if not found into a question mark.

*4 - Which in turn simplifies the LIST command a lot, as most tokens only need to be PRINTed without any further evaluation.

6
  • Some of the control characters are different in PETSCII, but with the exception of "£" versus "/", the printable seven-bit characters in PETSCII are identical to those of ASCII-1963.
    – Mark
    Sep 18, 2023 at 22:59
  • @Mark Sorry, but that's not true. PETSCII is even more screwed up than Sinclair - as mentioned above. For one, Screen codes are like with the ZX in different order than ASCII, so translation is needed. But BASIC codes are as well mixed up. While one charset replaces all lower case ASCII by graphics, the other does provide upper and lower case, but switches their places. Plus of course a few more punctuation in both charsets, like{|}~`.
    – Raffzahn
    Sep 19, 2023 at 1:57
  • ASCII-1963 doesn't have lower-case letters.
    – Mark
    Sep 19, 2023 at 2:36
  • @Mark Picking a very specific, under development version doesn't help making a point, especially when encoding still doesn't match.
    – Raffzahn
    Sep 19, 2023 at 8:40
  • ASCII-1963 wasn't an "under development version", it was a published standard, ASA X3.4-1963. The fact that it was superseded four years later by USAS X3.4-1967 doesn't make it a development version any more than the 1996 release of Unicode 2.0 made the 1991 release of Unicode a "development version".
    – Mark
    Sep 19, 2023 at 21:34
4

What the linked table does not show is that the letters with the high bit set are displayed in white on black rather than black on white. The equivalent table at Wikipedia is clearer in this respect.

If a program contains a SAVE instruction with a literal string such as

SAVE "CHECKERS"

When the instruction is executed, the high bit is set on the last character of the filename and so it then appears in the listing in reverse video.

2

why does it have all the letters, digits and punctuation in the table twice? For instance, A is in 38 as well as 166. What is the purpose of this?

In ZX Spectrum 0x90-0xA4 characters are actually UDG (User-Defined Graphics). By default they have the same apperance as 0x20–0x7F, but they are different symbols. Programmers can define these symbols to use as custom graphic tokens or national symbols.

1
  • 2
    What you are saying is true for the ZX Spectrum, but other answers (including mine) are about the ZX 81. That would point to the possibility that the question could be clearer which machine it's actually asking about Sep 15, 2023 at 9:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .