ZX Spectrum tokenisation

Question

The BASIC used in the early ZX Spectrum (and also its ZX predecessors) had this weird thing where every single BASIC token was printed onto the keyboard, and for example PRINT or LOAD were entered by a single keypress. In particular the ZX Spectrum had so many tokens on each key, meaning a great deal of shifting and entering various keyboard modes to enter some of the less common tokens such as ATAN or READ.

I am interested to see how and why this was easier to implement than putting a simple tokeniser and string comparer in the ROM like everyone else!

Answer 1

Contrary to other answers, obliging the user to enter BASIC tokens directly doesn't really save meaningful amounts of RAM. Many of its contemporaries such as the BBC Micro had BASICs where you typed keywords in full which were then immediately tokenised when you pressed enter. If anything, tokenised Sinclair BASIC generally had longer byte sequences than other BASICs due to parsing number constants and inlining the floating-point representation for performance. The latter is the source of bizarre memory-saving hacks on the ZX81 where e.g. PI-PI (3 bytes: PI, - and PI) would be used in preference to 0 (7 bytes: 0, 126 to indicate a floating-point constant, then the 40-bit constant).

Instead, it serves to save on ROM space. Remember that ZX BASIC started on the ZX80 which had just 4kiB of ROM for everything: Z80 control, I/O handling, character generation, and BASIC. Eliminating a tokeniser saved perhaps 500 bytes or so, which makes a real difference in such a small system. With the ZX Spectrum and its 16kiB ROM they would have had the option of supporting entering keywords in full, but chose not to. Partly perhaps because those 500 bytes could still do something more useful elsewhere, but most likely because the unusual input method was now a defining characteristic of ZX BASIC.

Another important reason would be marketing-related. Uncle Clive was very much a showman, and what better way for a machine to advertise what it was capable of than to label the keys with a treasure map of keywords begging to be explored?

Answer 2

A purely speculative answer: it's a user experience improvement.

The ZX80 is unable both to process keyboard input and to maintain a stable display. It has the Z80 itself step through display bytes and programmatically generate vertical syncs, which requires it to be in a loop that is tightly synchronised to the video signal. It's able to do a fixed-length test for whether a key is pressed within that loop, but as soon as it needs to branch out and process a key press, it enters variable-length code, and loses synchronisation. The user observes this as a little screen roll.

The keyboard itself is also pretty terrible: a flat membrane, no tactile feedback. Like a smartphone except that your eyes aren't anywhere near it.

Therefore minimising keystrokes means: less typing, less screen rolling.

The ZX81 has the same kind of keyboard and still rolls in fast mode, and the ZX81 ROM is backwards compatible, being installable in a ZX80 (though slow mode, which produces a steady picture even when typing, won't be available because it relies on an extra interrupt generator). The ZX Spectrum's ROM is a direct descendant of the previous two and, anyway, single-press entry was by then associated with the brand.

EDIT: it's from an emulator I wrote rather than a real machine, but it's getting this result by really emulating a CRT, not because I think it's cool, so I maintain that this is an accurate representation of the real ZX80 typing experience:

Answer 3

I wasn't necessarily easier - it was down to money.

Remember these were the days when the single most expensive component of a computer was its memory. For many micros of the time, a simple memory expansion kit would cost more than the base model machine itself. Sinclair was very clever at squeezing a working system into the smallest possible memory footprint.

Adding a tokeniser to read multiple keystrokes would have required extra ROM space for the software and extra RAM space to hold the keystrokes while a command was being built up. By using this form of key entry, the ZX80 managed to ship as a working machine with 1kb of RAM. The ZX81 and Spectrum followed on from this, deliberately undercutting the competition on price by minimising the use of expensive memory chips.

By the time the Sinclair QL came along, memory had reduced in price and so the drive to be so excessively frugal had reduced. Sinclair continued to undercut the competition by saving in other ways - for example Phloopy tapes instead of floppy disks.

Answer 4

I am in doubt it was due to a special technical reason. Finally I can not judge really but I am convinced there are a couple of advantages of this design.

My first computer/programming experience was with ZX Spectrum+. In contrast to its predecessors the keyboard of this model was outstanding till today. It was very well designed and very responsive.

For me to have all the available commands printed on the keyboard helped very much to make first steps in programming at all. Further you have been forced to enter syntax correct code. The editor by default only accepted correct commands syntactically. The system was easy to learn and not very complicated at all. I liked it very much.

Answer 5

The 128K model (and its Amstrad made successors) did use a tokenizer in 128 Basic mode.

In this mode, you had to type in the command lines in full text, then syntax checking was made upon line validation rather than on the fly as with 48 Basic, and if correct, then the line was tokenized, then either stored (if preceded by a line number) or executed.

So there actually WAS a tokenizer in Sinclair Basic. In 128K mode, involving an additional ROM bank.

Anyway, it meant an additonal step had to be added to analyze the syntax and translate keywords into tokens.

The token part itslef was necessary anyway, for several reasons : 1) shorten up the code ; 2) accelerate programs execution, as the necessary translation part was done at typing time, not execution time ; 3) since a lookup table is always needed to associate each machine code subroutine to each keyword, matching them at typing time once and for all is another way to accelerate programs execution.

So, yes, tokenization is always a good idea. BTW, whan you wanted to add new keywords, you just had to trigger an error that led to running your own code, using the existing "hook codes" mechanism originally intended for the Interface 1 additional keywords (that were printed on the keyboard, but not actually present in ROM).

There were two common ways to do so : either alter some of the existing commands syntax, or adding new parameters or new peripheral types (such as MGT disc interfaces), or leading new keywords by a character that could be typed in "K" cursor mode, for instance any symbol-shifted one (an "!" was usually used a header character).

The point was to trigger a syntax checking error, trap it and redirect the command to your own code, so it would have the system to accept the line when typing it, and run your own subroutine at runtime.

Of course new commands this style weren't tokenized, so they occupied more space and had to be reinterpreted at runtime, thus being slower than built-in ones (though since this was all done in machine code, I doubt you could notice anything).

Answer 6

My bet is because of small amount of RAM in BASIC. You need to take in mind there is the framebuffer, heap/variables and stack. If the code where represented as ASCII text it would be considerably longer. I often hit the memory limit in BASIC while writing small games in my early days of programming. For example this (first BASIC program I found in my archive):

 7901 BORDER 7: POKE 23693,56: CLS 
 7902 LET f=1: LET n= PEEK 23635+256* PEEK 23636: LET vars= PEEK 23627+256* PEEK 23628
 7903 LET k=16383: INPUT CHR$ 8;" LINE start: ";od;" koniec: ";do; AT 0,0;"Hlada sa: ";a$: CLS 
 7904 LET g=0: IF od <> ABS od OR do <> ABS do THEN LET g=1: LET od= ABS od: LET do= ABS do
 7905 IF n >=vars THEN STOP 
 7906 GOSUB 7917: LET n=n+1: GOSUB 7918: IF l<od THEN LET n=n+b+1: GOTO 7905
 7907 IF l>7900 AND l<7930 THEN LET n=n+b+1: GOTO 7907
 7908 LET t=n: LET n=n+1: IF l>do THEN STOP : STOP : STOP 
 7909 LET m= PEEK n
 7910 IF m= CODE a$ THEN GOSUB 7919
 7911 IF m= CODE """" THEN LET f=1-f
 7912 IF f AND (m= CODE ":" OR m= CODE " THEN ") THEN LET s=s+1: LET t=n
 7913 IF m=14 THEN LET n=n+5
 7914 LET n=n+1
 7915 IF m=13 THEN GOTO 7905
 7916 GOTO 7909
 7917 LET s=1: LET l= PEEK (n+1)+256* PEEK n: LET n=n+1: RETURN 
 7918 LET b= PEEK n+256* PEEK (n+1): LET n=n+1: RETURN 
 7919 PRINT PAPER 5;l;",";s;: PRINT TAB 8;: LET f=1: LET n=t
 7920 LET n=n+1: LET m= PEEK n
 7921 IF m= CODE """" THEN LET f=1-f

is 1058 bytes as ASCII and 867 bytes as a BASIC tokenized program file (including header). If I do the same for simple BASIC game like Zednik ma zizen (not mine but very popular back in the days) then:

ASCII:   45619 Byte
.P file: 40995 Byte

The 5KB difference might not look like much but at the time it was. This game stretches the limits as there is only 42240 bytes available, minus the BASIC variables, and as ASCII text would not fit just the code alone ...

When the memory size changed (like on ZX128) they added text representation too ... (that is why D40/D80 FDC+FDD for ZX48K does not work in such environment with BASIC as it use tokens too...)

Another reason is speed. As ZX has BASIC interpreter processing text is much slower than processing tokens. For compilers it does not matter much as after binary is created no further processing occur during runtime. But interpreters must process each line of code when it executes. So loops and GOTO/GOSUB invokes processing usually already processed lines again and again ...

Yes it is possible to use text editor and token representation together but you must realize that there where not much tools at that time and code that and properly debug it would probably take much longer than they wanted to spend on it. Also conversion back and forward between both representations take time and coding could get sluggish ...