Odd tokenizing in Integer BASIC

Question

While researching the underpinnings of Integer BASIC (IB), I came across an oddity I'm hoping someone can explain.

Unlike MS-derived systems, IB converted numeric constants to their 16-bit representations at edit-time rather than run-time. To indicate that the following data was a constant, it placed a single-byte token in front of the two-byte value. Nothing interesting so far, Atari BASIC did the same thing.

So here's the mystery: IB did not use a single token to indicate that a constant followed, but ten different ones. The token to use was selected by examining the first digit of the ASCII representation, so for instance, the number "10" starts with "1" and is prefixed with $B1, whereas "55" starts with "5" and is prefixed with $B5.

Can anyone offer an explanation for this oddity? I cannot find one on the 'net, and Woz never mentions it anywhere. It does not appear it saves times or memory in either the encode or decode, but maybe I'm missing something?

Answer 1

It does not appear it saves times or memory in either the encode or decode, but maybe I'm missing something?

Well, it kind of simplify decoding. The values $B0..$B9 are exactly what's uses to mark up numbers in literals and variable names. And here they as well simplify the interpreter.

In Integer BASIC all tokens are encoded with high bit off, while everything else, i.e. variables and literals have the high bit set - which is exactly like characters got stored in Apple II video RAM. In this content $B0..$B9 is simply the representation of ASCII $30..$39, the numeric characters.

So strictly, $B0..$B9 aren't tokens but ASCII as Integer BASIC sees it (with high bit set).

When the interpreter walks thru a program line, he will see the high bit set and collect the remaining characters to handle them as variable name, string literal or numeric literal. Except in case of a numeric one ($Bx) it saves the decoding and simply accepts the next two bytes as its preencoded value.

Due the use of high-bit-set ASCII, there is no saving in unifying to a single indicator, or using all 10 values equally, as they can't be used otherwise (variable names have to start with a letter). So why bother. Simply taking the first numeral literal saves at least an LDA :))

It as well keeps testing for a numeric literal the same as without optimized storage: >= '0' AND <='9' or >= $B0 AND <=$B9. Eventually more important, due the encoding in 'groups', the three basic items of a program line, tokens, variables and constants, can now be distinguished quite easy - maybe by using a single test instruction (*1).

---

I would speculate that he originally had numerical literals as well stored as their high-bit-set ASCII value and only added the optimisation later on. But that's just my own guess.

---

*1 - I wouldn't be surprised if distinction between tokens, variable names and constants would comes down to a simple three machine instruction sequence

• BIT the next byte in instruction stream
• BPL (high bit cleared) -> Token
• BVC (greater than $BF) -> Variable name
• otherwise numerical constant

(Just a wild but obvious guess, I haven't looked into disassembly, that would quite fit Woz' knack for bit fiddling - just compare this with the way the Apple II charset is formed by arranging a 64 character ROM into a sophisticated 256 code pattern to provide a main charset and two subsets)