The 128K model (and its Amstrad made successors) did use a tokenizer in 128 Basic mode.
In this mode, you had to type in the command lines in full text, then syntax checking was made upon line validation rather than on the fly as with 48 Basic, and if correct, then the line was tokenized, then either stored (if preceded by a line number) or executed.
So there actually WAS a tokenizer in Sinclair Basic. In 128K mode, involving an additional ROM bank.
Anyway, it meant an additonal step had to be added to analyze the syntax and translate keywords into tokens.
The token part itslef was necessary anyway, for several reasons : 1) shorten up the code ; 2) accelerate programs execution, as the necessary translation part was done at typing time, not execution time ; 3) since a lookup table is always needed to associate each machine code subroutine to each keyword, matching them at typing time once and for all is another way to accelerate programs execution.
So, yes, tokenization is always a good idea. BTW, whan you wanted to add new keywords, you just had to trigger an error that led to running your own code, using the existing "hook codes" mechanism originally intended for the Interface 1 additional keywords (that were printed on the keyboard, but not actually present in ROM).
There were two common ways to do so : either alter some of the existing commands syntax, or adding new parameters or new peripheral types (such as MGT disc interfaces), or leading new keywords by a character that could be typed in "K" cursor mode, for instance any symbol-shifted one (an "!" was usually used a header character).
The point was to trigger a syntax checking error, trap it and redirect the command to your own code, so it would have the system to accept the line when typing it, and run your own subroutine at runtime.
Of course new commands this style weren't tokenized, so they occupied more space and had to be reinterpreted at runtime, thus being slower than built-in ones (though since this was all done in machine code, I doubt you could notice anything).