Looking for a BASIC decruncher: inserting spaces between keywords

Question

Many dialects of 8-bit BASIC allow ‘crunched’ source; that is, most spaces can be ignored between keywords:

10 FORX=1TO10
20 PRINTX,X^2
30 NEXTX

After Microsoft BASIC-80 Rev. 5.x, however, spaces were required:

10 FOR X=1 TO 10
20 PRINT X,X^2
30 NEXT X

Other later BASIC interpreters (such as Locomotive BASIC on the Amstrad CPC) required spaces, while others didn't (for example BBC BASIC).

I'm looking for a program (Linux-capable preferred) that will reliably insert the necessary spaces into BASIC text. I'm not looking for the ability to detokenize: assume I have the text available.

Answer 1

Here's something that should do the trick.

It's written in AWK. Any language I picked will be the wrong language. awk (or gawk) is everywhere.

Copy and paste it into a file (say, expand.awk).

awk -f expand.awk crunched.bas > uncrunched.bas

The script:

# License: Creative Commons 0 https://creativecommons.org/share-your-work/public-domain/cc0/
# Assumes all uppercase, and inability to embed quotes in quoted strings.
# Basic philosophy is to break out the quoted strings, and then process
# the individual statements separated by colons. Then put it all back
# together again.
BEGIN {
    # list of keywords, EXCEPT for 'TO', it's handled special
    keywords="OPEN:CLOSE:CLEAR:INPUT:IF:THEN:ELSE:FOR:NEXT:GOTO:GOSUB:LINE:READ:DEF:DIM:PRINT";
    split(keywords, keylist, ":");
}

{
    # split out the quoted strings. They will be even fields when done.
    split($0, a, "\"");
    for(i = 1; i <= length(a); i++) {
        if (i % 2 == 0) {
            # quoted string, skip it
            continue;
        }
        # expand this chunk of the line
        a[i] = expand_line(a[i]);
    }

    # rebuild the line
    line = "";
    for(i = 1; i <= length(a); i++) {
        p = a[i]
        if (i % 2 == 0) {
            # quoted string, add the quotes back
            p = "\"" p "\"";
        }
        line = line p;
    }
    # clean up PRINT" and INPUT"
    # if these happen to be in quoted strings, kind of out of luck.
    gsub("PRINT\"", "PRINT \"", line);
    gsub("INPUT\"", "INPUT \"", line);

    # out it goes
    print line;
}

function expand_line(line, i, k, l) {
    # break up in to statements
    split(line, statements, ":");
    for(i = 1; i <= length(statements); i++) {
        # expand the keywords for each statement
        for(k = 1; k <= length(keylist); k++) {
            statements[i] = expand(statements[i], keylist[k]);
        }
        # because of GOTO, TO (of FOR TO NEXT) is handled special
        statements[i] = expand_to(statements[i]);
    }
    # rejoin with the :
    line = statements[1];
    l = length(statements);
    if (l == 1) {
        return line;
    }
    for(i = 2; i <= l; i++) {
        line = line ":" statements[i];
    }
    return line;
}


function expand(line, keyword) {
    # for each key word, check for space at each end
    rx=keyword "[^ ]";
    if (match(line, rx) > 0) {
        gsub(keyword, keyword " ", line);
    }
    rx="[^ ]" keyword;
    if (match(line, rx) > 0) {
        gsub(keyword, " " keyword, line);
    }

    return line;
}

function expand_to(line) {
    # hide any GOTOs
    gsub("GOTO", "xxx", line);

    # fix the TO
    if(match(line, "TO[^ ]") > 0) {
        gsub("TO", "TO ", line);
    }
    if(match(line, "[^ ]TO") > 0) {
        gsub("TO", " TO", line);
    }

    # put the GOTOs back
    gsub("xxx", "GOTO", line);
    return line;
}

Answer 2

If you're happy with a .net core solution you could take my MS Basic clone, load the code and then save it again.

https://github.com/peteri/ClassicBasic

You would need to remove the EditLine function in the GlassTeletype which uses Win32 interop to provide an line editor (invoked by edit nnnn where nnnn is the linenumber) and maybe give it a little tweak to get rid of the BOM when it saves.

Answer 3

Applesoft BASIC appears to not care about whitespace, but it gets a little weird with AT/ATN/TO. For example, you can write A TO X AT O, but if you crunch the whitespace out it will instead be parsed as A TO XA TO. For everything else, whitespace is ignored. ON ERR GO TO becomes ONERR GOTO. The trick when scanning is to find the longest match (ON is also a valid keyword).

I don't know how various dialects of BASIC compare in this respect. Just be aware that not everything that appears to ignore whitespace actually does.

I've written code that converts a text file with an Applesoft BASIC program to tokenized form, as well as code that converts the tokenized program to text. They're built into CiderPress. You can find the C++ source for the parser here, and the pretty-printer here. If your sources were in Applesoft you could launder them quite easily. For anything else some coding will be required.

(There's a comment in the parsing code with a chunk of the 6502 code from Applesoft, so you can see how it actually parses AT/ATN/TO.)