Why did BASIC programs tend to READ a redundant copy of DATA?

Question

Take for example this BASIC version of ELIZA which starts out (in lines 50–170) by a number of READ loops which copy DATA (lines 1340 and following) into a handful of arrays.

Isn't this rather wasteful on an 8-bit computer with perhaps anywhere from 4KB to 48KB of RAM?

That is, first the program itself, including all its lines of DATA statements must be loaded into RAM. Then when the program runs it proceeds to DIM additional RAM and READ it into a separate "working copy".

Now I suppose if that "working copy" needed to be modified, this data duplication would be useful to get a fresh copy that reset to its initial values on each RUN. But what inspired implementers of the various BASICs to build in a DATA/READ feature instead of some RAM-friendlier mechanism like a syntax for defining read-only arrays?

Alternatively, why wasn't it particularly common for programmers to assign array items directly — as this Apple 1 port of ELIZA (archive link) does — to leave more RAM free?

Answer 1

DATA is a kind of weird holdover from FORTRAN and batch-processing. It is not memory efficient, but it was part of the programming vocabulary of the day.

In FORTRAN, inline data can look like this:

  DIMENSION MARKS(10)
  DATA MARKS/31,32,33,34,35,36,37,38,39,40/

This fills the ten integer values of the MARKS array with [31,32,33,34,35,36,37,38,39,40] respectively. Since FORTRAN is compiled, there isn't any storage overhead. This is aside from run-time data parsing, where data cards were read by the FORTRAN program as it ran.

Dartmouth BASIC may have had less of an overhead using DATA statements, as it was dynamically compiled on the client-facing machine, and submitted to the actual processor as if it were a series of batch jobs. Most BASIC interpreters since the 8-bit days don't use this model, so DATA just sits about taking up memory.

You can eliminate redundant data storage by saving it to a separate (binary?) data file, but that requires file I/O and storage hardware. This isn't always evenly defined for systems running BASIC.

I know of only one interpreter that allows for inline array declaration, but there are likely others. BBC BASIC (since V5, so not on the 8-bit versions) allows you to do this:

10 DIM A(10)
20 A()=31,32,33,34,35,36,37,38,39,40
30 PRINT A(2)

This still has some redundant storage, but doesn't need code inside the program to interpret the data. It should be noted that — unusually for BASIC — this method fills the array from A(0) to A(9) rather than from A(1) to A(10). So A(2) here is equal to 33.

Answer 2

D'oh! I can provide at least a partial answer to my own question:

Alternatively, why wasn't it particularly common for programmers to assign array items directly — as this Apple 1 port of ELIZA does — to leave more RAM free?

What I was imagining here was that instead of something like:

 5 DIM A$(4)
10 FOR I = 1 TO 4 : READ A$(I) : NEXT I
20 DATA "ONE", "TWO"
30 DATA "THREE", "FOUR"

the programmer might "optimize" as:

10 DIM A$(4)
20 A$(1) = "ONE" : A$(2) = "TWO"
30 A$(3) = "THREE" : A$(4) = "FOUR"

so as to avoid having an "extra copy" from all the DATA statements. But of course this still has two copies of the data: one in the source code and one in A$().

So apart from some sort of language mechanism for referencing constant data, the READ/DATA combination is a bit less tedious than individually assigning each array index, and probably uses less RAM for storing the source code.

(The Apple 1 example accomplishes two distinct things: for one, the original Integer BASIC didn't include DATA/READ so the author had to go without regardless. And for two, it is somewhat optimized in that rather than DIMensioning full lists for all, some variables like R$ and K$ get assigned to a single value at a time and a computed GOSUB is used to "index" into the available values instead.)

Answer 3

READ and DATA were features of Dartmouth BASIC in 1964. Thus, anything that wanted to call itself BASIC really had to implement them.

The computing model of the day was "read some input; do some computing; write some output".

Normally, in other languages on other contemporary systems, you'd be using card or paper tape. The data were on cards or tape, probably immediately following the source program.

(An online multiaccess system that I used followed that model for Algol 60 programs, even though - luxury! - we had disc files for storage, rather than paper or cardboard. But the data could still be present in the same file as the source code, after the final end),

BASIC DATA follows that same structure, except that organization by line number means DATA can occur anywhere. But logically, it's "an input data file".

With the benefit of hindsight, I think DATA can achieve a useful separation between the program and the data that the program operates on, in a way that would not be possible if explicit assignment statements were used.

I'd say the ELIZA program followed that model so that the vocabulary could be tinkered with, without modifying code.

And, of course, consider the case where not all DATA are consumed at once.

10   READ A
20   IF A < 0 THEN 90
30   REM COMPUTE SOMETHING WITH A
40   ...
50   ...
80   GOTO 10
90   END

This is the difference between 'initialization' and actual data to be processed by this run of the program.

Writing a block of DATA with 100 values to be used consecutively is more convenient, and easier to update, than assigning 100 array elements.

Answer 4

TL;DR:

Simple answer: Because it's the way BASIC is defined.

Original BASIC had no way to access files at all. DATA lines were the only way to add predefined data to a program. The idea was essentially to have a stack of items - originally only numbers, strings where only added way later - that could be read like a stack of cards.

More Important: It is already close to an optimal approach within the limits set by BASIC.

---

In Detail:

Isn't this rather wasteful on an 8-bit computer with perhaps anywhere from 4KB to 48KB of RAM?

In what way wasteful? It's in fact less wasteful than any other way. Keep in mind that data that needs to be used need to be stored in source, so everything has to be present in source code anyway.

A Basic Issue About BASIC

It's important to remember that BASIC, at least for most micro computer implementations, is a source code interpreter (*1). The source code is not compiled into machine code, but is the 'code' to be executed. And this source code is to be preserved (*2).

The imperative of 100% retrieval of original source text does restrict abilities to reorganize.

DATA is Already the Most Compact Way of Storage.

Let's look at how data is represented/stored as DATA lines:

• line overhead,
• the DATA instruction (token),
• the data items themself and
• delimiters inbetween

Whatever form a more 'space savvy' may have, I guess we can agree that it needs at least to include the data items as written and delimiters. In addition such a form must of course be placed in a line, thus having line overhead. And finally such a line must start with some statement (token) marking it as non-executable.

In the end a DATA line is already the most compact representation possible within the limits of an (interpreted) BASIC's syntax. So storage in terms of source text can not optimized past that.

Can At Least Transformation (Loading) Be Optimized?

Of course also need to look at the loading routine, whose job is assigning data items to variables (single or array). In most cases that's a FOR/NEXT loop iterating over the data array. While being some code, it isn't a lot thereof:

10 FOR I=0 TO 20
20 READ A(I)
30 NEXT I

In MS-like BASICs that'S about 14 bytes plus line headers not really a big overhead space wise and executed only once. Any other method may need more. As it still must name the array to be filled plus optional offset to start and so on.

Using Direct Assignment

Alternatively, why wasn't it particularly common for programmers to assign array items directly — as this Apple 1 port of ELIZA does — to leave more RAM free?

The real reason why the Apple 1 Version uses this is rather obvious: There was no DATA statement.

Compared to DATA lines this will eat up considerable more space. Instead of having the raw source representation only delimited by a single comma plus one DATA statement per line, it now needs to be preceded by an additional variable name and an assignment operator. This adds up a lot. So that's at least one byte more per data item. For each additional item in the same DATA line saves two bytes. and that's only for single character numeric variable names. for each further character add one as well one if it's a string or an integer.

Bottom line: Already 5 single assignments of strings will eat up more memory than using DATA with a for loop to load an array.

Maybe a different assignment syntax (somewhat like FORTRAN) could help?

Using Multiple Assignments

Cool idea (and borrowed from BASICS parent FORTRAN). This could for example be done by introducing a multi assignment statement. For example like

LET A$(1)="A","B","C","D","E","F"

would load the letters A..F into A(1..6). Looks great, but needs more storage, as now each line also needs the target named:

LET A$(1)="A","B","C","D","E","F"
LET A$(7)="G","H","I","J","K","L"

after 3-4 lines this may eat up more source space than a corresponding FOR loop.

In addition this would remove the ability to structure data lines in a meaningful way, making them even harder to read. Just think a structure of corresponding items, like an address consisting of zip code, city, street and number. With classic data items this will look like

10 FOR I=1 TO 2
20 READ ZP(I), CY$(I), NR(I), ST$(I)
30 NEXT I
100 DATA  81739, "München", 34, "Waldheimplatz"
110 DATA  81541, "München", 6, "Schweiger Straße"

With multiple assignment it'll be more like

10 LET ZP(1)=81739, 81541
20 LET CY$(1)="München", "München"
30 LET NR(1)=34, 6
40 LET NR(1)="Waldheimplatz", "Schweiger Straße"

Which one is more pleasant to read? Not to mention that all these cryptic data lines now have to be at the begin of the program (or need calledas subroutine).

While it would be rather useful as well in situations outside of replacing DATA statements (*3), this would not only need more source code with lower readability (*4) but still have the internal representation (in data storage) in addition to source data.

So far the use of DATA statements seems the most space saving, but what about

A 'Special' Approach do DATA Data

One may of course think of a way to handle DATA statements like before, but on assignment only 'copy' a pointer to the data chunks within the DATA line. Could be a nice idea, but it comes with a hefty cost in code (*5). Now variable handling needs to distinguish between pointers into DATA statements and such into data storage and to handle copy on modification (*6)

Also, this would only make sense with strings, as data, including numeric is stored in DATA statements as ASCII (to be able to recreate source code). Converting them during each access from ASCII is less than great for performance, so they better get converted just once.

DATA is Already a (One Dimensional) Array

But what inspired implementers of the various BASICs to build in a DATA/READ feature instead of some RAM-friendlier mechanism like a syntax for defining read-only arrays?

DATA is already a read only array. It's a one dimensional array, starting with the first item, accessed sequentiallyvia consecutive READs. Thanks to RESTORE (*7) the read pointer can be reset any time to parse the data stack again. Not fast - in fact extreme slow - but the most space saving handling :)

---

Long Story Short

• There is no more efficient way than the existing within the context of BASIC as interpreter (and the restricted capacity of 8 bit systems).
• DATA statements are a way to handle a static input 'file' within the program.
• As soon as a system supports reading from data files (*8) that should be used (*9).

---

*1 - Funny considering the original BASIC being a compiler - which also explains why that duality didn't exist: The compiler could transform source DATA lines into a more appropriate intermediate format (especially as original BASIC had only numerical(FP) as its single data type).

*2 - Many BASICs employ methods to reduce storage need by using crunching methods like tokenization of keywords, some use (like Sinclair) use optimized constant storage to speed up execution (but increasing space), but all must handle the code in a way that the original text can be reconstructed down to the last dot and comma (e.g. when listed).

*3 - Something I'd really would welcome in a new BASIC.

*4 - Well, one could as well think of a multiple assignment with multiple variables as well on the left side, but I'd feel that would quite leave the path of BASIC. Not at least as it would need quite complex code, as until now there is always only a single left hand variable.

*5 - Another idea for a more modern BASIC where there is room for more code.

*6 - As MS does already for the ASCII content of strings - See Supercat's answer

*7 - Introduced with Third Edition BASIC in 1966

*8 - Introduced with 5th Edition BASIC in 1970.

*9 - The overhead consists of the same FOR loop plus opening and closing the file. In fact, most BASICs use INPUT for reading with a syntax quite like the READ instruction, so it really ends in moving the data lines to a file, removing lien numbers and the DATA statement and saving it. Quite what I call easy upward compatibility.

Answer 5

An additional workaround I've seen, in the book Writing Basic Adventure Programs For The TRS-80 went into this specifically; in an attempt to write a simple adventure-type game but stay within the restricted memory limits, the author recommended putting information into DATA blocks at the end of the code and then never reading it into variables, but just read it out directly using PEEK statements to find where it was stored in memory. It was really quite ingenious.

And as mentioned elsewhere, you could just populate your arrays and then BSAVE that area of memory into a binary file, and BLOAD it back in, which was the strategy that, for example, the game Odyssey used.

Answer 6

Simply, because most BASIC programs weren't sufficiently memory-constrained to need to do anything differently, and reading everything into memory made things simpler.

There are actually some BASIC programs that would not read all of DATA into an array, but rather just READ a single item (or small record of related items) that needed processing right now. (Typically these would be relatively large games, rather than other kinds of software.)

But this requires an additional statement (RESTORE) -- in modern BASIC this accepts an expression specifying the specific line number at which the next READ will start reading, allowing the program to use the DATA itself as a kind of indexable array.

Some older versions of BASIC did not allow specifying a line number in RESTORE, or only allowed a constant value rather than an expression. In these cases, to accomplish the same thing the program would need to RESTORE to the start of DATA (or a fixed location) and then READ and discard values in a loop until reaching the one that they actually wanted. And again, some programs did actually do this. (And even older versions of BASIC lacked the RESTORE statement at all.)

It should be obvious that both of these alternatives come at a performance cost vs. simply having all the data in a real array, so there is little incentive to do this unless you start running into memory limitations.

Answer 7

String variables and arrays store the length and address of the text. If a program computes a string value, then it will need to construct a string in memory which is filled from the top downward, but at least in Microsoft-derived BASIC for the 6502 and probably 8080, when a program performs a READ A$, the interpreter will store the address of the appropriate part of the DATA statement, rather than making a new copy of the string text.

That having been said, the design of DATA statements is pretty horrid for numeric information, both from a memory usage and execution time standpoint. I find it curious that back in the day people didn't routinely use a read/data loop to construct a machine-code routine that could process things much more efficiently (so that e.g. x=usr(12345),"ABCD1234" would store four bytes with values 0xAB 0xCD 0x12 0x34 to addresses starting at 12345.

Answer 8

The only alternative aproach to DATA is to assign initial values to initialized variables either directly (which doesn't save anything, the data is still present twice, both in the variables area and in the program code), load it from a binary file directly into the variable (a possibility on Apple BASIC, for example), or store initialized data together with the program on, for example, tape).

The last option was chosen by the designers of Sinclair Basic for the ZX 81 and ZX Spectrum computers: When you save a program, these computers will also save and restore the current contents of the variable area, which eliminates the need for DATA, READ and RESTORE completely. Typical space-saving programs would enter variable values either through direct commands or small "initializer" programs, then delete all program lines and enter the actual program that could then access the initialized values. SAVEing that new program would also save the initialised variables together with that, allowing the program to use these initialized values, which only existed once in memory now (as long as you didn't use CLEAR or otherwise change the values)

Answer 9

Basic programs "tended" to use Data and Read statements because these statements were defined parts of both the original Dartmouth implementation, and documented in the ANSI/ISO Minimal Basic and Full Basic Standards.

Given the generic mainframe computing paradigm of separate "program" and non-interactive "input data", one could concatenate together these two parts when feeding a Basic implementation, as long as care was taken to use separated ranges of line numbers for the 2 text files.

As for the "waste", this is an implementation detail, as there is no reason for a (non-self modifying) Basic implementation to read the full text of a program's text into memory. Applesoft and other *soft variants tokenized the actual Basic text, instead of storing the actual text. Other Basic implementation compiled the source into machine code, and ignored Rem statements. In other hypothetical implementations, the Data statement's contents could be compressed, or any Data statements could also be considered as just a separate file resource, and not read of off program storage media until actually needed by Read statements at run-time.

Answer 10

I think that if this turned into a problem, it could be mitigated with hacks. Some BASICs had commands for deleting a range of lines; that could be used to blow off the DATA section of the program. Even if that is not available, or doesn't liberate the memory in the right way, it could be hacked.

In developing a BBS on an Apple II machine, I used an overlay technique to fit a larger program into memory.

When the sysop (system operator) entered the administrative area of the BBS software, it loaded all that code from disk, and replaced a range of lines of the program with that code. Upon returning from the admin menus to the regular BBS area, it would load that again and discard the admin code.

I don't remember all the details, but I remember it involved a machine language routine which walked the linked list of the program's lines and did some splicing. It could have been that the loaded part of the program overwrote the memory starting at the right address, and then the machine language routine just repaired the linked list of the program.

The main point is that the BASIC program is data, making it possible to surgically remove some unneeded part of it, similarly in spirit to the way the Linux kernel discards functions only used for initialization. It may involve code that is specific to the BASIC implementation, depending on the details of the program's representation in memory and the memory layout.

In this particular situation, with regard to the DATA statements, we have to be careful. If the READ is implemented such that it's leaving the string data in place, and having the variables or array elements pointing to it, we cannot just remove that part of the program, because we end up with dangling pointers. (And in that case, we have less reason to anyway). We need to perform some string operation that will cause the string to be moved into the heap.