Simple answer: Because it's the way BASIC is defined.
Original BASIC had no way to access files at all. DATA lines were the only way to add predefined data to a program. The idea was essentially to have a stack of items - originally only numbers, strings where only added way later - that could be read like a stack of cards.
More Important: It is already close to an optimal approach within the limits set by BASIC.
Isn't this rather wasteful on an 8-bit computer with perhaps anywhere from 4KB to 48KB of RAM?
In what way wasteful? It's in fact less wasteful than any other way. Keep in mind that data that needs to be used need to be stored in source, so everything has to be present in source code anyway.
A Basic Issue About BASIC
It's important to remember that BASIC, at least for most micro computer implementations, is a source code interpreter (*1). The source code is not compiled into machine code, but is the 'code' to be executed. And this source code is to be preserved (*2).
The imperative of 100% retrieval of original source text does restrict abilities to reorganize.
DATA is Already the Most Compact Way of Storage.
Let's look at how data is represented/stored as DATA lines:
- line overhead,
- the DATA instruction (token),
- the data items themself and
- delimiters inbetween
Whatever form a more 'space savvy' may have, I guess we can agree that it needs at least to include the data items as written and delimiters. In addition such a form must of course be placed in a line, thus having line overhead. And finally such a line must start with some statement (token) marking it as non-executable.
In the end a DATA line is already the most compact representation possible within the limits of an (interpreted) BASIC's syntax. So storage in terms of source text can not optimized past that.
Can At Least Transformation (Loading) Be Optimized?
Of course also need to look at the loading routine, whose job is assigning data items to variables (single or array). In most cases that's a FOR/NEXT loop iterating over the data array. While being some code, it isn't a lot thereof:
10 FOR I=0 TO 20
20 READ A(I)
30 NEXT I
In MS-like BASICs that'S about 14 bytes plus line headers not really a big overhead space wise and executed only once. Any other method may need more. As it still must name the array to be filled plus optional offset to start and so on.
Using Direct Assignment
Alternatively, why wasn't it particularly common for programmers to assign array items directly — as this Apple 1 port of ELIZA does — to leave more RAM free?
The real reason why the Apple 1 Version uses this is rather obvious: There was no DATA statement.
Compared to DATA lines this will eat up considerable more space. Instead of having the raw source representation only delimited by a single comma plus one DATA statement per line, it now needs to be preceded by an additional variable name and an assignment operator. This adds up a lot. So that's at least one byte more per data item. For each additional item in the same DATA line saves two bytes. and that's only for single character numeric variable names. for each further character add one as well one if it's a string or an integer.
Bottom line: Already 5 single assignments of strings will eat up more memory than using DATA with a for loop to load an array.
Maybe a different assignment syntax (somewhat like FORTRAN) could help?
Using Multiple Assignments
Cool idea (and borrowed from BASICS parent FORTRAN). This could for example be done by introducing a multi assignment statement. For example like
would load the letters A..F into A(1..6). Looks great, but needs more storage, as now each line also needs the target named:
after 3-4 lines this may eat up more source space than a corresponding FOR loop.
In addition this would remove the ability to structure data lines in a meaningful way, making them even harder to read. Just think a structure of corresponding items, like an address consisting of zip code, city, street and number. With classic data items this will look like
10 FOR I=1 TO 2
20 READ ZP(I), CY$(I), NR(I), ST$(I)
30 NEXT I
100 DATA 81739, "München", 34, "Waldheimplatz"
110 DATA 81541, "München", 6, "Schweiger Straße"
With multiple assignment it'll be more like
10 LET ZP(1)=81739, 81541
20 LET CY$(1)="München", "München"
30 LET NR(1)=34, 6
40 LET NR(1)="Waldheimplatz", "Schweiger Straße"
Which one is more pleasant to read? Not to mention that all these cryptic data lines now have to be at the begin of the program (or need calledas subroutine).
While it would be rather useful as well in situations outside of replacing DATA statements (*3), this would not only need more source code with lower readability (*4) but still have the internal representation (in data storage) in addition to source data.
So far the use of DATA statements seems the most space saving, but what about
A 'Special' Approach do DATA Data
One may of course think of a way to handle DATA statements like before, but on assignment only 'copy' a pointer to the data chunks within the DATA line. Could be a nice idea, but it comes with a hefty cost in code (*5). Now variable handling needs to distinguish between pointers into DATA statements and such into data storage and to handle copy on modification (*6)
Also, this would only make sense with strings, as data, including numeric is stored in DATA statements as ASCII (to be able to recreate source code). Converting them during each access from ASCII is less than great for performance, so they better get converted just once.
DATA is Already a (One Dimensional) Array
But what inspired implementers of the various BASICs to build in a DATA/READ feature instead of some RAM-friendlier mechanism like a syntax for defining read-only arrays?
DATA is already a read only array. It's a one dimensional array, starting with the first item, accessed sequentiallyvia consecutive READs. Thanks to RESTORE (*7) the read pointer can be reset any time to parse the data stack again. Not fast - in fact extreme slow - but the most space saving handling :)
Long Story Short
- There is no more efficient way than the existing within the context of BASIC as interpreter (and the restricted capacity of 8 bit systems).
- DATA statements are a way to handle a static input 'file' within the program.
- As soon as a system supports reading from data files (*8) that should be used (*9).
*1 - Funny considering the original BASIC being a compiler - which also explains why that duality didn't exist: The compiler could transform source DATA lines into a more appropriate intermediate format (especially as original BASIC had only numerical(FP) as its single data type).
*2 - Many BASICs employ methods to reduce storage need by using crunching methods like tokenization of keywords, some use (like Sinclair) use optimized constant storage to speed up execution (but increasing space), but all must handle the code in a way that the original text can be reconstructed down to the last dot and comma (e.g. when listed).
*3 - Something I'd really would welcome in a new BASIC.
*4 - Well, one could as well think of a multiple assignment with multiple variables as well on the left side, but I'd feel that would quite leave the path of BASIC. Not at least as it would need quite complex code, as until now there is always only a single left hand variable.
*5 - Another idea for a more modern BASIC where there is room for more code.
*6 - As MS does already for the ASCII content of strings - See Supercat's answer
*7 - Introduced with Third Edition BASIC in 1966
*8 - Introduced with 5th Edition BASIC in 1970.
*9 - The overhead consists of the same FOR loop plus opening and closing the file. In fact, most BASICs use INPUT for reading with a syntax quite like the READ instruction, so it really ends in moving the data lines to a file, removing lien numbers and the DATA statement and saving it. Quite what I call easy upward compatibility.