Why did Commodore files not include metadata to say where in memory to load it?
They are there. Every saved memory content starts with two bytes noting the address it's taken from. No matter if disk or tape, if BASIC or machine code.
But since it doesn't really make sense to save a file in one way and load it again the other way, then the user needs to remember per file how to load it again.
It's not save it one way and load it another. Memory content, and that's true for machine or basic programs, are always saved with their start address stored as the first two bytes. And the way to load them to the same address they where saved is by using the secondary address
1 (*1). With this parameter the program gets loaded verbatim - exactly as saved. Without it gets relocated (and treated like a BASIC program).
There is no need to remember anything. Any BASIC program can also be loaded with or without - as long as it's on the same machine in the same configuration.
It seems much more sensible to me to have a bit in the metadata somewhere that specifies how to load the file.
And what would it be good for? With a fixed mechanism always loading a program into the area where they where saved isn't really flexible, is it?
Is anything known or understood about why this wasn't done?
It's all about making life easy for users by setting the machines BASIC start address as default load address. And as with all default mechanic, it is supposed to be the shorter notation.
Now users can load the most common files (BASIC programs) on any machine without fiddling with address changes. Different Commodore computers used different areas for BASIC programs (*2). With this default mechanism programs can (often) be exchanged between different machines (*3).
In fact, some machines even change the BASIC load address depending on the configuration. An unexpanded VIC20 loads programs at
$1000, while with a 3 KiB extension it's
$1200. With a fixed load address from the file saved people would no longer be able to use programs they've written before buying the 3 KiB expansion afterwards.
Beside changing the load address, loading without not only adapts the load address, but also relocates the program. Yes, there's a need for that, as Commodore BASIC is stored in tokenized memory format, and here every line is preceded by the address of the next line. To make a program work from a different start address the loader needs to go thru the program, after placing it in memory, and recalculate all line address (*4).
*1 - In fact it's not even a real secondary address, but only a marker for the loader to use the stored start address. The drive always gets opened with a secondary address of
*2 - For example classic PET at
$0400, VIC-20 at
$1000 or C64 at
*3 - Of course, only as long as only instructions are used that are common to both machines.
DLOAD from a BASIC 4.0 CBM will not compute on a straight C64.
*4 - That's also the reason why a disk drive is able to deliver a directory listing as pseudo BASIC program, without knowing anything about the computer handling of BASIC memory it is connected to: The 'program' gets relocated, so it can be listed on any Commodore. No matter if it's a PET with an 8050, or the same 8050 connected to a way later designed VIC-20