Here's an example program from the C64 Programmer's Reference Guide that plays a tune. It works by writing values to specific memory locations, which are mapped to the SID chip. You might want to get a C64 emulator and type it in yourself:
5 S=54272 sound registers start at S
10 FOR L=S to S+24:POKE L,0:NEXT clear sound registers
20 POKE S+5,9:POKE S+6,0 set ADSR
30 POKE S+24,15 set volume to max
40 READ HF,LF,DR read values HF,LF and DR from the DATA part
50 IF HF<0 THEN END quit on negative number
60 POKE S+1,HF:POKE S,LF write to frequency registers
70 POKE S+4,33 play note
80 FOR T=1 TO DR:NEXT wait
90 POKE S+4,32:FOR T=1 TO 50:NEXT release note and wait
100 GOTO 40 keep going
110 DATA 25,177,250,28,214,250
120 DATA 25,177,250,25,177,250
130 DATA 25,177,250,28,214,250
140 DATA 32,94,750,25,177,250
150 DATA 28,214,250,19,63,250
160 DATA 19,63,250,19,63,250
170 DATA 21,154,63,24,63,63
180 DATA 25,177,250,24,63,125
190 DATA 19,63,250,-1,-1,-1
That's three values per note, and all but one value is small enough to fit in a single byte. That's two bytes (a 16 bit value) for the frequency and one to specify the duration of the note.
For an actual product, it would also be necessary to specify the time between notes. So with this very simple encoding scheme, we need 4 bytes per note, which is quite economical.
Of course, we need a format that can alter the other registers on the SID chip, so we have some variety in the sounds we are producing. We can do this in a reasonably economical fashion by dividing the notes into sections of a bar or two. Each section would then have a list of notes and a list of register writes to perform before playing it.
The C code (though we would write this in assembler) might look something like this:
struct note{
char delay;
char frequency_high;
char frequency_low;
char duration;
};
struct sid_write{
char offset;
char value;
};
struct section{
note* notes;
sid_write* setup;
};
A nifty side effect of this format is that our arrays of notes and sid_writes can be reused. If the same few bars are repeated all over the piece, it only uses up a section for each case, which is just 4 bytes. And if many different sections use the same sound, they can use the same array of sid_writes.
This format is just an example, but you can see how easy it is to describe an 8 bit tune with a tiny amount of memory, especially if there is a lot of repetition. And 8 bit music generally has a lot of repetition.
If compression was used, what type of compression would that be?
Apart from the above, not much. Something like a Huffman decoder would itself require precious memory and music files were only a few kilobytes.
Because loading on a C64 was painfully slow, there might be some compression used on the whole program to speed up loading, but nothing specific to the sound files.
On a console like the NES, there would only be a tiny amount of memory to decompress the music into, and there would be far more important uses for that memory. It would therefore have to reside on the cartridge in a format that could be easily read while the game was running.