I don't think other answers have captured the essence here: Different RAM chips are connected to different parts of the computer.
You're looking at it from a software perspective: everything should be designed for flexibility. But have you considered how the hardware works?
Imagine you are designing a programmable sound circuit. Maybe it just reads out PCM sound data from RAM:
Timer => address incrementer => address register => RAM => DAC => analog filter => speaker
What makes up "RAM"? Well, a RAM chip. The address bus connects to the address register; the data bus connects to the DAC.
This is the core of your circuit - this is what makes the sound. Now you tack on some secondary bits to let the CPU access it. And you have to figure out how to actually do that. Maybe you say that when a sound isn't playing, the DAC is disabled and the CPU has access to the RAM's address and data bus, instead of the sound circuit.
CPU stuff =+=> start/stop flag -----------------------+-----------------\
| | | |
V V V V
Timer => address incrementer => address register => muxer => RAM <=+=> DAC => analog filter => speaker
^ | ^ |
| | | V
\---------------------/ +---> tristate switch
| data bus
Now you look at this design and you think "why is there sound RAM?" Well, it's obvious: this RAM is part of the sound circuit! The sound RAM is a RAM chip that's part of the sound circuit. The video RAM is a RAM chip that's part of the video circuit. In fact the video circuit might have more than one RAM chip: a Gameboy-like video circuit will have RAM to store pictures of game tiles, RAM to store sprites and RAM to store the arrangement of tiles on the screen, all in different places in the circuit!
In the arrangement shown above, the CPU has to turn off the sound to access the sound RAM. If it doesn't turn off the sound, the RAM's address bus will be connected to the sound address counter instead of the CPU's address bus, so the CPU won't be reading data from the address it expects to.
This is not the only way to design a peripheral with RAM. Many computers had the video circuit and the CPU take turns accessing the same RAM. Or the CPU's clock would be paused if it tried to access certain RAM chips on clock cycles where a peripheral was using them. Or just paused outright, regardless of what it was accessing. (See: Commodore 64 "bad lines")
There's also dual-ported memory with two separate address and data busses, but it's expensive.
On a modern computer, there is more than enough memory bandwidth to go around. The sound chip can contain a small FIFO buffer (because it does need predictable access) and refill it from the main memory with DMA whenever there is time. The video card, similarly. Retrocomputers did not have that luxury - everything had to be planned around memory cycles, and RAM was expensive, so you couldn't double-buffer your sound RAM to get around it.