CGA cards provide their own RAM to the address space. This memory lies in the hole between 640KB and 1MB. By convention, it was stated that the video memory (for CGA cards) would begin at segment B000 if the CGA card is being connected to a B/W monitor, and B800 if connected to a colour monitor. This way, two CGA cards would coexist in the same system, providing a means of multimonitor environment (text only for the monochrome monitor and graphics for the colour monitor in a typical CAD environment).
So it seems that the base memory location can be changed somehow. Actually, it is more flexible than that. In text video modes, scrolling text is a fast operation not because the CPU shifts blocks of memory to make room for new lines, but because the CPU changes the starting address of the video memory so that the user sees a new page of video memory. There were 4 pages of video memory for text on a basic 16KB CGA card using 80 columns mode, and 8 pages if using 40 columns mode. This was possible because the CGA (and subsequent compatible graphics cards) used the well known CRTC controller Motorola 6845, which provides registers to specify where in memory the video data should begin.