One does not simply make a hot-swappable cartridge connector.
For an electrical connector to handle hot-swapping well, it has to be specifically designed with that in mind. In particular, the contacts have to be designed to connect in a specific, predictable order to avoid erratic behaviour and electrical surges that could cause glitches, and in the worst case, fry the circuit outright. PS/2 sockets for PC keyboard and mice were somewhat notorious for this, although it got better in the later years, before they were entirely displaced by USB. For more details, I invite you to check out a similar question over at Electronics. While I have a hard time finding a good image of a GBA cartridge slot and connector, the images I do find show pretty clearly that it’s entirely symmetrical, with all pins of equal length. With such a design, seamless hot-swappability is a non-starter: there is no way to make it safe to the circuitry, never mind detect when it happens.
So that’s one reason why the GBA does not handle disconnecting a live cartridge more gracefully. One could ask: why didn’t Nintendo design the connector to allow hot-swapping then? I suppose there just wasn’t a compelling enough case for it. Nintendo didn’t have any interesting functionality in mind that would require carts to be hot-swappable; there is no point spending design effort on it just to display a lame error message. I guess it might also have complicated compatibility with the original Game Boy, and Nintendo may have been assuming that players would not do stupid things that could risk damaging their, after all, not all that cheap hardware.
These days they would have probably known better.
With all that out of the way: what happens when you ignore the dangers and rip the cartridge out of the handheld?
A cartridge slot is basically a direct extension of the CPU’s memory bus; it’s not so different from an expansion card slot in a desktop PC. When a cartridge is connected, the CPU can directly communicate with the devices present on the cartridge (which most of the time are simply a couple of ROM chips, sometimes some battery-backed SRAM for save states) simply by putting the address of the device on the memory address bus and then waiting until the device uses the data bus to either read the data the CPU sent its way or send the data the CPU wants. In normal operation, the CPU fetches instructions one after another as they are executed, either directly from the ROM, or mediated via a mapper chip on the cart. (Some emulator consoles that can read original cartridges are known to instead read the entire ROM in bulk into working memory and run it from there, leaving the cartridge an idle prop most of the time.)
When the cartridge is pulled, the CPU keeps fetching instructions from memory addresses that are no longer connected to a ROM. This situation, where a CPU accesses a memory address not answered by any device, is known as an open bus, and it’s infamous for causing lots of headaches for emulator developers. As the CPU hopelessly tries to fetch instructions from ROM that is no longer there, the game appears to freeze. The CPU keeps running, though, at least for a while; it just keeps executing garbage instructions obtained from the open bus. If the game happens to be running from built-in RAM, though, it may even keep actually working; at least, until it decides to read the cartridge ROM.
But that directly affects only the core logic. The RAM, which is installed in the handheld itself, is still present, and the video and sound hardware maintain their state: as such, the screen contents stay intact, and the sound keeps playing. Eventually though, the audio buffer runs out, and the audio hardware attempts to signal the CPU about it with an interrupt request; but since there is no game code to refill the buffers any more (or data to refill it from), the sound hardware just plays whatever garbage happens to be next to the audio data in memory, and eventually loops back to the start.
