Before we had graphics support in the Linux kernel, any program that wanted to enter graphics mode had to access the graphics card directly. Those programs were generally started with root privileges and asked the kernel to provide direct hardware access from user mode.
Besides the X Server, some tools used
SVGAlib, a lightweight library to acces (Super)VGA cards in graphics mode, and there was
SVGATextMode, a program to set up high resolution text modes. Also, the mis-named tool
dosemu (which is in fact a VM monitor that boots a real DOS) had a mode called "console mode", in which it claimed access to the graphics card and passed those access rights to the 8086 VM, so the native video BIOS of the graphics card could be used by DOS applications to set any mode supported by the card.
As the user application reprogrammed the graphics card without the kernel knowing about it, the user application also was responsible for re-setting the card to a sane text mode and telling the kernel that the graphics card is usable again. If an application crashed in graphics mode, the console was unusable. If the keyboard was not in raw mode (later Linux versions allowed Alt-SysRq-R to leave raw mode), one could try the script
textmode bundled with SVGAlib that reprograms the graphics card to a clean text mode, provided the text mode settings were saved with
savetextmode before messing up the graphics card state. Another option was to use
mode3, which used the video BIOS to reinitialize the graphics card to the standard text 80x25 mode (thanks ninjalj for the pointer).