When shopping for a VGA ISA card, how might I know if it would be compatible with an IBM 5162? Do all 8-bit and 16-bit ISA VGA cards work, or only some?
The 5162, more commonly known as the IBM PC XT/286, uses a 16-bit expansion bus. Any 8-bit “ISA” VGA card should work fine (but they are rare), and most 16-bit VGA cards should work too... Ironically, the 8-bit Hercules VGA card is known to cause issues. Some 16-bit cards designed for the PC AT won’t fit in the XT/286’s case and therefore can’t be used either.
Compatibility problems are mainly a concern with older systems, because of their 8-bit-only bus, and/or their 8088 CPU. A few 16-bit VGA cards work in IBM PCs and XTs; the Vintage Computer Forum has a list. Basically, older 16-bit VGA cards made by ATI, Cirrus Logic, Oak Technology, Trident, TSENG, Video Seven, and Western Digital (Paradise) are possibilities; some will work without configuration, others need jumpers to be set appropriately. This page lists 8- and 16-bit cards which are known to work in IBM 5150s.
As Brian Knoblauch points out, plugging a 16-bit card in an 8-bit slot means that part of the edge connector hangs off the end; this is perfectly safe as long as there are no components in the way on the motherboard, and as long as you don’t run into a slot with an edge that’s too thick to fit in between the connectors (it happens sometimes, so don’t force a card in). I couldn’t find an example photo of this in my brief search, but this works in the same way that a PCI-X card can usually be plugged into a PCI slot, with an edge connector hanging off the end (see this article for an example).
If you don't mind soldering and making/ordering custom PCBs, you can actually build your own VGA card. Sergey Malinov has schematics on his website: http://www.malinov.com/Home/sergeys-projects/isa-supervga
It is based around a Trident TVGA9000 chipset, which you can get as New Old Stock off of eBay or similar sites.
The card even has a fix for IBM XTs which do not implement the ALE line correctly.