The VGA design is inherited directly from the EGA design, though with a few extra features. Most considerations apply equally to both.
Although it is possible to treat the EGA or VGA memory as four banks of 65,536 octets, the EGA was designed to let it be treated more usefully as 65,536 groups of four octets, where each group had a single assigned CPU address. Within each group of eight pixels, each pixel would use one bit taken from each octet. The EGA and VGA have a 32-bit buffer that gets loaded from all four groups of pixels whenever an address is read, and each write will store to each group of 32 pixels a pattern which is based upon the value written, the value of some configuration and parameter registers, and the contents of the buffer.
This made it possible to perform many kinds of read-modify-write operations on a group of pixels at a time. The CPU would read a byte, ignore the result, and then write a value that indicated how the group of bytes should be modified. The display card would then combine the value in the buffer with the value that was read and store the modified result to all 32 bits. This allowed for bytes in display memory to be updated with a single read-modify-write sequence.
Alas, a couple aspects made this design far less useful than it could have been. They largely boil down to the fact that for various technical reasons, reading display memory is extremely slow, and every action on a group of 32 pixels would require four individual read-write sequences. If the computer had enough "main" RAM to maintain a copy of the screen, building the display contents in RAM and blindly outputting to the display would almost always be faster than trying to use the display's hardware to expedite the process. On slow machines the display hardware could offer some speed advantage, but as main-CPU speed increased the extra display hardware became more and more useless.
Still, the designers of the EGA and VGA had expected that the ability to update multiple bytes at once would be useful, and thus designed their architecture to facilitate it.