Short answer: Yes
Would it have been possible to plug suitably designed game cartridges into an S-100 slot?
Sure. After all, such a cart would be just another ROM/Memory board. Although, some configuration might be required to avoid address collision.
Admittedly most of the computers using that bus were designed as business machines and didn't have color graphics, but supposing you had one that did have color graphics?
Cromemco's TV Dazzler was already available in 1976. It got quite some publicity, I think I remember even a book about it back then. So colour wasn't as uncommon on S100 as people might believe.
There were several S100 systems offering out of the box colour abilities (like Poly88) - and even more when considering (home) computers that offered S100 as expansion boxes, like the Exidy Sorcerer, ok, B/W, but you get the picture.
The main issue here would not only be a non-game-console-like effort to configure a card (Address and I/O usage) but also the lack of a standard way to access various hardware - not to mention the merits of some abstraction layer. We all can rant a lot about MSFT and Windows, but it did provide a proper way to encapsulate even the most weird hardware, making old games run on new graphics cards (and vice versa ... well, ignoring the missing power :))
From a game system viewpoint, the S100 bus would have been a way expensive system. For one, the bus doesn't offer regulated power, but rather expects each card to have their own regulators. Very primitive, good for quick-starting a system, but expensive on the long run. Same for the board size. Cost goes with board size, and an S100 module needs to be at least as wide as the connector is. Mind you, there have been even VCS cartridges that tried to use odd shaped boards to minimize surface. It's a penny business. Even with such tricks like triangular shaped boards etc., they would cost several times what a standard VCS cartridge board was.
There's a reason something like this never happened, despite the zillions of developments done for S100.